VMware supported iSCSI HBAs have increased but my implementations have not
January 9th, 2009 | 
Author: iSCSI – Hardware or Software – How many TOEs do you have? is a post by Carlo Costanzo that really struck a chord with me. Carlo asks:
“More and more of my new implementations of VMware Infrastructure are being connected to iSCSI SANs (EMC, LeftHand, and Equallogic) and the question has come up about whether or not to spend extra dollars on TOE (TCPIP Offload Engine) Network cards.”
This made me realize that I have yet to implement an iSCSI SAN connection to an ESX Cluster with a hardware initiator. My customers have all used the native ESX iSCSI software initiator instead. So it made me wonder why am I not even considering the hardware iniator anymore, and should I present the option to my customers at all? Afterall, as Carlo points out, the number of supported TOE cards has increased from 2 to 16 (a table is provided at the end of this post)
I came up with a short list of reasons why I haven’t been using the iSCSI hardware initiators.
- the cost of iSCSI HBAs doesn’t justify the expected performance gains (around $200 for a 1GB nic versus over $1000 for a TOE per ESX host) unless you have a large number of heavy I/O VMs
- ESX 3.5 Clusters are designed with an N+1 number of hosts strategy so there is always extra available CPU cycles for the software initiator anyways.
- I did not know there are more iSCSI HBAs supported now!
I’m curious now about how prevalent the hardware initiator is used today. Maybe I’ll use this as a topic for a future VM /ETC poll.
In case you are wondering what a TOE or iSCSI HBA does, the following is from a slightly dated version of the vmware iSCSI design guide:
“iSCSI HBAs are interface adapters that provide high-performance offloading of both TCP and iSCSI
processing onto the adapter. Although this adds cost to the adapter, it delivers high-speed iSCSI transport with minimal CPU overhead.”
Be sure to check for yourself to get the current supported list, but here is a table of all the current VMware supported iSCSI HBAs taken (at the time of this post) from the online Hardware Compatibility Guide web application :
Notes:
- 1. Requires patch ESX-1001906.
2. Ethernet is not supported.
3. Supported on Intel E1000 and Broadcom running on a gigabit link only.
4. QLogic iSCSI adapters require minimum BIOS v1.09 and minimum firmware v2.00.00.38.
* Device Drivers:
1. megaraid_sas version 3.0.9
2. mptscsi_2xx version 2.06.48
3. qla4022 version 3.30
4. qla4022 version 3.24
5. qla4010 version 3.10
6. iscsimod version 3.6.3
7. iscsimod version 3.4.2
Posted in 
Tags: 
Rich,
Maybe you know this already (in which case I comment for the benefit of you google-ers out there that just found VMetc.com for the first time) but with the Hardware-based iSCSI, you can now get beyond the 1 Gig limit for VMFS datastores by assigning different LUNs to different HBAs.
With the software iSCSI initiator you limited to 1 Gig of storage bandwidth per ESX hosts, unless you complicate the setup and use guest-based initiators (which is not really a bad thing compared to SAN features you can now use if the data is on a native SAN LUN).
I’m going back to HW iSCSI for these reasons since it will give me the best of both worlds and prices should drop now that there is plenty of competition in the market.
-Sean
Twitter: vseanclark , b/c I’m too lazy to update my blog
I’m wondering why Dell PERC 6/E Adapter and HP SC44Ge HBA Adapter are on the list. They are SAS HBA adapters, and have hothing to do with iSCSI…
Correct me if i’m wrong.
Best Regards
Andrzej
Our primary VMware environment is a pair of HP Bladecenters, and it’s been difficult to track down Blade HBAs. HP doesn’t make their setup very clear as to whether their own NICs can double as an HBA, or whether the more expensive Qlogic hardware is required. We don’t have terribly high I/O VMs just yet, so it’s not too big of a concern, but I think it’s a lack of clear direction on the Blade side that’s keeping us from making the investment.
Great follow up post Rich. Sean brings up a good point with the 1 GB limitation of the SW iSCSI. Thanks to vMotion and DRS though, if the disk writes/reads are not being satisfied, VI will ‘move’ the VMs to another host. Of course only good if the reason for the delay was due to a collective request as opposed to the individual VM choking the pipe.
Sean,
Were you experiencing bottlenecks in your environment which is forcing you to move to the HW cards or are you just trying to stay ahead of the curve?
CARLO.
@Carlo
Customer might be experiencing a bottleneck soon as they are moving 400 user Groupwise installation (physical) to a virtualized Exchange 2007. They are on SW iSCSI now and I warned them that if we run into a bottleneck on VMDKs, that we’ll want to migrate a guest-based initiator to effectively use a different path to the storage than the VMkernel port is using.
Well, they did not like that idea at all because that takes away the very comfortable and easy to slice and dice VMFS option they are used to. They are small shop and hate having to enter the Celerra interface to do anything, let alone have to learn about guest-based initiators.
Soooo, I did more reserch into getting more than 1 NIC active with SW and hit brick wall. Only way to get more than 1 Gig of iSCSI traffic to VMFS volumes is to go HW iSCSI. So if their Exchange implementation and further VM growth, ends up going beyond 1 Gig of bandwidth for an ESX server, we will likely insert HW iSCSI adapters and pull 1 of the 2 quad port NIC cards..
Can’t wait till 10G becomes affordable.
Sean
The way I’ve always looked at it is that if I’m going to drop a thousand dollars on each of my servers for an HBA, it might as well be fibre channel. iSCSI is an option I usually exercise due to cost constraints…
@Carlo
Are you sure? I was under the impression that DRS is only going to intervene based on CPU & Memory and not Disk / Network. I could understand that if a VM was paging to it’s vSwap that DRS may step in but I don’t believe it works that way (although I could be wrong… happens on more than I’d like to admit)
-Tyler
Sean,
Good point about the 1 Gb bandwidth limit per LUN. But to use that design wouldn’t you require a few large VMFS LUNs with 12 to 16 VMs on each? How many iSCSI HBAs would you put in each ESX server? That becomes a problem for SAN replication or SRM designs, for example. Also, aren’t we crossing that grey line in the VMFS where NFS makes more sense too?
Andrzej,
I did a search for “iSCSI” using the compatibility guide web application, and to be honest I did not pay close enough attention to the results before I cut and pasted. It looks like VMware may have them both labeled wrong as type “iSCSI”. Thanks for pointing this out.
Carlo, Sean,
Great discussion, and yes I agree that 10gb ethernet will change the design pros and cons once again ..
Nick, Ian,
The cost of the hardware iSCSI adapters really do make you think twice. Of course, you have to have already implemented or licensed a FC SAN, so the $1k price tag is really just the “tip of that iceberg”
@Ian If you have the existing fibre channel infrastructure, I agree completely.
But if it’s a green field, iSCSI should be the 1st stop for small businesses looking at VMware. It’s in their budget and the performance worry can be alleviated if I can get closer to 2GB throughput per ESX host with HW iSCSI (and more with use of guest-based iSCSI initiators).
That being said, nothing beats the simplicity of VMFS on fast FC storage. Set it and forget it. Maybe add some more disks, then forget it again.
rich,
At VMworld I asked that same question about SRM of the Dell Equallogic guys. They said that as long as you replicate the data LUNs along with the VMFS LUNs containing the protected VMs, then you are golden. They also thought that nothing needed to be done to present the replicated data LUNs, but I have my doubts. You would probably need to the use the SRM feature to add manual steps in the recovery/test plans like “Call storage guy to verify data LUNs to DR ESX servers”. An Equallogic collection could also be used to guarantee the VMFS volumes and the data LUNS were “snapped” at same time.
**Disclaimer: I have not tried this yet, but I did stay at a Holiday Inn Express last year.
on the contrary, would smaller datastores with fewer VMs make it easier to split traffic 50/50. SVmotion with larger Datastores could accomplish same goal of balancing traffic.
I would recommend a single, dual port iSCSI HBA or 2 single port iSCSI HBAs if the money is there.
@Tyler – You are correct. Mem and CPU are the only catalysts.
There were posts on the forums about possibly expanding these to Disk and Network (which would be great) but I couldn’t find any additional information. Good Catch!
I wonder if a Maxed iSCSI pipe would bog down the vKernel which would hamper it’s ability to fulfill requests which would trigger a DRS event.. But that’s a streeetch huh?
According to this VMware KB article TOE features are NOT SUPPORTED!
http://kb.vmware.com/kb/1006143
Product Versions
——————
VMware ESX 3.0.x
VMware ESX 3.5.x
VMware ESXi 3.5.x Embedded
VMware ESXi 3.5.x Installable
Symptoms
———-
What is a TOE NIC and does ESX support any TOE NICs?
Resolution
———–
TCP Offload Engine or TOE is a technology used in network interface cards to offload processing of the entire TCP/IP stack to the network controller. It is primarily used with high-speed network interfaces, such as gigabit Ethernet and 10 gigabit Ethernet, where processing overhead of the network stack becomes significant. The term, TOE, is often used to refer to the NIC itself, although it more accurately refers only to the integrated circuit included on the card which processes the TCP headers.
ESX server does not support any NICs as TOE NICs as of this writing and we do not have any public commitments to support TOE NICs. If you find that some of the NICs listed under ESX HCL are TOE capable, those NICs will only be supported as regular NICs and TOE features available on those NICs will not be utilized by ESX server.
Isn’t 10Gb SW iSCSI supported with 10Gb NICs?
Re, the KB article.
This is no surprise — even Linux/Solaris don’t support TOE (full tcp offload), in general, these cards are not supported, except on specialized
platforms (Windows TCP Chimney, or open source OSes with major vendor-specific kernel patches).
But VMware ESX does support some iSCSI HBAs, which are TOE cards.
It supports the TOE function in conjunction with the full offload of
iSCSI, but ESX does not allow you to use the iSCSI HBA as an Ethernet interface. I.E. the TOE and even the Ethernet functionality is strictly limited to use of offloaded iSCSI, and other uses of the cards are not possible.
Most modern OSes _do_ support TCP checksum offload, and sometimes LSO (large segment offload), for compatible NICs with the feature.
I believe VMware supports at least the TCP checksum offload functions
of certain cards, also.
This is not the full TOE, but there are still speed enhancements to be found by picking the right vendor and model of NIC.
Hey Rich – A couple of quick comments to add for you. I have set up a TON of iSCSI both hardware and software based. We usually do it in the SMB market because a FC SAN will be too much. Sure, the iSCSI HBA are almost as much as a FC HBA, but what if you don’t already have the fabric in place? We have been setting up a couple of ESX servers in a cluster, hooked to the storage, all with iSCSI. Some things to remember:
1. The Q-Logic 40XX chipset (4022, 4052, and 4062 as of this writing) is the only TOE chipset supported today. I know there was a comment about blades above. That means the HP QMH4062 is the only iSCSI TOE card that will be supported with VMWare on HP Blades. More chipsets will be supported in the upcoming release of ESX.
2. DO NOT boot from iSCSI!! We have found that the pathing for ISCSI on ESX will not take a path hit like it will for FC. If you lose a path, even for a second, you will most likely PSOD the server if it is the boot path. Boot to a set of local drives and use the iSCSI (hardware or software) to go to the storage. The VMFS paths will take a hit fine.
Justin commented over at Carlo's blog clarifying the KB article and the supported iSCSI HBAs. I am quoting some of Justin's comment for VM /ETC readers here:
iSCSI HBA's are supported… (Qlogic, etc)
TOE enabled NICs are not supported, as in, none of the TOE features will work, they will function just as any old NIC would.
iSCSI HBA's (Hardware initiators) are supported and work great and they do perform offloading from the CPU.
They are not all $1000 either, the Qlogic 4050c is a single port adapter that you can find for $600-$700. I have two of those in every ESX Host I have.
If you are referring to a TOE card to mean a TOE enabled NIC you will get no offloading per the KB article.
If you are referring to a TOE card to mean an iSCSI HBA you will get all the benefits of offloading.
Wade also just posted some really great points to add to this conversation.
http://www.vmwareinfo.com/2009/01/iscsi-hardwar...
<—– Snip —->
To help clarify-
TOE, or TCP Off-load Engine, is a chipset NIC manufacturors put on the Cards to Off-Load the TCP Stack as well as the IP Stack from the OS, to improve performance. TCP off-load requires special drivers in the server network Stack. Thus you will commonly see that Windows (Chimney) has this, while most Linux OS variants do NOT have this. ESX included.
iSCSI HBA, is a different animal entirely. the iSCSI HBA, off-loads the IP, TCP, AND iSCSI processes onto Special Chipsets. These cards can and do reduce the amount of CPU needed by the host. iSCSI HBAs are supported by ESX, the forementioned QLA4050c, and 4052c(dual port) both work very well.
To the krux of the question around Hardware iSCSI HBA or ESX Soft iSCSI, this is a completely different argument. Both have legit use cases and are highly dependant upon your architecture.
Here are a few things to be aware of-
1- if you use a QLA405Xc they do NOT count against your NIC limit in ESX< thus if you are using a lot of NICs and dont' have them to spare they are a good choice.
2- you can only use 2 hardware iSCSI ports. (you can put 2 Dual port cards in a Host, but it will only see 2 of the ports.)
3- on the subject of failover, some people opt for 2 4050c single port cards to allow for card redundancy. NOTE- this is 2 PCI slots which can get really valuable in an ESX host. If you have HA, SRM, and soon FT configured correctly then a Single Dual port card is a nice option since a card failure is covered by HA at the Host level. (in my 15+ years in the storage business I can probably count the number of Card failures on a Single hand. The number of Cable and or Switch failures is far greater, yet covered nicely by a dual port card.)
4- in ESX 3.x path Failover is “active/passive” and thus if you have 2 iSCSI HBA ports you can set them up to provide fail-over the same way you do with FibreChannel. (in the next gen of ESX you start to see a more Active/Active or Round Robin approach)
5- If you use the software iSCSI you can simply use more than one NIC in the vswitch where your vmkernel port-group is located and if you have multiple iSCSI seesions (highly recommened) then you can see both NICs end up used for I/O.
6- If you want to Boot from SAN with NO drives in the ESX host, then the iSCSI HBA is the way to go. (Note- USB enable ESXi has lessened the use case here.)
7- with ESX 3.5, Jumbo Frames is supported in Both the Hardware and Software init.
8- If you are lucky enough to be running the 10G Converged Ethernet Adapters then the choice is a no-brainer!
9- Make sure your Network is correctly setup, as this will severely impact the performance of both HW and SW iSCSI inits.
10- The only area where you will typically see the iSCSI HBA do a better job is in LARGE BLOCK, Sequintial I/O… Small Block Random I/O not so much. ( if you have a VM doing a lot of Large block seq I/O, you have a different question to address.) If it's a lot of Small Block Random with a heavy latency requirement, you really need to pay close attention to the disk system you are using, because what's at the end of that wire will really matter.)
In most cases I have seen the Software init do just fine, and customer are quite happy with it. The CPU % need to run is typically so small it just doesn't make an impact and with todays Quad / Quad core machines the % just keeps falling.
I've seen both SW and HW hit wire speed 125MB/sec.
In the end it's up to you.
Wade O'Harrow
Sr. Manager, Sr. VMware Specialists
EMC, Corp.
<—– Snip —->
Wade also just posted some really great points to add to this conversation.
http://www.vmwareinfo.com/2009/01/iscsi-hardwar...
<—– Snip —->
To help clarify-
TOE, or TCP Off-load Engine, is a chipset NIC manufacturors put on the Cards to Off-Load the TCP Stack as well as the IP Stack from the OS, to improve performance. TCP off-load requires special drivers in the server network Stack. Thus you will commonly see that Windows (Chimney) has this, while most Linux OS variants do NOT have this. ESX included.
iSCSI HBA, is a different animal entirely. the iSCSI HBA, off-loads the IP, TCP, AND iSCSI processes onto Special Chipsets. These cards can and do reduce the amount of CPU needed by the host. iSCSI HBAs are supported by ESX, the forementioned QLA4050c, and 4052c(dual port) both work very well.
To the krux of the question around Hardware iSCSI HBA or ESX Soft iSCSI, this is a completely different argument. Both have legit use cases and are highly dependant upon your architecture.
Here are a few things to be aware of-
1- if you use a QLA405Xc they do NOT count against your NIC limit in ESX< thus if you are using a lot of NICs and dont' have them to spare they are a good choice.
2- you can only use 2 hardware iSCSI ports. (you can put 2 Dual port cards in a Host, but it will only see 2 of the ports.)
3- on the subject of failover, some people opt for 2 4050c single port cards to allow for card redundancy. NOTE- this is 2 PCI slots which can get really valuable in an ESX host. If you have HA, SRM, and soon FT configured correctly then a Single Dual port card is a nice option since a card failure is covered by HA at the Host level. (in my 15+ years in the storage business I can probably count the number of Card failures on a Single hand. The number of Cable and or Switch failures is far greater, yet covered nicely by a dual port card.)
4- in ESX 3.x path Failover is “active/passive” and thus if you have 2 iSCSI HBA ports you can set them up to provide fail-over the same way you do with FibreChannel. (in the next gen of ESX you start to see a more Active/Active or Round Robin approach)
5- If you use the software iSCSI you can simply use more than one NIC in the vswitch where your vmkernel port-group is located and if you have multiple iSCSI seesions (highly recommened) then you can see both NICs end up used for I/O.
6- If you want to Boot from SAN with NO drives in the ESX host, then the iSCSI HBA is the way to go. (Note- USB enable ESXi has lessened the use case here.)
7- with ESX 3.5, Jumbo Frames is supported in Both the Hardware and Software init.
8- If you are lucky enough to be running the 10G Converged Ethernet Adapters then the choice is a no-brainer!
9- Make sure your Network is correctly setup, as this will severely impact the performance of both HW and SW iSCSI inits.
10- The only area where you will typically see the iSCSI HBA do a better job is in LARGE BLOCK, Sequintial I/O… Small Block Random I/O not so much. ( if you have a VM doing a lot of Large block seq I/O, you have a different question to address.) If it's a lot of Small Block Random with a heavy latency requirement, you really need to pay close attention to the disk system you are using, because what's at the end of that wire will really matter.)
In most cases I have seen the Software init do just fine, and customer are quite happy with it. The CPU % need to run is typically so small it just doesn't make an impact and with todays Quad / Quad core machines the % just keeps falling.
I've seen both SW and HW hit wire speed 125MB/sec.
In the end it's up to you.
Wade O'Harrow
Sr. Manager, Sr. VMware Specialists
EMC, Corp.
<—– Snip —->