<—– Snip —->

To help clarify-

TOE, or TCP Off-load Engine, is a chipset NIC manufacturors put on the Cards to Off-Load the TCP Stack as well as the IP Stack from the OS, to improve performance. TCP off-load requires special drivers in the server network Stack. Thus you will commonly see that Windows (Chimney) has this, while most Linux OS variants do NOT have this. ESX included.

iSCSI HBA, is a different animal entirely. the iSCSI HBA, off-loads the IP, TCP, AND iSCSI processes onto Special Chipsets. These cards can and do reduce the amount of CPU needed by the host. iSCSI HBAs are supported by ESX, the forementioned QLA4050c, and 4052c(dual port) both work very well.

To the krux of the question around Hardware iSCSI HBA or ESX Soft iSCSI, this is a completely different argument. Both have legit use cases and are highly dependant upon your architecture.

Here are a few things to be aware of-
1- if you use a QLA405Xc they do NOT count against your NIC limit in ESX< thus if you are using a lot of NICs and dont' have them to spare they are a good choice.
2- you can only use 2 hardware iSCSI ports. (you can put 2 Dual port cards in a Host, but it will only see 2 of the ports.)
3- on the subject of failover, some people opt for 2 4050c single port cards to allow for card redundancy. NOTE- this is 2 PCI slots which can get really valuable in an ESX host. If you have HA, SRM, and soon FT configured correctly then a Single Dual port card is a nice option since a card failure is covered by HA at the Host level. (in my 15+ years in the storage business I can probably count the number of Card failures on a Single hand. The number of Cable and or Switch failures is far greater, yet covered nicely by a dual port card.)
4- in ESX 3.x path Failover is “active/passive” and thus if you have 2 iSCSI HBA ports you can set them up to provide fail-over the same way you do with FibreChannel. (in the next gen of ESX you start to see a more Active/Active or Round Robin approach)
5- If you use the software iSCSI you can simply use more than one NIC in the vswitch where your vmkernel port-group is located and if you have multiple iSCSI seesions (highly recommened) then you can see both NICs end up used for I/O.
6- If you want to Boot from SAN with NO drives in the ESX host, then the iSCSI HBA is the way to go. (Note- USB enable ESXi has lessened the use case here.)
7- with ESX 3.5, Jumbo Frames is supported in Both the Hardware and Software init.
8- If you are lucky enough to be running the 10G Converged Ethernet Adapters then the choice is a no-brainer!
9- Make sure your Network is correctly setup, as this will severely impact the performance of both HW and SW iSCSI inits.
10- The only area where you will typically see the iSCSI HBA do a better job is in LARGE BLOCK, Sequintial I/O… Small Block Random I/O not so much. ( if you have a VM doing a lot of Large block seq I/O, you have a different question to address.) If it's a lot of Small Block Random with a heavy latency requirement, you really need to pay close attention to the disk system you are using, because what's at the end of that wire will really matter.)

In most cases I have seen the Software init do just fine, and customer are quite happy with it. The CPU % need to run is typically so small it just doesn't make an impact and with todays Quad / Quad core machines the % just keeps falling.

I've seen both SW and HW hit wire speed 125MB/sec.

In the end it's up to you.

Wade O'Harrow
Sr. Manager, Sr. VMware Specialists
EMC, Corp.
<—– Snip —->

<—– Snip —->

To help clarify-

TOE, or TCP Off-load Engine, is a chipset NIC manufacturors put on the Cards to Off-Load the TCP Stack as well as the IP Stack from the OS, to improve performance. TCP off-load requires special drivers in the server network Stack. Thus you will commonly see that Windows (Chimney) has this, while most Linux OS variants do NOT have this. ESX included.

iSCSI HBA, is a different animal entirely. the iSCSI HBA, off-loads the IP, TCP, AND iSCSI processes onto Special Chipsets. These cards can and do reduce the amount of CPU needed by the host. iSCSI HBAs are supported by ESX, the forementioned QLA4050c, and 4052c(dual port) both work very well.

To the krux of the question around Hardware iSCSI HBA or ESX Soft iSCSI, this is a completely different argument. Both have legit use cases and are highly dependant upon your architecture.

Here are a few things to be aware of-
1- if you use a QLA405Xc they do NOT count against your NIC limit in ESX< thus if you are using a lot of NICs and dont' have them to spare they are a good choice.
2- you can only use 2 hardware iSCSI ports. (you can put 2 Dual port cards in a Host, but it will only see 2 of the ports.)
3- on the subject of failover, some people opt for 2 4050c single port cards to allow for card redundancy. NOTE- this is 2 PCI slots which can get really valuable in an ESX host. If you have HA, SRM, and soon FT configured correctly then a Single Dual port card is a nice option since a card failure is covered by HA at the Host level. (in my 15+ years in the storage business I can probably count the number of Card failures on a Single hand. The number of Cable and or Switch failures is far greater, yet covered nicely by a dual port card.)
4- in ESX 3.x path Failover is “active/passive” and thus if you have 2 iSCSI HBA ports you can set them up to provide fail-over the same way you do with FibreChannel. (in the next gen of ESX you start to see a more Active/Active or Round Robin approach)
5- If you use the software iSCSI you can simply use more than one NIC in the vswitch where your vmkernel port-group is located and if you have multiple iSCSI seesions (highly recommened) then you can see both NICs end up used for I/O.
6- If you want to Boot from SAN with NO drives in the ESX host, then the iSCSI HBA is the way to go. (Note- USB enable ESXi has lessened the use case here.)
7- with ESX 3.5, Jumbo Frames is supported in Both the Hardware and Software init.
8- If you are lucky enough to be running the 10G Converged Ethernet Adapters then the choice is a no-brainer!
9- Make sure your Network is correctly setup, as this will severely impact the performance of both HW and SW iSCSI inits.
10- The only area where you will typically see the iSCSI HBA do a better job is in LARGE BLOCK, Sequintial I/O… Small Block Random I/O not so much. ( if you have a VM doing a lot of Large block seq I/O, you have a different question to address.) If it's a lot of Small Block Random with a heavy latency requirement, you really need to pay close attention to the disk system you are using, because what's at the end of that wire will really matter.)

In most cases I have seen the Software init do just fine, and customer are quite happy with it. The CPU % need to run is typically so small it just doesn't make an impact and with todays Quad / Quad core machines the % just keeps falling.

I've seen both SW and HW hit wire speed 125MB/sec.

In the end it's up to you.

Wade O'Harrow
Sr. Manager, Sr. VMware Specialists
EMC, Corp.
<—– Snip —->

iSCSI HBA's are supported… (Qlogic, etc)

TOE enabled NICs are not supported, as in, none of the TOE features will work, they will function just as any old NIC would.

iSCSI HBA's (Hardware initiators) are supported and work great and they do perform offloading from the CPU.

They are not all $1000 either, the Qlogic 4050c is a single port adapter that you can find for $600-$700. I have two of those in every ESX Host I have.

If you are referring to a TOE card to mean a TOE enabled NIC you will get no offloading per the KB article.
If you are referring to a TOE card to mean an iSCSI HBA you will get all the benefits of offloading.

1. The Q-Logic 40XX chipset (4022, 4052, and 4062 as of this writing) is the only TOE chipset supported today. I know there was a comment about blades above. That means the HP QMH4062 is the only iSCSI TOE card that will be supported with VMWare on HP Blades. More chipsets will be supported in the upcoming release of ESX.

2. DO NOT boot from iSCSI!! We have found that the pathing for ISCSI on ESX will not take a path hit like it will for FC. If you lose a path, even for a second, you will most likely PSOD the server if it is the boot path. Boot to a set of local drives and use the iSCSI (hardware or software) to go to the storage. The VMFS paths will take a hit fine.

1. The Q-Logic 40XX chipset (4022, 4052, and 4062 as of this writing) is the only TOE chipset supported today. I know there was a comment about blades above. That means the HP QMH4062 is the only iSCSI TOE card that will be supported with VMWare on HP Blades. More chipsets will be supported in the upcoming release of ESX.

2. DO NOT boot from iSCSI!! We have found that the pathing for ISCSI on ESX will not take a path hit like it will for FC. If you lose a path, even for a second, you will most likely PSOD the server if it is the boot path. Boot to a set of local drives and use the iSCSI (hardware or software) to go to the storage. The VMFS paths will take a hit fine.

But VMware ESX does support some iSCSI HBAs, which are TOE cards.
It supports the TOE function in conjunction with the full offload of
iSCSI, but ESX does not allow you to use the iSCSI HBA as an Ethernet interface. I.E. the TOE and even the Ethernet functionality is strictly limited to use of offloaded iSCSI, and other uses of the cards are not possible.

Most modern OSes _do_ support TCP checksum offload, and sometimes LSO (large segment offload), for compatible NICs with the feature.

I believe VMware supports at least the TCP checksum offload functions
of certain cards, also.

This is not the full TOE, but there are still speed enhancements to be found by picking the right vendor and model of NIC.

But VMware ESX does support some iSCSI HBAs, which are TOE cards.
It supports the TOE function in conjunction with the full offload of
iSCSI, but ESX does not allow you to use the iSCSI HBA as an Ethernet interface. I.E. the TOE and even the Ethernet functionality is strictly limited to use of offloaded iSCSI, and other uses of the cards are not possible.

Most modern OSes _do_ support TCP checksum offload, and sometimes LSO (large segment offload), for compatible NICs with the feature.

I believe VMware supports at least the TCP checksum offload functions
of certain cards, also.

This is not the full TOE, but there are still speed enhancements to be found by picking the right vendor and model of NIC.