VM /ETC

rich,
At VMworld I asked that same question about SRM of the Dell Equallogic guys. They said that as long as you replicate the data LUNs along with the VMFS LUNs containing the protected VMs, then you are golden. They also thought that nothing needed to be done to present the replicated data LUNs, but I have my doubts. You would probably need to the use the SRM feature to add manual steps in the recovery/test plans like “Call storage guy to verify data LUNs to DR ESX servers”. An Equallogic collection could also be used to guarantee the VMFS volumes and the data LUNS were “snapped” at same time.

**Disclaimer: I have not tried this yet, but I did stay at a Holiday Inn Express last year.

on the contrary, would smaller datastores with fewer VMs make it easier to split traffic 50/50. SVmotion with larger Datastores could accomplish same goal of balancing traffic.

I would recommend a single, dual port iSCSI HBA or 2 single port iSCSI HBAs if the money is there.

@Tyler – You are correct. Mem and CPU are the only catalysts. There were posts on the forums about possibly expanding these to Disk and Network (which would be great) but I couldn’t find any additional information. Good Catch!

I wonder if a Maxed iSCSI pipe would bog down the vKernel which would hamper it’s ability to fulfill requests which would trigger a DRS event.. But that’s a streeetch huh?

According to this VMware KB article TOE features are NOT SUPPORTED!

http://kb.vmware.com/kb/1006143

Product Versions
——————
VMware ESX 3.0.x
VMware ESX 3.5.x
VMware ESXi 3.5.x Embedded
VMware ESXi 3.5.x Installable

Symptoms
———-
What is a TOE NIC and does ESX support any TOE NICs?

Resolution
———–
TCP Offload Engine or TOE is a technology used in network interface cards to offload processing of the entire TCP/IP stack to the network controller. It is primarily used with high-speed network interfaces, such as gigabit Ethernet and 10 gigabit Ethernet, where processing overhead of the network stack becomes significant. The term, TOE, is often used to refer to the NIC itself, although it more accurately refers only to the integrated circuit included on the card which processes the TCP headers.

ESX server does not support any NICs as TOE NICs as of this writing and we do not have any public commitments to support TOE NICs. If you find that some of the NICs listed under ESX HCL are TOE capable, those NICs will only be supported as regular NICs and TOE features available on those NICs will not be utilized by ESX server.

Isn’t 10Gb SW iSCSI supported with 10Gb NICs?

Re, the KB article.
This is no surprise — even Linux/Solaris don’t support TOE (full tcp offload), in general, these cards are not supported, except on specialized
platforms (Windows TCP Chimney, or open source OSes with major vendor-specific kernel patches).

But VMware ESX does support some iSCSI HBAs, which are TOE cards.
It supports the TOE function in conjunction with the full offload of
iSCSI, but ESX does not allow you to use the iSCSI HBA as an Ethernet interface. I.E. the TOE and even the Ethernet functionality is strictly limited to use of offloaded iSCSI, and other uses of the cards are not possible.

Most modern OSes _do_ support TCP checksum offload, and sometimes LSO (large segment offload), for compatible NICs with the feature.

I believe VMware supports at least the TCP checksum offload functions
of certain cards, also.

This is not the full TOE, but there are still speed enhancements to be found by picking the right vendor and model of NIC.

Hey Rich – A couple of quick comments to add for you. I have set up a TON of iSCSI both hardware and software based. We usually do it in the SMB market because a FC SAN will be too much. Sure, the iSCSI HBA are almost as much as a FC HBA, but what if you don’t already have the fabric in place? We have been setting up a couple of ESX servers in a cluster, hooked to the storage, all with iSCSI. Some things to remember:

1. The Q-Logic 40XX chipset (4022, 4052, and 4062 as of this writing) is the only TOE chipset supported today. I know there was a comment about blades above. That means the HP QMH4062 is the only iSCSI TOE card that will be supported with VMWare on HP Blades. More chipsets will be supported in the upcoming release of ESX.

2. DO NOT boot from iSCSI!! We have found that the pathing for ISCSI on ESX will not take a path hit like it will for FC. If you lose a path, even for a second, you will most likely PSOD the server if it is the boot path. Boot to a set of local drives and use the iSCSI (hardware or software) to go to the storage. The VMFS paths will take a hit fine.

Wade also just posted some really great points to add to this conversation.
http://www.vmwareinfo.com/2009/01/iscsi-hardwar…

<—– Snip —->

To help clarify-

TOE, or TCP Off-load Engine, is a chipset NIC manufacturors put on the Cards to Off-Load the TCP Stack as well as the IP Stack from the OS, to improve performance. TCP off-load requires special drivers in the server network Stack. Thus you will commonly see that Windows (Chimney) has this, while most Linux OS variants do NOT have this. ESX included.

iSCSI HBA, is a different animal entirely. the iSCSI HBA, off-loads the IP, TCP, AND iSCSI processes onto Special Chipsets. These cards can and do reduce the amount of CPU needed by the host. iSCSI HBAs are supported by ESX, the forementioned QLA4050c, and 4052c(dual port) both work very well.

To the krux of the question around Hardware iSCSI HBA or ESX Soft iSCSI, this is a completely different argument. Both have legit use cases and are highly dependant upon your architecture.

Here are a few things to be aware of-
1- if you use a QLA405Xc they do NOT count against your NIC limit in ESX< thus if you are using a lot of NICs and dont' have them to spare they are a good choice.
2- you can only use 2 hardware iSCSI ports. (you can put 2 Dual port cards in a Host, but it will only see 2 of the ports.)
3- on the subject of failover, some people opt for 2 4050c single port cards to allow for card redundancy. NOTE- this is 2 PCI slots which can get really valuable in an ESX host. If you have HA, SRM, and soon FT configured correctly then a Single Dual port card is a nice option since a card failure is covered by HA at the Host level. (in my 15+ years in the storage business I can probably count the number of Card failures on a Single hand. The number of Cable and or Switch failures is far greater, yet covered nicely by a dual port card.)
4- in ESX 3.x path Failover is “active/passive” and thus if you have 2 iSCSI HBA ports you can set them up to provide fail-over the same way you do with FibreChannel. (in the next gen of ESX you start to see a more Active/Active or Round Robin approach)
5- If you use the software iSCSI you can simply use more than one NIC in the vswitch where your vmkernel port-group is located and if you have multiple iSCSI seesions (highly recommened) then you can see both NICs end up used for I/O.
6- If you want to Boot from SAN with NO drives in the ESX host, then the iSCSI HBA is the way to go. (Note- USB enable ESXi has lessened the use case here.)
7- with ESX 3.5, Jumbo Frames is supported in Both the Hardware and Software init.
8- If you are lucky enough to be running the 10G Converged Ethernet Adapters then the choice is a no-brainer!
9- Make sure your Network is correctly setup, as this will severely impact the performance of both HW and SW iSCSI inits.
10- The only area where you will typically see the iSCSI HBA do a better job is in LARGE BLOCK, Sequintial I/O… Small Block Random I/O not so much. ( if you have a VM doing a lot of Large block seq I/O, you have a different question to address.) If it's a lot of Small Block Random with a heavy latency requirement, you really need to pay close attention to the disk system you are using, because what's at the end of that wire will really matter.)

In most cases I have seen the Software init do just fine, and customer are quite happy with it. The CPU % need to run is typically so small it just doesn't make an impact and with todays Quad / Quad core machines the % just keeps falling.

I've seen both SW and HW hit wire speed 125MB/sec.

In the end it's up to you.

Wade O'Harrow
Sr. Manager, Sr. VMware Specialists
EMC, Corp.
<—– Snip —->