VM /ETC

Rich,
Maybe you know this already (in which case I comment for the benefit of you google-ers out there that just found VMetc.com for the first time) but with the Hardware-based iSCSI, you can now get beyond the 1 Gig limit for VMFS datastores by assigning different LUNs to different HBAs.

With the software iSCSI initiator you limited to 1 Gig of storage bandwidth per ESX hosts, unless you complicate the setup and use guest-based initiators (which is not really a bad thing compared to SAN features you can now use if the data is on a native SAN LUN).

I’m going back to HW iSCSI for these reasons since it will give me the best of both worlds and prices should drop now that there is plenty of competition in the market.

-Sean
Twitter: vseanclark , b/c I’m too lazy to update my blog

I’m wondering why Dell PERC 6/E Adapter and HP SC44Ge HBA Adapter are on the list. They are SAS HBA adapters, and have hothing to do with iSCSI…

Correct me if i’m wrong.

Best Regards
Andrzej

Our primary VMware environment is a pair of HP Bladecenters, and it’s been difficult to track down Blade HBAs. HP doesn’t make their setup very clear as to whether their own NICs can double as an HBA, or whether the more expensive Qlogic hardware is required. We don’t have terribly high I/O VMs just yet, so it’s not too big of a concern, but I think it’s a lack of clear direction on the Blade side that’s keeping us from making the investment.

Great follow up post Rich. Sean brings up a good point with the 1 GB limitation of the SW iSCSI. Thanks to vMotion and DRS though, if the disk writes/reads are not being satisfied, VI will ‘move’ the VMs to another host. Of course only good if the reason for the delay was due to a collective request as opposed to the individual VM choking the pipe.

Sean,
Were you experiencing bottlenecks in your environment which is forcing you to move to the HW cards or are you just trying to stay ahead of the curve?

CARLO.

@Carlo
Customer might be experiencing a bottleneck soon as they are moving 400 user Groupwise installation (physical) to a virtualized Exchange 2007. They are on SW iSCSI now and I warned them that if we run into a bottleneck on VMDKs, that we’ll want to migrate a guest-based initiator to effectively use a different path to the storage than the VMkernel port is using.

Well, they did not like that idea at all because that takes away the very comfortable and easy to slice and dice VMFS option they are used to. They are small shop and hate having to enter the Celerra interface to do anything, let alone have to learn about guest-based initiators.

Soooo, I did more reserch into getting more than 1 NIC active with SW and hit brick wall. Only way to get more than 1 Gig of iSCSI traffic to VMFS volumes is to go HW iSCSI. So if their Exchange implementation and further VM growth, ends up going beyond 1 Gig of bandwidth for an ESX server, we will likely insert HW iSCSI adapters and pull 1 of the 2 quad port NIC cards..

Can’t wait till 10G becomes affordable.

Sean

The way I’ve always looked at it is that if I’m going to drop a thousand dollars on each of my servers for an HBA, it might as well be fibre channel. iSCSI is an option I usually exercise due to cost constraints…

@Carlo

Are you sure? I was under the impression that DRS is only going to intervene based on CPU & Memory and not Disk / Network. I could understand that if a VM was paging to it’s vSwap that DRS may step in but I don’t believe it works that way (although I could be wrong… happens on more than I’d like to admit)

-Tyler

Sean,

Good point about the 1 Gb bandwidth limit per LUN. But to use that design wouldn’t you require a few large VMFS LUNs with 12 to 16 VMs on each? How many iSCSI HBAs would you put in each ESX server? That becomes a problem for SAN replication or SRM designs, for example. Also, aren’t we crossing that grey line in the VMFS where NFS makes more sense too?

Andrzej,

I did a search for “iSCSI” using the compatibility guide web application, and to be honest I did not pay close enough attention to the results before I cut and pasted. It looks like VMware may have them both labeled wrong as type “iSCSI”. Thanks for pointing this out.

Carlo, Sean,

Great discussion, and yes I agree that 10gb ethernet will change the design pros and cons once again ..

Nick, Ian,

The cost of the hardware iSCSI adapters really do make you think twice. Of course, you have to have already implemented or licensed a FC SAN, so the $1k price tag is really just the “tip of that iceberg”

@Ian If you have the existing fibre channel infrastructure, I agree completely.

But if it’s a green field, iSCSI should be the 1st stop for small businesses looking at VMware. It’s in their budget and the performance worry can be alleviated if I can get closer to 2GB throughput per ESX host with HW iSCSI (and more with use of guest-based iSCSI initiators).

That being said, nothing beats the simplicity of VMFS on fast FC storage. Set it and forget it. Maybe add some more disks, then forget it again.

rich,
At VMworld I asked that same question about SRM of the Dell Equallogic guys. They said that as long as you replicate the data LUNs along with the VMFS LUNs containing the protected VMs, then you are golden. They also thought that nothing needed to be done to present the replicated data LUNs, but I have my doubts. You would probably need to the use the SRM feature to add manual steps in the recovery/test plans like “Call storage guy to verify data LUNs to DR ESX servers”. An Equallogic collection could also be used to guarantee the VMFS volumes and the data LUNS were “snapped” at same time.

**Disclaimer: I have not tried this yet, but I did stay at a Holiday Inn Express last year.

on the contrary, would smaller datastores with fewer VMs make it easier to split traffic 50/50. SVmotion with larger Datastores could accomplish same goal of balancing traffic.

I would recommend a single, dual port iSCSI HBA or 2 single port iSCSI HBAs if the money is there.