VMFS Storage Sizing for Maximum Performance
Based on best practices, this post is a “formula” for planning and sizing SAN storage for maximum VMFS performance. This is intended to be for all protocols where VMFS volumes are utilized ( FC, iSCSI ) and conservative enough to allow general sizing estimates while ensuring high performance of the running VMs.
WARNING: The storage design resulting from following these recommendations will not be the most cost effective solution. As storage performance generally requires the most spindles (disks) possible and this formula requires as many dedicated LUNs as possible, the cost for storage will be maximized. More often than not, compromises between performance and cost have to be reached that keep the design within the expected budget.
This post is split into 2 sections. The first section lists the VMFS Maximum Performance Rules while the second section uses a 25 server example to walk through the design rules.
VMFS Maximum Performance Rules
- Use multiple RAID array types to increase VM performance and split VM .vmdks across different RAID types.
- A VM example could be:
- C: on VMFS lun1 RAID 5 array contains operating system
- D: on VMFS lun2 RAID 5 array contains file data and application code
- E: on VMFS lun3 RAID 1+0 array contains transaction logs and database
- Create only 1 VMFS Volume per physical SAN Volume
- Create less than 16 .VMDK files (virtual disks) per VMFS Volume
- Connect less than 16 ESX hosts per SAN Volume / VMFS Volume
- Heavy use VMs should get their own SAN Volume / VMFS Volume or use RDMs to minimize disk contention and locking with other VMs (frequent power cycles, frequent vmotions, constant utilization).
- Examples of heavy use VMs might be:
- Development VMs
- Messaging
- Database
- Heavy use VMs should only be on VMFS volumes accessed by a maximum of 8 ESX hosts.
- If the client wants to be able to create ESX snapshots of VMs double the needed (or useable) storage size of the VMFS volumes.
- Size the disks in each array to maximize the spindal count
Sizing Example: client with 25 physical servers
This is a small and simple sizing example to illustrate the Maximum Performance Rules. Larger environments will be much more complex, but the following server table provides a realistic introduction on applying the rules to help design VMFS storage.
Example Client Physical Server Table
- Split VMs with multiple drives into multiple .vmdks across different RAID arrays
- Create 1 VMFS volume per physical SAN volume
- Volumes Needed
- Create less than 16 .vmdk files per VMFS volume
- Volumes Needed (using 16 VMs or .vmdks max per volume)
- Connect less than 16 ESX hosts per VMFS volume - In this example the customer will use 2 or 3 ESX servers so this rule is not applied
- Heavy use VMs should get their own VMFS volumes
- Messaging VMs - put c:, d:, and e: drives on separate volumes
- File VMs - put d: on separate volumes
- Heavy use VMs on LUNs accessed by less than 8 hosts - does not apply because we only have 3 ESX hosts max in this example.
- If customer wants to make ESX snapshots of .vmdks double the needed minimum VMFS Volume sizes - customer does not want to use snapshots in this example
- Size disks to maximize spindal count of the arrays - TBD by storage team?
The above example would need the following SAN VOLUMES (to be formatted as VMFS):
Related Posts
Tags: esx, SAN, virtualization, vmfs, vmware
June 11th, 2008 at 3:43 am
Hi Rich, thanks for your short design guide. Your recommended limitations for 16 resp. 8 hosts/volume intends to create separate clusters for specific types of VMs, am I right? If the environment is large enough in total, of course.
BR
Steffen
June 11th, 2008 at 5:33 am
Steffen,
Yes, the host limitations will result in multiple ESX clusters. Hopefully you will be able to combine VMs with different applications within the clusters to maximize DRS.
June 11th, 2008 at 7:49 am
I forgot to offer some supporting links. Here’s a good start:
http://communities.vmware.com/thread/147365
June 12th, 2008 at 10:00 am
[...] post begins to detail one alternative plan to my design - VMEtc No Comments so far Leave a comment RSS feed for comments on this post. TrackBack URI [...]
June 12th, 2008 at 10:07 am
Now I have to respectfully disagree with some of the assumptions here.
I would be impressed if you were able to demonstrate an I/O level where a 15 spindle Raid5 array (even Raid6 I bet) can’t handle the VM load from multiple LUNs spaced across the volume.
My environment is different as less than 10% of the systems are “production” and the rest are rapid build engineering and testing platforms but when we run scalability testing it does put some pretty significant load across the VMs AND my users are snapshot crazy.
My design looks more like this - two primary clusters with HA/DRS (5 Dell 4u’s 32GB ram+ with ~400 machines per cluster) hooked up via 4GB FC.
I can’t imagine a situation where Raid5 isn’t a good option (go with Flash Drives if the I/O is that high) but the real question we should be addressing is the use of SAS drives vs SATA drives in modern arrays. On that question I don’t have good answers yet. I am just starting to use SATA in our production environment and have good results so far.
June 13th, 2008 at 6:15 am
John,
I do not disagree with you. In fact, I end up using multiple LUNs per array just as you describe more times than not. You illustrate a great point - you have to know your disks, storage device, and even the protocol (FC or iSCSI) you use to find the most efficient performance design. This guide is intended to be a general starting point for any scenario.
June 17th, 2008 at 10:03 pm
[...] Brambley over at VM /ETC is on a roll with a couple of good articles, one on VMFS storage sizing for performance and one on downgrading the VM HAL after P2V. The title for that second article was “How to [...]
June 24th, 2008 at 7:14 am
[...] Bramley has written a very good article about VMFS storage performance advisories. Some of them are the kind of things you would do if [...]