vsphere_static_160x300
Badges

vexpert_logo_100x57

gestaltitbadge

follow-me-twitter

Subscribe to me on FriendFeed

Comments / DISQUS
Tag Cloud
Feedjit.com
«
»

Reasons For Using NFS With VMware Virtual Infrastructure

PostDateIcon May 1st, 2009 | PostAuthorIcon Author: Rich Brambley

A lot of companies are using NFS as the preferred protocol to shared storage for VMware Virtual Infrastructure. In my personal experience, The administrative options and convenience of NFS is unmatched, and the virtual machine (VM) performance is surprising.

For example, I recently helped migrate a company from ESX 2.X to new a installation of VI 3.5. Since the client did not have any additional space available on their fiber channel (FC) SAN for a new VMFS3 volume, we temporarily used a Windows Server 2003 R2 NFS share to host 2 dozen VMs until the existing FC volumes could be rebuilt and reconnected. The customer actually ran their production environment for 2 weeks in this configuration and was experiencing better performance. Newer hardware for the ESX hosts also contributed to this increase, but the point is that the NFS storage was not a bottleneck.

For those that are considering NFS, I was recently forwarded a list of links that provide sound arguments and testimonials on the unique advantages of using NFS with VMware. Although the published date of some of the posts that are referenced might be a bit dated, the content is still valid. Here is the list with quotes from some of the posts, but be sure to read the all in full for more information.

Dave at NetApp says:

“Since a VMDK is a virtual disk, I had assumed that block-based protocols like iSCSI and Fibre Channel would make more sense than NAS, so I asked several customers why they prefer NFS.

The answer is simple: Managing .vmdk files is much easier than managing LUNs. If you have 20 or 30 virtual machines, then VMFS is great for consolidating the VMDKs into a single LUN. But NAS is much easier and more scalable if you have hundreds or thousands of virtual machines.

The big advantage is that you can use all your file management tools. Group the VMDKs for Exchange servers in one folder, SQL servers in a second, virtual desktops in a third, and so on. Instead of backing up LUNs or virtual machines individually, simply backup a directory tree of VMDKs all at once. (This is much less expensive than buying a backup license for each virtual machine, and also easier to manage.) For disaster recovery, you can replicate the data for a whole group of virtual machines as a single unit.”

http://blogs.netapp.com/dave/2007/09/why-run-vmware-.html
—————————————————————————–
Chuck at EMC says:

“I think that, in the long term, we’ll find high-end NAS much more friendly for high-end VMotion / DRS farms than today’s SANs.  And I think that NAS has the potential to offer a few benefits that we might not find in the SAN world.

Not only does NAS deliver a flat name space that will probably make implementing VMotion far easier, you get other potential benefits that might not be obvious:
• You get to manage a file system, rather than a collection of LUNs
• You get some modicum of access control through the file system mechanisms
• You get access to advanced NAS features, like thin provisioning, snaps, replication, etc.”

http://chucksblog.typepad.com/chucks_blog/2006/12/vmware_virtual_.html
—————————————————————————–
Why VMware over Netapp NFS
http://viroptics.blogspot.com/2007/11/why-vmware-over-netapp-nfs.html

“NetApp NFS advantages:

  • You get thin provisioning by default with NFS. FC and iSCSI VMDKs are thick. This can save 50% of your disk space.
  • Adding NFS datastores are simple. Mount the NFS volume using the GUI and start creating VMs
  • Adding additional Netapp filers for datastores requires no down and no cabling changes.
  • You can have large datastores that span many disks. 16TB for Netapp.
  • You can use A-SIS to de-duplicate your datastores for a 50-80% reduction in disk space
  • You can expand AND decrease NFS volumes on the fly
  • You can use snapshots of the volumes to restore individual VMs
  • You can use snapmirror to backup VMware volumes to a DR site over a WAN
  • You don’t have to deal with VMFS or RDMs
  • You don’t have to deal with FC switches, zones, lun sizing, HBAs, and identical LUN IDs
  • You can restore multiple VMs, individual VMs, or files within VMs.
  • You can instantaneously clone (Netapp Flexclone), a single VM, or multiple VMs
  • You can also backup whole VMs, or files within VMs using NDMP or any other backup software
  • ESX server I/O is small block and extremely random which means that bandwidth matters little
  • No single disk I/O queue, so your performance is strictly dependent upon the size of the pipe and the disk array.
  • Failover to your SnapMirrored copies can be done in minutes. iSCSi/FC requires LUN resignaturing.
  • You can clone a single VM or create 100’s of VMs from a template in seconds”

—————————————————————————–
Why NFS?
http://communities.netapp.com/message/8904

  • Deduplication – possible to use deduplicated space savings with LUNs but MUCH more complicated (have to mess with fractional reserve, LUN thin provisioning, etc. — possible to get caught overprovisioning and have real issues)
  • VMware Datastore sizing — easy datastore growth (possible with VMFS) and shrinking (not possible with VMFS)
  • Larger datastores – no need to keep datastores smaller like with VMFS – up to 16 TB
  • Snapshots – can retrieve individual vmdk’s from snapshots and/or mount vmdk’s from snapshots for single file restore
  • SMVI – main benefit is ability to do faster VM restores (uses SnapRestore rather than LUN clone so can instantly restore a single VM to any previous snapshot)
  • VMDK Thin Provisioning
  • Ease of addition – somewhat easier than LUNs/VMFS
  • VMFS/RDMs – no need to deal with them
  • Single-file FlexClone (future feature) – can clone a vmdk instantly for fast provisioning
  • No single disk I/O queue as with iSCSI/FC so performance limitations are purely governed by pipe size and disk array size.
  • Faster failover to SnapMirror remote copies (less steps plus faster steps) – no need to do LUN resignaturing
  • ESX server I/O is small block and extremely random meaning that bandwidth is less important (i.e. GigE works well).
  • Can dump individual VM’s via NDMP
  • No FC zoning, switch cost, HBA’s, compatibility matrices, or LUN IDs

—————————————————————————–

You should also check out the following NFS related posts:

Pros and cons of using NAS NFS with VMware
http://searchstorage.techtarget.com/tip/0,289483,sid5_gci1351138,00.html?track=NL-57&ad=695051&asrc=EM_NLT_6173872&uid=8231339

VM File-Level Recovery with NetApp Snapshots
http://blog.scottlowe.org/2007/10/08/vm-file-level-recovery-with-netapp-snapshots/

Full VM Recovery with NetApp Snapshots
http://blog.scottlowe.org/2007/10/08/full-vm-recovery-with-netapp-snapshots/

VMware over NetApp NFS: A Customer’s Testimonial
http://blogs.netapp.com/storage_nuts_n_bolts/2008/01/vmware-over-net.html

Thanks to my Softchoice colleague Charles Tyler for providing this list. Charle’s expertise on NFS and all things NetApp is well known here in the Atlanta area. Follow Tyler on Twitter or watch for his occasional helpful comments on NetApp related blog posts!

Related Posts

PostCategoryIcon Posted in storage, vi3, vmware | PostTagIcon Tags: , , , , ,
«
»
  • Jason,

    Thanks for the large enterprise NFS example and volume sizing insight. The administrative convenience allowed by NFS is really amazing - on multiple levels. NetApp NFS takes the protocol to another level with performance, backups, and replication. Filer snapshots (along with replication bandwidth / data trabsfer) is definitely a good reason for still splitting the VM pagefiles from the .vmdk folders.
  • At my new environment, we are a very large NetApp shop. Much of our VMware VI runs on NetApp and we are moving our VMs from fibre channel SAN (EMC and Hitachi) to NetApp NFS on daily basis. The reasons for running on NFS are all outlined above. Our VMKernel swap is on NFS albeit separated from the VMs because we don't want swap snapped. Waste of disk space. By next year we'll have 6,000+ VMs. Probably 80% or more of running on NetApp NFS.

    Volume sizing of NFS versus SAN is quite different. With SAN we'll typically have somewhere around 500GB to 1TB LUNs. NFS volumes are 3.4TB. Quite a decrease in the amount of datastores to manage when they are that large. You wouldn't dare do that with SAN due to being forced to use extents (I still say avoid them like the plague) and you just don't put that many VMs on a single LUN unless you can guarantee low disk I/O or the VMs are large such that only 10 or less are going to fit on a 2+TB SAN volume anyway.
  • buckmaster
    Keep up the great work. Your site is the best, chalked full of USEFUL information.
  • Paul
    Thanks for the info! Seems the "new" best practice hasn't made it down stream yet to the support staff.
  • Paul
    Has anyone else experienced VMWare's "Dirty Little Secret" about NFS? We've just deployed a large environment using Netapp, we have years and years of NFS experience with Netapp so the fit was a match made in heaven, or so I thought.

    We started to experience random VM crashes and other odditys with VMotion. Opened a ticket with VMWare to be told that they recommend we run our "swap" on iSCSI! Apparently this as been the recommendation since VMWorld 2006! 2006!

    Sure, out Netapps can do iSCSI without breaking a sweat, and we have other platforms, running iSCSI (and even FCP), but the goal with this platform was to keep it simple and run only NFS - A configuration that both VMWare and Netapp stressed was "fully supported" when we were engineering the solution.

    Fast forward to now and we're now running hybrid iSCSI/NFS, and barely anyone at VMWare will give us any clue as to why this is required.

    Has anyone else faced this issue? I call this VMWare's "Dirty Little Secret" since they say NFS is supported, but if I have to run iSCSI too, I hardly call that supported. A few Google searches show that others have been told the same thing, I don't know about anyone else but this is pretty unacceptable to me.

    Paul
  • Paul,

    Splitting the VM swap files from the volume on the NFS share is no longer the recommendation from VMware. Check out this VI:OPS discussion: http://viops.vmware.com/home/message/1244#1244. In it Paul Manning from VMware clarifies the reason why it was previously recommended, and explains why it is now not necessary. Here is his comment from late November 2008:

    "Re: Using NFS as a data store.

    The current best practice for NFS is to not seperate the VM swap space from the VMhome directory on a NFS datastore. The reason for the originial recommendation was just good old fashioned conservitiveness. The thought was that if the IP traffic slowed the response time for swap space access that it could have a significant impact on the performance of the VM. However, the performance conern was not an issue that made this step of placing swap on antoher storage device a needed step.

    Further, it turns out that it is a more simple solution to address the concern of performance degredation is to not over subscribe the memory of the VM. However the performance of the access to NFS storage compared to FC storage for swap space is not considered to be a significant enough delta that would make the separation of VM swap from the VMhome worth it.

    So even though the KB article 1004082 hasreference to separating them, it is not longer considered best practice. And that KB article is slated to be updated to be consistent with our current best practice of keeping swap in the VMhome Directory for VMs on NFS datastores."
  • Sam,

    The biggest issues I can think of with using the same ip subnet for the ip storage and your production traffic is broadcasts and security.

    The NAS/NFS network interfaces will be listening and participating in all the noise between your clients and servers. This could reduce the performance of your ip storage.

    If your ip storage is on the public network, then you also need to take extra steps to harden the transmissions for security.
  • sampowers
    Some people say we should segregate our storage traffic on another vlan or even physical switch. But we don't. The NFS traffic runs over the same vlan on the same switch (a Cisco 3750) as several other types of traffic, including CIFS access to one of our filers and http traffic to several of our servers.

    I don't see a problem with this as NFS is just point to point TCP traffic, and the extra traffic on the vlan shouldn't be any cause for alarm, but are there any downsides to this approach?
  • CT,

    No really, thank you!
  • Charles Tyler
    Thanks Rich. Here's a story about AutoTrader moving from iSCSI to NFS on their NetApp storage.

    http://searchstorage.techtarget.com/news/articl...

    For the published topology performance comparison numbers, take a look at the "Multiprotocol Performance Test of VMware® ESX 3.5 on NetApp Storage Systems" doc here:

    http://www.netapp.com/us/library/technical-repo...
blog comments powered by Disqus
Hyper9 Cowabunga
Support VM /ETC
Support VMETC.com

Support VMETC.com

Free Business and Tech Magazines and eBooks
@rbrambley tweets
Advertisements
Facebook
VMETC
VMETC
Promote Your Page Too
VMTN Roundtable Podcasts
Subscribe



Add to Google Reader or Homepage
Subscribe in NewsGator Online
Add to netvibes
Add to Plusmo

License
Creative Commons License
www.vmetc.com by Richard Brambley is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

Contact Us | Terms of Use | Trademarks | Privacy Statement
Copyright © 2009 VM /ETC. All Rights Reserved.

Powered by WordPress and WordPress Theme created with Artisteer.