Virtualizing high performance SQL - VMFS or RDMs?

If you are considering virtualizing SQL, check out a great post from vm0.blogspot.com. Running SQL Server on ESX makes some great suggestions for planning CPU, RAM, and Disk I/O to allow for the highest performance possible of your database virtual machines (VM).

The section on disk I/O made me think - specifically about whether creating RDMs to raw disks was really a better choice than carefully planning VMFS LUNs. There is just too much convenience with using VMFS in my opinion, and I’ve never really been convinced that any report or testing has proven RDMs offer better performance than carefully planned VMFS. So, instead of a long comment on the vm0 post I decided to blog about it here at VM /ETC.

Be sure to read the entire post for the RAM and CPU recommendations, but here is the section that inspired me:

“I/O

I/O is by and large the biggest concern
when running a database on an ESX cluster. Your main concern here is
going to be ensuring that your database files are not impacted by I/O
operations on your other VMs. We can ensure this through Raw Device
Mappings.

Generally, for a high-performance database, you’ll want your data files, log files, and program/OS/backup files totempdb all live on separate spindles. To accomplish this, we do the following:

Data Files - RAID5 or RAID10 (RAID5 is more economical but you will suffer a hit on write performance).

Log Files - RAID10

TempDB - RAID10

OS/Program Files/Backup/File Share - This can be run from a standard VMDK.”

But do you really need RDMs? What about multiple, but dedicated and separate LUNS on the same, larger RAID5 or RAID10 arrays? Give the SQL server it’s own separate VMFS LUNs to spread out .vmdks across. The multiple .vmdks controlled by different virtual SCSI adapters combined with the fact that each LUN has it’s own I/O queue should provide for more than sufficient I/O performance. Add as many disks as possible to the larger arrays so you increase your spindle count.

Sure, if you have such a big data warehouse that tweaking the .vmdks and VMFS as described above won’t help it then probably the next best thing to do would be to dedicate an entire ESX host just for that “monster SQL VM”. Actually, dedicate 2 ESX hosts so you can have a “Monster SQL VM ESX Cluster”. You’d be looking at spending the cash for a multi-node physical cluster anyways, right? This way you get the mobility and conveninence of VMotion and all the administrative benefits of a VM.

As always, I ask, YOU, the vmetc.com readers to chime in with your opinions, preferences, and experience with heavy I/O virtual infrastructure design.

Tags: esx, planning, RDM, SAN, sql, storage, vmfs

This entry was posted on Monday, December 8th, 2008 at 7:55 am and is filed under RDM, SAN, storage, vmware. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Robert

One of our clients (an SMB) was advised by NetApp to put their SQL data on an ISCSI lun mapped using the MS initiator rather than RDM or VMFS. Similar story for their Exchange data.

This probably has something to do with them using NetApp snapmanager for backups. I'm not really sure, not quite enough experience with SAN's yet and we're not directly supporting the SAN anyway.

vseanclark

I'm sold. I do this for Exchange and bigger SQL VMs already. So far 100% of my installations are SMB, so I still haven't seen a monster SQL server yet. I have seen plenty of SQL Servers that the customer thought was a monster, but after reviewing the Capacity Planner reports were kind of embarrassed (but happy to be virtualizing their "monsters"!).

Like Robert's comment mentions, the only downfall to the VMFS-everything strategy is the application-aware snapshot features of the different SAN vendors (EQL, NetApp, other..). But as soon as vStorage APIs mature, it sounds like that object will go away, too.

Good post! -Sean

Ian Reasor

Rich, are you suggesting one SAN LUN made to a VMFS volume per VMDK? My reasoning in using RDMs rather than VMDKs was to make sure to segregate I/O from data, log and tempdb volumes from any other I/O. I\'ve seen situations where one physical array was carved up into two LUNs dedicated to SQL and Exchange. The contention on those spindles was massive and I\'ve tried to avoid sharing spindles between SQL and anything else since then...

rbrambley

Robert,

I gave not run into any scenario where the MS initiator (inside a VM OS?) was the recommendation. That just doesn't seem like the best option, but I'd have to understand the whole scenario like you suggest

Sean,

I agree. Everyone thinks they have monster SQL servers. It is surprising what Capacity Planner reveals.

Ian,

I'm not suggesting that using RDMs is not a good option. In fact, the extra layer of I/O separation you recommend is ideal - whether RDMS or VMFS. But, justifying the extra disks, arrays and LUNs is always tough when you don't have the monster SQL VM. I'm also suggesting that the convenience of VMFS is also an important consideration - to me anyways. Finally, I agree that 2 LUNs dedicated for SQL and Exchange on the same array is not a good scenario.

Ian Reasor

Right on. The steps in that article were geared towards monster SQL servers. I run quite a few servers with lesser demands on VMFS and have done quite well. Thanks for the link, by the way!

anon

Came across this - VMware VMFS Vs RDM http://malaysiavm.com/blog/vmware-vmfs-vs-rdm-r...

vseanclark

Rich, Have you seen this white paper? http://www.vmware.com/files/pdf/performance_cha... - It's a performance comparison between VMFS and RDM and it shows nearly identical performance. Yes, RDM is slightly better, but I think convenience and simplicity of VMFS is a smarter move most of the time.

I blogged about this, too. So if you see 2 extra hits this week, that was me, baby! sorry if those are the 2 hits that send you packing for different hosting provider. ;)

Vo

Like Robert above, we've been advised by our Equalogic vendor to put the SQL data on an iSCSI LUN using the MS Initiator. This has some interesting implication when using VMotion, as the Vmotion'ed guest simply reconnects the iSCSI LUN rather than Vmotioning over the VMFS volume. We're told it's the preferred way to go. I haven't tried yet.

Vo

Cameron Gocke

I wonder if there are any comparisons on running a similar environment using VMDK files over NFS datastores. Our storage is a NetApp system and we use either iSCSI or NFS for our ESX environment. Right now all of our OS/Program Files are on the NFS datastores and we use iSCSI RDMs for data volumes. I've often wondered if there was any real benefit for me even bothering with the RDMs there since the published papers comparing NFS to iSCSI on NetApp for VMDKs has often favored NFS.

rbrambley

Cameron,

True. NFS performance has been proven to be as equal as iSCSI in most
scenarios. So, without knowing your exact application mix, I would say
that the only benefits you are getting from RDMs is possible SAN
monitoring or the VMs native OS file system features. Probably not
significant performance benefits. Not sure if those benefits are worth
the complexity of RDMs?

Cameron Gocke

Well, I am finally getting around to running some performance benchmarks against my storage. The results I'm getting, if accurate, seem pretty crazy to me. My main problem, however, is that I don't have a storage system that is not being used in production and so I worry that the production traffic is skewing my numbers even though all I really care about are the relative comparisons not the raw values.

Basically, the test I am running is using SQLIO to test the storage performance of a VMDK file stored on an NFS datastore, vs. an RDM accessed via iSCSI, vs. the MS initiator run directly from the Guest OS accessing an iSCSI LUN. On my NetApp all 3 are stored on the same aggregate of disks. Because I do have production IO hitting those disks, the only good way I could think of to eliminate skewed data from workload changes throughout the day was to run all 3 IO tests simultaneously.

The results that I am seeing are as follows:
The VMDK stored on the NFS datastore dramatically beats out the other 2 options hands down for 8KB sequential IO both for reads and writes. The VMDK stored on NFS datastore also demonstrates much stronger performance on random writes 64KB and larger as well as on sequential reads of all sizes. As you get larger sequential reads, however, the RDM and MS initiator start to close the gap.

The MS initiator outperformed the other 2 storage volumes only on 8KB random write IO and the difference was substantial. For some reason from 64KB random writes and larger the VMDK on NFS then performed much better.

For random reads of all sizes the performance of all three volumes was close enough to be insignificant.

The only category where the RDM sustained higher IO was for 8KB random reads, but only by a very small margin.

The latency on the VMDK over NFS was always substantially lower than the other 2 with the RDM and MS initiator generally being pretty close to one another.

The throughput for the VMDK over NFS was much better for random writes, but for all other IO the results were very close for all 3.

I don't know if I'd be willing to 100% stand behind these results, and I really wish I had a dedicated NetApp to run this test against, but I thought I'd post the results here anyway.

rbrambley

Cameron,

NFS performance is surprising isn't it? Thanks for posting your NFS/iSCSI/RDM test results here!

Ed Saipetch

One thing to remember with the storage decisions comes compatibility with things like Site Recovery Manager (SRM), Fault Tolerance and other things that people may consider using in their environment. Not all features of VMware are supported on all storage types. I believe SRM still isn't supported over NFS at this point in time.

rbrambley

Ed,

Great point about the "big picture". Things are changing though ....

Cameron Gocke

Yes indeed. In fact at the vSpehere 4 launch event the presenter indicated that SRM support for NFS is due to come out in the second half of 2009.

Virtualizing high performance SQL - VMFS or RDMs?

Related Posts

www.vmetc.com by Richard Brambley is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.