VM /ETC

One of the engineers from our storage team asked me to join a discussion about whether or not restoring VMs from images made by any of the live VM backup utilities would be a reliable process for a client. I unknowingly joined what was really a battle about whether the client would be better off using SAN based snapshots, host based replication, or live VM imaging specifically for Exchange and SQL VMs. All are proven solutions, and depending on budget, recovery objectives, and service level agreements all solutions could be deployed with confidence. The customer, and this post, is focused on the live VM backup solutions and their reliability for Exchange and SQL. Based on cost of implementation and infrastructure required the VM backup tools are the most affordable and easiest to deploy, but they also leave you with the most uncertainty.

Before continuing I want to point out that normal backup agents still need to be used for Exchange and SQL VMs. Not one of the VM backup solutions can perform application maintenance or specialized backups, like the Exchange Information Store backup for example. Understand that the live VM backup tools only specialize in capturing a quiescent, working image of the VM operating system and application.

First let’s understand what we mean when we say quiescent data. In the latest VMware Virtual Machine Backup Guide quiescing is defined as:

A process of bringing the on disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations such as flushing dirty buffers from the operating system’s in memory cache to disk, or other higher level application specific tasks.

Now we can examine how some individual tools address quiescing the VM.

VCB

VMware Consolidate Backup (VCB) handles application quiescing by using the vmtools SYNC driver installed on the guest OS, and by allowing you to configure custom scripts to process the application during the backup process. The Virtual Machine Backup Guide explains this process as follows:

Running Custom Quiescing Scripts

When you use Consolidated Backup, your virtual machines are quiesced when you start the backup process. You can also run custom pre freeze and post thaw quiescing scripts to create a quiescent snapshot of your virtual machine. For example, use the scripts to achieve application consistent backups in Windows virtual machines. Deploy and run the custom quiescing scripts inside the protected virtual machine.

When running the scripts, you can use the SYNC driver, an optional feature that you can install when installing VMware Tools. If installed, the SYNC driver holds incoming I/O and flushes all dirty data to a disk, making file systems consistent.

• Creating quiesced snapshots using the SYNC driver requires waiting for I/O to drain in the guest operating system. This might affect time sensitive applications, such as DBHammer.
• If you do not install the SYNC driver when installing VMware Tools, you can avoid the delay caused by the I/O draining. However, your snapshots will be crash consistent only, unless you provide custom quiescing through pre backup and post backup scripts in the guest.

The SYNC driver is supported only on the 32 bit versions of Windows XP, Windows 2000, and Windows 2003. Because it is not supported on Windows NT 4.0 or Windows Vista guest operating systems, file system consistency is not guaranteed in virtual machines that run these guest operating systems.

Personally, I’ve never done this. Honestly, I do not want to. That’s a lot of scripting and customization for each VM.

DBHammer is a SQL benchmarking tool – I think. I am not sure why the VMware authors used DBHammer as an example of a time sensitive application, but the point is understandable.

vRanger

vRanger can also use VMware’s SYNC driver (but it is disabled by default) and/or be configured to call Microsoft’s Volume Shadow Copy Service (VSS). From Microsoft’s VSS page:

The Volume Shadow Copy Service provides the backup infrastructure for the Microsoft Windows XP and Microsoft Windows Server 2003 operating systems, as well as a mechanism for creating consistent point-in-time copies of data known as shadow copies.

The Volume Shadow Copy Service can produce consistent shadow copies by coordinating with business applications, file-system services, backup applications, fast-recovery solutions, and storage hardware. Several features in the Windows Server 2003 operating systems use the Volume Shadow Copy Service, including Shadow Copies for Shared Folders and Backup.

For Exchange and SQL, VSS is possible because a “writer” is built in to the application for the VSS process. vRanger initiates a VSS backup during the VM backup.

Vizioncore explains more about using VSS in their vRanger Pro User Guide:

The VSS process [omitted] leverages Microsoft’s Volume Shadow Copy Service and Vizioncore’s own VSS Service. While VSS is integrated with supported MS Operating Systems, the Vizioncore VSS Service needs to be installed on each guest for which you wish to leverage VSS. This can easily be accomplished via the Install VSS section of the Configuration menu …

For more info check out Chapter 9 of the current vRangerPro User Manual (linked above) for an explanation of how VSS is used in detail.

esXpress

PHD Technologies, makers of esXpress, uses neither VSS nor the VMware SYNC driver as part of the VM backup, but still is able to achieve quiesced backups for applications like Exchange and SQL. In fact, the only place you will find any mention of using VSS on their web site is in the Feature Request topic of their Forums. In that Forum is a post about why using VSS has not been added:

We have looked at VSS and still have not determined it’s value in a virtual environment. While it is a much needed relief for backing up physical Windows hosts, we haven’t seen any outstanding feature that requires its use in a virtual world.

VSS is a backup solution designed for Exchange/SQL servers running on physical hardware, where it is almost impossible to make a completely consistant, restorable backup. This is due to the design limitations of Windows, where it is impossible to make a 100% backup of a host due to Windows inability to handle open files. But because of the architecture of the VMware environment, when running Windows as a VM this is not an issue.

And due to the superior design of our product, we have never in 3 years had a corrupt Exchange or SQL VM reported to us.

Further down in the same esXpress post is a comment that summarizes not only this single vendor’s solution, but the overall advantage of full VM backups in general:

Another customer did their DR practice last year. One guy brought back like 30 VMs in the afternoon, 12 of them were MS SQL servers. (He keep calling us to give us an update and to ‘look’ like he was working, so the other team would not ask him to help). The physical team worked 3 days non stop with 3-5 people and only restored 14 physicals. And none of their MS SQL [restores] actually worked. After the DR drill, the rule was that all MS SQL servers have to be VMs.

Summary

Generally, the best solution for recovery of your VMs like Exchange and SQL is a mix of both agent based backups and full VM backups. As mentioned earlier in this post, you can not perform application specialized backups and restores (mailbox, database, system state, etc.) with the full VM backup tools. Those backups are still a necessity and can only be accomplished by the agent. On the other hand, a reliable, weekly full VM backup insures you can have a quick recovery point for the entire server. After recovering to a recent VM image you can then restore the appropriate agent based backups to get as close to the point of failure as possible.

Although feeling uncertain about how reliable your Exchange and SQL backups will be is understandable, the full VM backup tools have proven to be a solid and dependable solution when configured correctly.

Related Posts

• VCB in a Virtual Machine and other product enhancements (7)
• Live VM backups from a VM – esXpress VBA (0)
• P2V file servers? Consolidate to a CIFS share instead (3)
• Things that make you go hmmmm – March 28, 2008 (0)
• Small business P2V migrations (10)
• Is Citrix XenServer cheaper than VMware ESX Server? (16)
• Backup VMs to CIFS and NFS Datastores with vRanger Pro and Datadomain (0)
• Things that make you go hmmmm – April 25, 2008 (3)
• XenServer integrates everRun VM for HA features (2)
• Domain Controllers – to P2V or not to P2V (4)