How Long Would It Take To Troubleshoot And Recover?

Put yourself on the team that suddenly realized 88 VMs were gone! Where would you start? The storage  jumps out at me as a logical place to begin, but after your storage area network  is online, healthy, and normal then what? It’s time to try to crack open the VMware Black Box and scour event logs, alarms, permissions, and actions. Put that aside for a minute and think about how would you start the rebuild process and get the business reconnected!?

I don’t have an easy answer. My goal is asking you to think about this for yourself.

Warning! The Veeam Pitch

Since I work with Veeam products every day I’ll briefly suggest how they could help in this scenario. Decide for yourself what tools are best for your company. I’ll point out that

maintaining physical servers for monitoring and management, or a separate, limited access cluster of VMs would have to have been the implemented design to have helped here.

Veeam Monitor

Since the database is independent of vCenter a duplicate record of all events and actions would kept. Deleted VMs would remain in the GUI as italic, grayed objects, but there associated data would be kept. You can easily see the VMs “last few breaths”.

Veeam Reporter

vCenter permissions reports could be scheduled for delivery to the HR department and reviewed regularly. This betters the chance that the dismissed admin’s credentials would be removed from the virtual environment and not overlooked.

Configuration change reports would show what account was making changes, what properties were changed, and what the old and new values of those changed properties were/are.

Various reports would document the entire virtual infrastructure design and allow a rebuild as quick as possible. Everything from VM IP Addressing to datastore and vSwitch assignment would be easily available.

Veeam Backup and Replication

Instant VM Recovery would allow VMs to run from the backup files. In the time it takes to run the wizard and boot the VM(s) user’s would be reconnected to the guests. Migrations back to production storage could take place immediately or delayed and staggered in tiers as needed. If you have a DR site then failing over to replica VMs would also be an option.

Let’s call these new generation of projects V4DR (virtualization for disaster recovery) or V4BC (virtualization for business continuity).

The comparison

If I rewind 3 to 5 years ago in my career, capacity planning for server consolidation was a weekly project and topic of discussion. Customers were either in the process of converting physical servers to virtual machines or they were exploring the possibility to do so. In both cases, capacity planning scenario spreadsheets and reports were frequent “ground zero documents” to almost every project plan I was involved in.

Just like P2V projects, VM replication today also requires some of the same considerations for job scalability and times to complete – i.e. using multiple hosts as targets and making sure the network can support getting the job done as quick as possible. Not to mention ip addressing, VLAN assignments, and application connectivity after the fact. Thank goodness we no longer have to deal with hardware drivers and other unneeded software a second time. Hopefully, VM alignment is a thing of the past too!

I’m not seeing the same “ground zero documents” for replication projects, however.

Use the same capacity planning tools?

So, I’ll ask the question: 

are the existing capacity planning tools we have used previously for our P2V migrations good enough to help today with the V2V replications needed for our new V4DR and V4BC?

In theory they should be: monitor the virtual machines for several weeks and then determine the capacity and resources needed at the DR site. Leverage the capacity planning scenarios in these tools for reports like “What If I lose a Host?” or “What If I add X number of VMs?”.

Replication Capacity Planning

My point is, I think most look at today’s capacity planning tools as either no longer useful since the migration is over, or useful just for primary site capacity purposes. Looking at the output of these tools a little differently could also provide secondary site planning. A report like “What if I lose a host?” could also be viewed as “Can I consolidate my VMs on even less hosts for DR?”. “What if I add X number of VMs?” could also be viewed as “How much capacity do I need to purchase at my DR site?”.

Is what we have today good enough to be reused , or do we need a whole new version of capacity planning and virtualization assessment tools? Call the new tools Cloud / Hosting / DR/ BC / Replication Capacity Planners? Take your pick!

I’m curious. What tools that exist today do you think are already providing output that is useful for V4DR or V4BC planning? Leave a comment!

disclaimer: I work for Veeam Software. Veeam Backup and Replication is a VM replication solution. Veeam Reporter has Capacity Planning reporting capabilities.

disclaimer: I am a Systems Engineer for Veeam Software

Now, with VBR version 5 a FLR Appliance can be activated, auto registered, and configured for file level restores. Although the FLR appliance is only used when you select “Guest Files (other OS)” from the Restore options, the Linux based appliance VM can mount NTFS and most other non Windows file systems. By choosing to use this appliance you have the option to enable FTP access to the restored .vmdk thus making the recovered file system(s) easily available from any desktop. Therefore, end users or application owners can just open Windows Explorer, their favorite web browser, or a FTP client to browse, find, and download their recovered files themselves. No additional software needs to be purchased, and if you use the native OS web and file browsers nothing else even needs to be installed.

This means that the only time required by the VMware / Backup admin is the few minutes it takes to click through the Veeam restore wizard and verify the FLR appliance VM is running. Of course, a few extra seconds is required to click to close the FLR GUI when the user is done.

The rest of the post shows the many screen shots of using and configuring the Veeam FLR Appliance Restore option and a few simple, end user alternatives for accessing the restored files from the remote system. 

Settings / Using FLR Appliance To Restore Files

Click the Restore Button in the GUI and use the “Guest files (other OS)” option both for Windows and Other OS VMs

Please note that I have purposely skipped a few screens in the FLR wizard.

Select VM to recover files (I am selecting a Windows 2003 domain controller in the screenshot)

Select rollback point to restore from

Configure FLR Appliance with Customize button (bottom right corner of the Completing the Multi-OS FLR window)

Set Host, Resource Pool, VM Network, and choose between Dynamic or Static IP address. Check box at bottom to enable FTP server.

Complete Wizard and FLR Appliance automatically boots.

For me it takes less than a minute for the FLR appliance VM to be ready to use

The File Browser pops up on the Veeam Server desktop. This is where the VMware/Backup Admin can normally restore the file via a right-click “copy to” command.

Give the User the IP Address or DNS Record

Now you are ready to tell your end user to access the ip address of the FLR appliance for their self restore. If you choose to use a static ip address, you can create a new DNS record such as “self restore.yourdomain.com” mapped to the FLR Appliance VM to give to your users. The FLR Appliance will be available via this DNS name every time it is used.

Windows FLR User Self Restore Options

With Internet Explorer (or any other browser)

With Windows Explorer (Start/Run/”ftp://[FLR appliance ip address]

With FTP Client

Non Windows FLR User Self Restore Options

FTP with a Terminal / Console

ftp with Filezilla client (no username or password needed)

ftp:// with Firefox (or any browser)

That’s it! Tell the user to let you know when they are done, and then shut down the file browser on the Veeam server and the FLR appliance will auto power off and unregister from your VMware environment.

Just in case you are wondering, the FLR appliance is in a temporary read and write mode while operational, but any changes to the files on the mounted .vmdk are not permanent. Your backed up VMs do are still the same restore point when you are finished.

For a great overview of the entire U1 release check out Rick Vanover’s post vSphere 4.1 update 1 released from his Rickatron Blog and via his Servers and Storage Column/Blog at TechRepublic.

The rest of this post contains cut and pastes from the Release Notes and some commentary about them from me. I want to stress again that these are issues that have now been fixed!

Finally, I’ll point out the one huge VM backup issue (that I can think of right now) that still does not appear to be resolved.

• Cannot take quiesced snapshots of Microsoft Windows Server 2008 R2 virtual machine running vCenter Server 4.1
  When creating a snapshot of a Microsoft Windows Server 2008 R2 virtual machine that has vCenter Server 4.1 installed, the snapshot operation might fail to complete. This issue occurs on Microsoft Windows Server 2008 R2 virtual machines when the ADAM database is installed. The issue is resolved in this release.

The change from 2008 to 2008 R2 has caused quite a few applications to stumble, so it is good to see that VMware users can now install vCenter on the latest and greatest OS from Microsoft running as a virtual machine without having to worry about various hassles, not to mention coming up with a separate backup or business continuity solution for arguably one of the most important VMs in the environment.

• Creation of large .vmdk files on NFS might fail. When you create a virtual disk (.vmdk file) with a large size, for example, more than 1TB, on NFS storage, the creation process might fail with an error: A general system error occurred: Failed to create disk: Error creating disk. This issue occurs when the NFS client does not wait for sufficient time for the NFS storage array to initialize the virtual disk after the RPC parameter of the NFS client times out. By default the timeout value is 10 seconds. This fix provides the configuration option to tune the RPC timeout parameter using the esxcfg-advcfg -s <Timeout> /NFS/SetAttrRPCTimeout command.

Although specific to users of NFS storage, this one was definitely a head scratcher. Possible scenarios where this problem could rear it’s ugly head include full VM restores and even VM replication. Add the possibility of dedicating a large .vmdk as the backup repository for your backup server VM.

Let me know if I missed any other resolved issues specific to VM backups.

The issue still unresolved

Unfortunately, nothing in vSphere 4.1 U1 appears to address the loss of connectivity when committing snapshots for VMs using Change Block Tracking (CBT) while running on NFS storage.  This leaves VMware admins stuck between a rock and hard place – disable CBT or move VMs off of NFS. More about the problem can be found in VMware’s KB Article:

Virtual machine freezes temporarily during snapshot removal on an NFS datastore in a ESX/ESXi 4.1 host

Other Related articles

• VMware vSphere 4.1 Update 1 released (edugeek.net)
• vSphere 4.1 update 1 released (rickvanover.wordpress.com)
• Virtualization And Backup: VMs Need Protection, Too (informationweek.com)

From the post on the Virtualize Planet Blog: Replication Bandwidth Calculator | Virtualize Planet

ReplicaCalc

How many times do you get asked “how do I work out if VM Replication will work with my internet link” Well I wanted to demonstrate some way of providing a calculator without working it out in my head every time. So I made a Replication Calculation tool.  It is assumed that you provide it with 3 values:

1. Average Rate of Change.
2. The Link speed – this value should reflect the upload speed at the source site or the download speed at the target if this is less. So for example if the upload at the source is 6Mbs and 10Mbs download at the target then go for 6Mbs
3. Bandwidth % – which is the amount of bandwidth as % which achievable from the link speed specified.

Download and install ReplicaCalc from here > ReplicaCalc

Ricky describes the needed values in more detail in his post so be sure to read it all there. He also demonstrates how to use Veeam Backup and Replication to get the VM dynamic rate of change. The tool is useful whether using Veeam to replicate your VMs or not, however.

Thanks Ricky!

For more on this topic, I’ve blogged about using a bandwidth calculator to figure out whether your current WAN link is adequate or not for your replication jobs before.

I’m automatically pulling in the tweets of all involved, so I should have a good stream of both bloggers and sponsors represented here.

I’m using the following twitter lists:

• @TechFieldDay/tfd3-sponsors
• @TechFieldDay/tfd3-delegates

Look for my occasional commentary as well, but I’m not sure how much time I’ll actually have to participate. I’ll end the coverage when our time is up, which should be roughly 3 to 4 hours.

Click through the page break for the CiL widget and tune into the fun!

By the way, we are providing the first ever public demo of Veeam Backup and Replication 5.0 SureBackup!

 

Other Tech Field Day Seattle links:

Almost a week later and after a flurry of comments from most of the vendors leveraging CBT for virtual machine backups, VMware has published a KB article on the subject.

VMware KB: Reverting to a pre-existing snapshot under specific conditions can cause incremental backups based on CBT (Changed Block Tracking) to become inconsistent

The KB Article describes the exact scenario that causes the problem:

Four things need to occur in the following sequence before there is a possibility of this issue occuring:

1. A VM with hardware version 7 needs to have a snapshot present AND has been backed up previously by a backup product leveraging CBT
2. A backup product performs an incremental backup of VM and leverages CBT to determine changed blocks since last backup
3. After incremental backup is complete, user manually reverts snapshot on the VM
4. A backup product performs an incremental backup of VM and leverages CBT to determine changed blocks since last backup

This issue is caused by arguably a unique set of circumstances, but it is important for VMware administrators to be aware of none-the-less. I’ve blogged about (other bloggers blogging about) ESX snapshots being like a loaded gun before, and here is another example of why.

VMware offers the following resolution for now:

“The workaround for this issue is to do a full VM backup after a snapshot revert operation. If the backup application does not allow this as an option, you will need to remove the CTK files for that VM. The CTK files mentioned are stored with the virtual machine on the datastore and can be removed via the Datastore Browser. This delete operation can be safely done while the VM is running.”

Disclaimer: I work for Veeam Software, the creators of Veeam Backup and Replication.

Veeam Software has confirmed that Backup and Replication v4.1 successfully handles this issue without corruption in all but one specific scenario of manually caused events as described in the Veeam Forums here: http://www.veeam.com/forums/viewtopic.php?f=2&t=3699&p=15139#p15139.

The hotfix has been created for this one remaining scenario and is in "testing for validation" as of this writing.

In the same linked thread Veeam also recommends that until the patch is available workaround this last scenario by "disabling the use of changed block tracking in the Advanced job settings for all jobs which process VMs where manual snapshot reversal may happen, and triggering Full Backup on these jobs to heal the backup file (in case you believe you may have this scenario happened before for some VMs)."

vSphere Editions needed for vDR

A great place to start is by understanding what licensed versions of vSphere 4 vDR can be used with. Using VMware’s vSphere edition comparison table you can clearly see that vDR is only available for use with the Essentials Plus, Advanced, Enterprise, and Enterprise Plus versions.

Although vDR is available in the Enterprise and Enterprise Plus editions, the limitations described in the next section present some design challenges for deploying vDR in larger environments.

Disclaimer: I am a systems engineer for Veeam Software.

Special Considerations For Using vDR

The following list of vDR limitations was taken from the VMware Data Recovery 1.1 Administration Guide:

Data Recovery does not support:

• vCenter Server in Linked Mode.
• IPv6 addresses. IPv4 addresses are required for the Data Recovery appliance.
• NFS is only supported if the share is presented by an ESX Server and the VMDK is assigned to the Data Recovery appliance.
• Hot adding disks with versions of vSphere that are not licensed for hot add.
• Restoring linked clones.
  • Data Recover can backup linked clones, they are restored as unlinked clones.
• Backing up virtual machines that are protected by VMware Fault Tolerance.
• Backing up virtual machines that use VMware Workstation disk format.
• Backing up virtual machines with 3rd party multi-pathing enabled.
• Raw device mapped (RDM) disks in physical compatibility mode.
• Data Recovery has been tested for use with:
  • One backup appliance for each vCenter instance.
  • Each backup appliance protecting up to 100 virtual machines.
  • VMDK or CIFS based deduplication stores of up to 1TB.
  • Up to two deduplication stores per backup appliance.

Not being able to backup VMs using FT or 3rd party multi-pathing, being limited to backing up only 100 VMs, and being restricted to only one vDR appliance per vCenter instance are some examples that push the solution to be most appropriate for the Mid (depending on how Mid is defined) to SMB customer. This is consistent with VMware’s marketing message around vDR and the vSphere editions designed for the mid to SMB companies as well.

Also from the Admin Guide, the following description about the 100 VM limitation clarifies the design restrictions further.

“Each Data recovery backup appliance can protect a total of 100 virtual machines. It is possible to create backup jobs that are configured to protect more than 100 virtual machines, but the backup appliance only protects 100 virtual machines and any additional virtual machines are omitted. It is possible to protect more than 100 virtual machines by installing additional backup appliances, but different backup appliances do not share information about backup jobs. As a result, it is possible to establish unintended configurations. For example, two Data Recovery backup appliances could be configured to protect a folder containing 200 virtual machines, but it is likely that some of the virtual machines would be backed up twice and some would not be backed up at all.”

What Can vDR Do In The Right Use Case?

So, if you manage one of the specified editions of VI 3.x or vSphere 4 from a single vCenter hosting less than 100 VMs  there are some great features to take advantage of.

• vDR is a virtual appliance that integrates with vCenter / VirtualCenter. The appliance can be managed via a web interface or directly from the vSphere Client via a plug in. In either case the functionality is GUI based and straightforward.
• Data Recovery can concurrently back up a maximum of eight virtual machines, and concurrently restore a maximum of eight virtual machines.
• Backup jobs can be scheduled enabling “set it and forget it” functionality
• For virtual machines created in vSphere 4.0, the Data Recovery appliance creates a quiesced snapshot of the
  virtual machine during the backup.
• The backups use the changed block tracking (CBT) functionality on the ESX hosts.
  • If the VM has been upgraded to use virtual hardware version 7 CBT is enabled by default
  • Check if CBT is enabled via the ctkEnabled value in the Advanced Settings section of the VM’s hardware properties. Set it to “True” to enable CBT
• VSS is supported at various levels depending on which Windows OS is installed. See the following table from the Admin Guide:

• vDR leverages VMware’s own deduplication technology to save storage space consumed by backups.
• vDR has out of the box retention policies that pre configure back up jobs based on the retention cycle needed. As mentioned earlier, retention periods also enable “set it and forget it” functionality of backups.
  • Based on the CBT, dedupe and the retention policy features, VMware recommends that a vDR repository be created equal to the total space consumed by all VMs – not the total virtual disk space, but the actual disk in use sizes.
• Backup targets can be a CIFS share, a RDM off the vDR VM, or encapsulated within a secondary VMDK assigned to the vDR VM.
• vDR enables File Level Restores (FLR) via mounting indexed restore points as a volume on the computer with the FLR .exe installed
• vDR can verify the successful back up and restore of any VM with a Restore Rehearsal feature. A new VM is created based on a backup as opposed to overwriting the original VM. The new VM name contains “rehearsal” and the vNIC is not attached when powered on.

updated 050110 – added bullet about CBT enabled by default in VMs with hardware ver 7 and added disclaimer of my employment at Veeam