I’m automatically pulling in the tweets of all involved, so I should have a good stream of both bloggers and sponsors represented here.

I’m using the following twitter lists:

• @TechFieldDay/tfd3-sponsors
• @TechFieldDay/tfd3-delegates

Look for my occasional commentary as well, but I’m not sure how much time I’ll actually have to participate. I’ll end the coverage when our time is up, which should be roughly 3 to 4 hours.

Click through the page break for the CiL widget and tune into the fun!

By the way, we are providing the first ever public demo of Veeam Backup and Replication 5.0 SureBackup!

 

Other Tech Field Day Seattle links:

Almost a week later and after a flurry of comments from most of the vendors leveraging CBT for virtual machine backups, VMware has published a KB article on the subject.

VMware KB: Reverting to a pre-existing snapshot under specific conditions can cause incremental backups based on CBT (Changed Block Tracking) to become inconsistent

The KB Article describes the exact scenario that causes the problem:

Four things need to occur in the following sequence before there is a possibility of this issue occuring:

1. A VM with hardware version 7 needs to have a snapshot present AND has been backed up previously by a backup product leveraging CBT
2. A backup product performs an incremental backup of VM and leverages CBT to determine changed blocks since last backup
3. After incremental backup is complete, user manually reverts snapshot on the VM
4. A backup product performs an incremental backup of VM and leverages CBT to determine changed blocks since last backup

This issue is caused by arguably a unique set of circumstances, but it is important for VMware administrators to be aware of none-the-less. I’ve blogged about (other bloggers blogging about) ESX snapshots being like a loaded gun before, and here is another example of why.

VMware offers the following resolution for now:

“The workaround for this issue is to do a full VM backup after a snapshot revert operation. If the backup application does not allow this as an option, you will need to remove the CTK files for that VM. The CTK files mentioned are stored with the virtual machine on the datastore and can be removed via the Datastore Browser. This delete operation can be safely done while the VM is running.”

Disclaimer: I work for Veeam Software, the creators of Veeam Backup and Replication.

Veeam Software has confirmed that Backup and Replication v4.1 successfully handles this issue without corruption in all but one specific scenario of manually caused events as described in the Veeam Forums here: http://www.veeam.com/forums/viewtopic.php?f=2&t=3699&p=15139#p15139.

The hotfix has been created for this one remaining scenario and is in "testing for validation" as of this writing.

In the same linked thread Veeam also recommends that until the patch is available workaround this last scenario by "disabling the use of changed block tracking in the Advanced job settings for all jobs which process VMs where manual snapshot reversal may happen, and triggering Full Backup on these jobs to heal the backup file (in case you believe you may have this scenario happened before for some VMs)."

vSphere Editions needed for vDR

A great place to start is by understanding what licensed versions of vSphere 4 vDR can be used with. Using VMware’s vSphere edition comparison table you can clearly see that vDR is only available for use with the Essentials Plus, Advanced, Enterprise, and Enterprise Plus versions.

Although vDR is available in the Enterprise and Enterprise Plus editions, the limitations described in the next section present some design challenges for deploying vDR in larger environments.

Disclaimer: I am a systems engineer for Veeam Software.

Special Considerations For Using vDR

The following list of vDR limitations was taken from the VMware Data Recovery 1.1 Administration Guide:

Data Recovery does not support:

• vCenter Server in Linked Mode.
• IPv6 addresses. IPv4 addresses are required for the Data Recovery appliance.
• NFS is only supported if the share is presented by an ESX Server and the VMDK is assigned to the Data Recovery appliance.
• Hot adding disks with versions of vSphere that are not licensed for hot add.
• Restoring linked clones.
  • Data Recover can backup linked clones, they are restored as unlinked clones.
• Backing up virtual machines that are protected by VMware Fault Tolerance.
• Backing up virtual machines that use VMware Workstation disk format.
• Backing up virtual machines with 3rd party multi-pathing enabled.
• Raw device mapped (RDM) disks in physical compatibility mode.
• Data Recovery has been tested for use with:
  • One backup appliance for each vCenter instance.
  • Each backup appliance protecting up to 100 virtual machines.
  • VMDK or CIFS based deduplication stores of up to 1TB.
  • Up to two deduplication stores per backup appliance.

Not being able to backup VMs using FT or 3rd party multi-pathing, being limited to backing up only 100 VMs, and being restricted to only one vDR appliance per vCenter instance are some examples that push the solution to be most appropriate for the Mid (depending on how Mid is defined) to SMB customer. This is consistent with VMware’s marketing message around vDR and the vSphere editions designed for the mid to SMB companies as well.

Also from the Admin Guide, the following description about the 100 VM limitation clarifies the design restrictions further.

“Each Data recovery backup appliance can protect a total of 100 virtual machines. It is possible to create backup jobs that are configured to protect more than 100 virtual machines, but the backup appliance only protects 100 virtual machines and any additional virtual machines are omitted. It is possible to protect more than 100 virtual machines by installing additional backup appliances, but different backup appliances do not share information about backup jobs. As a result, it is possible to establish unintended configurations. For example, two Data Recovery backup appliances could be configured to protect a folder containing 200 virtual machines, but it is likely that some of the virtual machines would be backed up twice and some would not be backed up at all.”

What Can vDR Do In The Right Use Case?

So, if you manage one of the specified editions of VI 3.x or vSphere 4 from a single vCenter hosting less than 100 VMs  there are some great features to take advantage of.

• vDR is a virtual appliance that integrates with vCenter / VirtualCenter. The appliance can be managed via a web interface or directly from the vSphere Client via a plug in. In either case the functionality is GUI based and straightforward.
• Data Recovery can concurrently back up a maximum of eight virtual machines, and concurrently restore a maximum of eight virtual machines.
• Backup jobs can be scheduled enabling “set it and forget it” functionality
• For virtual machines created in vSphere 4.0, the Data Recovery appliance creates a quiesced snapshot of the
  virtual machine during the backup.
• The backups use the changed block tracking (CBT) functionality on the ESX hosts.
  • If the VM has been upgraded to use virtual hardware version 7 CBT is enabled by default
  • Check if CBT is enabled via the ctkEnabled value in the Advanced Settings section of the VM’s hardware properties. Set it to “True” to enable CBT
• VSS is supported at various levels depending on which Windows OS is installed. See the following table from the Admin Guide:

• vDR leverages VMware’s own deduplication technology to save storage space consumed by backups.
• vDR has out of the box retention policies that pre configure back up jobs based on the retention cycle needed. As mentioned earlier, retention periods also enable “set it and forget it” functionality of backups.
  • Based on the CBT, dedupe and the retention policy features, VMware recommends that a vDR repository be created equal to the total space consumed by all VMs – not the total virtual disk space, but the actual disk in use sizes.
• Backup targets can be a CIFS share, a RDM off the vDR VM, or encapsulated within a secondary VMDK assigned to the vDR VM.
• vDR enables File Level Restores (FLR) via mounting indexed restore points as a volume on the computer with the FLR .exe installed
• vDR can verify the successful back up and restore of any VM with a Restore Rehearsal feature. A new VM is created based on a backup as opposed to overwriting the original VM. The new VM name contains “rehearsal” and the vNIC is not attached when powered on.

updated 050110 – added bullet about CBT enabled by default in VMs with hardware ver 7 and added disclaimer of my employment at Veeam

Here’s the information from my email for those that did not sign up to be notified:

Breakthrough Technology: Test and Verify the Recoverability of EVERY Backup

Veeam is enhancing the SureBackup™ capabilities in Veeam Backup & Replication™ with a breakthrough technology that overcomes the final obstacle of image-level backups. IT professionals will have the ability to verify the RECOVERABILITY of EVERY backup of EVERY virtual machine EVERY time. It’s never been possible before.

SureBackup introduces a new patent-pending technology that allows IT professionals to run a virtual machine directly from a compressed backup file.

By publishing the content of backup files directly to ESX hosts, you can:

• Eliminate the need to extract backup files
• Test and verify EVERY backup in a matter of minutes, without affecting backup windows
• Perform recoverability testing without additional hardware or staff

The same technology also allows for item-level restore for any virtualized application, on any OS (patent-pending).

This exciting new technology will be made available in version 5.0 of Veeam Backup & Replication, scheduled for release this summer.

Sound too good to be true? Join us for a free webinar on Wednesday, March 24 at 2:00pm EDT to find out more. Live Webinar – Register Now!

Can’t wait? Go to www.veeam.com/surebackup NOW!

I was actually invited to Veeam’s pre release blogger briefing over the weekend, and the rest of this post is my notes and thoughts about what I learned during briefing.

Before continuing I want to be clear. I have not yet used Veeam SureBackup or a beta copy of Backup and Replication V5.

SureBackup is not available until Veeam Backup and Recover v5 is available

Veeam made it very clear to all of us on the call that for now SureBackup is just a technology announcement. Veeam feels that this breakthrough feature for full image VM backup deserved some time to be understood before it hit the market. Veeam told us SureBackup will be a feature of Veeam Backup and Replication version 5 which will be due sometime in Q3 2010.

What is it and how is it done?

Simply put, the magic of Veeam SureBackup is a wizard that walks an administrator through the configuration of mounting VMs in their compressed Veeam backup file format. This wizard, still to be named later, will create a NFS share from the Veeam Backup Server and automatically configure NFS access from an ESX host. All VM verification, credentials and ESX host information is identified and configured during the wizard. My understanding is that there is complete flexibility in choosing which virtual machines and hosts are used during each verification attempt.

Since the Veeam Backup Server is a Windows host, Veeam has developed it’s own proprietary NFS server. Multiple VMs can be mounted simultaneously and are isolated via a Veeam network fencing feature. The NFS mount is temporary and read only, and it will be removed once the administrator completes the verification process.

Why is that unique and helpful?

Imagine the ability to test your restores. Power on the full VM and it’s dependent VMs without shutting down the production originals. Start an isolated instance of your Domain Controllers so you can also start your Exchange server, for example. Know for a fact that your full VM backups can be successfully powered on when needed and your applications work as expected on the restored copies. Do all of this from the compressed backup files and not by transferring the VM back to your primary storage.

Is SureBackup competition for VMware SRM?

No it’s not. SRM requires SAN based replication. Veeam SureBackup requires full image backups from Veeam Backup and Replication. SRM is intended to leave VMs running at a DR site after failing over. SureBackup uses read only VMs that are designed to be used temporarily. Once the restore is verified the wizard ends and the NFS mount is removed.

I’ll point out that some interesting use cases may be possible. Can you clone or Storage VMotion a SureBackup copy of a VM? Could you replicate your Veeam Backup repository to a DR site and use another Veeam Backup and Replication Server to run the Verification wizard? I’m sure users will explore these unsupported possibilities and report their success on the Veeam Forums.

General Thoughts

By rolling up the administrator sleeves, other full image backup products could be used to test restores in an isolated virtual network on an ESX host. However, no other solution automates that process and makes the configuration as simple as a wizard. No other backup product would allow you to run the restored VMs from the actual backup file in the backup repository. Veeam definitely has a great idea with SureBackup.

Of course, VI environments will still need available capacity to run the verification copies.

Another cool use case will be the ability to start up any incremental backup of VM – not just the latest full image. This means you could go back as far as needed to find a version of a file or operating system prior to corruption, infection, or deletion. An admin will be able to drag the file from the verification VM to a restore location if needd.

“The purpose of this letter is to inform you of our vSphere backup product strategy, ongoing enhancements, and end of availability plans for VMware Consolidated Backup.

VMware Backup Product Strategy
VMware released vStorage APIs for Data Protection (VADP) with the vSphere 4.0 release in May, 2009. VADP is the next generation of VMware’s backup framework. We have also been working with several backup partners to integrate VADP into their solutions to make backup of vSphere Virtual Machines fast, efficient and easy to deploy compared to VCB and other backup solutions. Several of our major backup partners have already released VADP integrated backup products and we expect most of the major backup partners to have VADP integrated backup software by the upcoming feature release of the vSphere platform in 2010.

Future Product Licensing
Given the strong interest and adoption of VADP by our backup eco-system and the benefits offered by VADP compared to VMware Consolidated Backup (VCB), we are announcing the End of Availability for VCB starting with next vSphere feature release in 2010. Starting with the next vSphere platform feature release, VCB will be removed from vSphere platform. VADP integrated backup products (including VMware Data Recovery) will be the recommended option for efficient backup and restoration of vSphere Virtual Machines. This will allow us to focus new value added feature development on VADP instead of two backup frameworks (VCB and VADP).”

[omited]

I’ll go out on a limb and say that most of the VMware community will

help VCB “pack it’s bags” as quick as possible, but does that mean VMware’s vDR will be VCB’s replacement? If so VMware is going to have to expand the scalability of vDR’s current combination of virtual appliance and vSphere Client plugin, in my opinion.

The email doesn’t provide any hints about the future of vDR, but here are some thing to think about if you are considering vDR as VCB’s replacement:

(The following has been taken from http—www.vmware.com-pdf-vdr_11_admin.pdf)

• Virtual machines to be backed up and the backup appliance must both be running on ESX 4 or later or ESX 4i or later. Do not use Data Recovery with vCenter Servers running in linked mode.
• Recovery does not support:
• vCenter Server in Linked Mode.
• Pv6 addresses. IPv4 addresses are required for the Data Recovery appliance.
• NFS is only supported if the share is presented by an ESX Server and the VMDK is assigned to the Data Recovery appliance.
• Hot adding disks with versions of vSphere that are not licensed for hot add
• Restoring linked clones. Data Recover can backup linked clones, they are restored as unlinked clones.
• Backing up virtual machines that are protected by VMware Fault Tolerance.
• Backing up virtual machines that use VMware Workstation disk format.
• Backing up virtual machines with 3rd party multi-pathing enabled.
• Raw device mapped (RDM) disks in physical compatibility mode.
• Data Recovery has been tested for use with:
• One backup appliance for each vCenter instance.
• Each backup appliance protecting up to 100 virtual machines.
• VMDK or CIFS based deduplication stores of up to 1TB.
• Up to two deduplication stores per backup appliance.

In the immediate future it appears that third party VADP based products may be the best replacement for VCB, but I’ll be watching for future changes to VMware’s VDR as well.

Here is a general outline for automating the DR ip addressing with this method:

At the Primary Site

• For these instructions assume the production vSwitch at the primary site has a Portgroup named VM Network
• Build a new vSwitch and do not attach any physical NICs (local only isolated switch). Create a Portgroup named DR Network
• For each VM you need to fail over to a DR site, add an extra vNIC and attach it to the DR Network Portgroup

At the DR Site

• Create your DR site production vSwitch, attach physical NICs and add a Portgroup named DR Network.
• Create another vSwitch and do not attach any physical NICs (local only isolated switch). Create a Portgroup named VM Network

All you have to do for this to work is

configure the extra vNIC in each VM at the primary site with the correct VLAN and ip address of the DR site production network. This ip address configuration will be cloned in the replicated DR VM, and when it’s time to power on the VM at the DR site both vNICs will connect to their respective virtual networks. The DR Network Portgroup will now be on the vSwitch with the physical connectivity to the DR production network and the VM Network vNIC will be isolated to a local only vSwitch.

Vizioncore deserves the credit for showing me the idea, but there’s no reason why this won’t work with any VM replication or VM backup product. It’s not automated DR fail over workflow like VMware SRM, but using this extra virtual networking strategy eliminates some manual or even scripted ip re-addressing.

Simple!

This post provides some basic information on 2 free VCB tools – VCB Wrangler and vbcMC

VCB Wrangler

Robert Patton has started a new blog at virtualfoundry.blogspot.com, and one of his first posts is on his vbscript wrapper he created when his backup vendor did not integrate with VCB. Robert describes environments that could and could not use VCB Wrangler and gives an overview of it’s features in his post.

“Good environments for VCB Wrangler:

• VMs with smallish system partitions (60 GB or less)
• VMs on local SCSI, iSCSI or Fibre Channel SANs
• VMs with large data partitions are using virtual storage adapters (virtual NICS homed in the iSCSI network) or RDM to directly access SAN storage
• Backup software is handling incremental backups of large user data stores

[omitted]

Features:

+ Easy configuration using an XML file

+ Simple per-VM scheduling from one configuration file

+ Direct from SAN and over the network backup modes supported

+ Runs in scheduled task batch mode or interactive single backup mode from the same VBScript

+ Sends an email with details on each virtual machine backed up

+ Able to use passthrough authentication eliminating the need to store any passwords in files

+ Safely renames existing VCB images during backups and restores them if a backup operation fails, ensuring good images remain on disk

+ Written in VBScript and does not require anything be installed on the VCB proxy server

+ Freely customizable to meet the needs of any environment

If it sounds interesting, check out the VCB Wrangler SourceForge site, http://sourceforge.net/projects/vcb-wrangler/”

vcb Management Console 1.0.6 Beta

As a GUI front end installed on the VCB Proxy Server, the vcb Management Console (vcbMC) takes the scripting out of the VCB scripts.

As the tool’s creator, athlon_crazy, describes on his VMTN Communities blog:

“vcbMC is just a frontend for vmware consolidated backup tool and windows AT command. Meaning, I can backup all virtual machines with vcbMC manually, browse vm’s by name, register new vm’s and finally do weekly backup automatically. I don’t need to remember all the vcb commands such as vcbVmName.exe, vcbMounter.exe and all necessary parameters during the backup since vcbMC will do it for me.”

Athlon_crazy also provides some descriptions of the tools features:

A. BROWSE VIRTUAL MACHINE
This feature will let you to login into your ESX hosts, browse virtual machine and stored the list into
“vm.txt” for viewing purposes.

B. MANUAL BACKUP
This feature will let you to manually backup any virtual machine by using “source=name” only.

C. REGISTER VIRTUAL MACHINE
Before you can schedule any backup job, you need to register the virtual machine and generate the .bat files.

D. SCHEDULE VIRTUAL MACHINE
Once you registered the virtual machine, you can select the virtual machine .bat files and schedule it for
backup at any specific time given.

I personally have not tried either tool yet, but follow the links, download the tools, and try for yourself. I’m curious to know how well they work, so comments on this post to that effect will help us all.

If you did not know already, Bitpipe provides access to great virtualization industry articles and whitepapers on as well as many other technologies. I subscribe and get an email notification from Bitpipe full of great research links daily.

Here is more about the E-Guide from the download page:

ABSTRACT: Many organizations are reaping the benefits of virtual machine backup. However, in order to do so, it’s important to be aware of the key considerations and features available for virtual machine backup. This E-Guide illuminates critical aspects of virtual machine backup and explains how to avoid problems when backing up virtual machines.The three main topics discussed in this guide will be: Five things backup administrators should know about VMware Consolidated Backup Avoiding VMware virtual machine backup pitfalls Resources from esXpress(PHD Technologies Inc.) sponsored by PHD Virtual Technologies Inc.