I’ll get right to it. You can make a backup copy of physical SQL databases with Veeam BaR. Both vCenter and Veeam BaR have a SQL backend. You can’t schedule this as a job, but there are several scenarios where you could take advantage of a quick and easy, one time, manual backup  – before an upgrade or patch, for example.

Although I work for Veeam, this is not necessarily an intended or fully supported usage of the product. This is an easy alternative for the VMware admin to CYA , however.

I’ll start with a brief introduction on how the SQL U-AIR wizard is supposed to work, and then I will explain how you can use an admin switch to make a backup copy of SQL database whether on a VM or a physical server. VMware vCenter and Veeam BaR/Monitor/Reporter all have SQL back ends. 

The U-AIR Up There

To do this you can use the SQL U-AIR wizard. U-AIR stands for Universal Application Item Recovery, and there are 4 stand alone .exes for the various U-AIR wizards of Veeam BaR – AD, Exchange, SQL, and Universal. All of these wizards can be installed on the Veeam BaR server or on any Windows system that can communicate with Veeam. They could be installed on an admins desktop or the SQL, Exchange, or Domain Controller servers too.

Normally, the purpose of the U-AIR wizard is to request and kick off a workflow for a Veeam vPower Virtual Lab. Once the request is approved and managed by the VMware/Veeam administrator and the “Lab Manager–like” virtual lab is ready with the fenced off, running backup copy of the VM(s), the U-AIR wizards allow for the restore from the backup copy VM to the original production VM. For SQL VMs in particular, the restore options are shown in the following screen shot:

Watch this 4 minute video to see the normal SQL restore functionality of the wizard. This video skips the workflow request, skips the wait for approval and virtual lab start up, and just shows what is possible from a backup copy of a SQL VM. I also want to mention that this is an agentless solution. You do not need to install and manage agents anywhere with Veeam BaR.

Trick The System for Physical SQL backups

You can skip the workflow process of starting and using the vPower Virtual Lab if you use an undocumented (as far as I know) Admin Switch for the U-AIR wizards. I’ll focus on the SQL U-AIR wizard for the rest of this post, but it is the same for the Exchange and AD wizards as well.

To use the U-AIR Admin Switch do the following:

1. Download and Install the SQL U-AIR wizard on the Veeam BaR server, a SQL server, or your desktop.
2. Browse to the install folder – standard location is “C:\Program Files\Veeam\Application Restore\SQL Restore”
3. Create a shortcut – right click Veeam.AIR.SQLrestore.exe and choose “Create Shortcut”
4. Right click the new shortcut and choose Properties
5. Modify the “Target” field by adding “/admin” (see the screen shot to the right. click for a larger image)
6. Save the shortcut.
7. Move the modified shortcut to where ever you want (the desktop)

Now, when you run the wizard from the shortcut you go right to the restore options.

Making A Backup of a physical VMware vCenter SQL

Here’s where it gets interesting! Because the wizard prompts you for the SQL Server\Instance to connect to, you can choose any SQL instance that is configured for remote connections. Yes, SQL Express works too!

In my lab environment, vCenter is not included in a Veeam Backup job. It is a “virtual physical server”. I call it “virtual physical” because I use a VMware Workstation lab (3 VMs = 2 vSphere 4 ESXi + 1 vCenter). The Veeam BaR server is a nested VM along with the 5 other VMs in the environment which it backs up. So vCenter is the equivalent to a physical server in a real datacenter. Clear as mud?

To back up my vCenter database I can do the following:

• start the SQL U-AIR wizard from the shortcut with the /admin switch
• On the SQL Servers screen click the Browse button
• From the Choose SQL Server pop up window pick the vCenter instance (screen shot below. click for larger view. Note the Veeam BaR and Veeam One instances available too)

• For performing just a backup, copy the newly populated contents of the Backup server name field to the Production server name field (screen shot below, click for larger view)

• For the Restore Task pick “Restore Database Tables” (see screen shot at the beginning of this post)
• For the Restore Target pick “Bulk Copy Program (BCP)”

• From the Database screen pick the vCenter database.

• From the Objects to Restore screen leave all selected

• Pick an Output folder. (I created a Restore folder on the Desktop)
• Finish the wizard and export the database tables to the selected folder.

Now you have a CYA, backup copy of the vCenter database. In case you didn’t notice this was done while vCenter was running.

If you need help with the Microsoft SQL Bulk Import and Export process, check out the MSDN pages on the subject.

The Tip Of The U-AIR Backup / Restore Iceberg

Hopefully you can now see the potential of the Veeam BaR U-AIR wizards with the /admin switch? Imagine what else you can do by placing different SQL instances on your network in the Production and Backup server fields of the wizard. Once again, this functionality goes beyond the intent of the product, but the flexibility of it’s use is endless in the hands of the right admins!

Storm Clouds

I’ve also read the virtualization pundits’ predictions year after year. They usually go something like “this year is the year of VDI”, “sixty something percent of all servers can still be virtualized”, and “VMware’s market share will shrink to the advances of Microsoft and Citrix”. Is the record skipping? (does anyone know what a skipping record is anymore?). Personally, I’ve always felt a balanced market of hypervisor vendors would be the most likely prophecy for the datacenter, but VMware has always managed to innovate and stay ahead of the competition. Feature-wise, they continue to do so. But, the recent announcement of licensing changes may have changed things.

Whether right, wrong, misunderstood, reluctant to change, or just emotional, many VMware shops initially viewed the new licensing announcement like dark, thunder clouds approaching. Some reactions were as hot as a flash of lightning. Virtual warning sirens sounded across the community, but after a few days and some damage control from VMware, eventually calmer heads prevailed. But, like in the aftermath of any large storm, people began to build for the future. More so than ever before, public discussion of future plans seem to include a new possibility of alternative vendor virtual datacenters.

An Opening In The Clouds

My hunch is that current VMware shops will 

not jump on the newest version like in the past. They will wait to decide, and while they wait they will explore the alternatives.

Many will decide to stick with VMware. I’m guessing those with environments already right-sized would be least likely to actually switch, for example. I’m sure there will be plenty of others with costs that are favorable to stay where they are, but I am already hearing many proclaim they have decided, been ordered, or are now motivated to actually explore other virtualization solutions.

I do actually believe that many will now move away from VMware, however. I’m not proclaiming any solution is better or my favorite, but instead I am pointing out the market conditions are right for the advance of VMware’s competition. Some may be in the form of wholesale migration, but I also feel a lot will be in the form of maintaining the existing vSphere while expanding future VMs on Hyper-V or XenServer. I’m just being a realist. Features may lead the hypervisor market today, but if history has taught us anything, the total business benefits of any offering has incredible influence on the decision makers who write the checks. As much as the engineer in me hates to admit it, most shops can more than get by with a standard set of virtualization features that exist in all the hypervisor options available today.

Tipping Point

So, are we actually at the start of a significant change in the virtualization market? Time will tell. My point is that recent conditions seem to indicate to me that the time is right, more so than ever before.

For a great overview of the entire U1 release check out Rick Vanover’s post vSphere 4.1 update 1 released from his Rickatron Blog and via his Servers and Storage Column/Blog at TechRepublic.

The rest of this post contains cut and pastes from the Release Notes and some commentary about them from me. I want to stress again that these are issues that have now been fixed!

Finally, I’ll point out the one huge VM backup issue (that I can think of right now) that still does not appear to be resolved.

• Cannot take quiesced snapshots of Microsoft Windows Server 2008 R2 virtual machine running vCenter Server 4.1
  When creating a snapshot of a Microsoft Windows Server 2008 R2 virtual machine that has vCenter Server 4.1 installed, the snapshot operation might fail to complete. This issue occurs on Microsoft Windows Server 2008 R2 virtual machines when the ADAM database is installed. The issue is resolved in this release.

The change from 2008 to 2008 R2 has caused quite a few applications to stumble, so it is good to see that VMware users can now install vCenter on the latest and greatest OS from Microsoft running as a virtual machine without having to worry about various hassles, not to mention coming up with a separate backup or business continuity solution for arguably one of the most important VMs in the environment.

• Creation of large .vmdk files on NFS might fail. When you create a virtual disk (.vmdk file) with a large size, for example, more than 1TB, on NFS storage, the creation process might fail with an error: A general system error occurred: Failed to create disk: Error creating disk. This issue occurs when the NFS client does not wait for sufficient time for the NFS storage array to initialize the virtual disk after the RPC parameter of the NFS client times out. By default the timeout value is 10 seconds. This fix provides the configuration option to tune the RPC timeout parameter using the esxcfg-advcfg -s <Timeout> /NFS/SetAttrRPCTimeout command.

Although specific to users of NFS storage, this one was definitely a head scratcher. Possible scenarios where this problem could rear it’s ugly head include full VM restores and even VM replication. Add the possibility of dedicating a large .vmdk as the backup repository for your backup server VM.

Let me know if I missed any other resolved issues specific to VM backups.

The issue still unresolved

Unfortunately, nothing in vSphere 4.1 U1 appears to address the loss of connectivity when committing snapshots for VMs using Change Block Tracking (CBT) while running on NFS storage.  This leaves VMware admins stuck between a rock and hard place – disable CBT or move VMs off of NFS. More about the problem can be found in VMware’s KB Article:

Virtual machine freezes temporarily during snapshot removal on an NFS datastore in a ESX/ESXi 4.1 host

Other Related articles

• VMware vSphere 4.1 Update 1 released (edugeek.net)
• vSphere 4.1 update 1 released (rickvanover.wordpress.com)
• Virtualization And Backup: VMs Need Protection, Too (informationweek.com)

“The resource ’1′ is in use. DVS [vDS Name] port 1 is reserved by to entity <not-found.> vnic 4000, type:vmVnic”

The screen shot shows the actual error I received.

After thinking about that message, I decided to check to see if any ports were in fact being used on the vDS. Although PortID 1 dvPortGroups did not show it being used as expected, other PortIDs were, in fact, in use. See image below.

I tried messing around with disabling the ports (blocking was the terminology in the vSphere Client, I believe) but I still could not delete the vDS. It was Saturday and I had family stuff to do so I reluctantly gave up at the time. However, 3 days later I was able to easily remove it via the Remove link on vDS Configuration screen just like I was supposed to be able to do all along. Needless to say I was a bit perplexed.

After researching I discovered that

there is a 24 hour time out on the previously mentioned ports in use that was preventing the vDS removal. This is a guess because I am drawing a conclusion between my error message and the information from the following VMware KB article Changing the default timeout for locked dvPorts. The resolution section of this KB explains:

“Before you can remove a vNetwork Distribued Switch or dvPort group, all unused dvPort reservations must time out. You can either wait for the default 24-hour timeout period before removing the vNetwork Distributed Switch or dvPort group, or temporarily change the default timeout for dvPorts to free the ports before the default 24-hour timeout period. If you change the default timeout, be sure to change it back after the ports have been freed.”

The details section of the KB  is not exactly a clear match to my specific scenario, but this seems to explain what actually was preventing me from deleting the vDS.

The locked port timeout can be changed by following the KB articles instructions for changing the setting to 10 minutes:

To remove a vNetwork Distributed Switch or dvPort group with locked dvPorts

1. In vpxd.cfg, add the line <vpxd><dvs><portReserveTimeout>10</portReserveTimeout></dvs></vpxd> and save the file.
2. Restart vCenter. The default timeout is now set to ten minutes.

3. After the port reservation has timed out, remove the vNetwork Distributed Switch or dvPort group.

4. Reset the default timeout by removing the line you previous added to vpxd.cfg.

5. Restart vCenter.

A vDS is created on the vCenter 4 server, so the vpxd.cfg file was in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter on my vCenter 4. Although I used a custom install to change the default installation directory, I believe that is the default location for the vpxd.cfg file. If you have to search for it be sure to look for hidden files too.

You can see from the following screen shot I added the line in the already existing <vpxd> section just before the closing “</vpxd>” tag. Copy everything between the <vpxd></vpxd> tags in step 1 above. I’ll point out that I first placed the entire line as it’s own section just before the final </config> file, but that caused problems and the VMware VirtualCenter Service would not start. Don’t do that!

I tested the new config changes by creating a new vDS again, adding my hosts, migrating the VMs to use the vDS, powered on and off some VMs, performed a VMotion, migrated the VMs back to the standard vSwitch, removed the hosts from the vDS, and then deleted the vDS. Interestingly enough, I did not have to wait 10 minutes to delete it.

In the end this post may only be valuable for pointing out that a 24 hour period exists by default for locked portgroups on a vDS. I can’t say for sure that the vpxd.cfg change to 10 minutes means you can remove a stubborn vDS more quickly, but I’m drawing the conclusion that it helps. Let me know in the comments if you have a definitive explanation for this scenario!

Here is a general outline for automating the DR ip addressing with this method:

At the Primary Site

• For these instructions assume the production vSwitch at the primary site has a Portgroup named VM Network
• Build a new vSwitch and do not attach any physical NICs (local only isolated switch). Create a Portgroup named DR Network
• For each VM you need to fail over to a DR site, add an extra vNIC and attach it to the DR Network Portgroup

At the DR Site

• Create your DR site production vSwitch, attach physical NICs and add a Portgroup named DR Network.
• Create another vSwitch and do not attach any physical NICs (local only isolated switch). Create a Portgroup named VM Network

All you have to do for this to work is

configure the extra vNIC in each VM at the primary site with the correct VLAN and ip address of the DR site production network. This ip address configuration will be cloned in the replicated DR VM, and when it’s time to power on the VM at the DR site both vNICs will connect to their respective virtual networks. The DR Network Portgroup will now be on the vSwitch with the physical connectivity to the DR production network and the VM Network vNIC will be isolated to a local only vSwitch.

Vizioncore deserves the credit for showing me the idea, but there’s no reason why this won’t work with any VM replication or VM backup product. It’s not automated DR fail over workflow like VMware SRM, but using this extra virtual networking strategy eliminates some manual or even scripted ip re-addressing.

Simple!

• Increase Simultaneous VMotions as well as Increase Performance
• Guest blog entry: VMotion performance » broche.net – VMware …
• VMware Communities: ESX Tips: Increase number of VMotions per host
• Increase number of VMotions per host

One possible scenario for changing this setting would be to temporarily increase VMotions allowed in order to evacuate ESX hosts within a short maintenance window. I prefer to leave the setting at the default, so for this scenario be sure to change it back after the maintenance is complete. if you read the links provided above, others suggest they have changed the settings permanently.

This rest of this post contains a cut and paste of the steps necessary to make the configuration change with a brief explanation about setting the appropriate value. I am pasting from a VMware Partner PDF communication assembled by Michael White, VMware engineer.

“How can I increase the number of simultaneous VMotion activities?

It is possible to increase the number of simultaneous VMotions but it is something that you should be careful of. Do not due it lightly or on a regular basis but in fact use it on a project basis when required.

While it is not documented by VMware officially, it is in fact supported. Really. You only need to change a variable in the vpxd.cfg file and restart VC. Use the steps below:

1. Connect to your vCenter server.
2. Locate the vapid.cfg file – it is normally in C:\Documents and Settings\All Users\Application Data\VMware\VMware Virtual Center.
3. Make a back up copy of vapid.cfg.
4. Edit the file and insert the information below between the and sections. X
5. The value of X is what you need to calculate carefully.
6. A cold migration has a cost of 1, and a Hot Migration (VMotion) has a cost of 4. So a value of 16 for X would mean you could do 4 simultaneous VMotions (4 * 4 = 16).
7. Now save your changes.
8. Restart the VMware VirtualCenter Server service to make the changes live.”

It may be obvious, but I’ll point out the above setting also means you could do 16 simultaneous cold migrations (VMs powered off): 16 * 1 = 16

The following screen shots show the Ctrl+Shift keystroke combinations to move between the most common VI Client management views:

Other Ctrl+Shift Navigational shortcuts

VMware is saying VDR has improved performance and progress information during intgregity checks, enhanced CIFS support, and that the previous experimental support status for File Level Restores of Windows VMs has been elevated to full support.

Although I could find no mention of it on the VDR web page, the data sheet, or in the new Release Notes, VDR was originally targeted for virtual infrastructure that hosted up to 100 VMs. I’m not sure if this VMware support limitation is still in effect or not.

Still, combined with the already built in de-duplication of VDR VM backups, SMBs have a great VCB alternative that continues to improve.