I’ll get right to it. You can make a backup copy of physical SQL databases with Veeam BaR. Both vCenter and Veeam BaR have a SQL backend. You can’t schedule this as a job, but there are several scenarios where you could take advantage of a quick and easy, one time, manual backup  – before an upgrade or patch, for example.

Although I work for Veeam, this is not necessarily an intended or fully supported usage of the product. This is an easy alternative for the VMware admin to CYA , however.

I’ll start with a brief introduction on how the SQL U-AIR wizard is supposed to work, and then I will explain how you can use an admin switch to make a backup copy of SQL database whether on a VM or a physical server. VMware vCenter and Veeam BaR/Monitor/Reporter all have SQL back ends. 

The U-AIR Up There

To do this you can use the SQL U-AIR wizard. U-AIR stands for Universal Application Item Recovery, and there are 4 stand alone .exes for the various U-AIR wizards of Veeam BaR – AD, Exchange, SQL, and Universal. All of these wizards can be installed on the Veeam BaR server or on any Windows system that can communicate with Veeam. They could be installed on an admins desktop or the SQL, Exchange, or Domain Controller servers too.

Normally, the purpose of the U-AIR wizard is to request and kick off a workflow for a Veeam vPower Virtual Lab. Once the request is approved and managed by the VMware/Veeam administrator and the “Lab Manager–like” virtual lab is ready with the fenced off, running backup copy of the VM(s), the U-AIR wizards allow for the restore from the backup copy VM to the original production VM. For SQL VMs in particular, the restore options are shown in the following screen shot:

Watch this 4 minute video to see the normal SQL restore functionality of the wizard. This video skips the workflow request, skips the wait for approval and virtual lab start up, and just shows what is possible from a backup copy of a SQL VM. I also want to mention that this is an agentless solution. You do not need to install and manage agents anywhere with Veeam BaR.

Trick The System for Physical SQL backups

You can skip the workflow process of starting and using the vPower Virtual Lab if you use an undocumented (as far as I know) Admin Switch for the U-AIR wizards. I’ll focus on the SQL U-AIR wizard for the rest of this post, but it is the same for the Exchange and AD wizards as well.

To use the U-AIR Admin Switch do the following:

1. Download and Install the SQL U-AIR wizard on the Veeam BaR server, a SQL server, or your desktop.
2. Browse to the install folder – standard location is “C:\Program Files\Veeam\Application Restore\SQL Restore”
3. Create a shortcut – right click Veeam.AIR.SQLrestore.exe and choose “Create Shortcut”
4. Right click the new shortcut and choose Properties
5. Modify the “Target” field by adding “/admin” (see the screen shot to the right. click for a larger image)
6. Save the shortcut.
7. Move the modified shortcut to where ever you want (the desktop)

Now, when you run the wizard from the shortcut you go right to the restore options.

Making A Backup of a physical VMware vCenter SQL

Here’s where it gets interesting! Because the wizard prompts you for the SQL Server\Instance to connect to, you can choose any SQL instance that is configured for remote connections. Yes, SQL Express works too!

In my lab environment, vCenter is not included in a Veeam Backup job. It is a “virtual physical server”. I call it “virtual physical” because I use a VMware Workstation lab (3 VMs = 2 vSphere 4 ESXi + 1 vCenter). The Veeam BaR server is a nested VM along with the 5 other VMs in the environment which it backs up. So vCenter is the equivalent to a physical server in a real datacenter. Clear as mud?

To back up my vCenter database I can do the following:

• start the SQL U-AIR wizard from the shortcut with the /admin switch
• On the SQL Servers screen click the Browse button
• From the Choose SQL Server pop up window pick the vCenter instance (screen shot below. click for larger view. Note the Veeam BaR and Veeam One instances available too)

• For performing just a backup, copy the newly populated contents of the Backup server name field to the Production server name field (screen shot below, click for larger view)

• For the Restore Task pick “Restore Database Tables” (see screen shot at the beginning of this post)
• For the Restore Target pick “Bulk Copy Program (BCP)”

• From the Database screen pick the vCenter database.

• From the Objects to Restore screen leave all selected

• Pick an Output folder. (I created a Restore folder on the Desktop)
• Finish the wizard and export the database tables to the selected folder.

Now you have a CYA, backup copy of the vCenter database. In case you didn’t notice this was done while vCenter was running.

If you need help with the Microsoft SQL Bulk Import and Export process, check out the MSDN pages on the subject.

The Tip Of The U-AIR Backup / Restore Iceberg

Hopefully you can now see the potential of the Veeam BaR U-AIR wizards with the /admin switch? Imagine what else you can do by placing different SQL instances on your network in the Production and Backup server fields of the wizard. Once again, this functionality goes beyond the intent of the product, but the flexibility of it’s use is endless in the hands of the right admins!

“I wanted to remind everyone, of what I have already seen floating around the internet, but still important enough to remind.  Our next release of SRM is going to require a 64 bit OS.  This is the same as our next release of VC as it too will require a 64 bit host OS.  This change is required to support the increased capabilities of our products.  As we scale our products to match our customers needs, generally 1 – 2 years in advance of where they will need all the capabilities of a given product we have had to use a 64 bit OS.  This will show itself in increased numbers in things like more simultaneous vSphere client connections.”

To me these new operating system (OS) requirements mean we will see even more instances of vCenter as a VM (virtual machine). It only seems logical that a least path of resistance is to virtualize the management server in order to upgrade, especially considering all have already invested in 64 bit hardware for their hypervisors if they decided to upgrade to vSphere 4 in the first place. To go a step further, I’m willing to argue that it will be more common for an IT Department to justify the cost of additional ESX hosts, even if only dedicated for management, then it will to deploy new servers for physical instances of vCenter.

The looming transition to a console-less ESXi eventually means more management virtual appliances in the future too. Solutions which will continue to need a ESX console or similar will have to substitute their own appliance to operate with ESXi. This means even more justification for additional ESX/ESXi hosts and thus greases the decision to virtualize vCenter as well. I expect to see management clusters of ESX hosts become more common in the future than even the use of management networks today.

ESX hosts have bigger and badder hardware now than ever before allowing for higher consolidation ratios and larger applications to easily run in virtual machines, but it will be interesting to see if the vCenter as a VM best practices change over time. I personally feel that continuing to separate the database from the virtualized vCenter will continue to be a smart choice. Running a separate, and even virtualized, SQL instance ensures not only better performance of vCenter as a VM but enhances DR scenarios. In fact, those that already have the vCenter database on a remote instance will likely have a safer upgrade to the 64 bit vCenter.

The new 64 bit requirements will no doubt make for an interesting migration scenario, and I’m sure we will see some positive and negative opinions. Let me know your thoughts on a 64 bit vCenter as a VM in the future!

There are a few possible problems to consider. In this post I’ll first cover (with the help of a several others) general VM and vCenter vDS networking issues, but along the way I’ll explore thoughts about designing around a vDS for keeping vCenter as a VM.

The vDS Rock and A Hard Place

I’m not the first to recognize there are problems with using a vDS if vCenter goes down. Here’s a few posts that have already addressed the pitfalls to avoid.

VMware vSphere – VMware vNetwork Distributed Switch bug or limitation

“… if you lose virtual Center you will have no way in moving virtual machines between different port groups on the vNetwork Distribute Switch. In addition, you will not be able to get a virtual machine from the traditional virtual switch to a port group on the vNetwork Distributed Switch. Extra to that, you can’t move a VM to another VMware vNetwork Distribute Switch. So that means if you are using VMware vSphere vNetwork Distributed Switches & you lose virtual center you are almost disabled on the networking part. If you lose connectivity on the classic virtual switch & your adapter on the distributed switch are OK you still can’t move your virtual machines to that distributed switch till Virtual Center is back.”

See the entire blog post for screen shot examples of vDS and vSS portgroups available to VMs with and without vCenter.

Virtualizing vCenter With vDS Catch-22 is another post that explores what happened after taking down the vCenter VM for a routine migration.

“vCenter was shut down and unavailable, therefore, I had connected my vSphere client directly to the ESX4 host in which I transferred the VM to. When trying to configure the vCenter VM to use the vNetwork Distributed Switch (vDS) port group I had set up for all VM traffic, it was unavailable in the dropdown list of networks. The vCenter server was powered down and thus the vDS Control Plane was unavailable, eliminating my view of vDS networks.

This is a dilemma. Without a network connection, the vCenter server will not be able to communicate with the back end SQL database on a different box running SQL. This will cause the vCenter server services to not start and thus I’ll never have visibility to the vDS.”

Virtualizing vCenter with vDS: Another Catch-22 is from another blogger exploring the same problem that was inspired by the last post. 

VMware has a KB Article explaining Configuring vSwitch or vNetwork Distributed Switch uplinks from the command line in ESX 4 which helps manually migrate dvPortgroups back to a vSS from the Service Console when in trouble. Interestingly enough, I couldn’t find a similar article for ESXi. I’ll assume the same process is available via “unsupported mode” in ESXi, but the potential for having to perform these actions under fire must be considered.

Use a Hybrid Mix of vSS and VDS

Does this mean a virtual infrastructure design should keep a vSS around? I would say “yes!”. Perhaps it’s now more important to dedicate 2 of the ESX host’s pNICs for the ESX Service Console / ESXi Management VMKernel isolated as a vSS. The 2 pNICs are not only for redundancy anymore, but also to support one or more standby VM portgroups in case they’re needed as a recovery network for VMs normally using the vDS. Of course, that means creating the appropriate trunking, and VLANs ahead of time. Have everything ready for a quick and easy change of critical VMs when needed.

Therefore, a hybrid design using both a vSS and a vDS is a smart “safety net” to have. Especially when an admin has to point the vSphere client directly at an ESX/ESXi host. The “safety net” vSS portgroups will be available from each host and the VMs can be easily switched via the vSphere Client GUI.

BUT, Does VMware Support vCenter As A VM Using A vDS?

Even though VMware now fully supports running vCenter virtualized, the question is not whether to run vCenter as a VM, but instead it’s whether VMware even supports a vCenter VM using a vDS!

VMware Communities: Virtual vCenter and vNetwork ….

"I called support about running vCenter within a distributed switch and they said point blank, "it is not supported". They said because vCenter governs the distributed switch environment, you can’t have vCenter within the distributed switch."

OK, so VMware support has not always told customers the correct support policy based on actual technical capabilities in the past, but it’s something serious to consider. In fact, based on what has been already explained it makes sense they wouldn’t support it. Besides, it’s more like “putting all your eggs in one basket” then ever before when a vCenter VM is placed on vDS.

I’ve brought up the logistical argument about the brains of the virtual infrastructure running in the environment it is managing before. Don’t misunderstand – I am an advocate for and I virtualize vCenter Server all the time, but I make sure to adhere to the best practices. On that note, the VMware KB Article Running VirtualCenter in a Virtual Machine (updated as of Aug 09) serves as a pointer to the VMware tech note http://www.vmware.com/vmtn/resources/798 which then points to the old VI3 tech note on this topic: http://www.vmware.com/pdf/vi3_vc_in_vm.pdf. VMware definitely needs to update the tech notes to include best practice for vCenter 4 as a VM in an environment containing a vDS!

After all of that why do I want to use a vDS again?

Finally, here is a quick reminder of why the extra design considerations are worth the trouble.

Comparing the vDS to a vSS:

vNetwork Distributed Switch on ESX 4.x – Concepts Overview

• Can forward L2 frames
• Can segment traffic into VLANs
• Can use and understand 802.1q VLAN encapsulation
• Can have more than one uplink (NIC Teaming)
• Can have traffic shaping for the outbound (TX) traffic

• Can shape inbound (RX) traffic
• Has a central unified management interface through vCenter
• Supports Private VLANs (PVLANs)
• provides potential customisation of Data and Control Planes

VMware VirtualCenter 2.5 Update 6 is Generally Available

We are pleased to inform you that VMware VirtualCenter Server 2.5 Update 6 (English and localized) is generally available as of late night January 29, 2010.

VirtualCenter 2.5 Update 6 provides the following improvements:

• Guest Operating System Customization Improvements
• Support for Firefox 3.x Browsers with VirtualCenter Web Access
• Bug and security fixes

For details regarding the new fixes and improvements, please refer to the release notes.

VirtualCenter Server 2.5 Update 6 is available for download.

For details regarding compatibility, please view vSphere Compatibility Matrixes.

Thanks,
VMware Infrastructure Product Management Team

Some of the highlights for me when reading the Release Notes are:

From the Resolved Issues section:

• I/O error occurs when downloading a large file from the datastore browser or when exporting a virtual appliance to OVF (KB 1017614)

I’ve never experienced the issue with transferring .ISOs and other large files with the data store browser, but several VM /ETC readers have. VMware’s KB Article still doesn’t explain exactly when and why it happens, but it’s good to see there’s now a fix. This is critical for ESXi implementations without a service console and SSH access to transfer files. The data store browser will have to become the tool most often used to move files to and from ESX datastores.

From the What’s New section

Guest Operating Systems Customization Improvements

Customization support for the following guest operating systems has been added in this release:
For more complete information about supported guests included in this release, see the VMware Compatibility Guide.

• Windows 7 Enterprise (32-bit and 64-bit)
• Windows 7 Ultimate (32-bit and 64-bit)
• Windows 7 Professional (32-bit and 64-bit)
• Windows 7 Home Premium (32-bit and 64-bit)
• Windows Server 2008 R2 Standard Edition (64-bit)
• Windows Server 2008 R2 Enterprise Edition (64-bit)
• Windows Server 2008 R2 Datacenter Edition (64-bit)
• Windows Server 2008 R2 Web Server (64-bit)

Support for Firefox 3.x Browsers with VirtualCenter Web  Access

This release adds support for Firefox 3.x browsers with VirtualCenter Web Access. Firefox 3.x is not a supported browser for ESX Server 3.5 Update 5 Web Access with this release.

VirtualCenter 2.5 Update 6 also provides Update Manager Plug-In updates:

This release of the VMware Infrastructure 3 software suite includes an update to VMware Update Manager. For more information, see the VMware Update Manager 1.0 Update 6 for VirtualCenter 2.5 Release Notes. VMware Converter Enterprise Update 4 is compatible with VirtualCenter 2.5 Update 6.

Finally, Update 6 also improves security with updates to JRE and Tomcat

Security

• Updates JRE version to 1.5.0_22This release fixes the following security issues by updating the JRE version to 1.5.0_22:
  • The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the following names to the security issues fixed in JRE 1.5.0_22:
    CVE-2009-3728, CVE-2009-3729, CVE-2009-3864, CVE-2009-3865, CVE-2009-3866, CVE-2009-3867, CVE-2009-3868, CVE-2009-3869, CVE-2009-3871, CVE-2009-3872, CVE-2009-3873, CVE-2009-3874, CVE-2009-3875, CVE-2009-3876, CVE-2009-3877, CVE-2009-3879, CVE-2009-3880, CVE-2009-3881, CVE-2009-3882, CVE-2009-3883, CVE-2009-3884, CVE-2009-3886, and CVE-2009-3885.
  • The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the following names to the security issues fixed in JRE 1.5.0_20:
    CVE-2009-2625, CVE-2009-2670, CVE-2009-2671, CVE-2009-2672, CVE-2009-2673, CVE-2009-2675, CVE-2009-2676, CVE-2009-2716, CVE-2009-2718, CVE-2009-2719, CVE-2009-2720, CVE-2009-2721, CVE-2009-2722, CVE-2009-2723, and CVE-2009-2724.
  • The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the following names to the security issues fixed in JRE 1.5.0_18:
    CVE-2009-1093, CVE-2009-1094, CVE-2009-1095, CVE-2009-1096, CVE-2009-1097, CVE-2009-1098, CVE-2009-1099, CVE-2009-1100, CVE-2009-1101, CVE-2009-1102, CVE-2009-1103, CVE-2009-1104, CVE-2009-1105, CVE-2009-1106, and CVE-2009-1107.

• Updates Apache Tomcat version to 5.5.28This release of VirtualCenter Server updates the Tomcat server package from 5.5.27 to 5.5.28, to address multiple security issues. The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the names CVE-2008-5515, CVE-2009-0033, CVE-2009-0580, CVE-2009-0781, and CVE-2009-0783 to these issues. For more information on security vulnerabilities, refer to the Apache Tomcat 5.x Vulnerabilities page.

If you are still running VMware VI 3 you should consider upgrading to VirtualCenter 2.5 Upgrade 6 as soon as possible. ESX/ESXi 3.5 remains at Update 5 to the best of my knowledge at this time, however.

When you install Update Manager, you can select the location for downloading patches. To change the location after installation, you must manually edit the vci-integrity.xml
file.

Procedure

1. Log in to the Update Manager server as an administrator.
   
2. Stop the Update Manager service.
   
   1. Right-click My Computer and click Manage.
      
   2. In the left pane, expand Services and Applications and click Services.
      
   3. In the right pane, right-click the VMware Update Manager Service and click Stop.
      
3. Navigate to the Update Manager installation directory and locate the vci-integrity.xml file.
   
   1. The default location is C:\Program Files\VMware\Infrastructure\Update Manager.
      
4. Create a backup copy of this file in case you need to revert to the previous configuration.
   
5. Edit the file by changing the following fields:
   
   1. yournewlocation
   2. The default patch download location is: C:\Documents and Settings\All Users\Application Data\VMware\VMware Update Manager\Data\
      
   3. The directory path must end with \.
      
6. Save the file in UTF-8 format, replacing the existing file.
   
7. Copy the contents from the old patchstore directory to the new folder.
   
8. Restart the Update Manager service.
   

Note that integrating the Sysprep files are still required in all versions of vCenter to customize VMs. This includes vCenter 4 for vSphere. Sysprep is no longer used for Server 2008, however, but VMware has added native customization of Server 2008 VMs in vCenter 4 without adding any additional files.

KB Article 1005593 titled Sysprep file locations and versions not only provides download links and extract to locations but also explains the common symptoms when Sysprep is not installed correctly on vCenter.

• When attempting to customize the deployment of a virtual machine the radio buttons are disabled (greyed out).
• When a virtual machine (VM) is deployed from a Template, you find that the SID is always the same, despite the fact that you chose the option to generate a new SID during Template deployment and guest operating system customization.
• When attempting to create a new virtual machine from a Template in ESX v3.5 you receive the following error message

Warning: Windows customization resources were not found on this server

• Message in the guestcust.log:

deploy doesn’t contain known sysprep files

The KB article explains the cause

Microsoft has a different version of Sysprep for each release and service pack of Windows. According to Microsoft, “You need to use the version of Sysprep specific to the operating system you are deploying”. The differences are not immediately visible in the packaging and documentation of the service packs, so it is necessary to manually investigate.

Use either of the following methods to obtain the appropriate Sysprep files. All instructions in this post assume vCenter has been installed in the default location.

Follow the link above to find the table of Sysprep downloads for each Microsoft operating system. Extract those downloads to the exact locations provided in the same table. Basically, all Sysprep versions go in the C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\sysprep directory but in a separate sub folder specific for each OS.

At the time of this writing KB Article 1005593 has been updated as of August 14, 2009 so the download links seem current.

Obtain Sysprep from the Microsoft Install Media

Whether or not KB Article 1005593 remains updated or not, the good news is that you can always get the Sysprep files from the Windows Install CD/DVD too.

Assuming your CDrom drive is E:\

1. Use Windows Explorer to browse to E:\Support\Tools on the Windows (insert your OS here) CD/DVD and find the deploy.cab file
2. Double-click deploy.cab to open it (like a .zip file)
3. Copy all of the contents of the deploy.cab file
4. Paste the copied files on the vCenter server in the appropriate subfolder at C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\sysprep
   

The sub folders are named after the various Windows operating systems and are obvious, but use the table in KB Article 1005593 if you need to confirm the exact file location.

P2V and V2V Customizations with VMware Converter Stand Alone

The free VMware Converter Stand Alone version also let’s you integrate the Sysprep tools and customize multiple VM migrations from a single physical or virtual server. If you install Converter in it’s default location then obtain Sysprep in the same methods already discussed and place the files in the same locations as vCenter. Of course, this location is now on the server where you installed Converter, and you will only need to use the Sysprep version of the installed OS. You can’t install Converter on a Windows 2003 Server, put the Windows XP Sysprep tools in it’s correct directory, and then expect to P2V/V2V a new Windows XP VM!

Starting with VMware Converter 4 Stand Alone there is a Client/Server install option which allows Converter to utilized as a centralized tool for deploying new VMs. Creative administrators are able to duplicate most of vCenter’s deploy from template and VM cloning features using this install mode.

The following details were taken from the VMware VirtualCenter 2.5 Update 5 Release Notes:

What’s New

Support for High Consolidation in VMware HA Clusters – VirtualCenter 2.5 Update 5 includes significant performance and scalability improvements to VMware HA. Use VirtualCenter 2.5 Update 5 for environments with more than 35 virtual machines (VMs) per host in an HA cluster.

For information on the ESX Server host settings required for this scalability improvement, see ESX Server host settings required for environments with up to 80 virtual machines per host in an HA Cluster (KB 1012002).

Upgrading or Migrating to VirtualCenter 2.5 Update 5

This release supports upgrading from VirtualCenter 1.4.1, VirtualCenter 2.0.2 (including Update 1, Update 2, Update 3, Update 4, and Update 5), VirtualCenter 2.5, VirtualCenter 2.5 Update 1, VirtualCenter 2.5 Update 2, VirtualCenter 2.5 Update 3, or VirtualCenter 2.5 Update 4, to VirtualCenter 2.5 Update 5. Review the detailed upgrade and migration instructions and guidelines that are provided in the Upgrade Guide.

Following the above link to KB 1012002 explains that upgrading vCenter 2.5 to U5 is just the start. VI 3 admins also need to make some additional configurations on ESX/ESXi hosts to achieve the 80 VMs per host improvements.

“Starting with the VirtualCenter 2.5 Update 5 release, an ESX Server host in an HA cluster can support up to 80 virtual machines. For all virtual machines to power on on other hosts in the cluster, if hosts within the failover capacity limit fail, you need to ensure that the following parameters in the ESX Server hosts are set with the following values:

Parameter	Value	Applicability
Misc.RunningVCpuLimit	192vCPU	ESX Server 3.5 and ESX Server 3i Embedded
Service console memory	512MB	ESX Server 3.5
vim resource pool memory reservation	1024MB	ESX Server 3i Embedded
Host agent memory configuration	<hostdStopMemInMB>380</hostdStopMemInMB> <hostdWarnMemInMB>300</hostdWarnMemInMB>	ESX Server 3.5 and ESX Server 3i Embedded

For specific instructions on setting each of these configurations on ESX/ESXi 3.5 hosts read KB 1012002.

It also appears that this KB offers some interesting tweaks for improving heavy VM consolidation on ESX/ESXi 3.5 hosts even if not managed by VC2.5 U5. Why would it not benefit admins to make these changes on hosts regardless of VI 3.5 Cluster membership?

Although not stated, I assume ESX Clusters with greater than 80 VMs per host should be upgraded to vSphere 4 in order to insure the best performance and scalability features.

• view hosts/clusters
• find virtual machines
• view VM properties
• edit VM settings
• view events and event details
• view tasks and task details
• view triggered alarms and triggered alarm details
• view host summaries and manage hosts

Although I have not had a chance to test it personally, my take on Mobile Admin is that it appears to be a great tool with features tailor made for the administrator on call and responsible for providing remote or after hours support.

Here are some screen shots of the Windows Mobile Client

Not only does Mobile Admin work with VMware, but it also provides phone based access to common Windows infrastructure support tasks, remote desktop sessions, and remote management of  Blackberry, Exchange, SQL, Citrix, and Lotus servers just to name a few. See the entire list of functions on Rove’s web page.

Rove will be hosting a webinar titled “Introducing Mobile Admin” on March 25. Register here to find out more about the product. You can also download a 14 day trial from their web site.