How Long Would It Take To Troubleshoot And Recover?

Put yourself on the team that suddenly realized 88 VMs were gone! Where would you start? The storage  jumps out at me as a logical place to begin, but after your storage area network  is online, healthy, and normal then what? It’s time to try to crack open the VMware Black Box and scour event logs, alarms, permissions, and actions. Put that aside for a minute and think about how would you start the rebuild process and get the business reconnected!?

I don’t have an easy answer. My goal is asking you to think about this for yourself.

Warning! The Veeam Pitch

Since I work with Veeam products every day I’ll briefly suggest how they could help in this scenario. Decide for yourself what tools are best for your company. I’ll point out that

maintaining physical servers for monitoring and management, or a separate, limited access cluster of VMs would have to have been the implemented design to have helped here.

Veeam Monitor

Since the database is independent of vCenter a duplicate record of all events and actions would kept. Deleted VMs would remain in the GUI as italic, grayed objects, but there associated data would be kept. You can easily see the VMs “last few breaths”.

Veeam Reporter

vCenter permissions reports could be scheduled for delivery to the HR department and reviewed regularly. This betters the chance that the dismissed admin’s credentials would be removed from the virtual environment and not overlooked.

Configuration change reports would show what account was making changes, what properties were changed, and what the old and new values of those changed properties were/are.

Various reports would document the entire virtual infrastructure design and allow a rebuild as quick as possible. Everything from VM IP Addressing to datastore and vSwitch assignment would be easily available.

Veeam Backup and Replication

Instant VM Recovery would allow VMs to run from the backup files. In the time it takes to run the wizard and boot the VM(s) user’s would be reconnected to the guests. Migrations back to production storage could take place immediately or delayed and staggered in tiers as needed. If you have a DR site then failing over to replica VMs would also be an option.

http://kb.vmware.com/kb/1031106

The KB was updated 7.26.11

I’ll get right to it. You can make a backup copy of physical SQL databases with Veeam BaR. Both vCenter and Veeam BaR have a SQL backend. You can’t schedule this as a job, but there are several scenarios where you could take advantage of a quick and easy, one time, manual backup  – before an upgrade or patch, for example.

Although I work for Veeam, this is not necessarily an intended or fully supported usage of the product. This is an easy alternative for the VMware admin to CYA , however.

I’ll start with a brief introduction on how the SQL U-AIR wizard is supposed to work, and then I will explain how you can use an admin switch to make a backup copy of SQL database whether on a VM or a physical server. VMware vCenter and Veeam BaR/Monitor/Reporter all have SQL back ends. 

The U-AIR Up There

To do this you can use the SQL U-AIR wizard. U-AIR stands for Universal Application Item Recovery, and there are 4 stand alone .exes for the various U-AIR wizards of Veeam BaR – AD, Exchange, SQL, and Universal. All of these wizards can be installed on the Veeam BaR server or on any Windows system that can communicate with Veeam. They could be installed on an admins desktop or the SQL, Exchange, or Domain Controller servers too.

Normally, the purpose of the U-AIR wizard is to request and kick off a workflow for a Veeam vPower Virtual Lab. Once the request is approved and managed by the VMware/Veeam administrator and the “Lab Manager–like” virtual lab is ready with the fenced off, running backup copy of the VM(s), the U-AIR wizards allow for the restore from the backup copy VM to the original production VM. For SQL VMs in particular, the restore options are shown in the following screen shot:

Watch this 4 minute video to see the normal SQL restore functionality of the wizard. This video skips the workflow request, skips the wait for approval and virtual lab start up, and just shows what is possible from a backup copy of a SQL VM. I also want to mention that this is an agentless solution. You do not need to install and manage agents anywhere with Veeam BaR.

Trick The System for Physical SQL backups

You can skip the workflow process of starting and using the vPower Virtual Lab if you use an undocumented (as far as I know) Admin Switch for the U-AIR wizards. I’ll focus on the SQL U-AIR wizard for the rest of this post, but it is the same for the Exchange and AD wizards as well.

To use the U-AIR Admin Switch do the following:

1. Download and Install the SQL U-AIR wizard on the Veeam BaR server, a SQL server, or your desktop.
2. Browse to the install folder – standard location is “C:\Program Files\Veeam\Application Restore\SQL Restore”
3. Create a shortcut – right click Veeam.AIR.SQLrestore.exe and choose “Create Shortcut”
4. Right click the new shortcut and choose Properties
5. Modify the “Target” field by adding “/admin” (see the screen shot to the right. click for a larger image)
6. Save the shortcut.
7. Move the modified shortcut to where ever you want (the desktop)

Now, when you run the wizard from the shortcut you go right to the restore options.

Making A Backup of a physical VMware vCenter SQL

Here’s where it gets interesting! Because the wizard prompts you for the SQL Server\Instance to connect to, you can choose any SQL instance that is configured for remote connections. Yes, SQL Express works too!

In my lab environment, vCenter is not included in a Veeam Backup job. It is a “virtual physical server”. I call it “virtual physical” because I use a VMware Workstation lab (3 VMs = 2 vSphere 4 ESXi + 1 vCenter). The Veeam BaR server is a nested VM along with the 5 other VMs in the environment which it backs up. So vCenter is the equivalent to a physical server in a real datacenter. Clear as mud?

To back up my vCenter database I can do the following:

• start the SQL U-AIR wizard from the shortcut with the /admin switch
• On the SQL Servers screen click the Browse button
• From the Choose SQL Server pop up window pick the vCenter instance (screen shot below. click for larger view. Note the Veeam BaR and Veeam One instances available too)

• For performing just a backup, copy the newly populated contents of the Backup server name field to the Production server name field (screen shot below, click for larger view)

• For the Restore Task pick “Restore Database Tables” (see screen shot at the beginning of this post)
• For the Restore Target pick “Bulk Copy Program (BCP)”

• From the Database screen pick the vCenter database.

• From the Objects to Restore screen leave all selected

• Pick an Output folder. (I created a Restore folder on the Desktop)
• Finish the wizard and export the database tables to the selected folder.

Now you have a CYA, backup copy of the vCenter database. In case you didn’t notice this was done while vCenter was running.

If you need help with the Microsoft SQL Bulk Import and Export process, check out the MSDN pages on the subject.

The Tip Of The U-AIR Backup / Restore Iceberg

Hopefully you can now see the potential of the Veeam BaR U-AIR wizards with the /admin switch? Imagine what else you can do by placing different SQL instances on your network in the Production and Backup server fields of the wizard. Once again, this functionality goes beyond the intent of the product, but the flexibility of it’s use is endless in the hands of the right admins!

Everyone will be there!

Why should you attend? Check out the following samples of featured speakers, sessions, and labs:

Panel Discussion on Cloud Computing
(Rich Brambley, Mike Laverick, Jason Nash, Scott Lowe, Mike Dipetrillo)

Pros & Cons of Stretched Cluster Designs
(Scott Lowe, Industry Speaker)

Cloud in the Real World
(Mike Dipetrillo, VMware)

vCloud Director and VMware View
(Varrow Lab)

Automating vSphere with PowerCLI a Primer
(Aaron Miller, VMware)

VMware View Reference Architecture
(Mac Binesh, VMware)

Of course there will be various VMware and sponsor keynotes and presentations mixed throughout the day. Be sure to catch our Veeam Software session at 10:00 am. The event’s full agenda can be reviewed here.

Panel Discussion – The Sequel

I’m once again honored to get the invite to moderate the Carolina Summit’s featured Panel Discussion at 2:30 pm. I’ll be participating in an open and unscripted “state of the union” conversation about real world cloud computing with Mike Dipetrillo, Scott Lowe, Jason Nash, and Mike Laverick. Like last year, I view my role in this session to be like a co host of a live podcast. The entire VMUG audience, however, is real discussion driver. So, bring your questions for this expert panel!

Can’t make it in person?

Train Signal has partnered with the VMUG organizers and will be interviewing speakers and videoing sessions for those that can’t make it. They will also be live streaming the Keynotes at 8:45 am and 12:45 pm (EST). Find out more be checking out this post:  2011 Charlotte VMUG: Coming to You May 5th. Hopefully Train Signal will decide to live stream the Panel Discussion at 2:00pm as well! I assume vExpert David Davis will be in attendance? Help me reach out to him to include the Panel in Train Signal’s coverage!

Other Coverage

For more great coverage of the 2011 Carolina Summit VMUG also check out:

2011 Carolina VMware User Summit Coming Up – Scott Lowe

On the Road Again: Charlotte, North Carolina Summit – Mike Laverick

VMUG Carolina Summit: Be There or Don’t…See if I Care. No Really. Do Be There – Dustin Pike

VMware Regional Summit in Charlotte, NC! – Jason Nash

Let me know if you can make it!

Veeam recently published a Veeam Reporter Quickstart Guide that can be downloaded here: NEW! Quick Start Guide: Veeam Reporter Dashboard

In short, It was written (by me!) in order to help Veeam customers, evaluators, and free version users understand how to create and save useful VMware reports and use the output of that dynamically updating content for day to day VI Management. This new guide helps you get up and running fast, and provides a primer for building valuable dashboards (click image to the right for a sample dashboard) from those saved reports!

This post contains information for building one of the reports from the new guide – VM Report (By IP Address)

TiVo For Your VMware Infrastructure

The Quickstart Guide gets right to the point by explaining how to create an agentless collection job that constantly updates your reports. Much like if you had a premium subscription with your cable TV provider,

Reporter’s  collected data not only lets you “tune in” to what you need to see, but it also lets you look at “any episode” you need to see it from. In other words, seeing the latest data is always helpful, but being able to generate any report or a Visio diagram (for example) of your environment from last month, when the latest problems weren’t around yet, is even better. The guide also explains how you can create and save reports for all the VMware API “channels” available to you.

Useful Report Examples

Veeam Reporter is based on Report Packs, and each pack has it’s own categories with “out of the box”, ready to run reports and dashboard widgets. This Quickstart focuses just on the Base Report Pack Infrastructure Reports, however. Although these reports provide the most flexibility and unique customization, they have been the most difficult for Veeam Reporter users to understand how to use. Therefore, the Guide provides a handful of how to examples of common, everyday VI infrastructure reports that administrators are more than likely already maintaining manually.

The “VM Report (By IP Address)” is one of the reports explained in the guide (click for a larger view).

Help For the “Database Challenged” 

Once you save the Report you can use the data to make the dashboards. If your mind can’t easily comprehend data series, categories, fields, and chart types (like mine!), the Quickstart Guide gives you suggested widget content, format, and a table of data configuration choices that lets you just fill out the wizard with the provided information. So, for the VM IP Report shown above:

You can build these widgets (click for a larger view)

From this table (click for a larger view)

Download the full PDF from the link above!

Let’s call these new generation of projects V4DR (virtualization for disaster recovery) or V4BC (virtualization for business continuity).

The comparison

If I rewind 3 to 5 years ago in my career, capacity planning for server consolidation was a weekly project and topic of discussion. Customers were either in the process of converting physical servers to virtual machines or they were exploring the possibility to do so. In both cases, capacity planning scenario spreadsheets and reports were frequent “ground zero documents” to almost every project plan I was involved in.

Just like P2V projects, VM replication today also requires some of the same considerations for job scalability and times to complete – i.e. using multiple hosts as targets and making sure the network can support getting the job done as quick as possible. Not to mention ip addressing, VLAN assignments, and application connectivity after the fact. Thank goodness we no longer have to deal with hardware drivers and other unneeded software a second time. Hopefully, VM alignment is a thing of the past too!

I’m not seeing the same “ground zero documents” for replication projects, however.

Use the same capacity planning tools?

So, I’ll ask the question: 

are the existing capacity planning tools we have used previously for our P2V migrations good enough to help today with the V2V replications needed for our new V4DR and V4BC?

In theory they should be: monitor the virtual machines for several weeks and then determine the capacity and resources needed at the DR site. Leverage the capacity planning scenarios in these tools for reports like “What If I lose a Host?” or “What If I add X number of VMs?”.

Replication Capacity Planning

My point is, I think most look at today’s capacity planning tools as either no longer useful since the migration is over, or useful just for primary site capacity purposes. Looking at the output of these tools a little differently could also provide secondary site planning. A report like “What if I lose a host?” could also be viewed as “Can I consolidate my VMs on even less hosts for DR?”. “What if I add X number of VMs?” could also be viewed as “How much capacity do I need to purchase at my DR site?”.

Is what we have today good enough to be reused , or do we need a whole new version of capacity planning and virtualization assessment tools? Call the new tools Cloud / Hosting / DR/ BC / Replication Capacity Planners? Take your pick!

I’m curious. What tools that exist today do you think are already providing output that is useful for V4DR or V4BC planning? Leave a comment!

disclaimer: I work for Veeam Software. Veeam Backup and Replication is a VM replication solution. Veeam Reporter has Capacity Planning reporting capabilities.

disclaimer: I am a Systems Engineer for Veeam Software

Now, with VBR version 5 a FLR Appliance can be activated, auto registered, and configured for file level restores. Although the FLR appliance is only used when you select “Guest Files (other OS)” from the Restore options, the Linux based appliance VM can mount NTFS and most other non Windows file systems. By choosing to use this appliance you have the option to enable FTP access to the restored .vmdk thus making the recovered file system(s) easily available from any desktop. Therefore, end users or application owners can just open Windows Explorer, their favorite web browser, or a FTP client to browse, find, and download their recovered files themselves. No additional software needs to be purchased, and if you use the native OS web and file browsers nothing else even needs to be installed.

This means that the only time required by the VMware / Backup admin is the few minutes it takes to click through the Veeam restore wizard and verify the FLR appliance VM is running. Of course, a few extra seconds is required to click to close the FLR GUI when the user is done.

The rest of the post shows the many screen shots of using and configuring the Veeam FLR Appliance Restore option and a few simple, end user alternatives for accessing the restored files from the remote system. 

Settings / Using FLR Appliance To Restore Files

Click the Restore Button in the GUI and use the “Guest files (other OS)” option both for Windows and Other OS VMs

Please note that I have purposely skipped a few screens in the FLR wizard.

Select VM to recover files (I am selecting a Windows 2003 domain controller in the screenshot)

Select rollback point to restore from

Configure FLR Appliance with Customize button (bottom right corner of the Completing the Multi-OS FLR window)

Set Host, Resource Pool, VM Network, and choose between Dynamic or Static IP address. Check box at bottom to enable FTP server.

Complete Wizard and FLR Appliance automatically boots.

For me it takes less than a minute for the FLR appliance VM to be ready to use

The File Browser pops up on the Veeam Server desktop. This is where the VMware/Backup Admin can normally restore the file via a right-click “copy to” command.

Give the User the IP Address or DNS Record

Now you are ready to tell your end user to access the ip address of the FLR appliance for their self restore. If you choose to use a static ip address, you can create a new DNS record such as “self restore.yourdomain.com” mapped to the FLR Appliance VM to give to your users. The FLR Appliance will be available via this DNS name every time it is used.

Windows FLR User Self Restore Options

With Internet Explorer (or any other browser)

With Windows Explorer (Start/Run/”ftp://[FLR appliance ip address]

With FTP Client

Non Windows FLR User Self Restore Options

FTP with a Terminal / Console

ftp with Filezilla client (no username or password needed)

ftp:// with Firefox (or any browser)

That’s it! Tell the user to let you know when they are done, and then shut down the file browser on the Veeam server and the FLR appliance will auto power off and unregister from your VMware environment.

Just in case you are wondering, the FLR appliance is in a temporary read and write mode while operational, but any changes to the files on the mounted .vmdk are not permanent. Your backed up VMs do are still the same restore point when you are finished.

Davis explained more about the video training in an email I received:

In this course, I educate viewers on how to troubleshoot vSphere networking, storage, management processes (and much more), from the GUI but especially from the CLI.

The new vSphere Troubleshooting course will be part of an upcoming VCAP-DCA training package so I have covered all topics related to troubleshooting on the DCA blueprint. Even if viewers aren’t interested in the VCAP-DCA certification, I designed the course so that it is an excellent resource to learn how to solve virtual infrastructure trouble.

I was honored that Mike Laverick agreed to be the tech editor for the course.

Go to the vSphere Troubleshooting course product page where you can see the entire course outline.

Check out the following sample videos from the new training, and then be sure to get a full copy for your own library.