How Long Would It Take To Troubleshoot And Recover?

Put yourself on the team that suddenly realized 88 VMs were gone! Where would you start? The storage  jumps out at me as a logical place to begin, but after your storage area network  is online, healthy, and normal then what? It’s time to try to crack open the VMware Black Box and scour event logs, alarms, permissions, and actions. Put that aside for a minute and think about how would you start the rebuild process and get the business reconnected!?

I don’t have an easy answer. My goal is asking you to think about this for yourself.

Warning! The Veeam Pitch

Since I work with Veeam products every day I’ll briefly suggest how they could help in this scenario. Decide for yourself what tools are best for your company. I’ll point out that

maintaining physical servers for monitoring and management, or a separate, limited access cluster of VMs would have to have been the implemented design to have helped here.

Veeam Monitor

Since the database is independent of vCenter a duplicate record of all events and actions would kept. Deleted VMs would remain in the GUI as italic, grayed objects, but there associated data would be kept. You can easily see the VMs “last few breaths”.

Veeam Reporter

vCenter permissions reports could be scheduled for delivery to the HR department and reviewed regularly. This betters the chance that the dismissed admin’s credentials would be removed from the virtual environment and not overlooked.

Configuration change reports would show what account was making changes, what properties were changed, and what the old and new values of those changed properties were/are.

Various reports would document the entire virtual infrastructure design and allow a rebuild as quick as possible. Everything from VM IP Addressing to datastore and vSwitch assignment would be easily available.

Veeam Backup and Replication

Instant VM Recovery would allow VMs to run from the backup files. In the time it takes to run the wizard and boot the VM(s) user’s would be reconnected to the guests. Migrations back to production storage could take place immediately or delayed and staggered in tiers as needed. If you have a DR site then failing over to replica VMs would also be an option.

From the post on the Virtualize Planet Blog: Replication Bandwidth Calculator | Virtualize Planet

ReplicaCalc

How many times do you get asked “how do I work out if VM Replication will work with my internet link” Well I wanted to demonstrate some way of providing a calculator without working it out in my head every time. So I made a Replication Calculation tool.  It is assumed that you provide it with 3 values:

1. Average Rate of Change.
2. The Link speed – this value should reflect the upload speed at the source site or the download speed at the target if this is less. So for example if the upload at the source is 6Mbs and 10Mbs download at the target then go for 6Mbs
3. Bandwidth % – which is the amount of bandwidth as % which achievable from the link speed specified.

Download and install ReplicaCalc from here > ReplicaCalc

Ricky describes the needed values in more detail in his post so be sure to read it all there. He also demonstrates how to use Veeam Backup and Replication to get the VM dynamic rate of change. The tool is useful whether using Veeam to replicate your VMs or not, however.

Thanks Ricky!

For more on this topic, I’ve blogged about using a bandwidth calculator to figure out whether your current WAN link is adequate or not for your replication jobs before.

For example, the post explores several options:

”

• Option A: is to literally just stretch the storage fabric and have the storage at one side – but man, you better have insane connectivity between sites, and this is at severe risk to “smoking hole” site failure – say bye-bye to your array.
• Option B: is to use something like EMC Invista, Yotta Yotta or the like, and have a distributed synchronous LUN with two storage arrays.
• Option C: some vendors (Lefthand Networks comes to mind) have a neat trick where they are in essence doing software RAID across x86 platforms on which they run software – so the RAID mirror node can be remote, and “take over” if you lose a site.
• TIP: do the math. Regardless of which method is used, and even if you use compression – it’s a huge amount of bandwidth. My favorite analogy is to ask a customer to stick in a USB flash drive into their laptop, copy a big file and looking at the throughput. “

Chad goes on to point out that he feels stretching ESX Clusters is a bad idea in general and lists 4 solid reasons to support why. Check out all the whole post at the link above.

BTW, Chad revealed that he is the originator of the ESX home lab hardware shopping list I posted about back in March, and you can also see his original post titled Building a Home VMware Infrastructure Lab at his Virtual Geek blog as well.

“Because these snapshots are not, by default, integrated in any way with VMware ESX Server, we have to perform a few extra steps to ensure consistently reliable and usable storage array snapshots.”

Read all of Scott’s tip at the link to the article above.

My “2 cents” on this is that trying to configure the combination of the two snapshots manually might not

really be worth the complexity or the effort. I would lean toward using one or the other, and the storage array snapshots would be the preferred method.

Not to mention if another one of your objectives is to replicate the LUNs to a secondary location for disaster recovery. In that case not only do you have to configure or script a synchronized schedule of snapshots between ESX and the SAN, you also introduce the likelihood of an increase in the amount of data to be replicated nightly.

Of course, VMware’s answer to part of this whole delimma is Site Recovery Manager (SRM). Part of the advantage of SRM is the single point of control of both the SAN and the ESX hosts. With the help of a storage vendor provided SRM Agent, VirtualCenter with the SRM plug-in is able to coordinate the ESX snapshots with the SAN snapshots in order to provide site replication. Wouldn’t it be cool if a future version of SRM or a “sister” application was able to become a Snapshot Recovery Manager locally within the same site on the same storage device?

As Scott points out in his cross linked post at blog.scottlowe.org, NetApp may be the only storage vendor with an application today that can centrally manage the ESX and array snapshot process within a single storage device.

“NetApp, for example, has SnapManager for Virtual Infrastructure; this product is specifically designed to address this problem (among other problems).”

Finally, Scott and I have exchanged emails about an important additional design consideration related to the snapshots topic. That is, you need to strongly consider VM to LUN assignment. In other words, the traditional concept of creating large VMFS volumes to store multiple VMs is not the best design for using snapshots or site replication. Multiple ESX snapshots on a single LUN combined with a storage array’s snapshot of the entire LUN is bad news for consistency. Possible supporting LUN strategies could be:

• Using a 1:1 VM to LUN ratio
• Creating separate virtual disks for each OS partition of your VMs and Splitting those disks across separate LUNs

The 1:1 ratio is straight forward. The second option means that, for example, the C:, E:, F:, etc. drives of a Windows VM would each be different .vmdk files that would be placed on different LUNs. This allows you to snapshot or replicate only the LUN with the F: partition most frequently. The C: and E: partitions can be snapped or replicated less frequently. Grouping multiple VMs of the same application type or similar snapshot / replication needs on these same LUNs in this same manner is the ultimate design goal.

Be sure to read both of Scott’s posts, and feel free to add your thoughts and comments.

Quoting from the Best of VMworld (more on this award later in this post) white paper downloadable from the everRun link above:

” everRun creates a paired virtual machine environment that appears as a single virtual machine environment for both administration and application operations. The environment presents a single IP address and hostname to the application network so users never have to make client-side changes when failures occur. With everRun ComputeThru technology, protection can be configured so that component and even server failures go unnoticed. You simply keep computing through the failure, rather than failing over between servers.”

“The protected virtual machine appears and is managed just like a standard Windows server. Disk data is mirrored synchronously to redundant storage and network and server operations are protected from failure. The administrator loads and configures the application in the protected virtual machine as though it was being loaded onto a physical server and then simply walks away. There is nothing else to do!”

“Failure detection is automatic and recovery policy is embedded. There is no need to create and test complex fail over scripts. everRun works equally well with all classes of storage devices, unlike clusters that require expensive shared storage. Configuration is simple; everRun presents a single protected virtual machine to the administrator and transparently manages redundant resources for availability, removing opportunities for operator-induced errors.”

The everRun VM virtual appliance is called the everRun Availability Manager, and it is accessible and configured via a web interface. The browser based interface provides single glance access to events and status.

Although everRun VM is not clustering, it is like clustering in that it requires a shared private network between XenServer hosts. The technology is loaded as a virtual appliance on each host that monitors the VMs under it’s protection. For designs with more than 2 hosts the private network becomes a “token ring” -like network for XenServer hosts to communicate the status of VMs. The key requirement to the solution is latency of the private network.

Also unlike clustering, it should also be stressed that the everRun VM solution is application independent. All Microsoft applications are supported (yes, Exchange and SQL). Since the mirroring takes place at the VM level the application is irrelevant to the process.

The everRun VM solution can be scaled for one to many VM protection scenarios or even used as remote site disaster recovery fail over. For remote data center fail over with the previously mentioned latency requirement was recommended by Michael Bilancieri at the Atlanta event to be less than 10 ms.

There are three levels of protection for a VM offered by everRun:

1. Level 1 – High Availability (Basic fail over)
2. Level 2 – Continuous Availability (VM-level assured)
3. Level 3 – Fault Tolerance (Uninterrupted computing)

My notes and my memory from the event are unclear about the difference between Level 1 and Level 2, but “big red button” fail over (power off crash consistent recovery) like VMware ESX’s feature falls somewhere between these first 2 levels. Level 3 is live fail over like, for lack of a better example, clustering. everRun offers the ability to set the level of protection for each VM independently.

Marathon won an award in the New Technology Category from SearchServerVirtualization.com while at VMworld 2007. Interestingly, Marathon provided a demo in their booth at VMware’s conference running on XenServer. Marathon and XenSource were one of 2 winners in the category from over 100 exhibitors at VMworld.

Finally, everRun is not a solution only for XenServer. It will work with VMware ESX. everRun’s HA features can compliment or replace current VI3 HA functionality by offering separate protection to VMs, or offer an alternative to SRM as a Ethernet based solution for DR site fail over.

Platespin Forge

If you are looking for an entire VI DR environment in a box then Platespin Forge is your answer.

PlateSpin Forge is a purpose-built consolidated recovery hardware appliance that protects both physical and virtual server workloads using embedded virtualization technology. In the event of a production server outage or disaster, workloads can be rapidly powered on in the PlateSpin Forge recovery environment and continue to run as normal until the production environment is restored.

The PlateSpin Forge appliance ships with prepackaged storage, consolidated recovery software and virtualization technology that is ready to go out-of-the-box. The standard configuration protects 25 server workloads up to a total of 2.5 terabytes of data. For larger implementations, multiple appliances can be deployed.

By dramatically reducing the time and specialized technical resources required to plan, provision, deploy and test a recovery environment, PlateSpin Forge puts workload protection and recovery within reach for small and medium-sized businesses as well as departments or branch locations within larger enterprises. With PlateSpin Forge, organizations can begin reliably protecting their physical and virtual workloads in a matter of hours as opposed to months.

You can go to Platespin’s Forge web page where you can check out a webinar and find out more about the appliance. Interestingly, the products specifications page describes the appliance software as “embedded virtualization technology”.

Platespin PowerConvert

If you are like me, you probably first heard about Platespin’s PowerConvert as an alternative for doing p2v migrations. In reality, PowerConvert is can be more than just a one time conversion tool. Along with Platespin’s imageing capabilities, server recovery can be accomplished using PowerConvert to continuously replicate data to a virtual machine.

PowerConvert provides a range of workload protection alternatives in a single product for maximum flexibility. Data centers can choose between flexible image backup and hardware-independent restore or consolidated recovery using virtual infrastructure and whole server workload replication. By using virtual infrastructure as a recovery platform, organizations can better protect a larger percentage of data center workloads without having to invest in costly duplicate hardware or redundant operating system licenses. In addition to file-based replication, high-speed block-level transfer enables enterprise customers to protect transactional workloads, such as mail and database servers, where point-in-time transactional information, configuration settings and data must be maintained. Efficient incremental transfers ensure that only changes to source data files are replicated to the target environment, minimizing network usage.

Symantec Backup Exec System Recovery

Symantec has combined their former Livestate Recovery product with Backup Exec and added some new recovery options for recovery to a virtual environment.

Leverages the power of virtualization, enabling seamless physical to virtual (P2V) and virtual to physical (V2P) conversions for VMware ESX Server, VMware Server (formerly GSX Server), VMware Workstation, and Microsoft Virtual Server disk formats. Convert entire systems at once or selective volumes at a time.

• Easy-to-use, intuitive virtual conversion wizard simplifies the conversion process
• Now supports VMware ESX Server and Microsoft Virtual Server
• Perform preflight testing of patches, installations, etc. in the virtual environment before applying changes to production systems

Vizioncore vRanger P2V-DR

I previously posted about vRanger’s new feature P2V-DR. Vizioncore’s new module captures the server image while the server is live, and those images are converted for restoring the image to a VM.

P2Vbackup

Frane Borozan over at p2vbackup.com sent me an email last week asking me to check out his site that explains a pretty slick implementation of physical server disaster recovery using the free versions of a few VMware tools. Using VMWare Server, VMWare Converter, and some scripting Fran outlines a detailed implementation of his p2vbackup solution. Be sure to check out his physical to virtual backup and restore site.

Summary

Of course there are many other possibilities and products not discussed in this post – For example clustering, or host based replication products like Double Take and Marathon, are some quick ideas that come to mind. The flexibility of disaster recovery to a virtual environment can be achieved relatively inexpensively compared to the cost of building a similar physical infrastructure.

Using the Data Replication Minimum Bandwidth Requirements chart provided by NSI, makers of Double-Take, You can see that the 100 GB falls into the LAN 10Mb/s bandwidth category (in the 10% column). Click the thumbnail image to the left for a better view of the chart. We’ve already proved that this company needs better than a T1, but it’s close enough not to convince those that think their data change will be lower than 10%.

The real “gotcha” is that companies never consider how long it will take to replicate the data.

Most WAN replication solutions are scheduled to replicate once an amount of data change has occurred or a defined amount of time has elapsed since the last replication. Obviously, the more frequent the replication schedule the less restore work will be needed at the DR site to achieve up to date fail over. Let’s assume in our example that this company is scheduled to replicate every hour, and so, on average this company will need to replicate 1 GB of data.

I used an online connection speed calculator to estimate the transfer time at T1 bandwidth. Click on the thumbnail image to the right to see the cropped results. 1000 MB transferred across T1 1.544 MB will take over 1 hour and 25 minutes.

There goes the hourly replication schedule. Sure, you could back down the schedule to 2 hours or greater. You could even have alternate schedules for different VMs. There is a lot you could do to make it work better with a T1, but do you really want to monitor and administer that complexity? What if you also want to leverage your DR site for development and testing? Now you’ve just added more data flow between your sites.

Upgrade the link between the sites. Using a T3 link the 1000 MB transfer takes 3 minutes.

So, this means that I can buy a single NetApp product and put it at my DR site and start replicating my VI for fail-over. I don’t even need to worry about building the ESX infrastructure right away. Although, the ability to test my DR fail-over requires I have ESX servers at my secondary site.

I asked for general pricing for a small office SAN. I guestimated about 3TB of data would be needed. Although they wouldn’t give me a firm quote I was told pricing should be somewhere in the $10k – $15K range.