VM /ETC

The problem surfaced from a ESX 3.5 U2 host fiber connected (FC) to a NetApp filer. An AutoSupport email was generated (NetApp’s filer “phone home” feature) with the following information:

This AutoSupport indicates that there is a configuration issue in the FCP partner path.

Information:
============

[hostname: scsitarget.partnerPath.misconfigured:error]: FCP Partner Path Misconfigured.
[hostname: scsitarget.partnerPath.misconfigured:error]: FCP Partner Path Misconfigured – Host I/O access through a non-primary and non-optimal path was detected.

The AutoSupport email even explained what causes the misconfigured path.

This message occurs when the system detects that host I/O access to logical units (LUNs) is not through a primary path. NetApp clustered storage controllers allow access to LUNs through primary, optimized paths, and secondary, non-optimized paths.  Secondary(non-primary) paths provide access to LUNs through the partner storage controller’s FCP target ports. Under normal operating conditions, a host should not perform I/O to LUNs using a non-primary path. Access through a non-primary path should only occur when a host’s MPIO software detects a failure of all primary paths.

The mystery for me was “what caused a failure of the primary path?“. Believe it or not,

it turns out that the cause was an ESX reboot. The lesson learned was that the NetApp ESX Host Utilities must ALWAYS be installed on the Service Console of an ESX host when using FC or iSCSI connectivity to NetApp.

I’ve always been able to rely on the native ESX multipathing and keep the Service Console free from a SAN vendor’s agent. So, of course I did not have the NetApp Host Utilities installed. Now I know. A looming implication of this necessity is ESXi. Manual multipathing reconfiguration MUST ALWAYS OCCUR after every host reboot because ESXi does not have a Console OS. (See the link to kb44784 at the end of this post)

Scott Lowe and Duncan Epping helped me understand this scenario better via a combination of tweets and emails, and Scott has also since written a good explanation of the issue with ESXi and NetApp FC storage on his blog. Here’s a quote from Scott’s post explaining why it happens:

“… setting preferred paths from VirtualCenter is apparently not persistent across reboots because the VML LUN name is not used. Hence, when working with ESXi, neither method of setting the preferred path—vicfg-mpath or VirtualCenter—can provide persistent settings. Hence, the recommendation in the NetApp KB article.

But what about VMware ESX? The issue about using VirtualCenter remains, but esxcfg-mpath allows users to use the VML LUN name during the command to ensure that paths are persistent across reboots. In fact, there’s a note in the help screen of esxcfg-mpath (“esxcfg-mpath -h”) that mentions the fact that VMHBA names are not persistent across reboots, and users should use VML LUN names to be sure of consistency.”

Installing the NetApp Host Utilities avoids the VirtualCenter problem and eliminates the need to use the esxcfg-mpath Console command.

The following links are the relevant NetApp NOW documents I found while troubleshooting this issue. You will need a NOW account to access these articles.

• NetApp ESX Host Utilities 3.1 http://now.netapp.com/NOW/knowledge/docs/san/
• How to Verify VMWare ESX Fibre Channel Configurations with Multipathing I/O (MPIO) https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb30679
• What do FCP Partner Path Misconfigured messages mean? https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb30541
• Recommended Settings for VMware ESXi 3.5 Update 2 when connected to NetApp Storage Systems https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb44784

Also check out the VMTN Communities Forum thread Netapp host utilities for esxi when are they released?

Related Posts

• Storage VMotion now supported for iSCSI (0)
• Avoid Hot VMware Snapshots When Using Storage Array Snapshots (1)
• 11 best practices for upgrading to ESX 3.5 and VirtualCenter 2.5 (0)
• VMware Site Recovery Manager Overview (2)
• P2V file servers? Consolidate to a CIFS share instead (4)
• VCB in a Virtual Machine and other product enhancements (11)
• Is Citrix XenServer cheaper than VMware ESX Server? (26)
• Create a NFS share for VM ISO files with Windows 2003 Server R2 (13)
• Replicate your VMFS partitions – NetApp (2)
• Live VM backups from a VM – esXpress VBA (0)