Deployment of VMware vRealize Operations Manager

vRealize Operations Manager (vROM/vROPS) gives you visibility into the performance and health of your virtual infrastructure. The bad thing about virtualization is how easy it is to accumulate over-provisioned virtual machines. When your environment grows the numbers can really be staggering. With vROM you can find these VMs and really tweak your environment making it more lean and resource efficient. vROM also has the ability to ensure compliance with IT policies, regulatory requirements, and smart alerting. Basically vRealize Operations Manager will take your structure data like performance metrics and give you a unified interface of analytics. Here is my documented deployment of vRealize Operations Manager 6.0.1 in my lab environment.

vRealize Operations Manager Logo

Planning and Reading:

Read moreDeployment of VMware vRealize Operations Manager

Lost connectivity to the device backing the boot filesystem

One of our HP Gen 9 blades had the following configuration error:

Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disk/mpx.vmhba32:C0:T0:L0. As a result, host configuration changes will not be saved to persistent storage.

The boot filesystem is on an internal 8 GB SD Card. I logged into iLo, went to Diagnostics and found the system no longer saw the SD Card as mounted:

iLo 4 - Issue with media

We evacuated the host and reseated the SD Card. It mounted without issue but to be safe we went ahead and installed a replacement SD Card. Issue reoccurred. This is caused by the version of iLo running on the server.

iLo 4 - Issue Resolved

EDIT (9/14/2015): There seems to be a issue with the iLO firmware version 2.20 causing this issue. The firmware should be updated to version 2.22. HP should be releasing version 2.30 in the next few weeks that hopefully be a full fix for this issue! Big thanks to Mike B. for reporting his findings from HP!!!

EDIT (10/2/2015): The new HP Service Pack for Proliant (SPP) (dated version 2015.10.0) contains updated iLO firmware 2.30! We are rolling this through our environment over the next week.

EDIT (12/4/2015): The 5+ blades we upgraded to iLo 2.30 haven’t had the issue repeat.

 EDIT (12/10/2015): Multiple reports of the issue reoccurring on iLo 2.30 firmware have been received. Though 2.30 firmware seems to have helped the issue it isn’t a permanent fix. iLo firmware 2.22 should be used until we hear back from HP.

EDIT (2/5/2016):  Bjorn reports HP gave him iLo 2.40 firmware to load. Just checked HP Downloads and it isn’t available yet. Awaiting to read the release notes to see if it specifically addresses this issue.

EDIT (4/20/2016): HP released iLo 2.40 firmware on April 1st. Release notes do not specify this issue specifically, only IPv6 enhancements. We still have not had the issue reoccur on any of our blades running iLo 2.30 firmware. We are going to continue on 2.30 as the enhancements with 2.40 isn’t worth the gamble of a new version.

Deprecated VMFS volume(s) found on the host

One of my ESXi 6 hosts had the configuration issue message stating: “Deprecated VMFS volume(s) found on the host. please consider upgrading volume(s) to the latest version“. I only have two LUNs presented to my hosts and both are VMFS5:

Storage is VMFS

After some brief searching I found VMware KB article 2109735 which states the possible cause:

This issue occurs because at the time of initial detection, the version of the filesystem is not known. Therefore, comparing it against the list of valid filesystems does not return a match.

Looking through the hostd log I did find the entry mentioned in the KB article. It appears that after I renamed my StarWind LUN it did not report the filesystem to the host fast enough which caused the error to occur:


HOSTD Log Entry

The resolution was to restart the management agents on the host. I put the host in maintenance mode then restarted the management agents: restart restart

After the management services restarted the error cleared:

Error cleared

Leave comments below if you received this message and this KB article resolved your issue!


Upgrading Windows based vCenter 5.x to 6

For this post I will go through upgrading my Windows based vCenter from 5.5 to vCenter 6. VMware has really made the installer process lean. I would even go out on a limb and say “easy”! vCenter 6 introduces the Platform Services Controller which changes the architecture of vCenter down to two components. The graphic below shows my environment before the upgrade and after:

Upgraded Embedded Architecture

My setup is very simple as I have all vCenter 5.5 components on a single Server 2012 virtual machine. I have ran through the upgrade multiple times and have yet to run into anything major. To find what your environment will look like check out the VMware vCenter Server 6.0 Deployment Guide.

Read moreUpgrading Windows based vCenter 5.x to 6

vCenter Server Appliance Upgrade from 5.x to 6.0

The upgrade process of vCenter Server Appliance (vCSA) version 5.x to 6.0 isn’t truly a upgrade but more of a migration. You are basically deploying a second vCSA that will migrate the hosts/inventory/IP address from the old to the new. You can only upgrade vCSA versions 5.1U3 and 5.5. The 6.0 vCSA is still based on SUSE Linux Enterprise 11 and now has the the same mins/max as the Windows installed version.

vCenter 6 Appliance vs Windows

Before You Start:

There is some prep work you need to perform before doing the migration:

  • Pick a host and ensure it has a standard switch with a vmnic uplink. Because Distributed Switches is a vCenter function it wants to perform the upgrade on a standard switch.
  • Rename your current vCenter VM in the inventory so the new VM can be created as the same name. Otherwise the vCSA folders on the LUNs will not match.
  • Have at least two available LUNs, one for your current vCSA and one for the new vCSA. Since you will want to deploy the new one with the same name they have to be on different LUNs
  • The installation media is a .ISO instead of a .OVA. You will need to burn it to a disc, extract it, or mount it.

Read morevCenter Server Appliance Upgrade from 5.x to 6.0

vCenter Server Appliance 6.0 – A Fresh Install

The VMware vCenter Server Appliance (vCSA) is a security hardened SUSE Enterprise 11 operating system baked with the vCenter server function. With vSphere 6.0 the appliance now has the same mins/max as the Windows installed version. This makes it very appealing to move over to the appliance!

vCenter 6 Appliance vs Windows

Before You Start:

  • Pick a host and ensure it has a standard switch with a vmnic uplink. Because Distributed Switches is a vCenter function it wants to perform the upgrade on a standard switch.
  • The installation media is a .ISO instead of a .OVA. You will need to burn it to a disc, extract it, or mount it.
  • If you want to upgrade your environment instead of fresh deployment, check out my posts Upgrading Windows based vCenter 5.x to 6 and vCenter Server Appliance Upgrade from 5.x to 6.0


PluralSight has amazing video courses on VMware vSphere. If you haven’t checked out PluralSight it’s an amazing service! Highly recommended!


The VCSA is no longer a .OVA but instead a .ISO image so burn/extract/mount it on your computer. First we need to install the VMware Client Integration 6.0 Plugin. In the vcsa folder there is the executable named VMware-ClientIntegrationPlugin-6.0.0.exe.  The installation is simple, once installed proceed to the next step:

Install the VMware Client Integration Plugin 6

Now the plugin is installed, open the vcsa-setup.html in the root of the ISO. This will launch the vCSA installer splash screen:

1 vCSA Upgrade - Mount the ISO

Click Install:

1 vCenter Server Application - Splash Screen

Accept the EULA and click Next:

2 vCSA Install - License Agreement

Enter the FQDN or IP address and root credentials of the ESXi host for the VCSA to be deployed on. Click Next:

VCSA 6 - Connect to ESXi host

Enter the name that you want to call your appliance and set a root account password. Click Next:

4 vCSA Install - Name the appliance

For a small environment/lab, you will want to stay with the embedded platform services controller. If you are in a large environment that has multiple vCenters you may want to separate vCenter from the platform services controller. If you want to read more about separating them read about it in the VMware vCenter Server 6.0 Deployment Guide. Click Next:

5 vCSA Install - Platform Services Controller

If you have a SSO domain already enter the information below. Otherwise create a new SSO domain. Click Next:

Note: Do not use the same Active Directory domain name as your SSO domain. You will have major issues!

6 vCSA Install - Set SSO information

Here you can pick how much resources your vCSA will get.

Tiny: 2 vCPU , 8 GB memory , 120 GB disk space

Small: 4 vCPU , 16 GB memory , 150 GB disk space

Medium: 8 vCPU , 24 GB memory , 300 GB disk space

Large: 16 vCPU , 32 GB memory , 450 GB disk space

Pick your appliance size and click Next:

7 vCSA Install - Appliance Size

Pick a datastore to deploy the appliance to, click Next:

8 vCSA Install - Select Datastore

The vCSA supports using a external Oracle database. I want to continue using the embedded Postgres database. Click Next:

9 vCSA Install - Configure Database

Select the standard switch network you want to use, enter the IP address, FQDN, subnet mask, gateway, and DNS server. Check the box if you want to enable SSH (you can enable this later if you don’t check it now).  Click Next:

10-1 vCSA Install - Network Settings 10-2 vCSA Install - Network Settings

Look over the summary page. If everything looks correct click Finish:

11 vCSA Install - Ready to Complete

Once the VM is created you can monitor the console while it works. Nothing really happened but was neat comparing the console progress bar with the installation.

12 vCSA Install - Can monitor progression

And it’s finished! Click Close:

13 vCSA Install - Installation Complete

VCSA is ready for configuration!

Configuring the VCSA to use Active Directory:

Below is how to configure the VCSA to use Active Directory users and groups. First access the VCSA web client via https://FQDN or IP ADDRESS/vsphere-client/

Log in using the administrator SSO domain you configured above and password.

1 VCSA 6 - Login

On the home page click System Configuration

2 VCSA 6 - System Configuration

Click Nodes -> Your Node Name ( -> Manage -> Active Directory -> Join

3 VCSA 6 - Join to Active Directory

Enter your domain name and a user name/password that has at least SystemConfiguration.Administrators domain permissions

(More information about this step can be found here in the vSphere 6.0 Documentation:

4 VCSA 6 - Join AD Creds

Once you click OK and do not see an error, reboot your VCSA. It will not give you a success message.

To reboot right click your node name and click Reboot

5 VCSA 6 - Reboot

Once reboot log back into the web client. On the left menu click Administration. Now in Active Directory you should see your domain showing up with the Leave button enabled.

7 VCSA 6 - Successfully joined to domain

Now the VCSA is talking to the domain we need to configure it as a Identity Source. Click Home then on the left menu click Administration

6 VCSA 6 - Administration Menu

Click Configuration -> Identity Sources -> then the green + sign

8 VCSA 6 - SSO IdentitySource

Since we are configured the VCSA for Active Directory use the first option Active Directory (Integrated Windows Authentication)

Enter your domain name

Select Use Machine Account then click Ok

(More information about this step can be found here in the vSphere 6.0 Documentation:

9 VCSA 6 - Add Identity Source

Select your domain in the list and click Set as Default Domain

10 VCSA 6 - Set as Default Domain

(Default) will show up beside your domain. If your domain isn’t selected as the default I have had issues with the “Use Windows Session Credentials” option.

11 VCSA 6 - New default domain

Now that we have the Identity Source configured we are ready to assign permissions to a domain user.

On the left menu click Global Permissions -> Manage -> click the green + sign

12 VCSA 6 - Global Permissions

Click the Add button

13 VCSA 6 - Global Permissions - Add

Change the drop down menu to your domain, you can search or browse for users and groups, select them then click Add. Once finished click Ok

14 VCSA 6 - Global Permissions - Select Users and Groups

On the drop down menu select which role you want to assign the users/groups. Click Ok when finished.

15 VCSA 6 - Global Permissions - Finish

You now have a fresh install of the VCSA and configured it to use Active Directory users and groups for permissions.

If you did not deploy the latest version check out my post on Upgrading to Update 2!

Additional Resources:

VMware vCenter Server 6.0 Deployment Guide:

Windows 10 coming Summer 2015!

Most of the community expected Windows 10 to be released fall 2015, however Microsoft made a announcement on their blog that it will be released summer 2015 instead. There isn’t a date set in stone yet but months sooner than expected is welcomed!

Windows 10 will be a free upgrade for Windows 7 & 8 users for the first year as Microsoft introduces their subscription based plan. There hasn’t been a price set by Microsoft for this subscription plan yet.

Read the announcement blog post here:

Unable to kill DCUI – ESXi 5.1

One of our ESXi 5.1 hosts entered a disconnected state with reason unknown. When I logged into the console of the server and checked the network settings everything checked out. Then I went into the ESXi shell to see what the network interfaces looked like and here laid the problem… the management vmkernel interface was not enabled.

esxcli network ip interface list

1 ESXi Shell - Interface List

If you need to see what IP address is assign to each vmk, run this command:

esxcli network ip interface ipv4 get

ESXi Shell - Get Interface

Okay, no big deal…. that is a easy fix! But when I tried to enable it I received a “Unable to kill DCUI” error.

esxcli network ip interface set -e true -i vmk0

2 ESXi Shell - Enabled Interface - failed

I could not find any information about this error anywhere. With this production ESXi host disconnected and roughly 40 virtual machines still running on this server I admitted defeat and opened a support case with VMware.

The VMware Support Engineer referenced internal KB article 2052878 with the fix below:

First we need to find the processor ID of the DCUI.

ps | grep -i DCUI

3 ESXi Shell - Find DCUI process ID

Note: The number to the right of the Unable to kill DCUI error is NOT the PID. Use the command above

Now kill that PID, it will not return anything if successful.

kill -9 PID

4 ESXi Shell - kill the dcui pid

This should now let you enable the vmk interface

esxcli network ip interface set -e true -i vmk0

5 ESXi Shell - Enable interface sucess

Perform a interface list and that disabled vmk should be enabled. Check to see if your host is pingable again.

esxcli network ip interface list

6 ESXi Shell - Interface List with vmk0 enabled

If anyone else received this error, please comment with your scenario and results!

VMware vSphere 6.0 available for download!!

The wait is over!!! VMware vSphere 6, vSOM 6, vCloud Suite 6, SRM 6, and VSAN 6 are available for your downloading pleasure!! VMware has a notice message referencing KB2110293 with some scenario upgrade paths to get you started:

VMware Warning - KB 2110293

HP already has their customized ESXi 6 image available, however at the time of posting this Dell hasn’t released their version.

Watch for vSphere 6.0 documentation coming up right here in 🙂

Additional Reading:

– What’s New in VMware vSphere 6.0?

VMware Announced vSphere 6!!!