Lost connectivity to the device backing the boot filesystem

One of our HP Gen 9 blades had the following configuration error:

Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disk/mpx.vmhba32:C0:T0:L0. As a result, host configuration changes will not be saved to persistent storage.

The boot filesystem is on an internal 8 GB SD Card. I logged into iLo, went to Diagnostics and found the system no longer saw the SD Card as mounted:

iLo 4 - Issue with media

We evacuated the host and reseated the SD Card. It mounted without issue but to be safe we went ahead and installed a replacement SD Card. Issue reoccurred. This is caused by the version of iLo running on the server.

iLo 4 - Issue Resolved

EDIT (9/14/2015): There seems to be a issue with the iLO firmware version 2.20 causing this issue. The firmware should be updated to version 2.22. HP should be releasing version 2.30 in the next few weeks that hopefully be a full fix for this issue! Big thanks to Mike B. for reporting his findings from HP!!!

EDIT (10/2/2015): The new HP Service Pack for Proliant (SPP) (dated version 2015.10.0) contains updated iLO firmware 2.30! We are rolling this through our environment over the next week.

EDIT (12/4/2015): The 5+ blades we upgraded to iLo 2.30 haven’t had the issue repeat.

 EDIT (12/10/2015): Multiple reports of the issue reoccurring on iLo 2.30 firmware have been received. Though 2.30 firmware seems to have helped the issue it isn’t a permanent fix. iLo firmware 2.22 should be used until we hear back from HP.

EDIT (2/5/2016):  Bjorn reports HP gave him iLo 2.40 firmware to load. Just checked HP Downloads and it isn’t available yet. Awaiting to read the release notes to see if it specifically addresses this issue.

EDIT (4/20/2016): HP released iLo 2.40 firmware on April 1st. Release notes do not specify this issue specifically, only IPv6 enhancements. We still have not had the issue reoccur on any of our blades running iLo 2.30 firmware. We are going to continue on 2.30 as the enhancements with 2.40 isn’t worth the gamble of a new version.