Hardware RAID arrays on LSI/Broadcom MegaRAID controllers are managed from a command-line tool. The array is invisible to the operating system as individual disks, so lsblk and smartctl alone will not tell you whether a disk has failed — the controller has to be asked.
Which tool
StorCLI (storcli64) is the current tool and covers all recent controllers. MegaCLI is its predecessor, still present on many older servers and still functional. Both are downloaded from Broadcom's support download page — LSI was acquired by Avago and then Broadcom, so the old lsi.com download links no longer exist.
The download is a zip containing an RPM. On RHEL-based systems install it directly; on Debian and Ubuntu convert it with alien, or use the packages provided in the archive. StorCLI installs to /opt/MegaRAID/storcli/storcli64.
On a dedicated server with us, you do not need any of this to have a failed disk replaced — open a ticket and we will check the array and swap the disk. Physical replacement is done by us in any case. The commands below are for those who want to inspect or manage the array themselves.
Checking the array
This is what you need in almost every case.
StorCLI:
storcli64 /c0 show storcli64 /c0/vall show storcli64 /c0/eall/sall show
MegaCLI:
megacli -AdpAllInfo -aAll megacli -LDInfo -LAll -aAll megacli -PDList -a0
What to look at:
- State of each virtual drive —
Optimalis healthy;Degradedmeans a disk has dropped out and the array is running without redundancy;Partially DegradedorOfflineis worse. - Firmware state of each physical disk —
Online, Spun Upis normal;Failed,Unconfigured(bad)or a missing slot indicates a fault. - Media Error Count and Other Error Count — a count that is growing warns of a disk on its way out before it fails outright.
- BBU / cache status — a failed battery on the controller switches write caching off, and the array becomes noticeably slower for no obvious reason.
Details of one disk:
storcli64 /c0/e252/s4 show all megacli -pdInfo -PhysDrv[252:4] -a0
The two numbers are the enclosure ID and the slot number. List them with:
megacli -PDlist -a0 | grep -e '^Enclosure Device ID:' -e '^Slot Number:'
Rebuild progress
After a disk is replaced the array rebuilds, during which it is slower and still vulnerable:
storcli64 /c0/e252/s4 show rebuild megacli -PDRbld -ShowProg -PhysDrv[252:4] -a0
Reading SMART through the controller
smartctl cannot see the disks directly, but it can reach them through the controller:
smartctl -a -d megaraid,9 /dev/sda # SAS disk, device ID 9 smartctl -a -d sat+megaraid,9 /dev/sda # SATA disk
List the device IDs:
megacli -PDlist -a0 | grep '^Device Id:' | awk '{print $3}'
To check every disk in turn:
#!/bin/sh
for id in $(megacli -PDlist -a0 | grep '^Device Id:' | awk '{print $3}')
do
smartctl -d sat+megaraid,${id} -l devstat /dev/sda
done
For continuous monitoring with smartd, comment out DEVICESCAN in /etc/smartd.conf and add a line per disk:
/dev/sda -d sat+megaraid,0 -a -s L/../../3/02 /dev/sda -d sat+megaraid,1 -a -s L/../../3/03
The trailing values stagger the self-test times so that the disks are not all tested at once.
See How to check disk health using S.M.A.R.T. for how to interpret the output.
Replacing a failed disk
Silence the alarm — this affects the current alert only, not the alarm permanently:
megacli -AdpSetProp -AlarmSilence -a0
Identify the disk physically by lighting its indicator:
storcli64 /c0/e252/s4 start locate megacli -PdLocate -start -PhysDrv[252:4] -a0
On some chassis the locate LED does not work; using the activity light instead usually does:
megacli -AdpSetProp \{UseDiskActivityforLocate -1\} -aALL
If the controller has not already marked the disk as missing:
megacli -PDMarkMissing -PhysDrv[252:4] -a0 megacli -Pdgetmissing -a0
After the disk has been physically swapped, stop the indicator and check the new disk:
megacli -PdLocate -stop -PhysDrv[252:4] -a0 megacli -pdInfo -PhysDrv[252:4] -a0
A used disk may carry metadata from another array (Foreign Configuration), which the controller will not accept. Check and, if you are certain, clear it:
megacli -CfgForeign -Scan -aALL megacli -CfgForeign -Clear -aALL
Then bring it into the array and start the rebuild:
megacli -PdReplaceMissing -PhysDrv[252:4] -Array0 -row4 -a0 megacli -PDRbld -Start -PhysDrv[252:4] -a0
Creating and clearing configurations
The commands in this section destroy data. Clearing a configuration or deleting a virtual drive makes the array's contents unrecoverable. Run them only on a server with no data you need, and check the adapter and drive numbers twice — there is no confirmation prompt.
Delete logical drives, or clear the configuration entirely:
megacli -CfgLdDel -Lall -aAll megacli -CfgClr -aAll
Create a RAID-0, 1 or 5 array — substitute -r1 or -r5 for -r0:
megacli -CfgLdAdd -r0 [252:0,252:1] WB ADRA Cached CachedBadBBU -a0
Create RAID-10 from three mirrored pairs:
megacli -CfgSpanAdd -r10 -Array0[252:0,252:1] -Array1[252:2,252:3] -Array2[252:4,252:5] -a0
Show how the disks ended up arranged:
megacli -CfgDsply -a0
Initialise a new array and watch progress:
megacli -LDInit -Start -full -L0 -a0 megacli -LDInit -ShowProg -L0 -a0
JBOD
Some controllers need JBOD support enabled before individual disks can be passed through:
megacli -AdpGetProp enablejbod -aALL megacli -AdpSetProp EnableJBOD 1 -aALL megacli -PDMakeGood -PhysDrv[252:0] -Force -a0 megacli -PDMakeJBOD -PhysDrv[252:0] -a0
The same commands are available in the controller's pre-boot CLI, reached with Ctrl+Y during boot, but without the megacli prefix — for example AdpGetProp enablejbod -aALL. Use the console to reach it: Working with the IPMI interface.
CacheCade
Create an SSD-based cache for virtual drive 0 and assign it:
megacli -CfgCacheCadeAdd -r1 -Physdrv[252:6,252:7] WB -assign -L0 -a0 megacli -CfgCacheCadeDsply -a0
Worth setting up
A degraded array keeps working, which is the point of RAID — and also why a failed disk can go unnoticed for months until the second one fails. Configure monitoring that alerts you on a state change, rather than checking by hand occasionally.
If a check shows anything other than Optimal, open a ticket from your client area with the output of the status commands, and we will arrange a replacement.
