Problem with check_openmanage plugin and storage
Nic Bernstein
nic at onlight.com
Tue Jun 18 18:03:32 CEST 2013
We've recently been experimenting with Trond Hasle Amundsen's
check_openmanage on a large network with about a hundred Dell servers of
various ages, capabilities, etc. Mostly PE-2950, R210, R410 and R720.
Much thanks to Trond for all his great work on Nagios plugins and other
projects, by the way.
We've hit a wall, however, with the storage monitoring aspects of this
plugin.
For example, here's a quite specific case. This is a new PE R720, in debug:
onlight at monitor:~$ check_openmanage -H host -C secret -d
System: PowerEdge R720 OMSA version: 7.1.0
ServiceTag: ####### Plugin version: 3.7.9
BIOS/date: 1.2.6 05/10/2012 Checking mode: SNMPv2c UDP/IPv4
-----------------------------------------------------------------------------
Storage Components
=============================================================================
STATE | ID | MESSAGE TEXT
---------+----------+--------------------------------------------------------
OK | 0 | Controller 0 [PERC H310 Mini] is Ready
WARNING | 0:0:1:0 | Physical Disk 0:1:0 [Ata ST2000DM001-9YN164, 2.0TB] on ctrl 0 is Online, Not Certified
WARNING | 0:0:1:1 | Physical Disk 0:1:1 [Ata ST2000DM001-9YN164, 2.0TB] on ctrl 0 is Online, Not Certified
OK | 0:0 | Logical Drive '/dev/sda' [RAID-1, 1862.50 GB] is Ready
OK | 0:0 | Connector 0 [SAS] on controller 0 is Ready
OK | 0:1 | Connector 1 [SAS] on controller 0 is Ready
OK | 0:0:1 | Enclosure 0:0:1 [Backplane] on controller 0 is Ready
-----------------------------------------------------------------------------
Chassis Components
=============================================================================
STATE | ID | MESSAGE TEXT
---------+------+------------------------------------------------------------
OK | 0 | Memory module 0 [DIMM_A1, 4096 MB] is Ok
OK | 1 | Memory module 1 [DIMM_A2, 4096 MB] is Ok
OK | 2 | Memory module 2 [DIMM_A3, 4096 MB] is Ok
OK | 3 | Memory module 3 [DIMM_A4, 4096 MB] is Ok
OK | 0 | Chassis fan 0 [System Board Fan1 RPM] reading: 1200 RPM
OK | 1 | Chassis fan 1 [System Board Fan2 RPM] reading: 1080 RPM
OK | 2 | Chassis fan 2 [System Board Fan3 RPM] reading: 1200 RPM
OK | 3 | Chassis fan 3 [System Board Fan4 RPM] reading: 1080 RPM
OK | 4 | Chassis fan 4 [System Board Fan5 RPM] reading: 1080 RPM
OK | 5 | Chassis fan 5 [System Board Fan6 RPM] reading: 1080 RPM
OK | 0 | Power Supply 0 [AC]: Presence detected
OK | 0 | Temperature Probe 0 [System Board Inlet Temp] reads 26 C (min=3/-7, max=42/47)
OK | 1 | Temperature Probe 1 [System Board Exhaust Temp] reads 33 C (min=8/3, max=70/75)
OK | 2 | Temperature Probe 2 [CPU1 Temp] reads 49 C (min=8/3, max=83/88)
OK | 0 | Processor 0 [Intel Xeon E5-2603 0 1.80GHz] is Present
OK | 0 | Voltage sensor 0 [CPU1 VCORE PG] is Good
OK | 1 | Voltage sensor 1 [System Board 3.3V PG] is Good
OK | 2 | Voltage sensor 2 [System Board 5V PG] is Good
OK | 3 | Voltage sensor 3 [CPU1 PLL PG] is Good
OK | 4 | Voltage sensor 4 [System Board 1.1V PG] is Good
OK | 5 | Voltage sensor 5 [CPU1 M23 VDDQ PG] is Good
OK | 6 | Voltage sensor 6 [CPU1 M23 VTT PG] is Good
OK | 7 | Voltage sensor 7 [System Board FETDRV PG] is Good
OK | 8 | Voltage sensor 8 [CPU1 VSA PG] is Good
OK | 9 | Voltage sensor 9 [CPU1 M01 VDDQ PG] is Good
OK | 10 | Voltage sensor 10 [System Board NDC PG] is Good
OK | 11 | Voltage sensor 11 [CPU1 VTT PG] is Good
OK | 12 | Voltage sensor 12 [System Board 1.5V PG] is Good
OK | 13 | Voltage sensor 13 [PS2 PG Fail] is Good
OK | 14 | Voltage sensor 14 [System Board PS1 PG Fail] is Good
OK | 15 | Voltage sensor 15 [System Board BP1 5V PG] is Good
OK | 16 | Voltage sensor 16 [CPU1 M01 VTT PG] is Good
OK | 17 | Voltage sensor 17 [PS1 Voltage 1] reads 114 V
OK | 0 | Battery probe 0 [System Board CMOS Battery] is Presence Detected
OK | 0 | Amperage probe 0 [PS1 Current 1] reads 0.6 A
OK | 1 | Amperage probe 1 [System Board Pwr Consumption] reads 56 W
OK | 0 | Chassis intrusion 0 detection: Ok (Not Breached)
OK | 0 | SD Card 0 [vFlash] is Absent
-----------------------------------------------------------------------------
Other messages
=============================================================================
STATE | MESSAGE TEXT
---------+-------------------------------------------------------------------
OK | ESM log health is Ok (less than 80% full)
OK | Chassis Service Tag is sane
This run exits with 1 (WARNING).
We're not sure we agree with the decision to make the fact that a disk
is not Dell Certified a Warning, but we can at least understand that.
So, what if we exclude storage, with --no-storage?
onlight at monitor:~$ check_openmanage -H host -C secret -d --no-storage
System: PowerEdge R720 OMSA version: 7.1.0
ServiceTag: ####### Plugin version: 3.7.9
BIOS/date: 1.2.6 05/10/2012 Checking mode: SNMPv2c UDP/IPv4
-----------------------------------------------------------------------------
Chassis Components
=============================================================================
STATE | ID | MESSAGE TEXT
---------+------+------------------------------------------------------------
OK | 0 | Memory module 0 [DIMM_A1, 4096 MB] is Ok
OK | 1 | Memory module 1 [DIMM_A2, 4096 MB] is Ok
OK | 2 | Memory module 2 [DIMM_A3, 4096 MB] is Ok
OK | 3 | Memory module 3 [DIMM_A4, 4096 MB] is Ok
OK | 0 | Chassis fan 0 [System Board Fan1 RPM] reading: 1080 RPM
OK | 1 | Chassis fan 1 [System Board Fan2 RPM] reading: 1080 RPM
OK | 2 | Chassis fan 2 [System Board Fan3 RPM] reading: 1200 RPM
OK | 3 | Chassis fan 3 [System Board Fan4 RPM] reading: 1080 RPM
OK | 4 | Chassis fan 4 [System Board Fan5 RPM] reading: 1080 RPM
OK | 5 | Chassis fan 5 [System Board Fan6 RPM] reading: 1080 RPM
OK | 0 | Power Supply 0 [AC]: Presence detected
OK | 0 | Temperature Probe 0 [System Board Inlet Temp] reads 26 C (min=3/-7, max=42/47)
OK | 1 | Temperature Probe 1 [System Board Exhaust Temp] reads 33 C (min=8/3, max=70/75)
OK | 2 | Temperature Probe 2 [CPU1 Temp] reads 49 C (min=8/3, max=83/88)
OK | 0 | Processor 0 [Intel Xeon E5-2603 0 1.80GHz] is Present
OK | 0 | Voltage sensor 0 [CPU1 VCORE PG] is Good
OK | 1 | Voltage sensor 1 [System Board 3.3V PG] is Good
OK | 2 | Voltage sensor 2 [System Board 5V PG] is Good
OK | 3 | Voltage sensor 3 [CPU1 PLL PG] is Good
OK | 4 | Voltage sensor 4 [System Board 1.1V PG] is Good
OK | 5 | Voltage sensor 5 [CPU1 M23 VDDQ PG] is Good
OK | 6 | Voltage sensor 6 [CPU1 M23 VTT PG] is Good
OK | 7 | Voltage sensor 7 [System Board FETDRV PG] is Good
OK | 8 | Voltage sensor 8 [CPU1 VSA PG] is Good
OK | 9 | Voltage sensor 9 [CPU1 M01 VDDQ PG] is Good
OK | 10 | Voltage sensor 10 [System Board NDC PG] is Good
OK | 11 | Voltage sensor 11 [CPU1 VTT PG] is Good
OK | 12 | Voltage sensor 12 [System Board 1.5V PG] is Good
OK | 13 | Voltage sensor 13 [PS2 PG Fail] is Good
OK | 14 | Voltage sensor 14 [System Board PS1 PG Fail] is Good
OK | 15 | Voltage sensor 15 [System Board BP1 5V PG] is Good
OK | 16 | Voltage sensor 16 [CPU1 M01 VTT PG] is Good
OK | 17 | Voltage sensor 17 [PS1 Voltage 1] reads 112 V
OK | 0 | Battery probe 0 [System Board CMOS Battery] is Presence Detected
OK | 0 | Amperage probe 0 [PS1 Current 1] reads 0.6 A
OK | 1 | Amperage probe 1 [System Board Pwr Consumption] reads 56 W
OK | 0 | Chassis intrusion 0 detection: Ok (Not Breached)
OK | 0 | SD Card 0 [vFlash] is Absent
-----------------------------------------------------------------------------
Other messages
=============================================================================
STATE | MESSAGE TEXT
---------+-------------------------------------------------------------------
OK | ESM log health is Ok (less than 80% full)
OK | Chassis Service Tag is sane
OOPS! Something is wrong with this server, but I don't know what. The global
system health status is WARNING, but every component check is OK. This may
be a bug in the Nagios plugin, please file a bug report.
This yields exit code 3 (UNKNOWN).
Now, just for argument's sake, let's say we obviate the check for
certified drives, by commenting out the "workaround for OMSA 7.1.0
bug" code (just a handy little short-cut). Here's what we get then:
onlight at monitor:~$ check_openmanage -H host -C secret -d
System: PowerEdge R720 OMSA version: 7.1.0
ServiceTag: ####### Plugin version: 3.7.9
BIOS/date: 1.2.6 05/10/2012 Checking mode: SNMPv2c UDP/IPv4
-----------------------------------------------------------------------------
Storage Components
=============================================================================
STATE | ID | MESSAGE TEXT
---------+----------+--------------------------------------------------------
OK | 0 | Controller 0 [PERC H310 Mini] is Ready
WARNING | 0:0:1:0 | Physical Disk 0:1:0 [Ata ST2000DM001-9YN164, 2.0TB] on ctrl 0 is Online
WARNING | 0:0:1:1 | Physical Disk 0:1:1 [Ata ST2000DM001-9YN164, 2.0TB] on ctrl 0 is Online
OK | 0:0 | Logical Drive '/dev/sda' [RAID-1, 1862.50 GB] is Ready
OK | 0:0 | Connector 0 [SAS] on controller 0 is Ready
OK | 0:1 | Connector 1 [SAS] on controller 0 is Ready
OK | 0:0:1 | Enclosure 0:0:1 [Backplane] on controller 0 is Ready
-----------------------------------------------------------------------------
Chassis Components
=============================================================================
STATE | ID | MESSAGE TEXT
---------+------+------------------------------------------------------------
OK | 0 | Memory module 0 [DIMM_A1, 4096 MB] is Ok
OK | 1 | Memory module 1 [DIMM_A2, 4096 MB] is Ok
OK | 2 | Memory module 2 [DIMM_A3, 4096 MB] is Ok
OK | 3 | Memory module 3 [DIMM_A4, 4096 MB] is Ok
OK | 0 | Chassis fan 0 [System Board Fan1 RPM] reading: 1080 RPM
OK | 1 | Chassis fan 1 [System Board Fan2 RPM] reading: 1200 RPM
OK | 2 | Chassis fan 2 [System Board Fan3 RPM] reading: 1200 RPM
OK | 3 | Chassis fan 3 [System Board Fan4 RPM] reading: 1080 RPM
OK | 4 | Chassis fan 4 [System Board Fan5 RPM] reading: 1080 RPM
OK | 5 | Chassis fan 5 [System Board Fan6 RPM] reading: 1200 RPM
OK | 0 | Power Supply 0 [AC]: Presence detected
OK | 0 | Temperature Probe 0 [System Board Inlet Temp] reads 26 C (min=3/-7, max=42/47)
OK | 1 | Temperature Probe 1 [System Board Exhaust Temp] reads 33 C (min=8/3, max=70/75)
OK | 2 | Temperature Probe 2 [CPU1 Temp] reads 48 C (min=8/3, max=83/88)
OK | 0 | Processor 0 [Intel Xeon E5-2603 0 1.80GHz] is Present
OK | 0 | Voltage sensor 0 [CPU1 VCORE PG] is Good
OK | 1 | Voltage sensor 1 [System Board 3.3V PG] is Good
OK | 2 | Voltage sensor 2 [System Board 5V PG] is Good
OK | 3 | Voltage sensor 3 [CPU1 PLL PG] is Good
OK | 4 | Voltage sensor 4 [System Board 1.1V PG] is Good
OK | 5 | Voltage sensor 5 [CPU1 M23 VDDQ PG] is Good
OK | 6 | Voltage sensor 6 [CPU1 M23 VTT PG] is Good
OK | 7 | Voltage sensor 7 [System Board FETDRV PG] is Good
OK | 8 | Voltage sensor 8 [CPU1 VSA PG] is Good
OK | 9 | Voltage sensor 9 [CPU1 M01 VDDQ PG] is Good
OK | 10 | Voltage sensor 10 [System Board NDC PG] is Good
OK | 11 | Voltage sensor 11 [CPU1 VTT PG] is Good
OK | 12 | Voltage sensor 12 [System Board 1.5V PG] is Good
OK | 13 | Voltage sensor 13 [PS2 PG Fail] is Good
OK | 14 | Voltage sensor 14 [System Board PS1 PG Fail] is Good
OK | 15 | Voltage sensor 15 [System Board BP1 5V PG] is Good
OK | 16 | Voltage sensor 16 [CPU1 M01 VTT PG] is Good
OK | 17 | Voltage sensor 17 [PS1 Voltage 1] reads 114 V
OK | 0 | Battery probe 0 [System Board CMOS Battery] is Presence Detected
OK | 0 | Amperage probe 0 [PS1 Current 1] reads 0.6 A
OK | 1 | Amperage probe 1 [System Board Pwr Consumption] reads 56 W
OK | 0 | Chassis intrusion 0 detection: Ok (Not Breached)
OK | 0 | SD Card 0 [vFlash] is Absent
-----------------------------------------------------------------------------
Other messages
=============================================================================
STATE | MESSAGE TEXT
---------+-------------------------------------------------------------------
OK | ESM log health is Ok (less than 80% full)
OK | Chassis Service Tag is sane
Again, as with the original case, exit code is 1 (WARNING).
Is there any way around this? Should I be disabling global health
checks? Here's a run to test that, and it works:
onlight at monitor:~$ check_openmanage -H host -C secret -b pdisk=all
OK - System: 'PowerEdge R720', SN: '#######', 16 GB ram (4 dimms), 1 logical drives, 2 physical drives
Interestingly, when combining the blacklist with debug ("-d -b
pdisk=all"), the exit code is 3 (UNKNOWN), but with debug off, it's 0 (OK).
So, I guess what I'm wondering is why we need to blacklist the physical
disks (pdisk) instead of using --no-storage? Shouldn't --no-storage
also cause globalstatus to be ignored?
I can furnish SNMP walk output if that's useful.
Cheers,
-nic
--
Nic Bernstein nic at onlight.com
Onlight, Inc. www.onlight.com
219 N. Milwaukee St., Suite 2a v. 414.272.4477
Milwaukee, Wisconsin 53202
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-lists.org/archive/users/attachments/20130618/544e0af9/attachment.html>
-------------- next part --------------
------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:
Build for Windows Store.
http://p.sf.net/sfu/windows-dev2dev
-------------- next part --------------
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
::: Messages without supporting info will risk being sent to /dev/null
More information about the Users
mailing list