Nagios custom script checklist

Host

#cd /usr/lib64/nagios/plugins
- create script with EXIT 0/1/2  ie "check_raid_disks"
#nano /etc/nagios/nrpe.cfg
- define command[check_raid_disks]=/usr/bin/sudo /usr/lib64/nagios/plugins/check_raid_disks
#/etc/init.d/nrpe restart

* please notice that command definition includes sudo, normally nrpe scripts do not require this, but here we will be running custom executable inside our script. If sudo is not used nrpe responds with "NRPE: Unable to read output". Sudo-ers workaroud follows:

#nano /etc/sudoers (add nrpe user rule, comment out TTY requirement)
   nrpe ALL=(ALL) NOPASSWD:/usr/lib64/nagios/plugins/check_raid_disks
   # Defaults    requiretty
#setenforce 0 (disable selinux)

Nagios server

#/usr/lib64/nagios/plugins/check_nrpe -H 172.30.2.191 -c check_raid_disks
#nano /etc/nagios/servers/awsdb.cfg  (server name file)
- define service{
        use                             generic-service
        host_name                       awsdb
        service_description             Raid Disks
        check_command                   check_nrpe!check_raid_disks
         }
#/etc/init.d/nagios restart

Example script

#!/bin/sh

# Exit codes

STATE_OK=0

STATE_WARNING=1

STATE_CRITICAL=2

STATE_UNKNOWN=3

# server awsDB - virtual drives (vd's) count: 3

# server awsDB - physical drives count: 8

hds=$(/root/bin/MegaCli64 -PDList -aALL | grep state | grep Online -c)

# echo $dds

# Check for missing parameters

if [[ -z "$hds" ]]; then

  echo "Script error, check setup, environment or parameters"

  exit 2

fi

if [[ "$hds" = "8" ]]; then

  echo "OK - virtual disk count: $hds"

  exit 0

fi

if [[ "$hds" -lt "8" ]]; then

  echo "CRITICAL - not all disks are Online: $hds/8"

  exit 2

fi

echo "WARNING - Script error"

exit 1

Nagios debug command notes

#/usr/lib64/nagios/plugins/check_nrpe -H 172.30.2.246 -c check_cpu

#nagios -v /etc/nagios/nagios.cfg

Server monitoring recipe with SNMP: Observium + Nagios

Objective

Everyone having at least couple of servers, even a single server, would want to monitor it eventually. Some time ago I used MRTG for all that, but as the needs expanded I could do less and less with it and in the end it even became too complicated to use. MRTG is powerful, yet vulnerable to simple server restarts - you have to remap your pins.

Recipe

Will jump to it right away: the best option currently is Observium + Nagios. Will tell about the first one in a separate paragraph, it might suffer from an early death some day, but currently its a good tool for the job. I ended up using two tools because Nagios has a very good alerting system, but lacks interfaces and as you probably guessed already Observium has interfaces, but lacks alerting system.

Observium

The peckers behind this tool are pretty questionable. Some time ago they had a fundraising campaign to collect some doe and implement an alerting system. After funds were raised - they removed the promised functionality from the Community release and made it part of their paid version. Money is money, but hey, Internet knows everything. Further more I tried to communicate with them on Facebook - all my page messages and comments where removed and all PM's ignored.
Nevermind the folks, their tool is good for one thing - drawing nice charts:

Nagios

Where Observium fails - Nagios can help. Its an open source project, no need to tell more. It lacks interfaces and historical information (excluding payed plugins and extensions), but it has a powerful alerting system. Just setup a couple of users with emails and you are done:

Conclusions

Nagios allows you to receive an email in the middle of the forest when your backup drive hits a warning limit while Observium helps you analyze and plan you infrastructure, workloads and record historical events. I can now see that admin still hasnt added memory to our webserver and I asked for that a week ago.