Connection refused or timed out

[Michael Saldivar]

What should I look for when troubleshooting a "Connection refused or timed
out" when checking via NRPE to a Linux machine?

I'm having this problem on two targets.  So far, I've installed NRPE on 5
machines, and 3 worked without a hitch, and these two are giving me
headaches.   Some info about the targets.

10.5.10.50 is my Nagios server, dfslit1
10.5.10.90 is the target client, dfslrdb01. The second problem machine
produces identical output to this one, aside from a different CentOS
version.

Both (well, all 5) are running this version of NRPE:

NRPE - Nagios Remote Plugin Executor
Copyright (c) 1999-2008 Ethan Galstad (nagios at nagios.org)
Version: 2.12

Client OS versions are:
CentOS release 5.3 (Final)
CentOS release 5.2 (Final)

>From /var/log/messages when I restart the NRPE:

Aug 19 13:51:08 dfslrdb01 nrpe[13249]: Starting up daemon
Aug 19 13:51:08 dfslrdb01 nrpe[13249]: Listening for connections on port
5666
Aug 19 13:51:08 dfslrdb01 nrpe[13249]: Allowing connections from:
127.0.0.1,10.5.10.50
Aug 19 13:51:53 dfslrdb01 nrpe[13249]: Caught SIGTERM - shutting down...
Aug 19 13:51:53 dfslrdb01 nrpe[13249]: Cannot remove pidfile
'/var/run/nrpe.pid' - check your privileges.
Aug 19 13:51:53 dfslrdb01 nrpe[13249]: Daemon shutdown
Aug 19 13:51:53 dfslrdb01 nrpe[13266]: Starting up daemon
Aug 19 13:51:53 dfslrdb01 nrpe[13266]: Listening for connections on port
5666
Aug 19 13:51:53 dfslrdb01 nrpe[13266]: Allowing connections from:
127.0.0.1,10.5.10.50

(10.5.10.50 is my Nagios server)
I look in /var/run/nrpe.pid and the file is gone when NRPE stops, and is
re-created when it starts.  When it's running, permissions are:

[root at dfslrdb01 ~]# ll /var/run/nrpe.pid
-rw-r--r-- 1 root root 6 Aug 19 14:09 /var/run/nrpe.pid

Checking to see if the port's open:

[mikes at dfslrdb01 ~]$ netstat -l | grep 5666
tcp        0      0 *:5666                      *:*
LISTEN

Command definition from /etc/nagios/nrpe.cfg:

command[check_disk]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p
/dev/sda1

Running the command on the target machine:

[mikes at dfslrdb01 ~]$ sudo su - nagios
-sh-3.2$ /usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /dev/sda1
DISK OK - free space: / 294574 MB (45% inode=99%);|
/=358494MB;550890;619751;0;688613


Running it at the command line from the server; result is the same with or
without -n:

nagios at dfslit1:~$ /usr/lib/nagios/plugins/check_nrpe -H 10.5.10.90 -c
check_disk
Connection refused or timed out

nagios at dfslit1:~$ tcptraceroute 10.5.10.90 5666
Selected device eth0, address 10.5.10.50, port 36013 for outgoing packets
Tracing the path to 10.5.10.90 on TCP port 5666, 30 hops max
 1  10.5.10.90  2.965 ms !C  0.841 ms !C  0.744 ms !C

nagios at dfslit1:~$ telnet 10.5.10.90 5666
Trying 10.5.10.90...
telnet: Unable to connect to remote host: No route to host

I installed from Dag's repository:
http://dag.wieers.com/rpm/FAQ.php#B

[root at dfslrdb01 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere            icmp any
ACCEPT     esp  --  anywhere             anywhere
ACCEPT     ah   --  anywhere             anywhere
ACCEPT     udp  --  anywhere             224.0.0.251         udp dpt:mdns
ACCEPT     udp  --  anywhere             anywhere            udp dpt:ipp
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ipp
ACCEPT     all  --  anywhere             anywhere            state
RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:smtp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:nfs
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:https
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:http
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:sunrpc
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:terabase
ACCEPT     udp  --  anywhere             anywhere            state NEW udp
dpt:terabase
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:newoak
ACCEPT     udp  --  anywhere             anywhere            state NEW udp
dpt:newoak
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:pxc-spvr-ft
ACCEPT     udp  --  anywhere             anywhere            state NEW udp
dpt:pxc-spvr-ft
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:rquotad
ACCEPT     udp  --  anywhere             anywhere            state NEW udp
dpt:rquotad
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp
dpt:mysql
REJECT     all  --  anywhere             anywhere            reject-with
icmp-host-prohibited

My problem seems similar to this, but neither my nor Google Translate's
German is good enough to figure out what fixed the problem.  Perhaps it's in
suiss-allemagne, but high school French is doing me no good today.
http://www.nagios-portal.org/wbb/index.php?page=Thread&threadID=12885

What should I look at next?

Thanks,
-Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-lists.org/archive/users/attachments/20090819/9bb20a67/attachment.html>
-------------- next part --------------
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
-------------- next part --------------
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null