nagios server networking glitch..?

[Andreas Ericsson]

Patrick Friedel wrote:
> I know this isn't really Nagios related, but since it causes Nagios to 
> freak out when it happens, I hoped the list might have an idea of where 
> to check.  Occasionally, not more than once per day, my Nagios box gets 
> a route for an IP address that sends the traffic away from the intranet 
> WAN out to the internet, causing the packet to die and Nagios begins to 
> page me.  Caught it happening today and logged some information:
> 
> Remote IP: 204.75.219.254
> Nagios server: 199.242.227.113
> Intranet WAN gateway: 199.242.227.253
> Internet WAN gateway: 199.242.227.254
> 
> When it's failing, it goes to:
> pjf at jord:~$ ip route get to 204.75.219.254
> 204.75.219.254 via 199.242.227.254 dev eth0  src 199.242.227.113
>    cache <redirected>  mtu 1500 advmss 1460 hoplimit 64
> 
> The correct route is:
> pjf at jord:~$ ip route get to 204.75.219.254
> 204.75.219.254 via 199.242.227.253 dev eth0  src 199.242.227.113
>    cache  mtu 1500 advmss 1460 hoplimit 64
> 
>  Standard Sarge Debian distro, not running any funny routing daemons.  
> netstat is, IIRC, completely ignorant of the new route. The default 
> route sticks at .253, like it should, and no other entries in the 
> netstat routing table.  None of the other hosts are affected, so it's 
> not a global issue, it's usually highly specific to a single IP address. 
> (the problem host, however, seems to rotate, it's not a single IP 
> problem _that_ way.) 98% of my traffic goes through the intranet, only a 
> small percentage goes out the internet link.  I _suspect_ it's something 
> weird on the nagios monitor box, as my usual first reaction is to ping 
> the dead host from my workstation, where it works fine, then have pings 
> fail from the nagios box.  The only thing I can think of is that the 
> monitor box gets an ICMP REDIRECTED packet from the intranet router for 
> one of the internet monitored hosts and it sticks somehow.
> 

This would, if it's what actually happens, be a kernel-bug, as redirects 
are per target IP's.

If the nagios box is reachable from the internet somehow (apparently it 
is, since you're checking things there and the possibility for black 
IP-magic is nigh endless), some malicious person could also be 
redirecting your traffic on purpose.

>  Ideas?
> 

Add firewall rules that prevents sending packets through the internet 
unless they're destined for the hosts on your DMZ, and add an iptables 
rule to log all inbound ICMP-packets from the default gateway.
iptables -I INPUT -p icmp -s gatewayIP -j LOG
should do the trick. Then you can start debugging it properly.

It might also help to run mtr (http://www.bitwizard.nl/mtr/) while this 
is happening. mtr is available from just about any apt-repository. It 
sends a lot of ICMP echo-requests with low TTL's which is fairly useful 
when debugging misbehaving routers.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Lead Developer


-------------------------------------------------------
This SF.Net email is sponsored by Yahoo.
Introducing Yahoo! Search Developer Network - Create apps using Yahoo!
Search APIs Find out how you can build Yahoo! directly into your own
Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null