check_by_ssh question

Paul L. Allen pla at softflare.com
Mon Mar 29 03:54:32 CEST 2004


[Anyone who isn't Andreas should probably skip this, or die of boredom] 

Andreas Ericsson writes: 

> Paul L. Allen wrote:
>>>> Andreas Ericsson writes:
>>>> I haven't seen anything that doesn't have local exploits.
>>> 
>>> I have. It's called Owl, and you can find it at www.openwall.com/Owl
>> 
>> You can add TeX to that list.  The number of commonly-used server
>> applications that have never had a local or remote exploit is zero. 
>> 
> Wrong. I've written plenty of programs that are unexploitable. Small ones, 
> I'll give you that, but programming ideology scales infinitely.

Yeah, I've written "Hello World" programs too.  Your understanding of
scaling problems is woefully lamentable. 

>>> And most patches come out because a problem has been detected.
>> 
>> Which is why I said all you can hope is that you're not one of the
>> unlucky ones that gets hit before there's a patch.
>
> A downright lie. I've patched ntp, postfix and vsftpd BEFORE running them, 
> which means I proactively blocked one or more exploits from ever 
> happening.

OK, you patched problems you managed to detect before running them.  Unless
you are God, you cannot be confident that you have found ALL of the
problems.  I don't claim to be God, so I understand that there are likely
to be things I won't pick up on that somebody else does. 

>> You can vet the
>> sources of critical stuff, but even then you're likely to miss
>> something.  You can detect obvious stuff with automated tools, you
>> can detect less obvious stuff with human inspection, but you can't
>> detect stuff that you never imagined could be exploited that way.
> 
> This is where it becomes obvious to me that you're not a programmer (at 
> least not a very good one).

Ah, you're descending into ad hominem - a sure sign of defeat.  I
remember when race hazards impinged upon programmer awareness.  I hadn't
realized the problem, nor had many others, but it existed and was
exploited.  Unlike you, I am not so foolish as to believe we have found
all exploitable mechanisms.  These days everyone knows about buffer
overflows and race hazards (most kernels have been fixed to eliminate
race hazards running scripts).  What next?  It is incredible hubris
to exclaim that every possible exploitable mechanism is now known. 

>> We don't have any local users on our servers who don't already know the
>> root password.  I wouldn't dream of having a server that is also a
>> general box for ordinary users to play on.  Even apart from the security
>> issues, I don't want an ordinary user hogging the CPU or doing something
>> else that degrades the service. 
>> 
> Couple this with any remote exploit to get a shell as the 'nobody' (or 
> 'anybody', really) user, and voilá. All the sudden there's a user on the 
> system that doesn't know the root password. End of this part of 
> discussion, ok?

No, not OK.  As I have REPEATEDLY explained to you, our setup is such
that somebody who can use a remote exploit on our monitoring machine
can also use that same remote exploit on our monitored machines.  Actually,
our monitoring machine runs fewer services than our monitored machines so
is less likely to be remotely exploited in the first place. 

>> There are always remote exploits.
>
> Not really, no. The proper way to run networking services that need 
> root-access to bind ports, devices, etc. etc is to have them behave as 
> follows;
[...]
Which blocks all the mechanisms you know of but may not block mechanisms
you have not thought of. 

BTW, when TCP/IP came out, it was all in userspace except for the hardware
driver for the NIC.  Efficiency concerns forced most of the stack into
the kernel.  These days you could argue that TCP/IP is as important as
being able to read disk drives and so it now belongs in the kernel.  Either
way, there is a lot of networking code in kernel space that can be invoked
from user space.  Have you vetted the entire stack yet? 

>> You're right that the monitored servers *could* be compromised that
>> way.  But with no "ordinary" local users then the monitoring server
>> could only be hacked using a remote exploit which is very likely to
>> be usable on all the monitored servers too.
>
> However, the monitored servers doesn't contain a perfect map of the 
> network that the nagios server supplies. Also, the 'other' servers don't 
> have the key required to do it all without hoping for the exploitable 
> network services to be running.

But the monitoring server runs fewer services than the monitored servers,
and some of those services are not accessible externally.  The monitoring
server does not run IMAP, POP3 or SMTPD.  The monitoring server does run
bind, but that's not accessible from the outside. 

In an *ideal* world you might be right.  In the real world, you're not.
At least not for my situation.  If somebody has a remote exploit for
my monitoring server they have that same exploit for the monitored servers
so closing off a mechanism that allows them to use the monitoring server
to exploit monitored servers is utterly futile.  There comes a point where
"you must close all possible exploits" has to be tempered with common
sense. 

> This is where logic goes out the window. In less than ten lines of 
> shell-script, I could automate setting up perfectly invisible backdoors on 
> every last one of your monitored servers. In the 'exploit' scenario I'd 
> most likely have to do 'manual' labour for each server I wanted to 
> backdoor.

You're right, logic (yours) has gone right out of the window.  Apart from
the fact that most script kiddies don't have that sort of knowledge,
slowing people down is NOT a solution.  If it can be done it can be done.
The fact that you can slow somebody down is NOT security.  Either you
can stop them or you cannot.  One of us does not understand security
fully, and I don't think it's me.  You lambast people for not taking
steps that reduce functionality but only slow attackers down.  One of
us is out of touch with reality. 

>> But that doesn't protect you from all the other possible local exploits.
>> Far safer not to have ordinary users who don't know the root password on
>> servers.  In some ways, safer still not to have any local users because
>> the only reason for connecting to those boxes in the first place is to
>> do something as root. 
>> 
> And run network services (which can be exploited) as pseudo-users. Those  
> pseudo-users CAN get shell-accounts if the network services are exploited.

That's the standard approach.  It has flaws.  Everything has flaws. 

> Solution; Remove setuid/setgid bits on ALL programs on the server, and 
> have everybody do everything as root. Without the setuid programs, local 
> non-root users can only spawn shells as themselves when exploiting local 
> programs.

Ummm, so you're telling me to run Apache as root so it doesn't need
setuid?  I don't think so.  Apache has had many exploits, and suexec
is subject to race hazards, but bad as it is I don't want Apache running
as root for everything just so I can get rid of setuid programs. 

>>> (and that I for one would try for the fun of it).
>> 
>> Whereas I would not.  Not only because of the legal penalties but also
>> because I have a degree of respect for others.  Crackers would do it
>> for the hell of it, I wouldn't expect any sysadmin with a sense of
>> ethics to.  And I wouldn't trust any software with a team member who
>> said he'd try hacking into somebody else's machine "for the fun of it",
>> let alone security software.  I think I'll give owl a miss... 
>> 
> What I meant was to try it against my own network. Neither unethical nor 
> illegal, but equally fun.

It wasn't how it read to me.  It read like you were willing to attack
my servers for fun.  That's not something I would do to anyone.  It's
not even something I would say in jest.  Assume I said on this list, in
jest, that I'd like to attack your servers.  Then your servers got
attacked.  Who would you look to blame?  The police would take away
my computer and any computer I had access to in order to try to find out
if I'd really hacked your machines.  At the end of it they might drop
the case but I'd be unemployable because people would think I was too
clever at disguising my activities.  My employer would be bankrupt because
all their machines would be confiscated for weeks or months while they
were investigated. 

This is not something I joke about.  You did.  I have to admit that if
our machines get hit, your name is going to be top of my list of suspects.
If you were joking, it was a silly thing to say... 

>> I worry about buffer overflows in everything.
>> But again you're talking about needing a remote exploit first in order
>> to be able to use a local exploit in order to be able to use the key
>> exploit.
> 
> This is generally the usual path every attacker takes to rooting a box. 
> The key exploit just makes it a whole lot easier to root (and backdoor) 
> more servers on the network.

Again, knowing our situation, if they have an exploit for the monitoring
box they have an exploit for the majority of the monitored boxes without
needing the key exploit. 

>> The monitoring server is slightly more vulnerable than the
>> others because only it runs the NCSA daemon.  But since so few machines
>> run an NCSA daemon compared to machines that run apache, bind, etc.,
>> black-hats are less likely to spend time on it.
> 
> You're hoping for things that might not be true.
> Remember; Assumption is the mother of all fuckups.

The monitoring box runs fewer services than the monitoring boxes.
Because I've switched from check_by_ssh to NCSA I've eliminated your
key exploit.  If the NCSA daemon is exploitable, the monitoring box
is the only one running it and so is the only one at risk, and if
compromised through that route it does not lead to the compromise of
other machines.  In addition, because NCSA is so rare, it is less likely
to be compromised in the first place.  In addition, rebuilding the
monitoring machine causes us less problems than rebuilding all the
monitored machines. 

It's not possible to be 100% secure and still do anything useful.  What
is possible is to minimize the risks and to minimize the consequences in
an intelligent fashion. 

> Rooting through physical access IS impossible to safeguard against, and 
> it's supposed to be so I'm guessing it won't change.

It's impossible.  VAXes back in the 80s had keyswitches to prevent
certain types of operation (such as rebooting).  Of course, they all
had the same key, and any idiot who could open the cabinet could hot-wire
the switch.  You can come up with all sorts of schemes such as flash ROM
on the processor holding a public key and the private key has to be
entered from floppy, but replace the hardware and you get in.  You might
just achieve the objective with encrypted disk drives. 

>> UK law makes it a criminal offence as long as one end (attacker or
>> attacked machine) is in the UK.  Most countries have extradition laws. 
>> 
> Conducting an international investigation is not a small thing to do, and 
> will probably only happen if the crime has serious implications (my turn 
> to assume). Also, it's worth noting that script-kiddies usually don't know 
> anything about laws, and care about them even less.

I don't even hope to get script kiddies using remote exploits with the
law.  What I hope is that the law deters those with physical access. 

>> And why is there a large team of
>> programmers vetting all open-source sources for this sort of problem?
>> Did none of them know that you've already vetted EVERYTHING and they
>> could have just asked you? 
>> 
> I haven't gone through everything. Networking programs and setuid/setgid 
> binaries have priority. The rest of them can't really be used to elevate 
> privileges as long as the system is set up properly.

Oh dear. Two assumptions that are not supportable.  The first is that
systems are set up properly.  The second is that you know all possible
exploit mechanisms.  I would still bet that in years to come an exploit
mechanism will appear that will cause you to say "I didn't realize you
could do that." 

>> You're not running a fully-audited version.  You never will, because
>> new versions will appear faster than you can vet all changes to 
>> everything installed unless you're superhuman.
> 
> This is a problem we're keenly aware of.
> Owl currently uses glibc-2.1.3, which is two minor versions below the 
> latest current one. This is a result of the fact that auditing the code 
> for it took well over a year to complete.

So please do not call people idiots because they have examined the
weaknesses of their systems and evaluated the likely exploit mechanisms
and the costs of blocking them and reached conclusions that differ from
the ones you reached after looking at your own systems. 

> The result is that the vast majority of our patches now can't easily be 
> applied to the current glibc. We are working on it, and the glibc team 
> take in every patch they think is good and doesn't break standards.

It's not easy to do things right.  You get locked into historical errors.
You get blocked by people who are hostile to your changes because it
causes too much work on their part.  Welcome to reality, where you have to
make compromises... 

>> The day you have everything audited and can keep on top of
>> changes is the day you can corner the market in distros and probably
>> topple Microsoft too.  Until then you have to worry about local exploits
>> and remote exploits, just like the rest of us. 
>> 
> Yes, probably so. But the downside of totally secure systems is lower 
> userfriendliness. Most users want things to be easy, so they decrease 
> security in favour of usability.

Even higher up the food chain, people want things to be possible.  So if
it's the choice between running Apache knowing there are vulnerabilities
or telling our clients their websites no longer work, I don't have much
choice.  In an ideal world Apache would have no vulnerabilities or if it
did I could tell our clients their websites have to go down for a couple
of months while I vet the code.  In the real world, we have to run
web servers. 

>> Then there are all the kernel exploits.  You cannot hope to check
>> everything by yourself and you cannot check for exploits that rely
>> upon principles you never thought of.
> 
> That's why Owl is developed by a team, and not by me alone.

That's good, but how do you know that the team has thought of EVERYTHING?
I know my limitations.  I know I perceive things differently from my
co-workers but that they have their own limitations.  No matter how large
the team, there is no guarantee that somebody outside of the team can't
think of something nobody in the team thought of.  To suggest otherwise
is to adopt the sort of probabalistic argument you dislike me using. 

>> BTW, did you disassemble gcc?  There is no other way of being sure
>> that you don't have a Ken Thompson back-door unless ALL owl development
>> was done on machines that have never connected to the Internet before
>> the standard password routines were replaced by Owl on those machines.
>> Recompiling gcc from source, patched or not, does not get rid of a
>> Ken Thompson back-door.
> 
> Yes it does. In fact, simply renordering the variable declarations is 
> enough to get rid of the kt-backdoor, since it's only capable of applying 
> code to copies of itself, and not other programs (reordering the variable 
> declarations generates different assembly and object code, which is enough 
> for it to 'miss' on the recognition).

Are you SURE of that?  It's not the behaviour I'd give it, and KT is a
hell of a lot smarter than I am.  I don't recall anything in his
description of it which said that it relied on the order of variable
declarations.  Some implementations might, but I'd be very reluctant to
say that they all have to. 

> The nagios server is the hackers primary target, since it provides him 
> with a map of the network, and a listing of (most) of the services running 
> on each host which means he doesn't have to run portscans or random 
> connection attempts which might be picked up by any NIDS.

Which means that hacking the monitoring server to get a list of monitored
servers is more important than being able to run the key exploit.  Many,
perhaps most, of the monitored servers will be vulnerable to the same
exploit as the monitoring server.  But even that is relatively unimportant,
because most hacks simply seach IP blocks looking for vulnerable services. 

> Also, outgoing traffic from the nagios server is less likely to be 
> detected as intrusive,  since it already shoots packets high and low to 
> every monitored part of the network.

Now there, finally, I'll agree that you hit something I hadn't considered.
But switching to passive service checks defeats it. 

> Any passwordless key with or without restrictions can be used to obtain 
> shell-access as the remote key-holders user.

So patch remote exploits as soon as you know of them. 

> In an environment where all the local users have access to the 
> root-password, there's no need for setuid/setgid binaries.

But eliminating setuid/setgid binaries may mean that some daemons
have to run as root without dropping privileges, with the problems
that entails. 

In fact, what you suggest is the worst of all worlds.  If you have local
users with bad passwords and there is a local exploit, you're screwed.
Any daemons that would otherwise need setuid or setgid become far more
vulnerable to remote exploit since there doesn't need to be a local
exploit to get root.. 

> There's no need for any networking daemon to run as root, since the port 
> it listens to can easily be forwarded to one outside the privileged range 
> with a few simple iptables rules. Coupled with chroot jails where 
> possible, this can turn access granting exploits into mere DoS attacks. 
> Still annoying, but not by a longshot as dangerous to business.

Yeah, we could do that with Apache.  And thereby lose the ability to
have it change UID/GID when accessing virtual hosts.  So then any of
our clients can write a CGI to walk all over any of our other clients,
reading and/or corrupting private data. 

And yes, I know there are race hazards in suexec.  I have to live with
them, knowing that the alternatives are worse.  I live in the real world... 

-- 
Paul Allen
Softflare Support 




-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null





More information about the Users mailing list