In order to know when your box has been hacked, you have to know what your box is like under normal conditions. Something's got to trigger the "that ain't right" reflex. [grin] There are times when the hack is really obvious -- things like your website being replaced by "Chinese hackers overcome pitiful US defenses!", to paraphrase a recent common phenomena. Other times, it's not so obvious. One of my servers was hacked this past summer, with surpassing subtlety. If I hadn't been really paranoid (and somewhat lucky), I never would have realized they were there.
So, what do you look for? Well, let's start with the list of services that your box is running. Most of the time, successful remote hacks are exploits against a service that your box is running. (I'm talking attacks that give the black-hat priviliges and access to your server, not ones that will just drop your server off the Internet map for a while.) So it makes sense not to run more network services than you need. Running everything under the sun increases your chances of a hole being found in one of those services, and your box becoming vulnerable to attack.
On most Linux systems, you'll have a command called netstat. Simply put, it lets you know the status of your network. Running just the bare command, like so:
[root@linuxbox /root]# netstat Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 www.linuxbox.:2944 astro.dal.net:ircd ESTABLISHED tcp 0 0 www.linuxbox.o:ssh dragonhome.humanit:1594 ESTABLISHED tcp 0 256 www.linuxbox.o:ssh scarlett.demon.di:39735 ESTABLISHED Active UNIX domain sockets (w/o servers) Proto RefCnt Flags Type State I-Node Path unix 0 [ ] DGRAM 157112 /dev/log unix 0 [ ] DGRAM 272shows you the open sockets -- the connections between a local IP/port pair and a remote IP/port pair. It doesn't include the servers, which is what we're really interested in here. But we can see that someone on this box is connected to dal.net for IRC, and that two users are ssh'd in -- one from dragonhome.humanit[something], port 1594, and one from scarlett.demon.di[something], port 39735. (The latter is me connected to the box to use it as an example.) You can see that the remote hostnames are truncated if they're too long. Both of these are.
Much more helpful is to run netstat with the -pl options. This shows you the server processes that are listening too, and the processes that are associated with them.
[root@linuxbox /root]# netstat -pl (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 *:ftp *:* LISTEN 532/tcpserver tcp 0 0 *:smtp *:* LISTEN 462/tcpserver tcp 0 0 *:www *:* LISTEN 436/httpd tcp 0 0 *:ssh *:* LISTEN 421/sshd raw 0 0 *:icmp *:* 7 - raw 0 0 *:tcp *:* 7 - Active UNIX domain sockets (only servers) Proto RefCnt Flags Type State I-Node PID/Program name PathSo we can see that this server is running ftp, mail, www, and ssh. I'm using tcpserver to run my ftp and mail daemons, and Apache and ssh are running standalone, without a superserver.
There are lots more options out there -- check out the man page for netstat for a rather exhaustive list. If anyone feels like posting the netstat info from their system for comment, we can go over what you should and shouldn't see here. For the most part, if you don't know what it is, you probably shouldn't have it listening on a port here.
Try killing and restarting some of your services. You'll see the ports appear and disappear from netstat if they're running standalone. (Some superservers answer for the service they maintain whether it's up or not.)
Really common things that are hackable that you often see here by default:
- Rpcbind and other rpcservices -- portmap. If you're deliberately running them, make them only accessible to your internal network or whoever needs them with a firewall. If you don't know what they are, you probably don't need them.
- Sendmail. Make sure you have a recent version, and keep it patched. Securing sendmail is a whole other week's worth of discussion.
- Bind. Older versions are notorious for having holes. If you don't need it, turn it off.
- Xwindows services. Unless you're acting as an X server to remote Xwindows clients, X doesn't need to be listening on a port. You can forward X connections over ssh if you need to -- it's much safer.
If you want to run these services but don't want to have them listening on a port, there's usually some way to do that within the configuration for the service itself. We can get into detail for those if anyone has a specific service in mind.
There are other programs that do the same sort of thing -- lsof, for example. But one thing that's really important is to check your system both from the inside and from the outside. Use nmap or a similar tool to portscan yourself. You should see the same list of services available. If you see a service in one and not the other, start to be suspicious. (I'll cover nmap in another mail -- this one's getting long.)
Troubleshooting
I realize that I should say a word about basic troubleshooting procedures, though I'm sure that most of you already know this. Whenever possible, make only one change at a time. That way if something breaks, you know what the likely culprit is. So if you're not sure if you need a bunch of these services, turn one off, make sure everything you need that might have been using it still works, then turn the next one off, and so forth.
What's raw mode?
Our discussion of netstat output sparked a discussion about raw sockets.
Selected responses to netstat output
X
> tcp 0 0 *:x11 *:* LISTEN 2751/XWith X11, you don't want to stop X, you just want to stop it from listening on a port for other machines to connect to. (I'm assuming that you don't want to serve X to any machine other than your local one.) I actually did a Web page about this a few months back, since this is such a common problem.
http://www.oneeyedcrow.net/tech/securex.html
identd
> tcp 0 0 *:auth *:* LISTEN 825/identdYou'll need it if you IRC from this box. Some people disable it as a "leaks information" security concern. It's not a big deal either way, I don't think. For a very locked-down system you'd want to turn it off, but for a home system you're probably okay.
sshd
> tcp 0 0 *:x11-ssh-offset *:* LISTEN 2371/sshd > tcp 0 0 *:6011 *:* LISTEN 2415/sshd > tcp 0 0 *:https *:* LISTEN 1052/httpd > tcp 0 0 *:6012 *:* LISTEN 2495/sshdDo you use X forwarding for ssh? That's what the 6010 (x11-ssh-offset), 6011, and 6012 ports are. If you don't use it, turn it off in /etc/sshd_config, or wherever such things live for your flavor of ssh. I use OpenSSH, so for me it's
X11Forwarding noin /etc/sshd_config.
A false sense of security?
> I'll also note that this system is behind a NAT router, so maybe > running unneeded services isn't a humongous risk, but I want to know > how to do it right anyway.NAT will save you from some things, but not everything. It's no replacement for a good firewall. Much depends on how your router does the NAT. If it's one-to-one address mapping, then you're not significantly safer. If it's port forwarding, that is considerably safer, because outside attackers are less likely to know your box is there. Your NAT device can still get portscanned, of course.
> This computer was SuSE 7.1, then upgraded to 7.3. Nothing special has > ever been done to make it secure because it runs behind a firewall. > It operates on a network where the majority of computer run NT.The firewall will protect you from external threats, but most hacks happen from inside the company. I've seen varying statistics for how many that is, but most estimates range from 60% to 90%. Internal security is almost always weaker than external security, so it's easier. And just think of how many people you know that hate their jobs/bosses/co-workers...
I'm a big believer in defense in depth. Yes, I have a firewall that protects my home network. My workstation has a firewall too. That way, if somehow my husband's Windows box gets compromised, I'm okay. I run logcheck (I agree with the plug; it is a great tool) so I'm aware of problems on all my boxes. I have set key files on my workstation so that they can't be changed, even by root. That way, if my box gets compromised, the black hat will have to reboot it to trojan ssh. I'll notice that reboot. Defense in depth basically means that you're constantly thinking, "what if, what if, what if" , and you have lines of defense to fall back to.
Of course, this also means that you spend a lot of time and effort on network security. That's worth it for me, because I think it's fun. It's worth it for my company -- they pay for me, after all. [grin] But many people just don't see it as that much of a priority. You just have to decide if the risk of being hacked is worth the cost of maintaining a relatively secure network.
> I use rlogin and rcp, on my local net -- we have a router/firewall (a > Linkys somethingorother) which doesn't pass most incoming ports, so I'm > under the illusion that I can get away with things like that) because > they're so darned convenient and I hate having to type passwords every > time I want to move a file someplace. (Yes, I even have a .rhosts with > the names of our local hosts in it.) Am I being foolish in trusting the > router/firewall?Only somewhat. [grin] Like most things in security, it depends on how paranoid you want to be. You're probably okay for most script kiddies and such with that, but if a skilled black hat decided to have a go at you for some reason, that might be trouble. A quick google for "Linksys Bugtraq" yields these...
http://online.securityfocus.com/archive/1/201390
http://msgs.securepoint.com/cgi-bin/get/bugtraq0201/73.html
httpd
> tcp 0 0 *:https *:* LISTEN 4299/httpd > > ### Why does httpd show up twice here with seemingly > identical entries? ###Look closely -- it's http and https. Ports 80 (TCP) and 443 (TCP), respectively. Your web server is listening for both sorts of connections, normal and secure.
> tcp 0 0 *:www-http *:* LISTEN 1234/httpdKeep it if you're meaning to run a Web server on this box. Otherwise turn it off. (Note to the newbies -- you don't have to run a Web server in order to see Web pages. You only need a Web client for that, like a browser. Netscape, Opera, etc.)
login
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name > tcp 0 0 *:login *:* LISTEN 1110/inetdAaaah! login! Kill it! Kill it! [grin]
Er, I mean, how do you normally log in to your box? If this is what you use, I would highly recommend using ssh remotely and/or a local console login that doesn't listen on a port instead. There are a good number of programs that transparantly use login or the r-tools (rlogin, rsh, things like that), but they are usually reconfigurable to use ssh instead.
time
> tcp 0 0 *:time *:* LISTEN 1110/inetdAnother one you probably don't need if you're not actively using it. Time services and NTP are things that I constantly dither about. They are insecure and hackable. But they're also really useful, especially in a larger network, and as far as I know there's not a good replacement for the insecure xntpd. So this one's a judgement call -- how paranoid do you feel like being?
ypbind
> tcp 0 0 *:sco-sysmgr *:* LISTEN 859/ypbindYpbind isn't necessary unless you're using Sun's yp directory info.
finger
> tcp 0 0 *:finger *:* LISTEN 1110/inetdAnother one that you probably don't need. It gives out information about you -- though some people put their PGP keys in their .plan file and then say "Finger me for my PGP key". There have been holes in it. I generally turn it off, but it's not a tragedy if you don't.
portmap
> tcp 0 0 *:sunrpc *:* LISTEN 531/portmapMore Sun networking. Turn it off if you're not using it.
rpc.statd
> tcp 0 0 *:723 *:* LISTEN 543/rpc.statdAgain, unless you're doing Sun networking, kill it.
telnet
> tcp 0 0 *:telnet *:* LISTEN 1110/inetdOooh! Kill it, and especially kill it if you haven't patched it recently. There was an exploit in pretty much all versions of telnet derived from the BSD code base found about six months to a year ago. This includes Linux, Cisco, and a bunch of other OSs. There are many exploit scripts out there for it. Even if telnet weren't a really bad idea from a security POV because of the cleartext passwords, the exploit means you really don't want to be running it.
cupsd
> tcp 0 0 *:ipp *:* LISTEN 597/cupsdIf you're not using print services, kill it.
sendmail
> tcp 0 0 *:smtp *:* LISTEN 823/sendmail: accepIf you're running a local mail server that needs to accept mail, keep it. If you just need to send mail using your local server, set sendmail not to listen on a port:
Add
O DaemonPortOptions=Addr=127.0.0.1to your sendmail.cf. That way it will only be listening to your machine, and not to the Internet. (In your netstat output, you'll see 127.0.0.1:smtp instead of *:smtp.)
You may also want to add
O PrivacyOptions=goawayto disable the VRFY command. (This keeps the SMTP server you send the mail to from learning about the other accounts on your server. Some SMTP servers need this to be on to accept mail, but I haven't run into many.)
rpc.mountd
> tcp 0 0 *:700 *:* LISTEN 944/rpc.mountdKill it.
xfs
> Do I need the font server? I had trouble > getting X to work without it, so I just ran it and everything was happy, > but I got the impression that it wasn't really needed for a personal > machine... maybe I misread the docs.You do want the font server, so that you can have fonts working properly for your local machine, just like you do want to keep the X server so that you can have X working for your local machine. This is mostly a problem grown out of the way that computing in Unix has evolved -- when Xwindows was written, a common setup was to have one machine doing most of the processing and a bunch of dumb terminals using its X and XFS power. Nowadays it's common to only have the one machine you're on needing to access X and XFS. So you do still need the server (as far as I know -- if someone knows a way to do it without, please do speak up) -- it's just that it's only serving the one machine it's on.
You can still run X without xfs as long as you're not sharing the Xfonts with other machines (XDMCP or dumb terminals, this doesn't apply if you run X through ssh)
look at your XF86Config (This is for XFree86 4.1 but works on all X that I know of), find the "Files" section
look for FontPath, if you see it looking for a socket or a port,you're using xfs, you can simply change it to list all your font directories. Here's my section as an example:
# Multiple FontPath entries are allowed (which are concatenated together),
# as well as specifying multiple comma-separated entries in one FontPath
# command (or a combination of both methods)
#
# If you don't have a floating point coprocessor and emacs, Mosaic or other
# programs take long to start up, try moving the Type1 and Speedo directory
# to the end of this list (or comment them out).
#
FontPath "/usr/X11R6/lib/X11/fonts/truetype/"
FontPath "/usr/X11R6/lib/X11/fonts/local/"
FontPath "/usr/X11R6/lib/X11/fonts/misc/"
FontPath "/usr/X11R6/lib/X11/fonts/75dpi/:unscaled"
FontPath "/usr/X11R6/lib/X11/fonts/100dpi/:unscaled"
FontPath "/usr/X11R6/lib/X11/fonts/Type1/"
FontPath "/usr/X11R6/lib/X11/fonts/Speedo/"
FontPath "/usr/X11R6/lib/X11/fonts/75dpi/"
FontPath "/usr/X11R6/lib/X11/fonts/100dpi/"
ftpd
> If one has a need for ftp, what's a good server? Redhat uses wu-ftp, > which I've heard is about the worst as far as security holes?That is pretty much the worst. I use Pure-FTPd (http://www.pureftpd.org/), which has a really good security history. It's not a terribly mature program, but I've had good luck with it for a couple of years. It also supports the ability to create its own "virtual users", if you wanted to have FTP-only users with no shell.
I know other sysadmins that hated the way Pure does virtual users, though. One such favors Pro-FTPd (http://www.proftpd.org/), which had a showstopper bug a year or so back, but that's been fixed now. (It was the same file globbing bug that was in wu-ftpd -- a long string of ls's gave the remote hacker root.)
There's also a program called nc-FTPd that's an FTP server coded with security in mind. It's not free (in either sense), so I haven't checked it out yet. http://www.ncftpd.com/ if that's something you're interested in.
Raven has already listed Pure-FTPd, Pro-FTPd, and nc-FTPd. One I've heard about in this connection is vsftpd. I am told the name Chris Evans is familiar to bugtraq readers. He's the author. http://ferret.lmh.ox.ac.uk/~chris/vsftpd/
Copyright (c) 2002 by Raven Alder. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0 or later (the latest version is presently available at http://www.opencontent.org/openpub/).