Is this a hacker?

13 posts / 0 new
Last post
#1 Thu, 01/13/2011 - 22:49
mrwilder

Is this a hacker?

Hi,

For the last few days, a few times a day, I have had my whole network get super slow.

Looking at webmin's list of processes, it lists apache log, and this:

./std 212.63.xxx.xxx 80

That doesn't look at all familiar. Can someone tell me what this is? The IP address is definitely not ours.

Also, how can i tell if someone is using our machine to send emails without permission?

thanks!

Thu, 01/13/2011 - 22:53
andreychek

Howdy,

One thing you may want to do is see what user is running that particular process... and then, using that, it may help you find where that "std" application is. You can then review it and figure out what it does.

However, I would agree that it's suspicious :-)

As far as email goes -- one way to see if you're server is sending mail without permissions is to look in the mail queue. Often, if a spammer is sending mail through your server, you'll end up with a lot of messages in the queue. You can view the mail queue by running the command "mailq | tail -1".

-Eric

Thu, 01/13/2011 - 22:58 (Reply to #2)
mrwilder

The owner is Apache, which made me think it could be a buffer overflow, php injection type thing, however, I thought it could also just be that Apache or some stats program looking up IP addresses for log files (if so I wish I could turn that off!).

How would I narrow down the location of the file?

I'm gonna run the mailq command now and see what turns up. Thanks!

Thu, 01/13/2011 - 23:05
andreychek

Well, it's hard to know where a file executed by Apache might be stored... you may simply want to look across your entire server for it. Something like this can accomplish that:

find / -name std

You may want to verify that all your system updates are in place, as some distributions have released fixes for PHP vulnerabilities recently.

-Eric

Thu, 01/13/2011 - 23:10 (Reply to #4)
mrwilder

NASTY.

"find" found it in /tmp/.a/floodsend/std and /tmp/a/floodsend

SOOooooo, what's a good rootkit finder

Thu, 01/13/2011 - 23:25 (Reply to #5)
mrwilder

oh, never mind... I just found several obvious root kits and compromises in the tmp folder.

"orbit-root", "virtual-root", "floodsend", "multimech startup mech by dorin"

So, with a machine this compromised, what is the easiest fix? build a new machine from scratch, copy over only the absolutely important domains,and HOPE that the new php fixes whatever (probably) caused the injection?

Errr... uh, what?

Thanks

Thu, 01/13/2011 - 23:23
andreychek

Well, I'd be less concerned about a rootkit being installed, and more concerned about determining what was used to break in.

Breakins through web apps aren't uncommon, and if the rest of your system security is in good shape, it typically leads only to what you saw... some strange processes running by non-privileged users.

The key is to kill off the processes, clean up the files, and close the hole they broke in through.

If the files your finding in /tmp are all owned by apache, it may be a little harder to track down, and probably means that it was created by a Virtual Server that's running it's PHP scripts using mod_php.

You'd then want to go through the web apps installed on your server, and see if any are old. You'd also want to make sure all your system updates are applied.

You can (and should) run root kit checkers such as rkhunter and chkrootkit, but I have a suspicion that they'll come up clean. More commonly, root isn't what's compromised, it's just some of the web apps.

-Eric

Thu, 01/13/2011 - 23:30
mrwilder

"orbit-root", "virtual-root", "floodsend", "multimech startup mech by dorin" look like root kits to me... but I agree these were probably put there through some leaky old php code.

do you think if I do a new build and simply upgraded the code to RUN on php 5.3 that will prevent this type of injection, or, uh... what?

For now, then, delete all of these files, run a rootkit checker, change the root password, and see what happens?

Thu, 01/13/2011 - 23:30
andreychek

The existence of those files/directories doesn't imply root was compromised.

It's wise to research it root was compromised of course, but I wouldn't assume all is lost quite yet :-)

Attackers often put files into /tmp when they don't have permissions to put them elsewhere.

So, you can run some root kit checkers (such as the two I mentioned above), and work from there :-)

-Eric

Thu, 01/13/2011 - 23:36
mrwilder

Yeah, I'm sure I already have chkrootkit installed, I'll run that ... but I notice that a "pulse audio" exploit was taken advantage of... and I bet pulse audio probably wasn't installed with the default webmin installation.

I would never have knowingly installed such a thing

Fri, 01/14/2011 - 04:25
Locutus

Some notes on this from my end:

Coming from the Windows world, malware infected customer PCs are "daily business" here. Happens way more often in Windows than Linux. From experience I'd have to say that after a system was compromised with malware, there is no guarantee that any attempts at cleaning them up you might make are 100% successful.

Theoretically, the intruder can have replaced any executables that he attained write access to. If it was a web app, then I'd probably be most notably Apache and PHP. And depending on what the attacker managed to do, it can be hard to repair a compromised system from within. As I usually put it: A leaking ship cannot be repaired out at sea, it needs to go to the dry-dock.

I'm not saying that this is necessarily the case, but you should keep in mind that if after your cleanup attempts are complete you still see suspicious behavior, you should take the safe road and reinstall the server (after making a backup), and afterwards put the domains back on.

If you're using mod_php you might consider switching to FCGId, since then PHP scripts run in context of the user and not Apache (which has other security implications though - the user might have even more access rights than Apache, you need to check that beforehand :) ).

Sat, 01/15/2011 - 22:03
mrwilder

So far, I ran chkrootkit, ran rkhunter, deleted everything in the /tmp folder, deleted the cron job owned by apache they had installed (I'm impressed), yum remove pulseaudio, changed the root password, and wrote a cronjob to clean the temp folder of EVERYTHING once every minute for now.

I know that I need to find the vulnerability where they are injecting their content. I am ASSUMING this is a buffer overflow exploit/code insertion on some of the php code at one of the sites. This is a pre php 5.3 box and the code is also older than dirt and obviously written not-so-securely.

I THINK this will be useful until I find the problems in the code.

Now, my question is do any programs, such as spamassassin, use the tmp folder to store job specific info they need... info which will be destroyed by my cron'ed shell script, thus possibly screwing up legitimate processes? One would think that's not a great way to write code, but obviously I'm no linux guru.

Thanks for your input...

Tony

Sun, 01/16/2011 - 09:00
andreychek

Howdy,

The version of PHP you're using isn't likely the issue... most distributions backport security fixes into the PHP version they provide. Chances are, then, that if you have all the security updates for your distribution, that all the known PHP security issues are patches.

Chances are that if the breakin occurred via a web application, it was due to a vulnerability in the web app itself.

It's not uncommon for web apps to have security issues. They generally get them fixed up, but it requires keeping all the web apps installed on your server fully up to date. Which I know can be quite tricky, but it's also the only way to prevent bad guys from breaking into your server :-)

Most folks aren't running PHP 5.3 now... and you don't need to be using PHP 5.3 to have a secure server.

As far as your question goes -- in most cases, it's pretty safe to delete the files in /tmp. If you delete files a running process is using, you may confuse it and have to restart the daemon. But outside of that, it shouldn't be a problem and is generally safe.

-Eric

Topic locked