So I'm pretty sure I've been hacked in some manner but I'm too much of a noob to know how it happened or how to stop it. It came to my attention because the bandwidth on my VPS has been way over quota by 2x and from what I can tell it seems like it came from a hack located at this site, which itself seems to have been hacked to hold that file.
http://alisonmcleastudio.com/jpg/wpa.bwe that may need to be hidden so others don't use it or something
here is the results of the error log located at /var/log/httpd/errorlog, which is line after line after line of this type of stuff:
--2013-12-09 10:13:02-- http://alisonmcleastudio.com/jpg/wap.bwe Resolving alisonmcleastudio.com... 50.63.101.1 Connecting to alisonmcleastudio.com|50.63.101.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 40802 (40K) Saving to: `wap.bwe'
0K .......... .......... .......... ......... 100% 17.4M=0.002s
2013-12-09 10:13:02 (17.4 MB/s) - `wap.bwe' saved [40802/40802]
[Mon Dec 09 10:13:39 2013] [warn] mod_fcgid: process 5739 graceful kill fail, sending SIGKILL
rm: cannot remove *alisonmcle*': No such file or directory
--2013-12-09 10:15:42-- http://alisonmcleastudio.com/jpg/wpa.bwe
Resolving alisonmcleastudio.com... 50.63.101.1
Connecting to alisonmcleastudio.com|50.63.101.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 40803 (40K)
Saving to:wpa.bwe'
0K .......... .......... .......... ......... 100% 5.87M=0.007s
2013-12-09 10:15:42 (5.87 MB/s) - `wpa.bwe' saved [40803/40803]
--2013-12-09 10:15:42-- http://alisonmcleastudio.com/jpg/wap.bwe Resolving alisonmcleastudio.com... 50.63.101.1 Connecting to alisonmcleastudio.com|50.63.101.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 40802 (40K) Saving to: `wap.bwe'
0K .......... .......... .......... ......... 100% 12.1M=0.003s
2013-12-09 10:15:42 (12.1 MB/s) - `wap.bwe' saved [40802/40802]
[Mon Dec 09 10:16:19 2013] [warn] mod_fcgid: process 5743 graceful kill fail, sending SIGKILL
rm: cannot remove *alisonmcle*': No such file or directory
--2013-12-09 10:18:51-- http://alisonmcleastudio.com/jpg/wpa.bwe
Resolving alisonmcleastudio.com... 50.63.101.1
Connecting to alisonmcleastudio.com|50.63.101.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 40803 (40K)
Saving to:wpa.bwe'
0K .......... .......... .......... ......... 100% 16.7M=0.002s
2013-12-09 10:18:51 (16.7 MB/s) - `wpa.bwe' saved [40803/40803]
--2013-12-09 10:18:51-- http://alisonmcleastudio.com/jpg/wap.bwe Resolving alisonmcleastudio.com... 50.63.101.1 Connecting to alisonmcleastudio.com|50.63.101.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 40802 (40K) Saving to: `wap.bwe'
0K .......... .......... .......... ......... 100% 18.0M=0.002s
2013-12-09 10:18:51 (18.0 MB/s) - `wap.bwe' saved [40802/40802]
Does anyone know exactly what this hack is doing, and how can I stop it?
Thanks
Howdy,
It sounds like you've done a lot of the hard part -- determining which site/account is the source of the problem.
It's not simple to determine what the hack is doing -- but I'll offer that whatever it is, we don't want it doing that :-)
The first thing I'd suggest doing is reviewing all the software installed onto that account, and making sure it's fully up to date. If it's using WordPress, for example, make sure that it's running the most recent WordPress version.
You'd also want to look for a file named "wpa.bwe", possibly in the "jpg" directory of that domain.
One thing you may want to verify is to see if there's any email in the mail queue, as some breakins send spam.
You can determine that by running this command:
mailq | tail -1You may also want to see if there are any processes running as this user, and if so, you may want to kill them.
-Eric
Hi Eric,
The domain I mentioned is not associated with my server at all, that's on a foreign site, but somehow my server is connecting to it and repeatedly downloading files, though it doesn't look like they get stored anywhere. However, the hack is eating up tons of bandwidth and it's going to cost me a lot of money to cover the overages if I can't get this figured out quickly.
I ran the mail command you gave and it doesn't appear that any mail is being sent, so I'm not sure exactly what all the bandwidth is being used for.
Thanks
Hmm, normally, log messages go into the individual domain's error logs, located in $HOME/logs/error_log.
That may mean that there's some sort of web app setup on a default VirtualHost on your server.
Do you see any files in "/var/www/html/"?
-Eric
Same here. Exact same domain, same IP. I block the domain 50.63.101.1. But that's not the perfect solution. Someone else who can help out>
You might first want to use tools like "iftop", "netstat -tpn", "atop" to find out which process is doing the requests and which user it is doing them as. From there we can progress further.
It is the /sbin/syslogd process. Done by the Root user.
As you can see in the script from the startpost; http://alisonmcleastudio.com/jpg/wpa.bwe > there is the /sbin/syslogd name defined
The "syslogd" name might be a fake done by an exploit, or the executable itself is an exploit. According to your process list, syslogd is run by apache, not root. And I doubt the syslog daemon would run as apache.
I didn't get your last line. What is "defined" where?
Oh, I see it now. Yeah this "wpa.bwe" seems to be a bot (PERL script) that hides itself as "/sbin/syslogd" and participates in IRC activity (connects to a CNC I suppose). Well I suppose someone exploited a weakness in some web software and had Apache download and run multiple copies of that script.
First step would be killing these processes, rename (not delete, keep it for analysis) the script and review all software installed in your web sites. Software like "Linux Malware Detect" can help with that. Make sure all web software is up-to-date. Remove any insecure software or plugins.
Review logs in /var/log for suspicious activity, also do a "last" which shows logins. Review CRON job tables. Change passwords of web/email users that might be compromised. Change the root password for good measure. Use rootkit detection software to see if there's a deeper reaching compromise.
The file "wpa.bwe" might be related to Wordpress, maybe a known vulnerability/exploit. Make sure that all your Wordpress installations are up-to-date.
http://pastebin.com/9Nj8Pgv5 (found via Google, this is not a paste from me)
So it definitely looks like it's some exploit related to one of my wordpress installations, I'm not sure if it's a plugin or a theme issue though.
The plugins I have installed are all very popular and well known plugins like All in One SEO Pack and Form Maker. I know that forms are a particular vulnerability but I think that that plugin is pretty secure.
So that leaves a possible vulnerability with some of the theme files. I use the SAICO theme on this particular site, which did have some exploits when it was released but were fixed with an update later, though the developer has stopped supporting it. I think I'll see about changing the theme and seeing if that does the trick. Unfortunately it does require me to redesign the site quite a lot.
Below is an image of the hack process /sbin/syslogs in the webmin process details. It says it's a child process of "init" is that something I need to go root around for more so that it doesn't automatically start up when I restart the server?
Also, how can I determine the IP address that the attack is coming from and just block that IP as well?
Also, I went back and checked in /var/logs/httpd/error_log and found that apache is connecting to a different thing now. This deal was downloading a new index.php and some supporting files into my wordpress installation and putting up a weird "dark web" site.
These are the log files, it did this same process several times over so I just copied one cycle of it. It saved the files to wp-content/uploads/2013/04
I ran a last command and no one else has logged into the server directly so it seems all the attacks are through the one website. The rogue processes are all running as the user for that particular domain and not as root or anything that indicates the entire server is compromised.
I still don't understand exactly how the site is compromised but I updated wordpress to the newest 3.8 version and updated all plugins.
--2013-12-18 11:47:58-- http://yanaselandscaping.com/blog/wp-content/themes/echea/js/cufon-fonts... Resolving yanaselandscaping.com... 72.41.38.176 Connecting to yanaselandscaping.com|72.41.38.176|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 273165 (267K) [image/jpeg] Saving to: `header.tot'
100K .......... .......... .......... .......... .......... 56% 790K 0s 150K .......... .......... .......... .......... .......... 74% 742K 0s 200K .......... .......... .......... .......... .......... 93% 28.7M 0s 250K .......... ...... 100% 501K=0.4s
2013-12-18 11:47:59 (638 KB/s) - `header.tot' saved [273165/273165]
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 266k 100 266k 0 0 468k 0 --:--:-- --:--:-- --:--:-- 528k sh: /usr/bin/xdg-open: No such file or directory --2013-12-18 11:48:01-- http://yanaselandscaping.com/blog/wp-content/themes/echea/js/cufon-fonts... Resolving yanaselandscaping.com... 72.41.38.176 Connecting to yanaselandscaping.com|72.41.38.176|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 43216 (42K) [image/jpeg] Saving to: `logo.tot'
2013-12-18 11:48:01 (217 KB/s) - `logo.tot' saved [43216/43216]
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 43216 100 43216 0 0 133k 0 --:--:-- --:--:-- --:--:-- 167k sh: /usr/bin/xdg-open: No such file or directory --2013-12-18 11:48:02-- http://yanaselandscaping.com/blog/wp-content/themes/echea/js/cufon-fonts... Resolving yanaselandscaping.com... 72.41.38.176 Connecting to yanaselandscaping.com|72.41.38.176|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 3841 (3.8K) [text/html] Saving to: `tot.tot'
2013-12-18 11:48:02 (302 MB/s) - `tot.tot' saved [3841/3841]
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 101 3841 101 3841 0 0 30252 0 --:--:-- --:--:-- --:--:-- 60968
Yes, your processes screenshot shows the Perl script running as a server owner, so you know which website is the culprit.
You might want to run "Linux Malware Detect" on the web directory in question, maybe it finds something.
Also change all passwords for the Wordpress in question, change the "admin" username to something not easily guessable, and put .htaccess protection on the backend directory.
So I ran Linux Malware Detect and the scan file reads thusly:
malware detect scan report for ip-50-62-213-134.ip.secureserver.net: SCAN ID: 121913-1211.24547 TIME: Dec 19 12:12:14 -0700 PATH: /home/antiquemoney TOTAL FILES: 30432 TOTAL HITS: 1 TOTAL CLEANED: 0
NOTE: quarantine is disabled! set quar_hits=1 in conf.maldet or to quarantine results run: maldet -q 121913-1211.24547 FILE HIT LIST:
{HEX}gzbase64.inject.unclassed.14 : /tmp/maldetect-1.4.2/files/clean/gzbase64.inject.unclassedLinux Malware Detect v1.4.2 < proj@rfxn.com >
I did a grep in that directory for "base64_decode(" and got the following hits, all of which seem to have legitimate uses, not doing the normal base64 decode hack stuff I've seen before. Anymore ideas as to what may be the particular vulnerabilities that allowed the site to be hacked?
/home/antiquemoney/public_html/wp-includes/class-feed.php /home/antiquemoney/public_html/wp-includes/ID3/module.audio.ogg.php /home/antiquemoney/public_html/wp-includes/class-IXR.php /home/antiquemoney/public_html/wp-includes/SimplePie/Sanitize.php /home/antiquemoney/public_html/wp-includes/class-smtp.php /home/antiquemoney/public_html/wp-admin/includes/file.php /home/antiquemoney/public_html/wp-content/plugins/all-in-one-seo-pack/OAuth.php
Thanks for all the help so far