These forums are locked and archived, but all topics have been migrated to the new forum. You can search for this topic on the new forum: Search for Apache all of a sudden very slow on the new forum.
Web Hosting and Cloud Computing Control Panels
Howdy,
Well, a couple of thoughts:
I might look in the apache error logs, to see if there's any clues there (/var/log/httpd/error_log or /var/log/apache2/error_log)
You may want to look at your network -- try pinging your box for awhile from another server, and make sure that you aren't loosing any packets. And try doing a traceroute to make sure nothing in between the systems is causing trouble.
If it's not network related, and no clear Apache issues, you might consider installing a tool like "mytop" in order to see what's going on with the database. A "harsh" database query could really slow things down, and mytop will help clarify what all your database is doing.
That's just a few things to get you started -- feel free to discuss your findings, and hopefully we can figure out what's going on!
-Eric
Eric mentioned network, in addition I would first check DNS on the domains (www.intodns.com)
I would also look to how much RAM is on the system
is it a VPS?
What kind of webpages are slow? static, php?
If php, are they a cms like Joomla! or custom made scripts?
in TOP you see no oddities....so apache doesn't use a lot of ram nor cpu?
the thing is, when something goes slow, the lot goes slow. One major site is a joomla site (so php) and one other major site is custom made scripts. However, that one creates static pages and only becomes 'dynamic' again when mod rewrite cant find the static files, which only happens when a page has been added to the site. As soon as page is dynamically served then, it is written away as a static file until a new page gets added.
Those two sites get more or less 4000 visitors daily, nothing that should bring a server down I guess.
There are 4 other sites on there (wordpress, joomla). Those only get a few hundred visitors daily.
server (ubuntu 8) has 2 gb ram.
apache doesn't seem to be using a lot of cpu or ram...
Off puzzling again.
Mytop:
It seems mytop shows nothing weird, but this is done when the server is already hardly responding.
MySQL on localhost (5.0.67-0ubuntu6) up 0+00:47:12 [10:14:32] Queries: 18.1k qps: 7 Slow: 0.0 Se/In/Up/De(%): 54/00/00/00 qps now: 0 Slow qps: 0.0 Threads: 2 ( 1/ 6) 00/00/00/00 Key Efficiency: 98.8% Bps in/out: 0.1/ 9.5 Now in/out: 8.4/ 1.3k
Mysqltuner:
-------- General Statistics -------------------------------------------------- [--] Skipped version check for MySQLTuner script [OK] Currently running supported MySQL version 5.0.67-0ubuntu6 [OK] Operating on 32-bit architecture with less than 2GB RAM
-------- Storage Engine Statistics ------------------------------------------- [--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster [--] Data in MyISAM tables: 46M (Tables: 635) [!!] InnoDB is enabled but isn't being used [!!] Total fragmented tables: 8
-------- Performance Metrics ------------------------------------------------- [--] Up for: 2m 40s (3K q [19.944 qps], 702 conn, TX: 4M, RX: 288K) [--] Reads / Writes: 97% / 3% [--] Total buffers: 58.0M global + 2.6M per thread (100 max threads) [OK] Maximum possible memory usage: 320.5M (15% of installed RAM) [OK] Slow queries: 0% (0/3K) [OK] Highest usage of available connections: 8% (8/100) [OK] Key buffer size / total MyISAM indexes: 16.0M/8.2M [OK] Key buffer hit rate: 98.3% (45K cached / 756 reads) [OK] Query cache efficiency: 28.0% (647 cached / 2K selects) [OK] Query cache prunes per day: 0 [OK] Sorts requiring temporary tables: 0% (0 temp sorts / 75 sorts) [!!] Temporary tables created on disk: 28% (51 on disk / 178 total) [OK] Thread cache hit rate: 98% (8 created / 702 connections) [!!] Table cache hit rate: 1% (64 open / 5K opened) [OK] Open file limit used: 12% (128/1K) [OK] Table locks acquired immediately: 99% (1K immediate / 1K locks)
-------- Recommendations ----------------------------------------------------- General recommendations: Add skip-innodb to MySQL configuration to disable InnoDB Run OPTIMIZE TABLE to defragment tables for better performance MySQL started within last 24 hours - recommendations may be inaccurate Enable the slow query log to troubleshoot bad queries When making adjustments, make tmp_table_size/max_heap_table_size equal Reduce your SELECT DISTINCT queries without LIMIT clauses Increase table_cache gradually to avoid file descriptor limits Variables to adjust: tmp_table_size (> 32M) max_heap_table_size (> 16M) table_cache (> 64)
IntoDNS.com:
NSs have same SOA serial
Looks like your nameservers do not agree on the SOA serial. Ths SOA records as reported by your nameservers: 88.208.232.11 -> 2 213.171.223.34 -> 5 This can cause some serious problems that is why you should fix this asap.
I can't find the error log of apache, so maybe I'll have to set it in a config file. I'll look into that now.
Pinging the network works fine, no lost packages.
The DNS error, I remember I've already looked into this, and it was kind of impossible for me that is, to set it right on both servers. The other server comes with an admin panel by fasthosts.co.uk. Maybe I should install virtualmin on that server too... But: although I've had this dns problem for quite a lot months, the server has been working fine.
I'll start with the apache error log as the server has stopped responding again.
Thanks for your help people!
the apache error log says:
root@server2:~# tail -f /var/log/apache2/error.log [Tue Aug 11 10:03:25 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting [Tue Aug 11 10:05:07 2009] [notice] caught SIGTERM, shutting down [Tue Aug 11 10:05:09 2009] [notice] Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.2 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g configured -- resuming normal operations [Tue Aug 11 10:05:14 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting [Tue Aug 11 10:18:56 2009] [notice] caught SIGTERM, shutting down [Tue Aug 11 10:20:31 2009] [notice] Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.2 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g configured -- resuming normal operations [Tue Aug 11 10:20:37 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting [Tue Aug 11 10:37:23 2009] [notice] caught SIGTERM, shutting down [Tue Aug 11 10:37:26 2009] [notice] Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.2 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g configured -- resuming normal operations [Tue Aug 11 10:37:31 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
so I guess I'll look at maxclients first and will then try to find what sigterm means
I changed the max clients from 150 to 500 and after i did that, the server has been running fine. However, I'm not convinced yet. Might be that if someone was trying to attack the server or something (what do i know...) that they've simply stopped doing that at more or less the same I changed the setting.
I've no clue either how to optimise apache or mysql, so I guess there's a learning task too for me.
but far from perfect. Sometimes it runs fast, but now for example, I've got to wait ten seconds before a new page loads.
(Just to let you know it's not really solved.)
Well, knowing that I'm just sort of speculating here, the two thoughts that initially come to mind are:
You have a popular website! Nice work :-) If bumping that up to 500 works, and you have enough CPU and RAM, that's great.
It could be a DoS attack of some kind. There's been a few of those going around lately. I'd use a tool like netstat to try and diagnose that, running "netstat -an". It's hard to say what to look for, as it'd be different things depending on the attack. A lot of outstanding "SYN" requests, or perhaps "TIME_WAIT" connections, could indicate one kind of DoS.
You might want to look in System Settings -> Bandwidth Monitoring -> Show Usage Graph, you can see if any hosts are using a particularly large amount of bandwidth -- which might help you figure out a little more about what's going on.
-Eric
thanks, i've now enabled bandwidth monitoring.
will have a look at netstat now.
do you know anything about the popularity of your website? If it is not an attack, then it must be popular. That would indeed be a good thing.
(thought my latest post would end up down here, but it didn't, so here's the important bit:)
Those two sites get more or less 4000 visitors daily, nothing that should bring a server down I guess.
There are 4 other sites on there (wordpress, joomla). Those only get a few hundred visitors daily.
it's still no good.
In the morning and early evening, top shows speeds from around 1.00 going down and down while nothing seems to happen. Apache on the normal webserver has to be restarted for the sites to be seen again. Within a few minutes, it can happen again.
Apache on webmin/virtualminruns fine at the same time however, as do ssh and ftp etc.
Can you verify in Webalizer (ie, in http://domain.com/stats/) that there hasn't been an unusual amount of traffic recently?
Also, can you take the results of a "netstat -an", and attach it to a forum post here so we can take a look?
-Eric
Hi Eric, yes i did check the stats of the different domains on the server: nothing weird. (Luckily!)
Attached is the netstat file.
Hmm... well, first off, you don't have a particularly large amount of odd traffic there.
But it is unusual how many outstanding requests there are from 145.50.39.11.
Do you recognize that IP?
If you run netstat again, do you still see that IP with a lot of "SYN_RECV" entries?
Also, in Apache, I often change the "Timeout" parameter from the default of 300 down to something like 30 to make sure Apache closes up shop a little quicker and doesn't hang on to slow connections.
-Eric
I'll have a look at the timeout thing. In fact, I'll have to learn how to (fine)tune apache and mysql.
didn't see the 145... ip just now, I'll attach it for you to see.
Hrm, yeah, that all looks fairly normal now.
So, the problem may be elsewhere :-)
-Eric
indeed. So what would your next step be?
Right now, the server goes down within a minute of restarting apache (and/or mysql and/or the whole machine). So I did your netstat thing again, and it's full of these listings of one ip address:
IP address 77.165.42.1 Hostname ip4da52a01.direct-adsl.nl ISP ADSL41 Country Netherlands
I'll keep checking when the server has problems, but it does seem a bit dodgy, doesn't it?
Is there a way to see what exactly they're trying to look at (on the server that is obviously, not "their screen" or so ;) ?
Now the server reacts ok, and that ip address is gone from netstat output.
And now (I'm really sorry for this) its slow again, and netstat is full of two ip addresses:
193.191.210.10 and 193.191.210.2
they belong to the department of finances of the belgian government:
mineco.fgov.be and mail.mineco.fgov.be
Maybe the fact that one is mail and the other (seems) normal is a bit strange? I mean, on the server a lot at the same time.
maybe this is normal, but it's just what caught my eye.
Yeah, it kind of sounds like some hosts are hitting you pretty hard.
I'm not really sure why that variety of systems are finding you, but perhaps you may want to look into using some sort of anti-DoS tool.
You could just block a single host at a time using iptables or so, but it looks like they're rarely the same ones.
There's a few options out there -- you might look into some Apache modules such as mod_evasive or mod_qos.
There's also "Ddos deflate", which is a simple bash script that runs regularly and uses iptables to block hosts with more than N connections.
There's likely to be other great tools out there too, I'm not too familiar with this realm unfortunately.
Also, if you haven't already, I'd definitely lower the "Timeout" setting in Apache to at least 30.
Until you find a tool that works for you, you'd probably just want to block the hosts in question that appear to have an "excessive" amount of connections open.
-Eric
ok thanks for that. I did change the default apache setting from 300 to 30 as you mentioned earlier.
I'l check about those tools.
Right now I've got zillions of ...haarlem.nl in my netstat.
Is there a way to see if they're trying to contact a 'real thing' on the server, for example a file? (I know this sounds stupid but I'm just being curious)
The best you can do, I think, would be to look around in the Apache logs, and see if that particular IP is listed as having requested anything.
There are a variety of types of DoS attacks... the recently created slowloris attack just creates a bunch of connections to Apache, and never does anything with them.
However, in it's case, it doesn't typically create a high load on the server, it just makes Apache slow to respond.
-Eric
thanks for your answer yet again Eric. I've managed to install mod_evasive, it comes with a perl test script which showed it seems to work.
Curious what will happen now...
I didn't have a high load nor mem. usage, it was just apache indeed.
Still wondering why somebody would want to do it to one of my sites or server (I'm a nice guy you see :) :) and the sites are nothing special. Probably just boredom. (of the dos attacker, not of me making those sites)
I'll keep you updated.
and thanks for your clues and ideas in this thread, you've helped me a lot.
i don't think anyone will start ddossing out of boredom, it's too timeconsuming and costly.
I would contact my ISP to see if anyone on their network or even they themselves are a target and you suffer from some reminiscence. Or perhaps a switch over there is out of control...let them check
I worked with a big host who was regularly a target. A good ddos would kill all their Apache servers until the attack was properly mitigated..
I'll start rereading all the messages here, as today it seems to be happening again... But I thought I'd upload the netstat file already so you can have a look too. Going to have to block 165.72.200.11. Crazy thing is, I've got mod_evasive up and running since last time...
Does anyone know how i can find out what website on my server they're trying to 'attack'?
I'll try to learn how to block an ip.
Yeah, looks like a DoS attack originating from that IP (either that, or an extremely poorly behaved web browser!).
Using mod_evasive can be helpful, but it may not always do the trick, as you're seeing.
DoS attacks are a pain in the butt, as you're seeing :-)
There are additional ways to try and mitigate these sorts of attacks...
Some folks have suggested they had good luck with DDOS Deflate:
http://deflate.medialayer.com/
All that script does, though, is check the number of connections for any particular IP address at any given moment -- and if it exceeds a particular threshold, it uses iptables to block the IP.
The command line it uses is:
iptables -I INPUT -s IP_ADDRESS_HERE -j DROPThanks Eric, had found the iptables thing, and managed to get iptables up and running (and block that ip address). Set timout apache to 10 and server is serving again.
Will now look into ddos deflate
is there a way to see what site they were trying to attack?
I've installed ddos-deflate
thanks!