It's me again... (getting bored of myself)
Since 2 days, I've got a too high load on the server, nearly 20 all the time although it used to be around 1 normally. It could be a result of me trying to understand another (unsolved) problem that I have, which I posted here: https://www.virtualmin.com/node/34327
Anyway, using "top" the thing that surprises me is seeing "xargs" a lot, I don't remember seeing that so much in there. Fail2ban doesn't seem to be updating iptables as it did a few days ago, if that was how it worked. When I (over the last few months) typed iptables -L the huge list was kind of "live" produced in front of my eyes. That is no longer the case now.
I'll include stuff that should help pin pointing where the problem might be, but I don't see anything weird.
Again, sorry for all this asking for help. I'd prefer being out and taking photographs or so in stead of sitting here for hours with another server problem. So, all help is very much appreciated, as in REALLY!
(p.s. in "preview" I see my attachments here, but as soon as I hit save and see the real thing I don't see it. What am I doing wrong?)
Hi,
I've seen a few people saying they cannot attach stuff to the first post of a topic and I have seen this first hand as well. Simply making a second post in the thread should allow you to attach to said second post just fine.
Can you sort your "top" output by CPU load and post a screenshot or output of that?
-Dustin
I've asked Joe if he understands why this is occurring, but your attachments are there just fine -- but for some reason attachments aren't appearing when associated with the main post, only with comments. I can see them if I hit "Edit" though.
I reviewed your process list -- there's definitely a lot of them associated with MySQL... though that may be normal if MySQL were just restarted.
What is the output of this command:
ps auxw | grep mysqlAlso, what is the output of this command:
mailq | tail -1Thanks people!
here's what you asked for, and some of the others as well, all from the same moment more or less. I didn't restart mysql, this is what it is showing all the time since the high load.
p.s. a few hours ago, there were quite a lot of (1500 or so) mails in the queue. I deleted them, nearly all of them were from "lfd on penghosting.nl: Excessive resource usage", to me.
They'd be about www-data or mysql mostly
I checked the server-status but i didn't see things out of the ordinary. Well, that might be me as well of course, I don't know a lot about these things.
What does your my.cnf contain? It looks like MySQL is working really hard, so I'd want to make sure it has sufficiently large buffers configured.
You may also want to make sure your disk(s) is performing well. We've had significant problems keeping Virtualmin.com snappy because of poor disk write performance on our systems (we're moving to all SSD drives soon).
--
Check out the forum guidelines!
Hi Joe, here's my.cnf.
/dev/sda: Timing cached reads: 606 MB in 2.00 seconds = 302.84 MB/sec Timing buffered disk reads: 318 MB in 3.01 seconds = 105.49 MB/sec
Does this say anything?
In 'top' mysqld (only one instance) is using a lot of cpu resources: between 40 and 60 all the time. mysql there are quite a few, but further down (without the 'd') using a lot less cpu.
If I disable mysql the load shown in top goes down to normal, so Joe, I think you're right. There's something with mysql. I'll see if I can find something how to troubleshoot that.
Thanks for the hint!
Does anybody here have a good experience with a specific mysql help forum?
I can't get it solved on my own. It's a bit weird, early mornings, the server seems fine and top just shows a normal 1 load or so, and then, at 7 and something it shoots up to 18 or so.
I tried checking the tables, and most of them are fine. I used
myisamchk -r /var/lib/mysql/* / *.MYI(without the spaces around the / ) and in the end only remained with these errors (after solving the rest)
myisamchk: error: '/var/lib/mysql/bacs/bacs_k2_comments.MYI' is not a MyISAM-tablemyisamchk: File '/var/lib/mysql/gci/jos_sef_sm_cache_items.MYD' not found (Errcode: 2)
myisamchk: error: File '/var/lib/mysql/gci/jos_sef_sm_cache_items.MYI' doesn't exist
myisamchk: error: '/var/lib/mysql/lucia/jos_k2_comments.MYI' is not a MyISAM-table
I haven't solved those yet. Don't know if they can be the cause, but I better try.
In the syslog I see weird stuff, I'll inlcude the file here. But there's a lot of these:
Nov 19 08:07:36 penghosting /etc/mysql/debian-start[5235]: Upgrading MySQL tables if necessary.Nov 19 08:07:36 penghosting /etc/mysql/debian-start[5247]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Nov 19 08:07:36 penghosting /etc/mysql/debian-start[5247]: Looking for 'mysql' as: /usr/bin/mysql
Nov 19 08:07:36 penghosting /etc/mysql/debian-start[5247]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Nov 19 08:07:36 penghosting /etc/mysql/debian-start[5247]: This installation of MySQL is already upgraded to 5.1.72, use --force if you still need to run mysql_upgrade
Nov 19 08:07:36 penghosting /etc/mysql/debian-start[5305]: Checking for insecure root accounts.
Nov 19 08:07:36 penghosting /etc/mysql/debian-start[5333]: Triggering myisam-recover for all MyISAM tables
Nov 19 08:07:40 penghosting init: mysql main process (4435) terminated with status 1
Nov 19 08:07:40 penghosting init: mysql main process ended, respawning
Nov 19 08:07:40 penghosting kernel: [397853.822667] type=1505 audit(1416384460.719:55500): operation="profile_replace" pid=7608 name="/usr/sbin/mysqld"
What's the best place to go with this?
And thanks people, for being here and doing what you do!
Howdy,
I think what I'd do to troubleshoot that is to log into MySQL when that problem is happening, and to run this command:
show full processlistThat will show what's going on in MySQL at the time.
You may find that there's one particular domain you have that's doing something in particular with MySQL... the above command will help sort out what that is.
-Eric
Thanks Eric!
I get this mostly, don't know if all this NULL business is normal...
mysql> show full processlist;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 292
Current database: *** NONE ***
+-----+----------------------+-----------------+------+---------+------+------------------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+----------------------+-----------------+------+---------+------+------------------+-----------------------+
| 15 | root | localhost | gci | Sleep | 0 | | NULL |
| 280 | unauthenticated user | localhost | NULL | Connect | NULL | Reading from net | NULL |
| 282 | unauthenticated user | connecting host | NULL | Connect | NULL | login | NULL |
| 284 | unauthenticated user | localhost | NULL | Connect | NULL | Reading from net | NULL |
| 285 | unauthenticated user | localhost | NULL | Connect | NULL | Reading from net | NULL |
| 286 | unauthenticated user | localhost | NULL | Connect | NULL | Reading from net | NULL |
| 287 | debian-sys-maint | localhost | NULL | Sleep | 0 | | NULL |
| 288 | debian-sys-maint | localhost | NULL | Sleep | 0 | | NULL |
| 289 | debian-sys-maint | localhost | NULL | Sleep | 0 | | NULL |
| 290 | debian-sys-maint | localhost | NULL | Sleep | 0 | | NULL |
| 292 | root | localhost | NULL | Query | 0 | NULL | show full processlist |
+-----+----------------------+-----------------+------+---------+------+------------------+-----------------------+
11 rows in set (0.00 sec)
Will look more into it now. Thanks for the clue!