Ridicoulusly High CPU Usage From MySQL

11 posts / 0 new
Topic locked
Last post
#1 Fri, 10/28/2011 - 20:53
UWF

Ridicoulusly High CPU Usage From MySQL

MySQL is slowing down my whole server, it stops page from starting there loading and uses up to over 90% of the CPU at random spikes. I can't do anything like import, repair, optimize etc. mySQL databases. It only crashes and results in a 500 internal server error.

Fri, 10/28/2011 - 23:48
andreychek

Howdy,

You may want to run "top" to see what all is at the top of your resource usage.

But, if it really is MySQL -- you'd need to look into what applications are running that are causing MySQL to use so many resources.

If you log into MySQL as root from the command line (with "mysql -u root -p", you can run "show full processlist" to get a list of what queries are currently running in MySQL.

You can use that to determine which applications are causing the issues.

-Eric

Sat, 10/29/2011 - 07:00
UWF

Ok, well yes it's the process "mysqld" using very large amounts of CPU at spike.

Since I have an IPB forum with 300,000 posts and the DB is about 200mb I thought it might be that. I've tried repairing and optimizing all of my databases so I'll see if that helps and also try this.

Sat, 10/29/2011 - 07:54
UWF

Also, it seems awstats is doing the same thing. It's coming up and hitting 100% CPU at random times.

Sat, 10/29/2011 - 08:47
UWF

I disabled Awstats and the optimizing and repairing of the DBs seemed to bring the MySQL CPU usage down but still sometimes when I go to a page on my site the load takes 10 seconds or so to begin, once it begins it's fast and the next page loads are fast until I have another 10 second before load. This is the output I get running the mySQL tuning script.

      -- MYSQL PERFORMANCE TUNING PRIMER --
             - By: Matthew Montgomery -
 
MySQL Version 5.1.49-3 x86_64
 
Uptime = 0 days 0 hrs 10 min 32 sec
Avg. qps = 19
Total Questions = 12272
Threads Connected = 2
 
Warning: Server has not been running for at least 48hrs.
It may not be safe to use these recommendations
 
To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service
 
SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 10.000000 sec.
You have 0 out of 12293 that take longer than 10.000000 sec. to complete
Your long_query_time seems to be fine
 
BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/5.1/en/point-in-time-recovery.html
 
WORKER THREADS
Current thread_cache_size = 8
Current threads_cached = 5
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine
 
MAX CONNECTIONS
Current max_connections = 151
Current threads_connected = 2
Historic max_used_connections = 6
The number of used connections is 3% of the configured maximum.
You are using less than 10% of your configured max_connections.
Lowering max_connections could help to avoid an over-allocation of memory
See "MEMORY USAGE" section to make sure you are not over-allocating
 
INNODB STATUS
Current InnoDB index space = 9 M
Current InnoDB data space = 71 M
Current InnoDB buffer pool free = 0 %
Current innodb_buffer_pool_size = 8 M
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory
 
MEMORY USAGE
Max Memory Ever Allocated : 58 M
Configured Max Per-thread Buffers : 405 M
Configured Max Global Buffers : 42 M
Configured Max Memory Limit : 447 M
Physical Memory : 3.80 G
Max memory limit seem to be within acceptable norms
 
KEY BUFFER
Current MyISAM index space = 522 M
Current key_buffer_size = 16 M
Key cache miss rate is 1 : 39
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine
 
QUERY CACHE
Query cache is enabled
Current query_cache_size = 16 M
Current query_cache_used = 8 M
Current query_cache_limit = 1 M
Current Query cache Memory fill ratio = 52.30 %
Current query_cache_min_res_unit = 4 K
MySQL won't cache query results that are larger than query_cache_limit in size
 
SORT OPERATIONS
Current sort_buffer_size = 2 M
Current read_rnd_buffer_size = 256 K
Sort buffer seems to be fine
 
JOINS
Current join_buffer_size = 132.00 K
You have had 0 queries where a join could not use an index properly
Your joins seem to be using indexes properly
 
OPEN FILES LIMIT
Current open_files_limit = 1024 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine
 
TABLE CACHE
Current table_open_cache = 64 tables
Current table_definition_cache = 256 tables
You have a total of 1793 tables
You have 64 open tables.
Current table_cache hit rate is 0%
, while 100% of your table cache is in use
You should probably increase your table_cache
You should probably increase your table_definition_cache value.
 
TEMP TABLES
Current max_heap_table_size = 16 M
Current tmp_table_size = 16 M
Of 417 temp tables, 22% were created on disk
Created disk tmp tables ratio seems fine
 
TABLE SCANS
Current read_buffer_size = 128 K
Current table scan ratio = 19 : 1
read_buffer_size seems to be fine
 
TABLE LOCKING
Current Lock Wait ratio = 1 : 1514
You may benefit from selective use of InnoDB.
If you have long running SELECT's against MyISAM tables and perform
frequent updates consider setting 'low_priority_updates=1'
If you have a high concurrency of inserts on Dynamic row-length tables
consider setting 'concurrent_insert=2'.
Sat, 10/29/2011 - 08:55
andreychek

Howdy,

For any of your Virtual Servers with the awstats feature enabled -- there's an entry in cron for them to run stats collections once a day. That stats collection can be resource intensive -- using Webalizar rather than awstats can lower your resource usage.

You're welcome to attempt tuning MySQL. Generally though, when you're getting as poor performance as you're seeing -- there's a deeper problem than can be solved with just tuning. You may need to spend some time getting a better feel for the specific problem that's occurring.

The "mytop" program can show you what's going on inside MySQL, it's one way that you can troubleshoot things.

-Eric

Sat, 10/29/2011 - 09:52
UWF

Do you think it could be do to the fact that I have a lot of sites that mostly sit idle? So the processes have to start up on the first page load?

This is what it looks like when it happens: http://i44.tinypic.com/5ed990.png This is what it looks like when I test it again right after: http://gyazo.com/241af4cb046d57c9f6244d92c2c9a49d.png

Sat, 10/29/2011 - 15:05
andreychek

I don't think your issues are related to having idle sites, or too many sites.

I think, in order to troubleshoot this issue, that you need to use tools like top and mytop in order to determine what processes, and within MySQL, what queries, are causing the problems you're seeing.

You can also use "uptime" to see your system load, and "free -m" to see your used/available RAM.

-Eric

Sun, 10/30/2011 - 07:45
UWF

I get command not found for mytop and I tried logging into my mySQL server but I just get an arrow when I run that command.

mysql> show full processlist
    ->
Mon, 10/31/2011 - 09:14
andreychek

You'd need to install the "mytop" program, it's not installed onto most Linux distros by default.

When you type MySQL commands, they should end with a semicolon... so that command should look like this:

show full processlist;

Tue, 11/01/2011 - 08:21
UWF

I found a fix which has worked 100%. I added a value to my my.cnf with the following line:

wait_timeout = 1800

No more random slow downs, system runs 100% smoothly now.

Topic locked