Apache error log for domains grows out of control

Submitted by sebastianmorsony on Sat, 09/10/2011 - 10:00 Pro Licensee

Hi

I've seen the apache error log for a domain grow to 108GB, maxing out server capacity. How do I make sure this doesn't happen again?

Thanks Sebastian

Status: 
Closed (fixed)

Comments

Submitted by andreychek on Sat, 09/10/2011 - 10:19

Howdy -- well, it's pretty unusual that you'd see that, for two reasons... one, an application has to be logging a wild amount of stuff to generate that much text, unless perhaps log rotations aren't working. Second, quotas should prevent a user from taking up more space than they are allotted.

So a few questions -- what is the output of these commands:

  1. mount

  2. df -h

  3. If you cd into $HOME/logs for the domain you had problems with, what output do you get if you run "ls -la"

Submitted by JamieCameron on Sat, 09/10/2011 - 11:22

Also, try forcing log rotation with the command logrotate -f /etc/logrotate.conf

If this fails (perhaps due to a missing log file), that explains why your error_log hasn't been regularly truncated.

Submitted by sebastianmorsony on Fri, 09/16/2011 - 06:38 Pro Licensee

Hi again

It's the folder /var/log/virtualmin, that grows out of control. Would that be affected by the logrotate script as well?

I did have quota disabled, since I had some problems with it I needed to take a look at.

I did get some errors when I ran logrotate, that I fixed myself.

Warning: DocumentRoot [/home/andy2/public_html] does not exist Warning: DocumentRoot [/home/andy.wigf5/public_html] does not exist Warning: DocumentRoot [/home/benjamin.wigf5/public_html] does not exist Warning: DocumentRoot [/home/andy2/public_html] does not exist Warning: DocumentRoot [/home/andy.wigf5/public_html] does not exist Warning: DocumentRoot [/home/benjamin.wigf5/public_html] does not exist Warning: DocumentRoot [/home/andy2/public_html] does not exist Warning: DocumentRoot [/home/andy.wigf5/public_html] does not exist Warning: DocumentRoot [/home/benjamin.wigf5/public_html] does not exist Warning: DocumentRoot [/home/andy2/public_html] does not exist Warning: DocumentRoot [/home/andy.wigf5/public_html] does not exist Warning: DocumentRoot [/home/benjamin.wigf5/public_html] does not exist Warning: DocumentRoot [/home/andy2/public_html] does not exist Warning: DocumentRoot [/home/andy.wigf5/public_html] does not exist Warning: DocumentRoot [/home/benjamin.wigf5/public_html] does not exist Warning: DocumentRoot [/home/andy2/public_html] does not exist Warning: DocumentRoot [/home/andy.wigf5/public_html] does not exist Warning: DocumentRoot [/home/benjamin.wigf5/public_html] does not exist

Output from mount.

root@Webserver8:/var/log/virtualmin# mount /dev/mapper/Webserver8-root on / type ext4 (rw,grpquota,errors=remount-ro,usrquota) proc on /proc type proc (rw,noexec,nosuid,nodev) none on /sys type sysfs (rw,noexec,nosuid,nodev) none on /sys/fs/fuse/connections type fusectl (rw) none on /sys/kernel/debug type debugfs (rw) none on /sys/kernel/security type securityfs (rw) none on /dev type devtmpfs (rw,mode=0755) none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) none on /dev/shm type tmpfs (rw,nosuid,nodev) none on /var/run type tmpfs (rw,nosuid,mode=0755) none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) none on /var/lib/ureadahead/debugfs type debugfs (rw,relatime) /dev/sda1 on /boot type ext2 (rw)

Output from df -h

root@Webserver8:/var/log/virtualmin# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/Webserver8-root 129G 14G 109G 11% / none 2,0G 188K 2,0G 1% /dev none 2,0G 0 2,0G 0% /dev/shm none 2,0G 92K 2,0G 1% /var/run none 2,0G 0 2,0G 0% /var/lock none 2,0G 0 2,0G 0% /lib/init/rw none 129G 14G 109G 11% /var/lib/ureadahead/debugfs /dev/sda1 228M 67M 150M 31% /boot

Submitted by andreychek on Fri, 09/16/2011 - 10:18

Logrotate should be rotating your logs regularly, preventing them from getting too large. I don't see any errors in your output that would have broken the log rotation.

It's possible that there were some in the past but they've since been corrected.

We'd generally suggest using quotas if you're concerned about one user accidentally using up all the space on the system.

Log rotation should prevent most common causes of log files growing too large, but if an application goes a bit wild and sends too much information into the logs before they get rotated, without a quota that could potentially cause disk space issues on your server.

Submitted by sebastianmorsony on Sat, 09/17/2011 - 17:17 Pro Licensee

Thanks for your help. Did run logrotate and enabled quota again. :-)