lsof open files: thousands of log files open

4 posts / 0 new
Topic locked
Last post
#1 Thu, 06/17/2010 - 07:13
fatbear

lsof open files: thousands of log files open

Lately, services on my machine are failing (e.g., dovecot). One thing I've noticed is that there are many thousands of open files for logging. For each httpd process that is running, it seems that there is an error_log and an access_log file open for each VirtualHost on my system. This is amounting around 9,000 open files for logging.

I've seen this kind of thing before. My previous solution (before using Virtualmin) was to have only one Apache logfile. I would then process that logfile (e.g., via fetchlog or some other logfile analyzer/processor) that would take all logfile entries and append them to the appropriate logfile. For example, the logfile would live in /etc/httpd/logs/access_log and /etc/httpd/logs/error_log and the logfile analyzer would append content from these files to files such as /home/mydomain/logs/access_log, /home/domains/sub.mydomain.com/logs/access_log (and similarly with error_log files).

Of course, I'm using Virtualmin now and I don't want to break anything. Can anyone suggest how I can fix this problem so that I don't have thousands of open log files?

Thu, 06/17/2010 - 08:59
andreychek

Well, I'm curious if the issues are unrelated... there's a few things that can trigger Dovecot problems that I've seen.

What distro are you using? If it's Debian or Ubuntu, I'm curious if you see any errors in /var/log/mail.err.

-Eric

Tue, 06/29/2010 - 07:40
fatbear

This is on a Fedora 7 server. It's in its last year of operation, but I still need to deal with it.

My real need is to not have file handles open for each access.log and error.log file on the server. I've counted: I have 358 log files on my server:

   # /bin/ls -1d /home/*/logs/*_log /home/*/domains/*/logs/*_log | wc -l
   356
   #

I then counted the number of open file handles:

   # lsof | grep '/home/.*_log' | wc -l                                          
   5696
   #

I then counted the numer of open file handles per log file:

   # lsof | grep '/home/.*_log' | sed 's#.* /home#/home#' | sort | uniq -c
   16 /home/account1/logs/access_log
   16 /home/account1/logs/error_log
   16 /home/account1/domains/subserver1/logs/access_log
   ...

Note that 16*356 = 5696.

I then checked how many httpd processes are running:

   # ps ax | grep httpd | grep -v "grep httpd" | wc -l
   16
   #

So, it is pretty clear that the number of open files attributable to log files is equal to the number of httpd processes times the number of log files.

The problem is that this is not scalable and it is causing problems on my server. So, what I want to do is to have just two log files (access_log and error_log) into which all transactions go. Then, have these split out by a separate process to individual user log files. Or something like that. Something that does not involve having thousands of open files that is causing my system to be unstable.

So, the question is: how do I do this with Virtualmin?

Tue, 06/29/2010 - 09:04
andreychek

Howdy,

Well, I appreciate your desire to want to lower the amount of open logs. I suppose the only things I can offer are these two things:

  1. With the open file limits set to non-default values, I hadn't run into issues with open files before, even with hundreds of domains on one server

  2. There's no way built into Virtualmin to handle just one error log and access log for all the domains.

Exploring #2 above a bit -- it's not that you couldn't do this manually, but you'd lose your bandwidth monitoring.

My suggestion would be to just make sure your max open files is set a little higher if you're concerned that it's causing problems... however, if you want to reduce Apache logging to two open files, you'd change the logging that's setup in System Settings -> Server Templates -> Default -> Apache Website.

-Eric

Topic locked