Problems with "Too many open files in system"

4 posts / 0 new
Topic locked
Last post
#1 Tue, 02/01/2011 - 02:38
ummecasino

Problems with "Too many open files in system"

Since I run Virtualmin on my Debian 5 machine I have logs full of those messages. Mostly those messages appear on mail.err and on apache2 error log.

I never heard of this error before so it is a bit hard for me to debug it. I tried to list open files with lsof, I could do that yesterday and had about 1300 open files for apache. I tried to do the same some minutes ago got

lsof -n|grep -oE '^[a-z]+'|sort|uniq -c|sort -n
      8 logger
      9 init
     11 grep
     11 uniq
     16 cron
     17 freshclam
     18 xinetd
     19 lsof
     21 su
     22 lookup
     23 syslogd
     26 sort
     29 pickup
     29 qmgr
     30 anvil
     32 fail
     33 bash
     36 tlsmgr
     37 saslauthd
     40 proftpd
     44 mailmanct
     46 smtpd
     54 pop
     58 named
     76 miniserv
     89 dovecot
     93 imap
    110 sshd
    122 spamd
    128 master
    160 mysqld
    192 postgres
    328 python
    452 php
    860 apache

Sometimes I just get

lsof -n|grep -oE '^[a-z]+'|sort|uniq -c|sort -n
bash: /bin/grep: Too many open files in system
bash: pipe error: Too many open files in system

I changes the file limit for www-data to

www-data     hard    nofile             16384

but it seems like this won't help at all.

Can anyone help me?

Tue, 02/01/2011 - 07:25
andreychek

While that should work (you'd need to restart Apache afterwards), I haven't always had success getting Apache to see those limits correctly either.

Another option is to edit the Apache init script, /etc/init.d/apache2, and add this line in there somewhere:

ulimit -n 16384

That would ensure the Apache and it's children have plenty of files to play with.

Let us know if that does the trick for you!

-Eric

Tue, 02/01/2011 - 08:12
ummecasino

Thanks Eric, I'll try that and report!

cheers, Alex

Sun, 04/10/2011 - 16:23
chrismfz

Seriously, is this normal with apache and php ? :-) :

lsof -n|grep -oE '^[a-z]+'|sort|uniq -c|sort -n 
  3 iscsi
  3 iw
  3 kacpid
  3 kauditd
  3 kedac
  3 khelper
  3 khubd
  3 khungtask
  3 kmpath
  3 kpsmoused
  3 kseriod
  3 kstriped
  3 kswapd
  3 kthread
  3 loop
  3 rdma
  6 pdflush
  7 klogd
  8 sleep
  9 init
  9 mdadm
  9 uniq
 11 freshclam
 11 grep
 12 gpm
 12 kjournald
 12 portmap
 13 syslogd
 14 hidd
 15 brcm
 15 pcscd
 15 smartd
 17 udevd
 18 scsi
 20 md
 21 sh
 22 collectin
 23 dbus
 23 ntpd
 24 aio
 24 clamd
 24 cqueue
 24 events
 24 kblockd
 24 kmpathd
 24 kondemand
 24 ksoftirqd
 24 migration
 24 rpciod
 24 watchdog
 25 lookup
 27 ata
 28 lsof
 33 ib
 33 iscsid
 34 proftpd
 38 pickup
 38 qmgr
 40 trivial
 41 local
 48 mailmanct
 48 mingetty
 48 named
 53 rpc
 55 snmpd
 57 postdrop
 57 sendmail
 58 bash
 68 lfd
 72 rpmq
 78 dovecot
 78 imap
 78 pop
 80 crond
 84 cleanup
105 hald
111 smtp
125 master
128 spamd
133 miniserv
143 sshd
155 saslauthd
408 python

1820 mysqld 66991 php 91309 httpd

Almost 150.000 open files from php and apache when apache-status reports the same time: 89.2 requests/sec - 2.2 MB/second - 25.7 kB/request 316 requests currently being processed, 61 idle workers

System is Virtualmin on a Centos 5.5 64bit OS. Nothing else installed, only virtualmin's repo and nothing else "exotic".

Sites running using fast cgi with a 240 seconds timeout on apache. turning it on mod_php it is catastrophic, server load reached 750 (!!!) i couldnt even type on terminal and I couldnt even shutdown apache... Reboot saved me.

Running netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n to check connections per IP and total connections it was fine. Server isn't under DoS or DDoS attack. Normal traffic as apache status shows but it raises load without an obvious reason...

Topic locked