Apache appeared to hit OOM due to ungraceful restart

Submitted by warphost on Thu, 02/05/2015 - 10:41

Hi,

We have observed a strange behaviour on a customer's virtual server. In the Apache error logs, after every daily cronjob, there was this:

[Sat Jan 31 06:25:08 2015] [notice] Graceful restart requested, doing restart
[Sat Jan 31 06:25:08 2015] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)

It seems, from a bit of digging, that this could be related to htcacheclean and/or apachectl not properly closing off FCGId PHP processes, causing more to build up day after day.

As an interim measure, we have put in a full daily Apache restart at 7am. Ideally, of course, it'd be good to see if this issue can be resolved and remove the non-standard cron.

Could you suggest any pointers for investigating/fixing this?

Many thanks, Steve

Status: 
Active

Comments

Submitted by andreychek on Thu, 02/05/2015 - 11:00

Howdy -- hmm, I'm not quite sure what might cause that, though I might start by looking at two things. What is the output of these two commands:

dpkg -l 'apache*'
ls /etc/apache2/mods-enabled

Also, you do always have the option of switching to CGI rather than FCGID, if the performance difference isn't that great in your case. Many websites don't experience a noticeable performance change when switching from one to another.

Submitted by warphost on Mon, 02/23/2015 - 07:05

I'll switch to CGId and see how we go on that.

Here's the output:

# dpkg -l 'apache*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                            Version              Architecture         Description
+++-===============================-====================-====================-===================================================================
un  apache                          <none>                                    (no description available)
un  apache-common                   <none>                                    (no description available)
un  apache-utils                    <none>                                    (no description available)
ii  apache2                         2.2.22-13+deb7u4     amd64                Apache HTTP Server metapackage
un  apache2-common                  <none>                                    (no description available)
ii  apache2-doc                     2.2.22-13+deb7u4     all                  Apache HTTP Server documentation
un  apache2-mpm                     <none>                                    (no description available)
un  apache2-mpm-event               <none>                                    (no description available)
un  apache2-mpm-itk                 <none>                                    (no description available)
ii  apache2-mpm-prefork             2.2.22-13+deb7u4     amd64                Apache HTTP Server - traditional non-threaded model
un  apache2-mpm-worker              <none>                                    (no description available)
un  apache2-suexec                  <none>                                    (no description available)
ii  apache2-suexec-custom           2.2.22-13+deb7u4     amd64                Configurable suexec program for Apache 2 mod_suexec
ii  apache2-utils                   2.2.22-13+deb7u4     amd64                utility programs for webservers
ii  apache2.2-bin                   2.2.22-13+deb7u4     amd64                Apache HTTP Server common binary files
ii  apache2.2-common                2.2.22-13+deb7u4     amd64                Apache HTTP Server common files


# ls /etc/apache2/mods-enabled
actions.conf      authz_default.load    cgi.load      deflate.load    mem_cache.load    proxy_balancer.conf  reqtimeout.load  status.conf
actions.load      authz_groupfile.load  dav_fs.conf   dir.conf        mime.conf         proxy_balancer.load  rewrite.load     status.load
alias.conf        authz_host.load       dav_fs.load   dir.load        mime.load         proxy.conf           ruby.load        suexec.load
alias.load        authz_user.load       dav.load      env.load        negotiation.conf  proxy_connect.load   setenvif.conf
auth_basic.load   autoindex.conf        dav_svn.conf  fcgid.conf      negotiation.load  proxy_http.load      setenvif.load
auth_digest.load  autoindex.load        dav_svn.load  fcgid.load      php5.conf         proxy.load           ssl.conf
authn_file.load   cache.load            deflate.conf  mem_cache.conf  php5.load         reqtimeout.conf      ssl.load

Many thanks!

Steve

Submitted by andreychek on Mon, 02/23/2015 - 10:31

I don't see anything unusual about your Apache version or the modules that are running. Has that issue been an on-going problem, or did it just occur once?

Submitted by warphost on Mon, 07/20/2015 - 04:38

Hi,

Sorry for the mega-late reply. At the time, this issue hadn't persisted for a long time - it was just discovered over the course of a day or two and this "fix" put in place.

Since I set up this restart cron job, it's been running fine. I think it's probably worth leaving this as-is for now, rather than mess about with it. Clearly, it's been 5-odd months since the problem was discovered/fixed and no issues have occurred since.

Cheers, Steve

Submitted by andreychek on Mon, 07/20/2015 - 10:26

I'm glad it's been working for you! I'll go ahead and mark this as fixed, but feel free to let us know if you have any additional questions.