Apache stops randomly and replace with

Hi,

My Apache randomly stops and I cannot start it again. I said the address already in use. https://goo.gl/Osz905 I have to follow this post (first answer) to kill the process: http://askubuntu.com/questions/277162/apache-fails-to-start-address-alre... And then restart Apache by: service apache2 start

Then I follow the second answer to disable the port in ports.conf but the problem still exists. It show other error when I try to start via webmin: https://goo.gl/cKt31z

Please help me. It is very urgent for me.

Thanks, Giang Anh

Status: 
Active

Comments

Howdy -- it sounds like the process may have gotten stuck.

You should be able to correct that by running these two commands:

killall -9 apache2
service apache2 start

Hi,

Is this the permanent solution? Or I have to use these command every I have this issue?

Thanks, Giang Anh

Hmm, are you seeing that issue a lot?

That issue shouldn't be occurring frequently... if it is, we may need to try and troubleshoot what is causing the processes to lock up like that.

Yes. I have this issue about 3-4 day/time. I have just happened as well.

Ah, that sounds very bad, that should definitely not happen. Are you perhaps on a VPS? If so, what is the output of "free -m"?

Also, do you see any errors or interesting messages in /var/log/apache2/error_log around the time that Apache is stopping?

Hi,

# free -m
             total       used       free     shared    buffers     cached
Mem:         56360       4984      51376       1016        259       2791
-/+ buffers/cache:       1933      54427
Swap:            0          0          0

Do you see any errors or interesting messages in /var/log/apache2/error_log around the time that Apache is stopping?

And if this is a VPS, what kind of VPS is it? For example, KVM, Xen, or OpenVZ?

Hi,

The log here.

[Wed Apr 13 18:03:53.727260 2016] [core:notice] [pid 1857] AH00051: child pid 24041 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:05:09.810731 2016] [core:notice] [pid 1857] AH00051: child pid 23172 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:05:19.825293 2016] [core:notice] [pid 1857] AH00051: child pid 24039 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:05:38.846757 2016] [core:notice] [pid 1857] AH00051: child pid 20311 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:05:40.849122 2016] [core:notice] [pid 1857] AH00051: child pid 22663 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:13:03.324194 2016] [core:notice] [pid 1857] AH00051: child pid 24531 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:13:04.325337 2016] [core:notice] [pid 1857] AH00051: child pid 24530 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:14:05.393549 2016] [core:notice] [pid 1857] AH00051: child pid 24125 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:14:06.394841 2016] [core:notice] [pid 1857] AH00051: child pid 22666 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:14:30.425816 2016] [core:notice] [pid 1857] AH00051: child pid 24124 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:31:11.365460 2016] [core:notice] [pid 1857] AH00051: child pid 20943 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:31:15.370020 2016] [core:notice] [pid 1857] AH00051: child pid 24625 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:38:15.777484 2016] [core:notice] [pid 1857] AH00051: child pid 25897 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:38:16.778719 2016] [core:notice] [pid 1857] AH00051: child pid 25554 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:38:17.780041 2016] [core:notice] [pid 1857] AH00051: child pid 23177 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:41:17.977523 2016] [core:notice] [pid 1857] AH00051: child pid 24636 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:41:47.004606 2016] [core:notice] [pid 1857] AH00051: child pid 24624 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:42:01.017409 2016] [core:notice] [pid 1857] AH00051: child pid 23636 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:51:10.598754 2016] [core:notice] [pid 1857] AH00051: child pid 23178 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Apr 13 18:51:15.260912 2016] [mpm_prefork:notice] [pid 1857] AH00171: Graceful restart requested, doing restart
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
[Wed Apr 13 18:51:15.265141 2016] [core:notice] [pid 1857] AH00060: seg fault or similar nasty error detected in the parent process
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f2efc15eb30 ***

It's MS Azure Virtual Machine, not VPS.

Yikes, it looks like you're seeing some unusual Apache crashes.

What is the output of these commands:

dpkg -l 'apache*'
ls /etc/apache2/mods-enabled

Hi,

~# dpkg -l 'apache*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  apache2        2.4.7-1ubunt amd64        Apache HTTP Server
un  apache2-api-20 <none>       <none>       (no description available)
ii  apache2-bin    2.4.7-1ubunt amd64        Apache HTTP Server (binary files
ii  apache2-data   2.4.7-1ubunt all          Apache HTTP Server (common files)
ii  apache2-doc    2.4.7-1ubunt all          Apache HTTP Server (on-site docum
un  apache2-mpm-ev <none>       <none>       (no description available)
un  apache2-mpm-it <none>       <none>       (no description available)
un  apache2-mpm-pr <none>       <none>       (no description available)
un  apache2-mpm-wo <none>       <none>       (no description available)
un  apache2-suexec <none>       <none>       (no description available)
ii  apache2-suexec 2.4.7-1ubunt amd64        Apache HTTP Server configurable s
un  apache2-suexec <none>       <none>       (no description available)
un  apache2-utils  <none>       <none>       (no description available)
un  apache2.2-bin  <none>       <none>       (no description available)
un  apache2.2-comm <none>       <none>       (no description available)
root@JWeb:~# ls /etc/apache2/mods-enabled
access_compat.load  autoindex.conf  fcgid.load           proxy_connect.load
actions.conf        autoindex.load  filter.load          proxy_http.load
actions.load        dav_fs.conf     mime.conf            proxy.load
alias.conf          dav_fs.load     mime.load            rewrite.load
alias.load          dav.load        mpm_prefork.conf     setenvif.conf
auth_basic.load     dav_svn.conf    mpm_prefork.load     setenvif.load
auth_digest.load    dav_svn.load    negotiation.conf     slotmem_shm.load
authn_core.load     deflate.conf    negotiation.load     socache_shmcb.load
authn_file.load     deflate.load    php5.conf            ssl.conf
authz_core.load     dir.conf        php5.load            ssl.load
authz_host.load     dir.load        proxy_balancer.conf  status.conf
authz_svn.load      env.load        proxy_balancer.load  status.load
authz_user.load     fcgid.conf      proxy.conf           suexec.load

Hmm, that all looks fairly normal.

When did this problem start occurring?

I've been doing some research on your problem, and it sounds like some people who were seeing that issue were using a PHP cache, such as Zend. Are you using any PHP caching modules for any of your domains?

Hi,

It happened from around 2 months ago. I'm using PHP caching for my Joomla and Wordpress websites. Please let me know if you need root account of the server.

Thanks, Giang Anh

You appear to be seeing either an Apache bug, or a PHP bug.

What you might want to do is disable the PHP caching temporarily, to see if that resolves the problem that you're experiencing.

If it does resolve the issue, you may need to look into a different caching type.

Which type of PHP cache are you using, out of curiosity?

I have just disable OPcache in my php.ini file. Will keep you updated.

It still happens. 5 times today.

And just to verify, you removed those caching modules from all domains that are using them?

I disable it in php.ini file in my server. And yes, it will be removed on all sites.

Just to verify -- are you having any problems with any other services running on your server?

If not, then it's either an Apache bug, or a PHP bug, that you're experiencing.

There isn't an easy way to resolve that.

You could always post a note in the Ubuntu bug tracker regarding that, but they're going to ask what modules you're using, as it's very rare to see Apache crash like that.

So what you could always try, is to review both your Apache modules, as well as your PHP modules, and remove any that you don't need.

For starters, what is the output of this command:

dpkg -l 'php*'

Also, how many domains are running on your server there?

Hi,

I don't have any other issue and this problem happens randomly. Also I'm not familiar with Ubuntu nor Ubuntu bug tracker since from the first time I began with Linux, I use Virtualmin under your support :)

# dpkg -l 'php*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
un  php-apc        <none>       <none>       (no description available)
un  php-crypt-gpg  <none>       <none>       (no description available)
ii  php-pear       5.5.9+dfsg-1 all          PEAR - PHP Extension and Applicat
un  php-xml-util   <none>       <none>       (no description available)
un  php4-pear      <none>       <none>       (no description available)
ii  php5           5.5.9+dfsg-1 all          server-side, HTML-embedded script
ii  php5-cgi       5.5.9+dfsg-1 amd64        server-side, HTML-embedded script
ii  php5-cli       5.5.9+dfsg-1 amd64        command-line interpreter for the
ii  php5-common    5.5.9+dfsg-1 amd64        Common files for packages built f
ii  php5-curl      5.5.9+dfsg-1 amd64        CURL module for php5
un  php5-dev       <none>       <none>       (no description available)
un  php5-fpm       <none>       <none>       (no description available)
ii  php5-gd        5.5.9+dfsg-1 amd64        GD module for php5
ii  php5-json      1.3.2-2build amd64        JSON module for php5
un  php5-mhash     <none>       <none>       (no description available)
ii  php5-mysql     5.5.9+dfsg-1 amd64        MySQL module for php5
un  php5-mysqli    <none>       <none>       (no description available)
un  php5-mysqlnd   <none>       <none>       (no description available)
ii  php5-readline  5.5.9+dfsg-1 amd64        Readline module for php5
un  php5-suhosin   <none>       <none>       (no description available)
un  php5-user-cach <none>       <none>       (no description available)
un  php5-xcache    <none>       <none>       (no description available)
un  php5-xdebug    <none>       <none>       (no description available)
un  phpapi-2012121 <none>       <none>       (no description available)

I'm running around 6 websites under this server. However, this is a powerful server with 16 core strong CPU and 64GB RAM

What you're seeing there is very odd!

Ubuntu 14.04 is one of the more popular distributions for running Virtualmin, and we've never seen crashes like you're experiencing.

I was actually wondering if maybe it were a hardware issue, but if you aren't seeing problems with other services, that is unlikely.

I double-checked the Apache modules that are loaded, and noticed one that stood out -- the socache_shmcb module.

You may want to try disabling that to see if that helps... that can be disabled with this command:

a2dismod socache_shmcb

After running that, restart Apache.

Hi,

I tried to disable that module but it requires ssl module to be disabled. After disable those 2 modules. I cannot restart apache:

# a2dismod socache_shmcb
ERROR: The following modules depend on socache_shmcb and need to be disabled first: ssl
root@JWeb:~# a2dismod ssl
Module ssl disabled.
To activate the new configuration, you need to run:
  service apache2 restart
root@JWeb:~# a2dismod socache_shmcb
Module socache_shmcb disabled.
To activate the new configuration, you need to run:
  service apache2 restart
root@JWeb:~# service apache2 restart
 * Restarting web server apache2                                         [fail]
 * The apache2 configtest failed.
Output of config test was:
AH00526: Syntax error on line 91 of /etc/apache2/sites-enabled/adanhgia.com.conf:
Invalid command 'SSLEngine', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.

Okay, I didn't realize the SSL module depended on it. It sounds like you'll need to re-enable both socache_shmcb and ssl.

I'll do some digging to see if I can get any other ideas.

This Solve it for me: For Debian based systems :

Replace ' apache2 reload ' with ' apache2 restart ' in /etc/logrotate.d/apache2 file.

For RHEL based systems

Replace ' httpd reload ' with ' httpd restart ' in /etc/logrotate.d/httpd file.

If the above steps do not help on Debian/Ubuntu OSes, edit log rotation settings of apache in /etc/logrotate.d/apache2 file. Change " /etc/init.d/apache2 reload " to " /etc/init.d/apache2 restart ":

cat /etc/logrotate.d/apache2

.... /etc/init.d/apache2 restart > /dev/null;