Apache won't restart after usermin u0pdate

mlkoekenberg's picture
Submitted by mlkoekenberg on Mon, 03/29/2010 - 08:28

hello,

After an automatic UserMin update (Yum). Apache won't restart until I reboot the virtualmachine.

Martin Koekenberg

Status: 
Active

Comments

Submitted by andreychek on Mon, 03/29/2010 - 09:50

Well, Usermin shouldn't affect Apache, as Usermin is a standalone daemon and doesn't use or require Apache.

However, we'll gladly help to figure out what's going on.

When you try to launch Apache, do you see any errors in your Apache log, /var/log/httpd/error_log?

mlkoekenberg's picture
Submitted by mlkoekenberg on Mon, 03/29/2010 - 10:39

Apache went down on 04:00. On that moment a usermin update cron Job is running.

There is a 'no space on device' message. There is enough space ;-)

[Sun Mar 28 04:03:24 2010] [notice] Graceful restart requested, doing restart [Sun Mar 28 04:03:24 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket) [Sun Mar 28 04:03:24 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket) [Sun Mar 28 04:03:25 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:03:25 2010] [notice] Digest: done [Sun Mar 28 04:03:26 2010] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Sun Mar 28 04:03:27 2010] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations [Sun Mar 28 04:03:32 2010] [notice] mod_fcgid: call /home/sassendonk/public_html/index.php with wrapper /home/sassendonk/fcgi-bin/php5.fcgi [Sun Mar 28 04:03:39 2010] [notice] mod_fcgid: process /home/sassendonk/public_html/index.php(9655) exit(shutting down), terminated by calling exit(), return code: 0 [Sun Mar 28 04:03:48 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:48 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:49 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:49 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:50 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:50 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:03:50 2010] [notice] Digest: done [Sun Mar 28 04:03:52 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:52 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:03:52 2010] [notice] Digest: done [Sun Mar 28 04:03:52 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:53 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:53 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:54 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:54 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:55 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:55 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:03:55 2010] [notice] Digest: done [Sun Mar 28 04:03:55 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:55 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:03:55 2010] [notice] Digest: done [Sun Mar 28 04:03:57 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:57 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:58 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:58 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:59 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:03:59 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:00 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:04:00 2010] [notice] Digest: done [Sun Mar 28 04:04:00 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:00 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:01 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:01 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:02 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:04:02 2010] [notice] Digest: done [Sun Mar 28 04:04:03 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:04 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:04 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:04 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:05 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:04:05 2010] [notice] Digest: done [Sun Mar 28 04:04:05 2010] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Sun Mar 28 04:04:06 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:06 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:04:06 2010] [notice] Digest: done [Sun Mar 28 04:04:08 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:08 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:08 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:04:08 2010] [notice] Digest: done [Sun Mar 28 04:04:10 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:10 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:11 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:04:11 2010] [notice] Digest: done [Sun Mar 28 04:04:11 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sun Mar 28 04:04:11 2010] [notice] Digest: generating secret for digest authentication ... [Sun Mar 28 04:04:11 2010] [notice] Digest: done [Sun Mar 28 04:04:11 2010] [emerg] (28)No space left on device: mod_fcgid: Can't create global pipe mutex [Sun Mar 28 04:07:13 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed [Sun Mar 28 04:12:46 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed [Sun Mar 28 04:17:13 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed [Sun Mar 28 04:22:14 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed [Sun Mar 28 04:27:46 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed [Sun Mar 28 04:32:14 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed [Sun Mar 28 04:37:13 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed [Sun Mar 28 04:42:13 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed [Sun Mar 28 04:47:13 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed

Submitted by andreychek on Mon, 03/29/2010 - 10:58

At the end of your log file, I see entries such as this:

No space left on device: mod_rewrite: could not create rewrite_log_lock

Are you running low on disk space?

For example, what does this command show:

df -h

mlkoekenberg's picture
Submitted by mlkoekenberg on Wed, 03/31/2010 - 15:21

Hello,

I've enough diskspace.

This is the output of the command:

Filesystem Size Used Avail Use% Mounted on /dev/sda1 60G 39G 20G 67% / none 4.0G 0 4.0G 0% /dev/shm da-svr002:/var/backup 263G 155G 95G 63% /var/backup

Submitted by andreychek on Wed, 03/31/2010 - 19:02

What do you get if you type this on the command line:

ipcs -s

If Apache left too many semaphores laying around, that may be causing problems; you may need to clear some of them out in order to get Apache back online.

mlkoekenberg's picture
Submitted by mlkoekenberg on Fri, 04/02/2010 - 05:20

------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 4390912 apache 600 1 0x00000000 4358145 apache 600 1 0x00000000 4423682 apache 600 1 0x00000000 4456451 apache 600 1 0x00000000 4489220 apache 600 1 0x00000000 4521989 apache 600 1 0x00000000 4554758 apache 600 1 0x00000000 4587527 apache 600 1 0x00000000 4620296 apache 600 1 0x00000000 4653065 apache 600 1

Submitted by andreychek on Fri, 04/02/2010 - 08:24

Okay, so if Apache isn't running, there shouldn't be any semaphores open by Apache.

You can delete those by using "ipcrm" command. You can use omething like:

ipcrm -s SEMAPHORE_ID

In your case above, you could delete the first semaphore by typing "ipcrm -s 4390912".

After you delete those, you should be able to restart Apache.

mlkoekenberg's picture
Submitted by mlkoekenberg on Fri, 04/02/2010 - 09:39

If this happens again ill try this.

mlkoekenberg's picture
Submitted by mlkoekenberg on Sun, 04/11/2010 - 05:40

Hello,

Last night a 04:00 (Netherlands) Apache stopped again.

Can you login and look at the logfiles to find out what the problem is ???? Can I send you mij login details and the root password to check this.

I get a lot of questions from my customers.

I allready set all the apache processes and limits to the default value.

Martin Koekenberg

Submitted by andreychek on Sun, 04/11/2010 - 14:39

Well, I'd be curious what errors show up in the logs at 4am last night, as well as what "ipcs -s" shows when Apache isn't running.

However, if you'd like me to log in and take a look, I can do so.

You can either enable remote logins using the Support module, or you can email your root login information to eric@virtualmin.com.

Submitted by andreychek on Mon, 04/12/2010 - 09:02

Yeah, it seems to be the same error as you saw previously -- those semaphore related issues. It's occurring immediately after logrotate attempts to restart Apache with the "graceful" option.

I'm not entirely certain why those are occurring, though I'm also seeing a lot of errors related to mod_fcgid. I'd be curious if your problems went away if you were to switch from fcgid to straight cgi. Is that an option?

You could try changing one or two sites first to make sure it works properly for you.. and if so, you could try changing all your sites over to that.

You can change all sites at once using the command line tools... this command can do that for you:

virtualmin modify-web --all-domains --mode cgi

But again, I recommend changing one or two domains first, and testing that the domains continue to work in an acceptable manner for you :-)

mlkoekenberg's picture
Submitted by mlkoekenberg on Mon, 04/26/2010 - 03:16

Hello,

Last sunday morning there was nog apache down time.... I think it is solved now.

Martin

Submitted by andreychek on Mon, 04/26/2010 - 09:02

That's great -- thanks for the heads up!

mlkoekenberg's picture
Submitted by mlkoekenberg on Sun, 05/02/2010 - 05:24

This night it happens again. Again at 04:00.

I don't think it has somthing tot do with fcgi. But I'll change al the websites to straight cgi.

Martin

mlkoekenberg's picture
Submitted by mlkoekenberg on Mon, 05/03/2010 - 08:29

I tried to change the websites to straight cgi with the command:

sh modify-web.pl --all-domains --mode cgi

This is the output:

modify-web.pl: line 3: =head1: command not found modify-web.pl: line 5: Change: command not found modify-web.pl: line 49: contents: command not found modify-web.pl: line 50: address,: command not found modify-web.pl: line 52: =cut: command not found modify-web.pl: line 54: package: command not found modify-web.pl: line 55: syntax error near unexpected token {' modify-web.pl: line 55:if (!$module_name) {'

Submitted by andreychek on Mon, 05/03/2010 - 09:27

It looks like you're prepending "sh" to the command... rather than that, you'd run the "virtualmin" tool.

Using the parameters you shows above as an example, you could use this:

virtualmin modify-web.pl --all-domains --mode cgi

mlkoekenberg's picture
Submitted by mlkoekenberg on Mon, 05/03/2010 - 09:30

where is this virtualmin toollocated (Path to thistool) ?

Submitted by andreychek on Mon, 05/03/2010 - 09:31

I believe on most Linux distros, it's located in "/usr/sbin/virtualmin".

mlkoekenberg's picture
Submitted by mlkoekenberg on Mon, 05/03/2010 - 11:07

I changed all the virtualservers.

I just wait a few day and thesee if those erros are stilin thelogfiles.

mlkoekenberg's picture
Submitted by mlkoekenberg on Sun, 05/23/2010 - 06:47

Apache went down again. There are a lot of semaphore active. Due some reason apache isn't cleaning them. Is there a setting wrong ? Is there a option to clean the automaticaly?

mlkoekenberg's picture
Submitted by mlkoekenberg on Sun, 05/23/2010 - 06:59

I added this tot sysctl.conf: kernel.msgmni = 1024 kernel.sem = 250 256000 32 1024

And activated it in the kernel with sysctl -p

Could this be a solution ?

Martin

mlkoekenberg's picture
Submitted by mlkoekenberg on Tue, 08/31/2010 - 08:46

hello,

I found an other issue with the same problem. Log Rotation is causing this problem. Due a apache restart instead a grace full are the semaphore not cleared properly. This logrotation is happening at sunday morning 04:00. The moment that apache is going down.

The log rotate post action should be an apache gracefull instead an apache restart.

See: http://www.virtualmin.com/node/13303

Martin

Submitted by JamieCameron on Tue, 08/31/2010 - 11:37

That sort-of sounds like an Apache bug to me .. surely a restart should clear everything cleanly.