apache will not start. Kinda urgent........

15 posts / 0 new
Last post
#1 Sat, 12/15/2012 - 05:05
kevinrmanderson

apache will not start. Kinda urgent........

Acccidentaly upgraded to the latest wbm (??) package which is probably related to the php_admin_value engine issue that caused problems a week or so ago.

Now apache will not start. Kinda urgent........

Any ideas?

Sat, 12/15/2012 - 06:13
helpmin

please the check error logs of your (web) server and post the error message here.

Sat, 12/15/2012 - 09:23
andreychek

The error logs are located in /var/log/httpd/error_log, or /var/log/apache2/error_log, depending on your distro.

Let us know what errors show up when you're trying to start Apache, and we can help troubleshoot that.

-Eric

Sat, 12/15/2012 - 18:01 (Reply to #3)
kevinrmanderson

Hi,

It was 2130 that I noticed apache wasn't running. 21:36 to 22:00 are attempts to start apache all failing - as you can see not much in the logs. I tried changing from mod_php, cgi wrapper etc - no change. Posted the query at 2200 ish. A bit later I realized I had a backup of an httpd.conf from a few weeks ago and put that in place to get the server running, hence the final line indicating a start.

Regards kevin

[Sat Dec 15 17:53:52 2012] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Sat Dec 15 17:53:52 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fcgid/2.3.7 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips SVN/1.6.11 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
[Sat Dec 15 17:55:37 2012] [notice] Graceful restart requested, doing restart
[Sat Dec 15 17:55:37 2012] [notice] Digest: generating secret for digest authentication ...
[Sat Dec 15 17:55:37 2012] [notice] Digest: done
[Sat Dec 15 17:55:37 2012] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Sat Dec 15 17:55:37 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fcgid/2.3.7 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips SVN/1.6.11 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
[Sat Dec 15 18:49:24 2012] [notice] SIGHUP received.  Attempting to restart
[Sat Dec 15 18:49:24 2012] [notice] Digest: generating secret for digest authentication ...
[Sat Dec 15 18:49:24 2012] [notice] Digest: done
[Sat Dec 15 21:36:42 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Dec 15 21:37:17 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Dec 15 21:38:32 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Dec 15 21:39:04 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Dec 15 21:40:09 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Dec 15 21:42:38 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Dec 15 21:43:10 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Dec 15 21:57:15 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Dec 15 22:00:42 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Dec 15 22:07:31 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Dec 15 22:07:31 2012] [notice] Digest: generating secret for digest authentication ...
[Sat Dec 15 22:07:31 2012] [notice] Digest: done
[Sat Dec 15 22:07:32 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fcgid/2.3.7 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips SVN/1.6.11 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
Sat, 12/15/2012 - 22:50 (Reply to #4)
kevinrmanderson

Few more bits.

I have tried taking the virtualhost section from failing conf file and adding that to the start of the httpd.conf up to the first virtualhost from working conf file but that fails. I have managed to take the working httpd.conf and add the virtualhost section for a few domains (some from the failing conf file and some for an older working file) and have them working. Several https sites are broken and I have moved them to another server temporarily.

A diff between the working and failing httpd.conf shows the FollowSymLinks and SymLinksifOwnerMatch as the main changes. Difficult to see the diffs in some of the virtualhost sections since the order has changed with the editing to get things going.

Regards kevin

Sun, 12/16/2012 - 00:54 (Reply to #5)
kevinrmanderson

I really need to get this sorted out. We have several sites not working, due to upgrades some now moved to their prior working server have also failed.

Is it safe to reinstall http://software.virtualmin.com/gpl/universal/wbm-virtual-server-3.96.gpl... (last version) or even the one before that?? The discussion at https://www.virtualmin.com/node/12181 seems to indicate it is ok to go back to a prior version.

Does the install of the wbm-virtual-server do a rebuild of the httpd.conf since the one there at the moment is a mix of pieces from severs 'versions' and some bits from other servers. I can rename the one from last night that failed, will that get updated??

Regards kevin

Sun, 12/16/2012 - 01:55
kevinrmanderson

Just playing round trying to get some of the secure sites back working and this appears to be what is causing the crashing. The individual site error_log show the certificates aren't matching (presume the cert and the private key matching) and throws an error. Last night I installed the certs for one of the sites and they are ok (the modulus still matches) and virtualmin shows the correct expiry date/issuers etc.

As a test I just removed the virtualhost port 443 sections from last night failing httpd.conf and voila http runs ok. So appears to be an issue with ssl sites.

Edited one of the sites to enable ssl. OK with the self signed certs, copy back the correct godaddy cert (all four files, key, private, ca, csr) and httpd fails to start.

So to get it all back am removing the ssl for all the sites with ssl until I can get it sorted out.

Kevin

Sun, 12/16/2012 - 05:13 (Reply to #7)
kevinrmanderson

crock of sheeeiiitt. Any site that has had ssl is broken, have turned off SSL (as noted above) so only port 80 working. They now return an error of unable to access index.php or whatever. Now moving these sites back to the old servers yet again, some will be broken.

So back to the query re loading the older version of wbm-virtual-manager and all the queries in the earlier message..... We have an out of the box server with only virtualmin and yum updates so we can't be the only ones with this problem post update. Was this upgrade tested???

Why is life so hard. Now the two most recent upgrades to virtualmin and both have broken the server. The upgrade two weeks ago lost us one client, bet we will loose a heap with this one.

Thought we were going to have a quiet and potentially early night.

If we buy a commercial licence will it fix this issue, can we upgrade the GPL without reinstalling everything, will it work with goDaddy certs (as the GPL version did until yesterdays update).

Kevin

Sun, 12/16/2012 - 10:44
andreychek

Edited one of the sites to enable ssl. OK with the self signed certs, copy back the correct godaddy cert (all four files, key, private, ca, csr) and httpd fails to start.

There might be a problem with one of the certs for some reason... what you could try doing is to enable those SSL sites, one at a time... and when Apache fails to start, check the error logs for that particular Virtual Server that you just enabled SSL for. Those logs are in $HOME/logs/error_log.

It's possible that an SSL-specific error could show up there, rather than the system-logs.

We have an out of the box server with only virtualmin and yum updates so we can't be the only ones with this problem post update. Was this upgrade tested?

I know you're frustrated, and we'll certainly do our best to help. You're the only one reporting an SSL problem like this though.

The set of symptoms you're describing makes it sound like there's an SSL certificate issue going on -- and it's possible that the problem was the Apache restart, and not necessarily the Virtualmin upgrade. Though if there's a problem with the Virtualmin update, we'd certainly like to figure out what that is.

If we buy a commercial licence will it fix this issue, can we upgrade the GPL without reinstalling everything

You're not seeing a problem that would be fixed by having the Pro version.

We don't want folks using Virtualmin GPL or Pro to have problems, and we certainly wouldn't want anyone to have to pay money to not have problems. Virtualmin Pro does offer more features, not the least of which is the Premium Support -- but that's not at all related to the problems you're seeing here.

The key to resolving the issue you're seeing, is to determine what error Apache is throwing.

If it's not in the main Apache log, then you may want to review the Apache error log for the individual domains as mentioned above... if you can enable SSL one site at a time, that would help you figure out which logs to look at.

Sometimes, rather than using the init script, launching Apache by hand can show the error it's throwing on the console.

For example, if you're using CentOS, try running this command as root:

/usr/sbin/httpd

When doing that, do you see any errors being output?

-Eric

Mon, 12/17/2012 - 03:07
Locutus

Apache also has a command line parameter to perform a full configuration check and report any errors. Can't recall what it was (am not at my PC currently) , but it is there.

Mon, 12/17/2012 - 06:27
kevinrmanderson

Thanks for the comments.

The issue was I accidentally updated my primary production server. On the system information screen the position of the status and update is too close/disastrous, since one click on update and all over. I started to get to the server via an ssh session to reboot/stop update but that was too slow. From rading the forums It looks like other are also having issues although as you say, I can''t see any other ssl ones. Interestingly we are aware of a number of other SSL sites in the Hobart area (unknown where hosted) who have also suddenly had major issues with the infrastructure and are offline - not saying virtualmin is the issue, but coincidental.

Latest update: What I have since tried is to delete one of the old ssl servers and recreate with ssl (so self signed). That works and no 'unable to access' apache errors for http but still for https. Godaddy certs load ok, but same unable tpo access

back to your comments/queries, based on status yesterday: progressively try the certs. I disabled all the ssl sites then tried only one site with self signed and that started ok. Moved the verified godady cert and wouldn't restart. So backed it out (yet to try with the new load as per above). There were four sites with ssl certs (all godaddy) and all still verify the modulus as ok, so the cert combination is ok.

problem post upgrade etc. The server had all sites working perfectly. I accidentally did the wbm-virtual-server upgrade to the current and it was all over. I have some other production servers on the prior wbm... version and no issue (although they don't have ssl sites). Copy back a prior httpd.conf without ssl segments and works, so I see what looks like a 100% correlation to the upgrade. I worked out the SSL issue from the error logs in the individual domains. I also want to fix the issue, hence I am also doing all I can as well.

Whoops, haven;t indicated the server config - Intel i5, raided ssd, 6GB ram, Centos 6.3, all Centos patches/updates, was a new server approx 6 weeks ago.

I just tested the same godaddy certs in the newly recreated ssl based domain and apache starts ok. However access the ssl sites returns 'You don't have permission to access /home.php on this server.' Works perfectly, all content etc as a non-ssl site as it did previously.

re gpl verses commercial. It is far from clear to me what the commercial gives. However, we use this to help us make money and live so fairs fair and nd we need to buy so you can do the same.

The apache check command is in apachectl -t - ran it and shows a bit ominously...

 apachectl -t
[Mon Dec 17 23:13:13 2012] [warn] VirtualHost 27.50.84.xxx:443 overlaps with VirtualHost 27.50.84.xxx:443, the first has precedence, perhaps you need a NameVirtualHost directive
[Mon Dec 17 23:13:13 2012] [warn] NameVirtualHost 180.92.199.yyy:443 has no VirtualHosts
Syntax OK
 
 
regards
Mon, 12/17/2012 - 14:41
andreychek

From rading the forums It looks like other are also having issues although as you say

Well, the issue is that Virtualmin added config changes to Apache that are --

  1. Preventing PHP from executing in mod_php mode, when CGI or FCGID are selected. This shouldn't cause problems in most cases, except when some non-standard PHP configuration is being used. When this does cause problems, PHP apps will throw error messages when being accessed. That's not what you're seeing though, you're seeing a problem with Apache not starting, and errors accessing an SSL site.

  2. It also prevents the FollowSymlinks tag from being used in a .htaccess file. When this occurs, an error is thrown in the Apache logs stating that "FollowSymlinks isn't allowed". That's not what you're seeing either.

I'm a bit puzzled as to what you're seeing, as it doesn't match any problems we've run into before, I don't believe.

Latest update: What I have since tried is to delete one of the old ssl servers and recreate with ssl (so self signed). That works and no 'unable to access' apache errors for http but still for https. Godaddy certs load ok, but same unable tpo access

So what it sounds like you're saying is -- it allows you to enable SSL, but when trying to access any of the SSL sites, you receive an "Unable to access" error in your browser... is that correct?

The Apache config test you ran is throwing some suspicious warnings... have any IP addresses on your system changed recently?

It looks like multiple sites are attempting to use SSL on the same IP address... which is an unusual setup, and not supported by all browsers (it's called SNI).

If it's your intention to use that, you may want to edit your Apache config here:

/etc/httpd/conf/httpd.conf

And add this line to the end:

NameVirtualHost 27.50.84.xxx:443

And then restart Apache.

However, in most cases, you'd want one SSL certificate per IP address. But, adding the above config line may help in the meantime.

-Eric

Tue, 12/18/2012 - 03:21 (Reply to #12)
kevinrmanderson

Hi

you comment re enable ssl etc, correct.

IPs changed. When I deleted the ssl enabled site and reentered I had to re-add that IP, Added one other last Sat morning (before issues began) and others are the same for the last few weeks. Dont believe I have multiple on same ip (if so will fix/remove).

So still an unknown. I will check multiple ssl on same IP later today when back in the office.

Kevin

edit, later in day: Checked: SSL is on three non shared IPs. There is also a NameVirtualHost 27.50.84.xxx:443 line for the shared IP address. Also checked all directives in the virtualhost container and aside of IPCCommTimeout now being replaced with FcgidIOTimeout nothing else was particularly unexpected.

Still unable to access SSL sites, keep getting the unable to access error.

Tue, 12/18/2012 - 23:10
andreychek

What happens if you run " apachectl -t" again, does that show any warnings/errors?

Outside of that, I'm running low on ideas without getting into the specifics of your setup.

To be able to help further, you'll probably need to give us an actual domain name that isn't working, as well as the VirtualHost block in your Apache config for one of the domain's that isn't working.

You can always go back after we get things working and remove them from your post :-)

-Eric

Thu, 12/20/2012 - 23:57 (Reply to #14)
kevinrmanderson

apachectl -t Syntax OK

Status - is good now.

I spent some time going through the httpd.conf. I found the Options line was commented out from when the initial problem presented. I uncommented that line. I reset the permissions for all files in the entire subserver tree. I checked the virtualmin setup and the www/non-www were ok. Check the dns server (from different server) and the www was to a different server although the checks have all been against the non-www. However all the files also existed on the other server (we are migrating from that one to the new server). restarted apache and it is now working ok. Checked the sslcerts and they are the godaddy ones.

So the thing I did that appears to have fixed the issue was the Options directive.

I need to go back and recheck my notes for the godaddy/self signed and issues there. This is the one that still concerns me.

Needless to say keeping a close eye on it. Have one more ssl site to shift to the server tomorrow so will see if the sames issues arise.

Thanks for comments etc, still really none the wiser re the issues/root cause/fix.

Kevin

Think I have figured it out.

Scenario, site was working, some sort of problem, ssl site fails and certs remain in place. in this case the certs are valid (modulus checks ok on cert and key) but apache barfs (still to figure out why). Try to fix by unchecking SSL site in virtualmin and re-add. Next step is add SSL by checking the ssl server check box, however, the failing certs are still in in place, the 'add' appears to be successful but the port 443 directives (virtualhost port 443 block) for the specific site are not added to the httpd.conf. Result when accessing the https/port 43 is unable to access files.

I removed the offending ssl files, reselected the self signed cert, uncheck ssl, save, recheck ssl save, and all ok. SSL now works.

Now to figure out the problem SSL certs but this is now a different problem. Systems/SSL working.

Fix, disable SSL, Remove certs from server ()regardless of how valid), reset to self signed, drop SSL enablement, reset ssl enable and, reenable self signed, and then copy back the good certs. Restart httpd.

Topic locked