Reboot server. 403 error. Manually restart apache2 and it starts to work fine.

16 posts / 0 new
Last post
#1 Wed, 11/28/2012 - 12:41
Chris_C

Reboot server. 403 error. Manually restart apache2 and it starts to work fine.

I'm running Virtualmin - latest version 3.96.GPL

2 virtual servers. Only use 1 for now. Email and web are setup correctly - as far as I can see. csf firewall is running and gives 36/37 score for security.

Here's the problem.

When the server reboots, apache2 gives a 403 Forbidden error for any access to the web site.

I have to manually restart apache2 - with service apache2 restart - then the site works.

I want apache2 to work from the get-go - no 403 error - without having to manually restart it.

Help !

Wed, 11/28/2012 - 14:28
andreychek

Howdy,

That's really unusual... I'm not sure what kind of problem you might be experiencing upon a reboot that causes Apache to not work initially, but it would work after restarting the service.

Are you by chance using a VPS? If so, what kind of VPS is it?

Also, which distro are you using? And do you see any errors in the system-wide Apache error logs?

-Eric

Wed, 11/28/2012 - 14:48
Chris_C

Yes- it's a VPS- SolusVM, OpenVZ container.

As for distro- my container is running Debian 6 (squeeze) x64.

System wide apache error log : tail /var/log/apache2/error.log Could not open input file: /home/mysitenamewhichis18characterslong/public_html/components/com_hwdvideoshare/xml/autogenerate.php (repeated 10 times) I checked that folder, and that file doesn't exist. So it shouldn't be a problem. I think it's an optional file.

I looked around a bit in Virtualmin and noticed something weird: First, my virtual server name is a bit long - it's 18 characters long in fact.
The weird thing is, the "user" and "group" are missing from Webmin's list of "users and groups"- must be a bug where Webmin chokes on the string being too long? My OTHER virtual server name is 17 characters long, and THAT one has entries showing up in "users and groups". BUT, when I go to add the user and group for my 18-character virtual server inside Virtualmin, it says it already exists.
I verify by looking at the files in the public_html for this 18-character virtual server name, and they seem to all have their user and group ownership set to that 18-character name. Maybe this is nothing or maybe it's a clue to what's happening.

Wed, 11/28/2012 - 15:17
andreychek

Hmm, could you paste in the contents of your /proc/user_beancounters file?

-Eric

Wed, 11/28/2012 - 16:49
Chris_C

It's a wide output.. hope it gets formatted OK..

~# cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
      944:  kmemsize                 34365284            131456233           2147483646           2147483646                    0
            lockedpages                     0                  968               999999               999999                    0
            privvmpages                424442               797854               786432               786432                98883
            shmpages                     1722                15185               524288               524288                    0
            dummy                           0                    0                    0                    0                    0
            numproc                       159                  323               999999               999999                    0
            physpages                  196435               264236                    0           2147483647                    0
            vmguarpages                     0                    0               524288           2147483647                    0
            oomguarpages               196435               264260               524288           2147483647                    0
            numtcpsock                     96                  520              7999992              7999992                    0
            numflock                       65                   73               999999               999999                    0
            numpty                          1                    3               500000               500000                    0
            numsiginfo                      1                   40               999999               999999                    0
            tcpsndbuf                 1745408             10674816            214748160            396774400                    0
            tcprcvbuf                 1767904              9393120            214748160            396774400                    0
            othersockbuf               326720              2024000            214748160            396774400                    0
            dgramrcvbuf                     0                30176            214748160            396774400                    0
            numothersock                  219                  466              7999992              7999992                    0
            dcachesize                1058253              1883634           2147483646           2147483646                    0
            numfile                      4805                 9916             23999976             23999976                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            numiptent                     151                  152               999999               999999                    0
~#
Wed, 11/28/2012 - 18:34
andreychek

Howdy,

Every once in awhile, we run into a really strange, bizarre, issue... one that we think should never ever be able to happen.

And we troubleshoot for a bit, discovering that the issue is occurring on an OpenVZ-based VPS, with the user_beancounters showing RAM or network resource failures.

That may be what's going on here as well, unfortunately :-)

Your user_beancounters file is showing that 98,883 requests for RAM were made, but denied... and that sort of issue can cause some really strange behavior.

To solve that, you could ask your provider for more resources.

Or, you could try disabling some services on your system, so that less are being launched at boot time... it's possible that would help.

You may also want to disable any Apache modules that you don't need... that could help too. There's some thoughts here on running Virtualmin on low-memory systems:

http://www.virtualmin.com/documentation/system/low-memory

The only thing that has me scratching my head is that usually these issues occur after some time of the server being online, it's rare that they occur only during bootup.

So, just to make sure we're not overlooking another kind of issue... what is that domain's PHP Execution Mode set to? You can determine that in Server Configuration -> Website Options. If it's set to FCGID, you may want to give CGI a try. Or if it's CGI, try FCGID, and see if that makes a difference.

-Eric

Wed, 11/28/2012 - 21:54
Chris_C

I really appreciate the help.

I set it to CGI, saved, and rebooted the server.
Still the same bug - apache comes up with 403 Forbidden- I even let it settle for 1 minute- still the 403 Forbidden error- Until I manually do service apache2 restart- then apache starts to work immediately.

Could the user_beancounters privvmpages memory issue be related to php.ini memory_limit being set to 256M ?
I have 8 apache (prefork?) processes running as www-data + 1 running as root. I'm 99% sure that's the default number of processes for apache prefork in virtualmin.
And my VPS is 1.5 GB + 1.5 GB Swap, total says 3 GB.
The fail counter is the same now as it was earlier today, so no new out of memory conditions.
But if 20 people surf the site at the same time it might run into memory issues and bump the fail counter? I could reduce the 256M to 128M for a day, and see if that helps. There's no timestamped log from when those beancounters were failcnt'ed so it might've been all from a month ago when php.ini memory_limit was set in the 64M range.

Still stumped..!

Wed, 11/28/2012 - 22:17
Chris_C

Hold up, looks like I made some headway...

~# tail /var/log/virtualmin/mydomain.com_error_log
[Wed Nov 28 22:50:18 2012] [warn] RSA server certificate wildcard CommonName (CN) `*.mydomain.com' does NOT match server name!?
[Wed Nov 28 22:50:20 2012] [crit] [client 72.14.199.71] (13)Permission denied: /home/mydomain/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable
[Wed Nov 28 22:50:36 2012] [crit] [client 173.44.37.242] (13)Permission denied: /home/mydomain/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable
~# ls -la /home/mydomain/.htaccess
ls: cannot access /home/mydomain/.htaccess: No such file or directory
~# touch /home/mydomain/.htaccess
~# ls -la /home/mydomain/.htaccess
-rw-r--r-- 1 root root 0 Nov 28 22:53 /home/mydomain/.htaccess
~# tail /var/log/virtualmin/mydomain.com_error_log
[Wed Nov 28 22:52:01 2012] [crit] [client 76.248.147.117] (13)Permission denied: /home/mydomain/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable
~# ls -la /home/mydomain/.htaccess
-rw-r--r-- 1 root root 0 Nov 28 22:53 /home/mydomain/.htaccess
~# chown mydomain:mydomain /home/mydomain/.htaccess

~# chown www-data:www-data /home/mydomain/.htaccess

~# tail /var/log/virtualmin/mydomain.com_error_log
[Wed Nov 28 22:54:48 2012] [crit] [client 180.76.5.90] (13)Permission denied: /home/mydomain/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable
~# service apache2 restart
Restarting web server: apache2 ... waiting .
~# tail /var/log/virtualmin/mydomain.com_error_log
[Wed Nov 28 22:55:23 2012] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Wed Nov 28 22:55:23 2012] [warn] RSA server certificate wildcard CommonName (CN) `*.mydomain.com' does NOT match server name!?
~# ls -la /home/mydomain/.htaccess
-rw-r--r-- 1 www-data www-data 0 Nov 28 22:53 /home/mydomain/.htaccess
~#

First time I saw errors on a non-existent /home/mydomain/.htaccess

Although, putting the file there, and changing its ownership to the most likely user and group still didn't fix the 403 error. Still had to restart apache2 manually for the site to work..

Thu, 11/29/2012 - 08:29
andreychek

Ah, you're getting permission denied error when trying to read that .htaccess file.

If you go into System Settings and run the Re-Check Config, does it notice anything unusual?

Also, just to be sure there isn't some sort of permission issue, you may want to go into Limits and Validation -> Validate Virtual Servers -> Fix Directory Permissions, you may want to try resetting the permissions for that particular domain.

However, it's odd that any permission problem would be resolved by restarting Apache...

-Eric

Thu, 11/29/2012 - 09:32
Chris_C

I do the System Settings, Re-Check Config. Nothing unusual- it says all is OK.

I do the Limits and Validation, Validate Virtual Servers, Fix Directory Permissions, reset permissions for that domain. It resets the permissions.

Reboot server. 403 error accessing any page. Tail the domain's log. "Permission denied /home/mydomain/.htaccess pcfg_openfile unable to check htaccess file, ensure it is readable."

service apache2 restart and the site comes right up.

The weird thing is... /home/mydomain isn't even supposed to be accessible from the internet.
/home/mydomain should neither have nor need an .htaccess file in it.
The TOP folder for web access in my folder heirarchy should be /home/mydomain/public_html Could THAT be the reason why apache is choking? apache is told, through some configuration mistake, to serve the site from /home/mydomain but in fact apache's blocked for security reasons since /home/mydomain is outside of the allowed correct web-accessible home folder of /home/mydomain/public_html ???

Thu, 11/29/2012 - 10:12
andreychek

Howdy,

Well, although you don't serve content from there, you can still have a .htaccess file in /home/mydomain. Apache is checking there to see if you have one... but due to some sort of permission issue, it's not able to read there (which it should have permission to see).

Until, very oddly, after you restart Apache. At which point it works fine.

You aren't by chance using something like LDAP for users, are you?

-Eric

Thu, 11/29/2012 - 22:13
Chris_C

Excellent thinking. I'd attempted to get OpenLDAP for email users working- the idea was to have Zarafa store user info in LDAP instead of the operating system unix user accounts.. but it never quite got working.

It's probably messing up somewhere with the LDAP- maybe failing to authenticate, or invalid username-

EDIT: 1. Any suggestion on how to determine whether apache is using the "ldap users and groups" vs. standard unix users? And how to make apache successfully query the ownership/permissions of /home/mydomain/.htaccess , without breaking the current dovecot imap email login authentication (which is using standard unix users - I think ?!)? 2. FYI - I checked, and /home/mydomain has permissions 750 and ownership mydomain:mydomain - is this correct? Is mydomain:nobody more appropriate? The same/less/more secure?

Thu, 11/29/2012 - 22:17
Chris_C
Fri, 11/30/2012 - 08:48
andreychek

Howdy,

LDAP is unfortunately not really my area of expertise... but what you could do is try stopping the LDAP service from running, and then run the command:

id USERNAME

And make sure that it returns user and group information for your username.

Then, try this command:

groups www-data

That groups shown there would need to include your username... that is, the www-data user needs to be added to the group of your user (and all the other Virtual Server owners on the system).

If that doesn't work with LDAP stopped, that may mean some user data is being stored in LDAP.

The link you mentioned is indeed correct in that you're seeing a permission problem... however, you shouldn't be seeing one, and you even did the "Fix Directory Permissions". So I suspect the issue is something, well, non-standard :-)

The LDAP problem is definitely an intriguing possibility!

-Eric

Fri, 11/30/2012 - 15:00
Chris_C

Eric, I believe you've found it.

~# id mydomain
uid=1002(mydomain) gid=1002(mydomain) groups=1002(mydomain)

~# service slapd stop
Stopping OpenLDAP: slapd.
~# id mydomain
id: mydomain: No such user

~# service slapd start
Starting OpenLDAP: slapd.
~# groups www-data
www-data : www-data list otherdomain mydomain

~# service slapd stop
Stopping OpenLDAP: slapd.
~# groups www-data
www-data : www-data list otherdomain

~# service slapd start
Starting OpenLDAP: slapd.
~#

The best guess I can make, at why it got into this state (mismatch between unix users and OpenLDAP users) is because I first installed virtualmin, and added the virtual server "otherdomain". Then only later I activated the LDAP Users and Groups virtualmin/webmin module , and set it to "authenticate against ldap not against local unix users" (or however it's worded exactly), and added the virtual server "mydomain" (which is the one I'm having the trouble with - apache giving this 403 error until I restart apache2 service.)

My intention was to have apache2, dovecot, zarafa, everything, authenticate off openldap - and I assumed that's what the LDAP Users and Groups module's purpose is.

Could it possibly be a race condition between apache2 and OpenLDAP ?
Suppose apache2 completes starting up BEFORE OpenLDAP is fully up, and so apache2 thinks it should just use unix users, because OpenLDAP isn't alive (yet) ??
Maybe apache2 sees a LOCAL UNIX group for "www-data" and thinks it's OK to just go ahead and only look at the unix users/groups...? Except mydomain is NOT a member of www-data on the local unix groups. Neither does it exist as a user on local unix users. And then on the apache2 restart, apache2 obviously now sees OpenLDAP up and running, and just authenticates off that, and everything works because in OpenLDAP, mydomain exists as a user, and is a member of www-data.

What do you think ?

Tue, 12/04/2012 - 21:37
Chris_C

Bump.

Just wondering whether you had a chance to take a look at this.

Is my hypothesis that it's caused by apache2 starting before openldap, correct ? In your opinion ?

How to make it so that openldap starts first, then apache2 after ?

Topic locked