Emergency Apache2 Issue

All Shared hosting on one of my nodes is all of a sudden going to the most recent virtual server created and I can't figure out what is causing this.

We recently had our ISP setup a new rdns entry but I can't imagine that is causing this..

Any ideas? I got a lot of clients not pleased : )

~Jeremy

Status: 
Closed (fixed)

Comments

Howdy -- yeah, it's not likely related to the reverse DNS.

That sort of problem typically occurs when there's an IP address mismatch -- that there's some sort of problem between the IP address of your system, and the VirtualHost line being used in the .conf files in /etc/apache2/sites-enabled/.

There's some suggestions on what to look for here:

http://www.virtualmin.com/documentation/web/troubleshooting

If you're still stuck, what you could do is post the output of the command "/sbin/ifconfig", as well as the contents of the Apache config for the website that's being displayed -- that Apache config is in /etc/apache2/sites-enabled/domain.com.conf.

Hi Eric,

Thx for the quick response. I think I figured it out. I was playing around with Cloudmin transfers of virtual servers in Virtualmin earlier and what struck me was the website showing up for all of the websites addresses was the last one I had moved. Once I realized that, I opened Virtualmin and went to the Website Options and looked for "Default website for IP address?" it was set to Yes and did not have the option to change it. I promptly deleted this virtual server and all things started working again like they should.

So here is the info that I think is pertinent and might shed light on a possible NASTY bug. I am running Cloudmin 6.4 on Ubuntu10.04 LTS The Virtualmin node I transferred from is 3.93GPL Ubuntu 10.04 LTS The Virtualmin node I transferred to is 3.93PRO Ubuntu 10.04 LTS

When I transferred the virtual server I used the default settings as shown in my attachment.

Hope you can find the bug, I am not too eager to re-produce it ATM after the firestorm I just fielded : ).

Let me know if you need to login to my systems to get a better idea of what happened.

~Jeremy

Did the virtual server you were transferring have its own private IP address, or was it just using a shared IP?

Hi Jamie,

It was using a shared IP.

Were you running Virtualmin 3.93 on both of these systems?

Yes :

The Virtualmin node I transferred from is 3.93GPL Ubuntu 10.04 LTS

The Virtualmin node I transferred to is 3.93PRO Ubuntu 10.04 LTS

Do you happen to know what the domain's <virtualhost> block in the Apache configuration contained on the new system, and how that compared to existing domains?

Hi Jaimie,

I unfortunately do not have the conf files from the Mars host when the issue arose.

I am reluctant to try the transfer again to the host where the issues were caused during business hours. I did however try to restore it to another Virtualmin 3.93 PRO Ubuntu 10.04 LTS host and did not have any issues. I will attempt the transfer late tonight or Sat and see if I can reproduce it again on the Mars virtualmin host.

Something I did find interesting, on the GPL node, the "Default website for IP address?" setting in Website Options is set to Yes for the remaining Virtual Host on Venus without the option to change it. I attached a screenshot of this, is this normal behavior? This is exactly what I was seeing on Mars after I had transferred through Cloudmin.

Here is the Apache conf file :

<VirtualHost 69.33.169.230:80>
SuexecUserGroup "#1012" "#1002"
ServerName cascadebrass.net
ServerAlias www.cascadebrass.net
ServerAlias webmail.cascadebrass.net
ServerAlias admin.cascadebrass.net
DocumentRoot /home/cascadebrass.net/public_html
ErrorLog /var/log/virtualmin/cascadebrass.net_error_log
CustomLog /var/log/virtualmin/cascadebrass.net_access_log combined
ScriptAlias /cgi-bin/ /home/cascadebrass.net/cgi-bin/
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/cascadebrass.net/public_html>
Options -Indexes +IncludesNOEXEC +FollowSymLinks +ExecCGI
allow from all
AllowOverride All
AddHandler fcgid-script .php
AddHandler fcgid-script .php5
FCGIWrapper /home/cascadebrass.net/fcgi-bin/php5.fcgi .php
FCGIWrapper /home/cascadebrass.net/fcgi-bin/php5.fcgi .php5
</Directory>
<Directory /home/cascadebrass.net/cgi-bin>
allow from all
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} =webmail.cascadebrass.net
RewriteRule ^(.*) https://cascadebrass.net:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.cascadebrass.net
RewriteRule ^(.*) https://cascadebrass.net:10000/ [R]
RemoveHandler .php
RemoveHandler .php5
IPCCommTimeout 31
</VirtualHost>

~Jeremy

If you could, try another transfer (perhaps of a test domain) and let us know that the <VirtualHost> line for the transferred and existing domains contained on the new system.

Also, I would be interested to see any NameVirtualHost lines your Apache config contains.

Hi Jamie,

Here is the ports.conf from Mars (the system that exhibited the strangeness)

# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default
# This is also true if you have upgraded from before 2.2.9-3 (i.e. from
# Debian etch). See /usr/share/doc/apache2.2-common/NEWS.Debian.gz and
# README.Debian.gz

NameVirtualHost *:80
NameVirtualHost 69.33.169.228:443
NameVirtualHost 69.33.169.228:80
Listen *:80

<IfModule mod_ssl.c>
    # If you add NameVirtualHost *:443 here, you will also have to change
    # the VirtualHost statement in /etc/apache2/sites-available/default-ssl
    # to <VirtualHost *:443>
    # Server Name Indication for SSL named virtual hosts is currently not
    # supported by MSIE on Windows XP.
Listen *:443
</IfModule>

<IfModule mod_gnutls.c>
    Listen 443
</IfModule>

This system has been around for a while and survived through a GPL to PRO migration. I do not see similar NameVirtualhost in other FRESH installs of Virtualmin PRO/GPL.

~Jeremy

Ok, Finally 11pm rolled around and I reproduced the issue transferring the same domain from Venus to Mars.

The domain is jeremiahbrock.com

Here is the Apache2 conf file after the transfer :

<VirtualHost 69.33.169.228:80>
SuexecUserGroup "#1665" "#1216"
ServerName jeremiahbrock.com
ServerAlias www.jeremiahbrock.com
ServerAlias webmail.jeremiahbrock.com
ServerAlias admin.jeremiahbrock.com
DocumentRoot /home/jeremiahbrock.com/public_html
ErrorLog /var/log/virtualmin/jeremiahbrock.com_error_log
CustomLog /var/log/virtualmin/jeremiahbrock.com_access_log combined
ScriptAlias /cgi-bin/ /home/jeremiahbrock.com/cgi-bin/
ScriptAlias /awstats/ /home/jeremiahbrock.com/cgi-bin/
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/jeremiahbrock.com/public_html>
Options -Indexes +IncludesNOEXEC +FollowSymLinks +ExecCGI
allow from all
AllowOverride All
AddHandler fcgid-script .php
AddHandler fcgid-script .php5
FCGIWrapper /home/jeremiahbrock.com/fcgi-bin/php5.fcgi .php
FCGIWrapper /home/jeremiahbrock.com/fcgi-bin/php5.fcgi .php5
</Directory>
<Directory /home/jeremiahbrock.com/cgi-bin>
allow from all
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} =webmail.jeremiahbrock.com
RewriteRule ^(.*) https://jeremiahbrock.com:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.jeremiahbrock.com
RewriteRule ^(.*) https://jeremiahbrock.com:10000/ [R]
RemoveHandler .php
RemoveHandler .php5
<Files awstats.pl>
AuthName "jeremiahbrock.com statistics"
AuthType Basic
AuthUserFile /home/jeremiahbrock.com/.awstats-htpasswd
require valid-user
</Files>
PerlRequire /etc/webmin/virtualmin-google-analytics/apachemod.pl
PerlOutputFilterHandler Virtualmin::GoogleAnalytics
IPCCommTimeout 31
</VirtualHost>

Here is an existing conf file from the Mars host

<VirtualHost *:80>
ServerName xtremeservices.net
ServerAlias *.xtremeservices.net
DocumentRoot /home/xtremeservices.net/public_html
ErrorLog /var/log/virtualmin/xtremeservices.net_error_log
CustomLog /var/log/virtualmin/xtremeservices.net_access_log combined
ScriptAlias /cgi-bin/ /home/xtremeservices.net/cgi-bin/
ScriptAlias /awstats/ /home/xtremeservices.net/cgi-bin/
<Directory /home/xtremeservices.net/public_html>
Options All -Indexes IncludesNOEXEC FollowSymLinks +ExecCGI
allow from all
Action application/x-httpd-php5 /cgi-bin/php5.cgi
AddType application/x-httpd-php5 .php5
AddType application/x-httpd-php5 .php
</Directory>
HostNameLookups off
UseCanonicalName on
<Directory "/home/xtremeservices.net/cgi-bin">
AllowOverride None
Options +ExecCGI -Includes
Order allow,deny
Allow from all
</Directory>
PerlRequire /etc/webmin/virtualmin-google-analytics/apachemod.pl
PerlOutputFilterHandler Virtualmin::GoogleAnalytics
RemoveHandler .php
RemoveHandler .php5
SuexecUserGroup "#1126" "#1033"
<Files awstats.pl>
AuthName "xtremeservices.net statistics"
AuthType Basic
AuthUserFile /home/xtremeservices.net/.awstats-htpasswd
require valid-user
</Files>
</VirtualHost>
<VirtualHost 69.33.169.228:443>
ServerName xtremeservices.net
ServerAlias *.xtremeservices.net
DocumentRoot /home/xtremeservices.net/public_html
ErrorLog /var/log/virtualmin/xtremeservices.net_error_log
CustomLog /var/log/virtualmin/xtremeservices.net_access_log combined
ScriptAlias /cgi-bin/ /home/xtremeservices.net/cgi-bin/
ScriptAlias /awstats/ /home/xtremeservices.net/cgi-bin/
<Directory /home/xtremeservices.net/public_html>
Options All -Indexes IncludesNOEXEC FollowSymLinks +ExecCGI
allow from all
Action application/x-httpd-php5 /cgi-bin/php5.cgi
AddType application/x-httpd-php5 .php5
AddType application/x-httpd-php5 .php
</Directory>
HostNameLookups off
UseCanonicalName on
<Directory "/home/xtremeservices.net/cgi-bin">
AllowOverride None
Options +ExecCGI -Includes
Order allow,deny
Allow from all
</Directory>
PerlRequire /etc/webmin/virtualmin-google-analytics/apachemod.pl
PerlOutputFilterHandler Virtualmin::GoogleAnalytics
RemoveHandler .php
RemoveHandler .php5
SuexecUserGroup "#1126" "#1033"
<Files awstats.pl>
AuthName "xtremeservices.net statistics"
AuthType Basic
AuthUserFile /home/xtremeservices.net/.awstats-htpasswd
require valid-user
</Files>
SSLEngine on
SSLCertificateFile /home/xtremeservices.net/ssl.cert
SSLCertificateKeyFile /home/xtremeservices.net/ssl.key
SSLCACertificateFile /root/CA/wildcard/STAR_xtremeservices_net.ca-bundle
</VirtualHost>

What is very interesting is that on all the other FRESH Install Virtualmin hosts, they have

<VirtualHost IPADDRESS:80>

but on Mars each conf file has *:80 except for the SSL VirtualHosts :

<VirtualHost 69.33.169.228:443>
ServerName xtremeservices.net
ServerAlias *.xtremeservices.net
DocumentRoot /home/xtremeservices.net/public_html
ErrorLog /var/log/virtualmin/xtremeservices.net_error_log
CustomLog /var/log/virtualmin/xtremeservices.net_access_log combined
ScriptAlias /cgi-bin/ /home/xtremeservices.net/cgi-bin/
ScriptAlias /awstats/ /home/xtremeservices.net/cgi-bin/
<Directory /home/xtremeservices.net/public_html>
Options All -Indexes IncludesNOEXEC FollowSymLinks +ExecCGI
allow from all
Action application/x-httpd-php5 /cgi-bin/php5.cgi
AddType application/x-httpd-php5 .php5
AddType application/x-httpd-php5 .php
</Directory>
HostNameLookups off
UseCanonicalName on
<Directory "/home/xtremeservices.net/cgi-bin">
AllowOverride None
Options +ExecCGI -Includes
Order allow,deny
Allow from all
</Directory>
PerlRequire /etc/webmin/virtualmin-google-analytics/apachemod.pl
PerlOutputFilterHandler Virtualmin::GoogleAnalytics
RemoveHandler .php
RemoveHandler .php5
SuexecUserGroup "#1126" "#1033"
<Files awstats.pl>
AuthName "xtremeservices.net statistics"
AuthType Basic
AuthUserFile /home/xtremeservices.net/.awstats-htpasswd
require valid-user
</Files>
SSLEngine on
SSLCertificateFile /home/xtremeservices.net/ssl.cert
SSLCertificateKeyFile /home/xtremeservices.net/ssl.key
SSLCACertificateFile /root/CA/wildcard/STAR_xtremeservices_net.ca-bundle
</VirtualHost>

Just for grins, I set everything back to normal by deleting the transferred host and edited the xtremeservices.net.conf to be

<VirtualHost 69.33.169.228:80>

and restarted Apache. This reproduced the same behavior with all my virtual hosts with *:80 showing up in the browser as the xtremeservices site.

I am sure there is a link to the NameVirtualHost 69.33.169.228:80 and that same IP:port being specified in the .conf file. What I am not sure of yet is which method is the CORRECT way of doing things. Is the *:80 working as designed and I need to get my other Virtualmin hosts working with NameVirtualHost settings so that I can transfer between them, or should I go the other way around.

Sorry for this long post, this has been quite an adventure today.

Well it is 1AM and I will finish researching this tomorrow.

~Jeremy

Another Update,

I discovered that my Fresh Installs of Virtualmin have NameVirtualHost specified in the Apache2.conf instead of ports.conf.

My GPL System Apache2.conf has :

# Include the virtual host configurations:
Include /etc/apache2/sites-enabled/
NameVirtualHost 69.33.169.230:80
NameVirtualHost 69.33.169.230:443

My other PRO System Apache2.conf has :

Include /etc/apache2/sites-enabled/
NameVirtualHost *:80
NameVirtualHost 69.33.169.227:80
NameVirtualHost 69.33.169.227:443

I also read up and learned that VirtualHost *:80 is the usual setting for Apache 2.2.

Where can I modify Virtualmin to control how a new virtualserver.conf file is created so I can tell it to use *:80 instead of IP:80?

Thx,

~Jeremy

Ok, it looks like that virtualhost mismatch between * and the IP is the reason - Apache's method of choosing which virtualhost to select is really complex, and what seem like small differences like this can cause odd behavior.

If you create a completely new domain on the destination system, what does it's <virtualhost> line look like?

Also, did the domain being moved have SSL enabled?

Hi Jamie,

Yes, the domain that was being moved had SSL enabled and was using a shared IP.

I created a new virtual server (On Mars) with SLL enabled on a Shared IP and it built the conf with :

<VirtualHost 69.33.169.228:80>
SuexecUserGroup "#1666" "#1217"
ServerName anothertest.me
ServerAlias www.anothertest.me
ServerAlias webmail.anothertest.me
ServerAlias admin.anothertest.me
... etc

I also created a virtual server without SSL on the Shared IP and got the same results :

<VirtualHost 69.33.169.228:80>
SuexecUserGroup "#1667" "#1218"
ServerName anothertest.me
ServerAlias www.anothertest.me
...etc

This is really interesting, when I restart Apache2 I get :

root@mars:/etc/apache2/sites-enabled# apache2ctl graceful
[Sat Jul 28 15:35:15 2012] [warn] NameVirtualHost 69.33.169.228:80 has no VirtualHosts

~Jeremy

Good evening Jamie,

I played with things a bit more today and went through my Apache Cookbook 2nd Edition.

I decided to go ahead and use the defaults from a Fresh Install as my basis for moving forward on Mars

Here is what my ports.conf file looks like now :

NameVirtualHost 69.33.169.228:80
NameVirtualHost 69.33.169.228:443
listen 80
listen 443

And I updated all of my VirtualHost *:80 to IP:80 in sites-enabled/

Well anyways, everything is back to normal, thx for helping point me in the right directions.

~Jeremy

This can be marked as Fixed/Closed.