Latest Apache2 Update with Virtualmin failed and shot a hole in SUEXEC

Reading of some security issues I thought it would be a good idea to update packages with virtualmin. All packages failed to install. And since then my apache2 wouldn't start up any longer, bad thing on a live system. I could fix this manually, but now it does seem to lack SUEXEC support, which renders my wiki sites useless. Tried apt-get update, which complains about NO_PUBKEY. Thought about dist-upgrade which would give me apache2-suexec-custom, but when I run, it says "The following packages cannot be authenticated!" webmin ... and when I say "Install without verification" it stops while preconfiguring webmin packages with "find: `.': Bad address E: Sub-process /usr/bin/dpkg returned an error code (1)" Here is where I am stuck now.

Assume I gotta fix that authentication thing first of all...

Status: 
Closed (fixed)

Comments

Howdy -- it almost sounds as if there may be a third-party repository enabled that's causing some trouble.

Can you post the contents of your /etc/apt/sources.list file? Thanks!

Thanks for the quick reply, here's sources.list:

deb http://194.97.2.69/debian/ stable main contrib
# Uncomment the deb-src line if you want 'apt-get source'
# to work with most packages.
# deb-src http://194.97.2.69/debian/ stable main contrib
# uncommenting the following line will enable security updates
  deb http://security.debian.org/ stable/updates main contrib

# deb http://software.virtualmin.com/gpl/debian/ virtualmin-lenny main
# deb http://software.virtualmin.com/gpl/debian/ virtualmin-universal main

  deb http://000000:XXXXXXXXX@software.virtualmin.com/debian/ virtualmin-lenny main
  deb http://000000:XXXXXXXXX@software.virtualmin.com/debian/ virtualmin-universal main

If you try to start Apache with a command like :

/etc/init.d/apache2 start

what gets logged to /var/log/apache2/error.log ?

Right now it seems to operate normally, starting suexec wrapper, I'll post also the log lines from before installing apache2-suexec-custom:

Warning: SuexecUserGroup directive requires SUEXEC wrapper.
Warning: SuexecUserGroup directive requires SUEXEC wrapper.
Warning: SuexecUserGroup directive requires SUEXEC wrapper.
Warning: SuexecUserGroup directive requires SUEXEC wrapper.
Warning: SuexecUserGroup directive requires SUEXEC wrapper.
[Thu Aug 27 17:47:48 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Thu Aug 27 17:47:48 2009] [notice] Digest: generating secret for digest authentication ...
[Thu Aug 27 17:47:49 2009] [notice] Digest: done
[Thu Aug 27 17:47:50 2009] [notice] Apache/2.2.9 (Debian) DAV/2 SVN/1.4.2 mod_ruby/1.2.6 Ruby/1.8.5(2006-08-25) mod_ssl/2.2.9 OpenSSL/0.9.8g configured -- resuming normal operations
[Thu Aug 27 19:00:27 2009] [notice] caught SIGTERM, shutting down   
/etc/init.d/apache2 start
[Thu Aug 27 19:04:49 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Thu Aug 27 19:04:49 2009] [notice] Digest: generating secret for digest authentication ...
[Thu Aug 27 19:04:49 2009] [notice] Digest: done
[Thu Aug 27 19:04:50 2009] [notice] Apache/2.2.9 (Debian) DAV/2 SVN/1.4.2 mod_ruby/1.2.6 Ruby/1.8.5(2006-08-25) mod_ssl/2.2.9 OpenSSL/0.9.8g configured -- resuming normal operations

But still my sites configuration seems to be broken, cgi-bins for instance do not work for subdomains etc. Also remember I was stuck in the middle of dist-upgrade, so currently perl, dovecot et al are not operating properly. Webmin and Virtualmin are not processed by perl.

From the silence, must I assume it would be best to set up everything freshly, new debian or ubuntu, running install.sh? What can I do to avoid such a really bad situation on a live system, i.e. clicking update in virtualmin, ending up with a broken apache et al? Is debian etch a good choice for a fresh start? Or would a dist-upgrade before install.sh do better?

Just to make what happened more clear:

Clicking to perform all updates in virtualmin

Gives "Install failed" with all updates, after Authentication error

Apache is stopped

Apache restart fails, giving suexec wrapper error

Looking at the file system a bunch of *.dpkg files clutter the conf section

Setting old *.conf files in place again

Apache starts, still complaining about missing suexec wrapper

Virtual servers using subdomains not resolved, only domains show up

... and so on, until I got stuck with this halfhearted trial to dist-upgrade, just to get things, at least apache2, fully updated and fresh this way, which didn't help with what update had broken before (lost configuration etc.) - surely my bad.

For now subdomains are not being served, mail doesn't authenticate and so on.

Regarding Webmin/Virtualmin it brought me here:

HTTP/1.0 500 Perl execution failed Server: MiniServ/0.01 Date: Fri, 28 Aug 2009 07:37:51 GMT Content-type: text/html Connection: close
Error - Perl execution failed

Undefined subroutine &main::get_left_frame_width called at /usr/share/webmin/virtual-server-theme/index.cgi line 83.

So right now some advice would help:

Is a new install recommended or can I fix things?

If the latter is the case, I would like to use debian 5.0 plus virtualmin, which procedure would you recommend?

How can I avoid the always reoccuring gpg related authorzation troubles with sources.list?

Can I backup Maildirs and set them back into place on a fresh system?

How can I avoid this sort of update troubles in the future?

Sorry for the delay --

It sounds like something went awry during the initial update process. What you saw shouldn't occur and isn't typical behavior; I'm unfortunately not sure what to tell you differently next time, as I'm not sure what went awry. The errors your seeing are typical for when there's a third party repository setup in the sources.list file, but that doesn't appear to be the case here.

That said, you might consider choosing an alternate Debian mirror just to be ultra-sure that there isn't any trouble at the one you have listed.

Moving forward -- it sounds like thing to do in this case is to finish the dist-upgrade, and work from there.

To do that, I would review this howto on upgrading a Virtualmin server from Debian 4 to Debian 5:

http://www.virtualmin.com/documentation/id,upgrading_debian_etch_to_lenn...

Once you've gone through that, let us know if things are working better for you.

If they aren't, we'll figure out how to proceed from there :-)

http://www.virtualmin.com/documentation/id,upgrading_debian_etch_to_lenn...

actually this address is where I came from before I started this issue. Today I did - with a new sources.list - clean and autoclean to only get freshly loaded packages and again ended up with this:

Extracting templates from packages: 100%
Preconfiguring packages ...
find: `.': Bad address
E: Sub-process /usr/bin/dpkg returned an error code (1)

I don't seem to get any further than this.

Howdy -- can you paste in the new contents of your sources.list file?

Also, try running this command:

dpkg --configure -a

As that may assist in cleaning things up.

Sorry, I forgot to mention this, I have already been using dpkg --configure a, something I always do when some update with virtualmin wouldn't complete.

Here's sources.list

deb http://ftp.debian.org/debian/ lenny main contrib non-free
deb-src http://ftp.debian.org/debian/ lenny main contrib non-free
deb http://security.debian.org/ lenny/updates main contrib non-free

# deb http://software.virtualmin.com/gpl/debian/ virtualmin-lenny main
# deb http://software.virtualmin.com/gpl/debian/ virtualmin-universal main

deb http://xxxxxx:000000000@software.virtualmin.com/debian/ virtualmin-lenny main
deb http://xxxxxx:000000000@software.virtualmin.com/debian/ virtualmin-universal main

OK, I think going through the same over and over will get me nowhere. Three days with half of the services unavailable should be enough. I will have my ISP set up a fresh debian etch.

The one important remaining question is should I dist-upgrade to lenny before doing install.sh or should I do install.sh before doing dist-upgrade?

I do think all other remaining questions, like backing up configurations, mail, scripts, content and such are OS related, so no need to care.

Joe's picture
Submitted by Joe on Sat, 08/29/2009 - 06:11 Pro Licensee

It's always best to start with the OS you'll be running, rather than installing and then making major changes to the OS (in the case of etch to lenny, there are actually several changes to the way Virtualmin is configured and works, so it's even more dramatic than usual). Does your host not offer a lenny install? It'd be best of all to start from there.

So, yes, get a known good lenny install first, and then run install.sh

My provider seems to be a bit lame with putting up images for the most recent OS, so I gotta go from etch to lenny first and run install.sh last.

I'll do a report if something weired happens with the virtualmin part.

You seem to get up quite early in Mountain View - on a saturday morning ;-)

Thanks for your support Boris

Ok, here is what happened after setting up debian etch, doing dist-upgrade to lenny and such:

  • running dist-upgrade - went through without a glitch

  • getting /etc/host and /etc/hostname right

  • running install.sh - went through as well

  • creating virtual host in virtualmin - fails because /home has no disk quota set (looking after /etc/fstab shows that it was touched by install.sh, but doesn't have the correct entries)

  • correcting /etc/fstab from

LABEL=home  /home  xfs  grpquota,usrquota,rw  0  2

to

LABEL=home  /home  xfs  grpquota,usrquota,rw  1  1
  • reboot

  • creating virtual host - works

  • sarting webmin file manager - fails with

Failed to get language list : javax.net.ssl.SSLHandshakeException : java.security.cert.CertificateException: Java couldn't trust Server.

Didn't fool around with certificates, this is what happens when doing it from scratch.

  • trying to let webmin use the certificate from the created virtual host - with no avail

That is all as far as I can see. I assume the quota entry could be easily corrected, the cert/File Manager issue poses a severe problem, since I need to use it regularily, and it points to something going quite wrong.

Up to now all the rest seems to be fine.

BTW Virtualmin's behaviour of not cleaning up after something had failed, like with creating a virtual host, i.e. not deleting user and whatever was created in such process can be very much confusing :-) - and is something what brought me here in the first place with a broken apache2 update.

And did I mention that virtualmin is however a superb tool, second to none :-))

The file manager issue is actually a problem with Java on your client PC recognizing the self-signed certificate that Virtualmin provides by default. There is a setting somewhere in the Java control panel to change this ..

The file manager issue is actually a problem with Java on your client PC recognizing the self-signed certificate that Virtualmin provides by default. There is a setting somewhere in the Java control panel to change this ..

The file manager issue is actually a problem with Java on your client PC recognizing the self-signed certificate that Virtualmin provides by default. There is a setting somewhere in the Java control panel to change this ..

The file manager issue is actually a problem with Java on your client PC recognizing the self-signed certificate that Virtualmin provides by default. There is a setting somewhere in the Java control panel to change this ..

The file manager issue is actually a problem with Java on your client PC recognizing the self-signed certificate that Virtualmin provides by default. There is a setting somewhere in the Java control panel to change this ..

Nope, checked for it and tested on different machines. On all machines (Mac OS X and Windows XP) File Manager applet did work before, and File Manager applet still works on all machines with an older Server with only webmin installed.

Did you perhaps have a real SSL cert setup for Virtualmin's webserver on the old system? This can be done if you have an SSL domain with the same name as what you use to access Virtualmin on port 10000 ..

The old system made use of the self generated certificate from the install procedure, with no problems. I have ordered a trusted one today, might take until tomorrow to get it signed.

Meanwhile another problem is arising where SUEXEC might be involved, a bunch of apache SSIs which make use of XBitHack aren't being processed, but they were on the old sytem, poorly enough I don't remember what was the quirk with the old system's configuration.

Right now I can't see any suexec started entry in the error log (with debug mode set) after stop-starting apache. Doesn't have to mean anything, I think apache would complain anyway if there was no wrapper for suexec, since it is called for in the virtual host conf files.

I'll go after the SSi trouble first to get the static sites going. Right now using midnight commander over ssh - maybe I'll get used to it ;-) Will report tomorrow about the cert thingy.

Apache SSI doesn't invoke suexec at all, by the way ..

Hmm... http://httpd.apache.org/docs/2.0/suexec.html

Calling a script to include or calling a system command to include, according to the suexec documentation, is bringing suexec into play. That's where I use XBitHack, and thats where I'm stuck. Simple file includes work for sure with 'Options +Include'.

If you include a command's output from SSI, then yes suexec will be used .. but for just regular includes or other SSI directives, I don't think it is.

Yes, and that is what I need to do.

But I find that I have serious permission problems going on with all the system that, with apache, go something like this, no matter what I try:

[Tue Sep 01 14:40:04 2009] [error] [client 212.227.165.17] unable to include "inc/plink.txt" in parsed file /home/malenka-photographie/public_html/index.html
[Tue Sep 01 14:40:04 2009] [crit] [client 212.227.165.17] (13)Permission denied: /home/malenka-photographie/public_html/inc/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable

Note: There is no .htaccess that apache should read. This error comes with all virtual hosts, no matter if there is .htaccess files or not, and I also get a 403 with simple static pages on one of them. suexec users are configured correctly, apache user is member of all virtual hosts groups.

Next thing is SMTP authentication is not working.

Now I suspect the change I needed to make to /etc/fstabs to get virtual hosts installed in the first place might have broken permission settings in /home. Here's what it is set to:

LABEL=/ /       ext3    defaults        0       1                                                            
/dev/hda2       none    swap    sw                                                                           
LABEL=usr       /usr    xfs     defaults        0       2                                                    
LABEL=var       /var    xfs     defaults        0       2                                                    
LABEL=home       /home  xfs     defaults,grpquota,usrquota  0    2                                           
proc            /proc   proc    defaults        0       0                                                    
none    /tmp    tmpfs   defaults        0       0                                                            

where install.sh had left me with 'defaults' for /home, without usrquota and without grpquota.

Could it be that my installation is somehow incomplete? I remember the installation process ended with something like 'postfix blahblah ...succeeded' and sent me back to the prompt.

If so I'd prefer to set it all up freshly again, sticking to etch, since it was working fine until that update came only to eat up my precious time.

If you can, a re-install onto Etch will work fine. Upgrading to Lenny is nice, but not necessarily mandatory..

Lenny doesn't seem to have posed the problem on this.

Something else seems to go wrong. Installing onto etch left me with the same issues, and brought me additionally some strange things like, apache2, dovecot, bind and others now appear in the list of 'un-used modules' inside webmin.

  • DNS seems not to work correctly
  • Besides this, the /etc/fstab remained the same (needed to reboot, mount -a wouldn't do)
  • File manager still not accepted (certificate not accepted)
  • suexec trouble with SSI system command output

The list goes on.

I gotta think about this more thouroughly and will now again have my ISP set up a fresh debian etch to start from scratch.

Three questions before I start again:

  • Does the fact that I put ssh port to 8888 and disallow root login interfere with any install.sh operations?
  • What would be a proper /etc/hosts and /etc/hostname entry when the main domain on which I set up is nospitting.net?
  • What would be the proper setup with my ISP's nameservers, should they precede the ones I want to have set up by virtualmin (mine should be called something like ns1.nospitting.net and ns2.nospitting.net)?

It appears to me that my troubles lie somewhere inbetween these three questions. Any support with my next try would be very much appreciated :-)

Luckily most of the stuff on the machine is about our art production cooperative, so no cusomers are involved directly, but this story is now becoming a sad joke, since I am the person that has setup such systems by hand before, (well, ok, always with a little help by webmin) hmm...

For the record:

  • The initial, apache2 update trouble seems to be related to the fact that while having etch installed virtualmin's update procedure pulled a lenny update, this being related to the /etc/apt/sources.list calling "stable" instead of "etch". Don't know what virtualmin can do about a users sources.list entries being dangerously formed this way.
  • Second, trouble reinstalling system, reinstalling virtualmin, seems to be related to a bad /etc/hosts and /etc/hostname configuration, this being related to my ISP resetting network configuration to dhcp with every restart.
  • Third, doing my certificate by hand, and letting it sign by my "own" CA, convinced Java applet to accept File Manager, while I think that virtualmin is doing basically the same, don't know wherin the diffrence lies.
  • One or both of the latter seem to have caused the suexec trouble within the first two attempts of reinstalling from scratch.

Have now debian lenny, with virtualmin on top, all code working again, all fine.

One question remains: Does the virtualmin backup make use of "tar --preserve-permissions"? I had to restore everything from a Mac OS X backup which gave me back quite a mess permissions wise.

Thanks for the support...

No, it doesn't use the -p flag. I've found that this doesn't seem to matter, at least on my test systems .. tar restores the original permissions regardless.