webalizer/awstat issue

Submitted by mmoxnes on Sat, 09/11/2010 - 03:35 Pro Licensee

hi I have a installation on a centos 5.4 server, and after resore (migrating from Suse to centos) all old domains dont get any output from webalizer or awstats

it works for new domains... so I guess this is related to the old bug I found mentioned, but I didnt find the solution for it (the logfiles under /var/log/virtualmin is owned by domainOwner and has apache as group and when I try to tun the cron command it say "no output"

Status: 
Active

Comments

Submitted by JamieCameron on Sat, 09/11/2010 - 11:09

If you SSH in as root and run crontab -l , do you see cron jobs that run the command /etc/webmin/virtual-server/webalizer.pl with each domain's log file as a parameter?

If so, try running one of those commands and let us know what it outputs.

Submitted by mmoxnes on Sat, 09/11/2010 - 18:22 Pro Licensee

yes, the crontab jobs are there

output from command: Logfile /var/log/virtualmin/_acces config file does not exist at /usr/libexec/webmin/webalizer/webalizer.pl line 8.

Submitted by mmoxnes on Sat, 09/11/2010 - 18:23 Pro Licensee

there should be a domain name before the _access

Submitted by JamieCameron on Sat, 09/11/2010 - 21:10

Do you mean the domain name was missing in the error message, or it just wasn't included in your posting above?

Submitted by mmoxnes on Sat, 09/11/2010 - 21:28 Pro Licensee

the domain name not included in my posting (since its the same error for all domains)

Submitted by JamieCameron on Sun, 09/12/2010 - 00:22

Looks like virtualmin didn't properly handle the switch from logs between ~/logs and /var/log/virtualmin on the two systems.

The fix is to SSH in as root and run the following commands :

virtualmin disable-feature --all-domains --webalizer
virtualmin enable-feature --all-domains --webalizer

Submitted by JamieCameron on Sun, 09/12/2010 - 00:22

Looks like virtualmin didn't properly handle the switch from logs between ~/logs and /var/log/virtualmin on the two systems.

The fix is to SSH in as root and run the following commands :

virtualmin disable-feature --all-domains --webalizer
virtualmin enable-feature --all-domains --webalizer

Submitted by mmoxnes on Sun, 09/12/2010 - 01:42 Pro Licensee

tried that,

still same output from the webalizer.pl

can it be that some of the files have wrong ownership, wrong permission?

Submitted by mmoxnes on Sun, 09/12/2010 - 01:43 Pro Licensee

tried that,

still same output from the webalizer.pl

can it be that some of the files have wrong ownership, wrong permission?

Submitted by mmoxnes on Sun, 09/12/2010 - 01:50 Pro Licensee

tried that,

still same output from the webalizer.pl

can it be that some of the files have wrong ownership, wrong permission?

Submitted by mmoxnes on Sun, 09/12/2010 - 01:52 Pro Licensee

hi, tried that , still the same error

can it be wrong ownership ??or something?

Submitted by mmoxnes on Sun, 09/12/2010 - 01:53 Pro Licensee

sorry for multiple posting, but I got a server 500 error...

Submitted by JamieCameron on Sun, 09/12/2010 - 11:29

So after doing the disable and then enable, try running crontab -l | grep webalizer again and see if the command has changed. And if so, try running the new command..

Submitted by mmoxnes on Mon, 09/13/2010 - 19:03 Pro Licensee

tried running the new command, no error, but no update of the statistic page either

Submitted by lvsys on Sat, 09/18/2010 - 00:54

I am having the same issue. But I found what the problem is: the apache log files are restored under the wrong user ids and user names.

To find that out, I did what Jamie suggested: going into the cron. Then I followed the trace to the awstats.pl wrapper under /usr/share/webmin/virtualmin-awstats and replaced the output of NULL to /home/myfolder/output.log to see what it says for a restored domain (for which awstats is failing like all other restored domains on the machine)

So then I re-run awstats.pl for the domain that I restored from a ubuntu 8 box. The log file is pretty clear: gzip cannot unzip the access files in /var/log/virtualmin... because guess what....... the restored domain names have their log files belonging to the wrong users!!

Here's an excerpt of my /var/log/virtualmin:

-rw-rw---- 1 lbctanning     www-data    13643 2010-09-12 06:25 tcpartners.com_access_log.1.gz
-rw-rw---- 1 lbctanning     www-data    12040 2010-09-05 06:25 tcpartners.com_access_log.2.gz
-rw-rw---- 1 lbctanning     www-data    11360 2010-08-29 06:45 tcpartners.com_access_log.3.gz
-rw-rw---- 1 lbctanning     www-data    14246 2010-08-22 06:45 tcpartners.com_access_log.4.gz
-rw-rw---- 1 lbctanning     www-data    11156 2010-08-15 06:45 tcpartners.com_access_log.5.gz
-rw-rw---- 1 lbctanning     www-data     3786 2010-09-17 10:51 tcpartners.com_error_log
-rw-rw---- 1 lbctanning     www-data       20 2010-09-05 06:28 tcpartners.com_error_log.1.gz
-rw-rw---- 1 lbctanning     www-data       20 2010-08-29 06:49 tcpartners.com_error_log.2.gz
-rw-rw---- 1 lbctanning     www-data       20 2010-08-22 06:49 tcpartners.com_error_log.3.gz
-rw-rw---- 1 lbctanning     www-data       20 2010-08-15 06:46 tcpartners.com_error_log.4.gz
-rw-rw---- 1 lbctanning     www-data       20 2010-08-08 06:44 tcpartners.com_error_log.5.gz
-rw-rw---- 1 lvcity              www-data     6664 2010-09-17 22:35 lvcity.com_access_log
-rw-rw---- 1 lvcity              www-data    25296 2010-09-17 22:35 lvcity.com_error_log
-rw-rw---- 1 mynewhouse            www-data  8215703 2010-09-17 22:35 mynewhouse.com_access_log

Why do tcpartners.com_access_log belong to lbctanning ??????? Seems to me the restore process is not working properly.

All the restored domains are actually mismatched to the wrong log files. But all new virtual servers - as lvcity and mynewhouse can attest in the above excerpt - are working properly.

How do we correct this?

Submitted by JamieCameron on Sat, 09/18/2010 - 01:31

lvsys - do tcpartners and lbctanning perhaps have the same UID in the /etc/passwd file?

You could check by running :

grep -e tcpartners: -e lbctanning lbctanning: /etc/passwd

and looking at the 3rd column.

Submitted by lvsys on Sat, 09/18/2010 - 01:37

No. They don't - that's the first thing I checked. Everyone is very different.

But, here's what I just did that worked:

I used chown to restore the correct owners of the log files in /var/log/virtualmin, and now /etc/webmin/virtualmin-awstats/awstats.pl is working properly

Hope you find this useful. But I can tell you for sure I just made a big note in my backup procedure to restore proper ownership to these files in the future, should something happen.

I have the same problem on another box.

Who created the log files? --- I checked the apache config and the UID for apache is correct. I am wondering that because the log files are writable by the group www-data, we don't see the problem of apache having problems to log to the files....

I am telling you, this must have happened during the restore process. Do you restore the log files? And do you check that they are restored to the proper users??

Submitted by lvsys on Sat, 09/18/2010 - 01:50

Yep. That's your issue. The log files are restored under the wrong user ID. Then logrotate keeps rotating the log under the wrong user id, hence propagating the issue forever.

So here you go. The fix of the awstat failing to log restored virtual servers is to actually restore ownership of the /var/log/virtualmin files to their rightful owners, including the gziped files. Then logrotate will keep rotating the files under the owner of the xxx.log file ; and all will be better.

I cannot tell you how good I feel that I no longer have to manually run awstat (and forget about it)

That was an interesting one.

Cheers.

Submitted by JamieCameron on Sat, 09/18/2010 - 12:02

This still looks like a virtualmin bug, in that the logs were restored with the wrong permissions.

How long ago (or on which virtualmin version) did you restore those domains?

Submitted by lvsys on Sat, 09/18/2010 - 12:07

I restored on June 27 from the latest virtuamin at the time on ubuntu 8 to the latest virtualmin on ubuntu 10. I remember precisely making sure that both virtualmin were up to date on both machines - but I did notice that the Awstats config file where mentioning different awstats versions but had nothing different in them. Not sure if that's related, or can help.

But yeah, it was a migration from ubuntu 8 to ubuntu 10

Submitted by JamieCameron on Sat, 09/18/2010 - 12:12

lvsys - when you restored, was it via the web UI or the command line? If via the web, did you happen to change the "Re-allocate UIDs and GID?" setting?

Submitted by lvsys on Sat, 09/18/2010 - 12:16

via the Gui, but honestly I don't remember the UID/GID stuff... I must have left all defaults on the restore option. Everything else has been restored fine (maybe to the exception of some arcane php settings) - unfortunately, I don't have a backup from the ubuntu 8 times to help you out

Submitted by JamieCameron on Sun, 09/19/2010 - 19:02

Oh well .. unfortunately working out the original cause of this problem is going to be hard then, as I can't re-produce it on any of my test systems.

Submitted by lvsys on Sun, 09/19/2010 - 23:00

yeah, well, it's not like it's a difficult issue to resolve either since no programming is involved. It's just something for us to watch out. no big deal.