Preserve Awstats Historical Data When Migrating from cPanel / Where is Awstats When Logged In as Virtual Server Admin?

I'm noticing that none of the historical Awstats data seem to be available in Virtualmin after migration from cPanel. I was surprised to notice that because it does seem to be part of the migration.

There's also a gap in Webalizer data for the two months immediately prior to migration.

Also, when logged in as site admin (not root), Awstats is not available under Logs and Reports. It is when I am logged in as root. The Awstats feature is enabled in Enabled Features at both the Account Plan and Virtual Server configurations.

I'm still testing Virtualmin on a server hosting only my own sites (although it is online as a production server). I do need to know how to make these features available to users before switching to Virtualmin on client sites, however. Awstats is one of the few things the clients actually seem to care about.

Thanks,

Richard

Status: 
Active

Comments

RJM Web Design's picture
Submitted by RJM Web Design on Tue, 07/23/2019 - 10:35 Pro Licensee

Actually, some of the Awstats data is there, but there are large gaps that aren't the same from site to site. So one site may be missing more than two years of data, but another only five months.

In all cases, the gaps are for the months immediately prior to migration; but the time span during which they extend into the past are different for each virtual server. The gaps also begin during random days of the month rather than on the same days, and the months and years when the gaps start are different.

In a few cases, the bandwidth data is present (but incorrect -- much too low), but none of the detailed data is there. Very strange.

RJM Web Design's picture
Submitted by RJM Web Design on Tue, 07/23/2019 - 11:59 Pro Licensee

Title: Preserver Awstats Historical Data When Migrating from cPanel / Where is Awstats When Logged In as Virtual Server Admin? ยป Preserve Awstats Historical Data When Migrating from cPanel / Where is Awstats When Logged In as Virtual Server Admin?
RJM Web Design's picture
Submitted by RJM Web Design on Tue, 07/23/2019 - 12:02 Pro Licensee

Since it's probably relevant, I made the backups in cPanel with SCP to the Virtualmin server as the destination. Maybe I should try making it locally in /backup on the cPanel server and manually SCP'ing over to the Virtualmin server. I still have a few sites I haven't moved yet. Maybe something's getting hosed in the archive creation and transfer process.

RJM Web Design's picture
Submitted by RJM Web Design on Thu, 07/25/2019 - 08:37 Pro Licensee

I migrated four more accounts last night. For one, all the historical data was preserved. For another, none of it was preserved (zero hits since the account was established). The other two lacked recent months' data, but preserved older data. So that's still a problem, although I don't know on whose part. It's not a deal-killer, but it would be nice not to have it.

On the other issue, all that has to be done to enable the Awstats link in user accounts is to enable it in the template.

Would it be possible to get one of the backup files that didn't migrate properly?

RJM Web Design's picture
Submitted by RJM Web Design on Fri, 07/26/2019 - 06:44 Pro Licensee

Hi Jamie,

Do you have a way for me to get it to you securely? Some of the code includes MySQL credentials used as part of a malicious activity tracking system that gathers information from multiple sites and compiles it into blocklists, so I don't want to just post a public link. The file is ~108MB.

Also, I think I've narrowed down the problem a bit. Awstats on cPanel treats SSL and non-SSL versions of sites separately, and I was unable to open some of the logs for the SSL versions after extracting the archive unless I copied the files to someplace outside their containing folders. If I tried "Open With" while the file was in its containing folder (using Win10 Pro), nothing happened. No options to open the file were offered, and no request to select a program to open it was made by Windows. But if I copied the file to the desktop, I could open it with any text editor using "Open With."

Specifically, I was able to extract

\homedir\logs[domain.tld]-ssl_log-Jun-2019.gz

which extracts a folder named

\homedir\logs[domain.tld]-ssl_log-Jun-2019

containing a single file of the same name as the folder. I cannot open that file while it's in the folder. Windows neither errors out nor asks me how I want to open it. It just does nothing. But if I copy it to the desktop, I can open it using any text editor.

So my hunch is that part of the problem may stem from the fact that cPanel splits the Awstats data into SSL and non-SSL versions. That's as far as I got with this before I had other things to do.

Please let me know if there's some way I can securely send you the file, and I'll be happy to do so.

Yes, that could be the cause.

Could you upload the file to google drive, and share it with my account jcameron@webmin.com ?

RJM Web Design's picture
Submitted by RJM Web Design on Sun, 07/28/2019 - 20:20 Pro Licensee

Done, and thanks for your help.

Richard

RJM Web Design's picture
Submitted by RJM Web Design on Tue, 07/30/2019 - 10:12 Pro Licensee

I think the workaround for this post-migration, assuming that the site forced all traffic to SSL, would be to replace the

awstats[date].[domain.tld].txt

files for the missing months in

/home/[user]/awstats/

on the Virtualmin server with those in the cPanel migration subdirectory

backup-[date]_[username]/homedir/tmp/awstats/ssl/

Of course, this will only work (if it works at all) if all traffic were forced to SSL pre-migration, resulting in zero traffic in the files because they were for the non-SSL domain. One would just look for the empty months in Awstats and replace only those files.

Also, if the site were switched to SSL mid-month, the data for the time prior to the switch for that month would be lost; but that would be better than all the traffic data being lost.

I'll test it out later on. I'm a bit behind at the moment. But I think it will work. How to merge the historical records during the migration itself is a project for someone much smarter than I am.

RJM Web Design's picture
Submitted by RJM Web Design on Tue, 07/30/2019 - 13:24 Pro Licensee

Okay, my last idea did in fact work as a post-migration solution to recover the data for the months for which the stats were missing from Awstats due to cPanel's practice of separating the reports into SSL- and non-SSL versions. I just tried it on a different domain than the one I sent you, Jamie, that was missing data for the following months:

April through December of 2017

All of 2018

January through June of 2019.

Those months correspond to the time during which I'd forced SSL on the site, prior to migrating it to Virtualmin.

This is what I did to recover the data, step by step.

  1. SCP'd into the old server (I still have backup files there).

  2. Extracted the most recent backup.

  3. Downloaded /backup-[date]_[username]/homedir/tmp/awstats/ssl/ to my computer.

  4. Picked the .txt files for the months with missing data.

  5. SCP'd to the new (Virtualmin) server and uploaded those .txt files to /home/[user]/awstats/.

  6. Updated Awstats.

Voila. There were the records for the missing months.

I have AllowToUpdateStatsFromBrowser enabled for the domain in question, so I updated it from the Awstats report itself.

The only odd thing was that each year had to be separately updated for the recovered stats to render. In other words, I had to be viewing the stats for that year (the month didn't seem to matter), and run the update while viewing that year. I don't know if this would be the case had I run the update as root.

If nothing else, this workaround confirms that the problem with the stats disappearing during migration is due to the way cPanel separates SSL- from non-SSL Awstats data for the same domain. It also provides a post-migration workaround to recover those missing stats, except for part of the month of migration, and part of the first month when the site switched to SSL.

I guess someone smarter than I am could figure out a way to merge the files for the SSL- and non-SSL versions. But until then, I hope this helps as a post-migration workaround.