Doing a periodic check on an idle domain name hosted on one of my servers, I encountered an unusually high bandwidth usage, reported by Virtualmin.
Bandwidth Usage (per Month)
03/2010 58.14 GB
02/2010 334.63 kB
01/2010 528.34 kB
12/2009 163.59 kB
11/2009 286.87 kB
10/2009 263.09 kB
As you can see, this domain name never had a traffic of more than 0.5MB per month, when suddenly it went up to 58GB in March. Here's the daily analysis for March 2010:
Bandwidth Usage (per Day)
24/Mar/2010 784 bytes
23/Mar/2010 1.01 kB
22/Mar/2010 16.08 MB
21/Mar/2010 345 bytes
20/Mar/2010 0 bytes
19/Mar/2010 11 bytes
18/Mar/2010 334 bytes
17/Mar/2010 8.05 kB
16/Mar/2010 3.42 kB
15/Mar/2010 5.99 kB
14/Mar/2010 5.10 kB
13/Mar/2010 20.74 kB
12/Mar/2010 11.08 kB
11/Mar/2010 5.69 kB
10/Mar/2010 9.29 kB
09/Mar/2010 5.42 kB
08/Mar/2010 3.31 kB
07/Mar/2010 27.58 kB
06/Mar/2010 5.44 kB
05/Mar/2010 10.77 kB
04/Mar/2010 26.72 GB
03/Mar/2010 349.38 MB
02/Mar/2010 4.60 kB
01/Mar/2010 31.07 GB
I was of course alerted by the 31GB on the 1st and the 26.7GB on the 4th. I did a thorough check through the apache logs of those 2 days and the bandwidth usage of the rest of the domains hosted on the server. None of them had any weird patterns like this. The next thing was to check how many bytes went through the server's nic and voila, the number of bytes was 57.7GB smaller than all the domains' bandwidth usage reported by Virtualmin. I was then led straight to the problem, which is:
On the 1st and 4th, I had uploaded 2 compressed files (~400MB each), which I then asked a client to download using a download manager (Flashget - free), so he could resume if his connection was lost. Each download was also split in 10 parts (http requests). That resulted in many 206 requests and even though the download time for each file was less than 30 minutes, these requests were translated as 57.7GB of traffic in total. So I think the bandwidth usage module might have a bug in counting 206 requests as 200 ones, resulting in massive erroneous bandwidth usage.
You should be able to reproduce this bug by uploading a file and then downloading it using any download manager which splits the file in parts, and then checking the bandwidth usage.
I can provide more info (logins, etc) and raw access logs in private if needed.
Server Specs
Operating system: Debian Linux 5.0
Webmin version: 1.500
Virtualmin version: 3.76.gpl GPL
Kernel and CPU: Linux 2.6.26-2-amd64 on x86_64
Comments
Submitted by JamieCameron on Wed, 03/24/2010 - 15:44 Comment #1
Would it be possible for you to attach the raw Apache logs for the problem days to this bug report, or email them to me at jcameron@virtualmin.com ?
Also, when you look at the bandwidth usage graph by day, it shows a breakdown by web/mail/FTP data transferred. Is the excessive traffic classified as web, or does it appear as mail or FTP?
Submitted by blehhh on Fri, 03/26/2010 - 20:25 Comment #2
Jamie I emailed you two screenshots from the domain name bandwidth usage and the apache log file for the 3 problematic days.
In my bug report, I mentioned this happening on 2 days, but on the 24th I repeated the same procedure, only to find out that this bug added some 50GB more to the account's total traffic. As you will see, I speculate that the problem is with the way the script counts the 206 (partial requests).
Since both the NIC report in bytes, and the webhost's traffic charts do not report this traffic, it must be the bandwidth usage module of virtualmin.
All of the excessive traffic is classified as web (http).
Submitted by JamieCameron on Sat, 03/27/2010 - 13:58 Comment #3
Thanks for those logs - I see the cause now. In the access_log, the partial requests with the 206 status code have the full file size as the size, which is not really correct. I will have Virtualmin ignore those in the next release (3.78).
Submitted by blehhh on Mon, 03/29/2010 - 18:33 Comment #4
Glad I could be of some help. Looking forward for the 3.78 update.