Security Update cache eating up all my disk space

18 posts / 0 new

Topic locked

#1 Fri, 01/14/2011 - 00:31

byrnef87

Security Update cache eating up all my disk space

Hi all,

I have Virtualmin installed on my server running Ubuntu 10.04. I am having an issue with the cache files in the directory /etc/webmin/security-updates eating up all the disk space on the server. More and more files appear in this folder until the server crashes and I have to go in and manually delete them all.

Is there any way to periodically delete these files automatically?

FYI the files look like this "available.cache1.webmintmp.9914" where the number at the end just keeps counting up.

Thanks for your help!

Francis

#2 Fri, 01/14/2011 - 04:13

Locutus

Hmm... for me, that directory looks like this, and Virtualmin is running for quite a while there:

root@orion:/etc/webmin/security-updates# ls -l
total 13984
-rw-r--r-- 1 root root    81236 2011-01-14 10:12 available.cache0
-rw-r--r-- 1 root root 14011435 2011-01-14 10:12 available.cache1
-rw-r--r-- 1 root root       13 2011-01-05 21:07 config
-rw-r--r-- 1 root root   164076 2011-01-14 11:08 current-all.cache
-rw-r--r-- 1 root root    20800 2011-01-14 11:08 current.cache

Those "webmintmp" files sound to me like Webmin was unable to complete some "store stuff temporarily, then later rename/merge/delete files" operation.

What's the permissions of the directory and files for you? Maybe Webmin can't write them. Check with ls -l and with lsattr.

You might also turn on Webmin's debug logging to see if any operations with those files fail.

#3 Fri, 01/14/2011 - 12:45 (Reply to #2)

JamieCameron

All those extra files are rather surprising - Virtualmin does create temp files in that dir, but should delete them almost immediately.

If you delete all the *cache* files in that directory, do the temp files re-appear?

#4 Sun, 01/16/2011 - 19:51

byrnef87

Thanks for the replies.

@Locutus: The file permissions are like this:

-rw-r--r--   1 root root 13590528 2011-01-12 09:49 available.cache1.webmintmp.9895
-rw-r--r--   1 root root 13578240 2011-01-15 04:04 available.cache1.webmintmp.9904
-rw-r--r--   1 root root 13590528 2011-01-09 16:49 available.cache1.webmintmp.9914
-rw-r--r--   1 root root        0 2011-01-09 21:14 available.cache1.webmintmp.9941
-rw-r--r--   1 root root 13578240 2011-01-11 22:24 available.cache1.webmintmp.9959
-rw-r--r--   1 root root       13 2011-01-07 09:24 config
-rw-r--r--   1 root root   147257 2011-01-17 12:39 current-all.cache
-rw-r--r--   1 root root    20142 2011-01-17 12:39 current.cache

Note: this is a sample of files, there are waaay more in the folder. lsattr yields these results:

------------------- ./available.cache1.webmintmp.29959
------------------- ./available.cache1.webmintmp.30092
------------------- ./available.cache1.webmintmp.30222
------------------- ./available.cache1.webmintmp.16010
------------------- ./available.cache1.webmintmp.6078
------------------- ./available.cache1.webmintmp.29697

The directory permissions:

drwxr-xr-x 2 root root 163840 2011-01-17 12:44 security-updates

Seems like the same as yours, so I don't know if it's a permissions error, unless the task that is doing the merging/deleting isn't run by root?

@JamieCameron: When I delete all the cache files, they just reappear after a day or two and keep piling up again.

#5 Sun, 01/16/2011 - 22:34 (Reply to #4)

JamieCameron

If you delete all those *cache* files and then run :

/etc/webmin/virtual-server/collectinfo.pl

as root from the shell, does it output any error messages?

#6 Mon, 01/17/2011 - 20:51 (Reply to #5)

byrnef87

I tried that just now and it simply runs the script then exits, no error messages, no output whatsoever. I checked the files in the directory before and after, and they were the same.

Thanks again for your patience.

#7 Mon, 01/17/2011 - 21:15 (Reply to #6)

JamieCameron

I don't suppose your system is low on disk space or inodes?

You can check with the commands :

df -h
df -i

#8 Tue, 01/18/2011 - 00:40

byrnef87

Nah, there's like 6GB of free space (after I delete all those cache files).

#9 Tue, 01/18/2011 - 11:20 (Reply to #8)

JamieCameron

One possible cause is that available.cache1 is not writable. What is the output from :

ls -l available.cache1
lsattr available.cache1

#10 Wed, 01/19/2011 - 03:59

Locutus

According to his previous screen dump, that file didn't even exist for him... Might that be the problem?

#11 Wed, 01/19/2011 - 12:32 (Reply to #10)

JamieCameron

Yes, that could explain it..

Perhaps /etc/webmin/security-updates isn't writable?

OP - if you SSH in as root, can you create a file in that directory with a command like touch /etc/webmin/security-updates ?

#12 Sun, 01/23/2011 - 07:18

byrnef87

The file does exist, I was only showing part of the thousands of available.cache... files that were in the folder. Here is the permissions of available.cache1:

#ls -l available.cache1
-rw-r--r-- 1 root root 13582299 2011-01-23 12:30 available.cache1
# lsattr available.cache1
------------------- available.cache1

And the permissions on /etc/webmin/security-updates: drwxr-xr-x 2 root root 163840 2011-01-24 00:09 security-updates

Looks to me like they are both writable (by root anyway). I can touch /etc/webmin/security-updates as well.

I was thinking maybe that the script is being run by something other than root. Does the following line of syslog imply that the collectinfo.pl script is being run by root?

Jan 24 00:09:01 squareoneit CRON[15375]: (root) CMD (/etc/webmin/virtual-server/collectinfo.pl)

I am totally stumped on this. I just keep having to delete all the excess files every couple of days. Very frustrating!

#13 Sun, 01/23/2011 - 15:04 (Reply to #12)

JamieCameron

Those permisions all look OK to me ..

I think to debug this further, I would need to login to your system myself and see what is going wrong.

Or if that is not possible, could you run :

strace -o /tmp/strace.out -f /etc/webmin/virtual-server/collectinfo.pl

and email me the strace.out file at jcameron@virtualmin.com

#14 Sun, 01/23/2011 - 19:34

byrnef87

Hmmm, this is strange. I just tried running /etc/webmin/virtual-server/collectinfo.pl again, and it gave me this error:

Out of memory!
Terminated

Every time I try re-running this script, including when I've deleted all the available.cache1.* files, I get that same error. I tried checking the processes using top in a separate terminal whilst running the command and it appears that the memory usage for collectinfo.pl spikes at one point, then quits with that error.

Is it a memory leak in the script itself?

Here is the contents of /etc/webmin/virtual-server/collectinfo.pl:

#!/usr/bin/perl
open(CONF, "/etc/webmin/miniserv.conf");
while(<CONF>) {
        $root = $1 if (/^root=(.*)/);
        }
close(CONF);
$ENV{'PERLLIB'} = "$root";
$ENV{'WEBMIN_CONFIG'} = "/etc/webmin";
$ENV{'WEBMIN_VAR'} = "/var/webmin";
chdir("$root/virtual-server");
exec("$root/virtual-server/collectinfo.pl", @ARGV) || die "Failed to run $root/virtual-server/collectinfo.pl : $!";

#15 Sun, 01/23/2011 - 23:08 (Reply to #14)

JamieCameron

Yes, it looks like the collectinfo.pl was crashing due to lack of RAM, causing incomplete temp files to be left behind. The next version of the security-updates module for Virtualmin will reduce RAM usage significantly in this case, and will fix the problem. We plan to release this new version this week ..

#16 Mon, 01/24/2011 - 06:26

Locutus

So I was quite right with my initial assumption "Those "webmintmp" files sound to me like Webmin was unable to complete some "store stuff temporarily, then later rename/merge/delete files" operation.", even though for a different reason. ;)

#17 Mon, 01/24/2011 - 12:30 (Reply to #16)

JamieCameron

Yes - whenever Webmin writes to any important file, it will first create a file with webmintmp in the name, write to that, and then rename it over the original file. This avoids the problem of a file being partially written due to lack of disk space - not fun if it happens to /etc/passwd .

#18 Sun, 01/30/2011 - 19:06

byrnef87

Hi,

I just wanted to reply and confirm that everything has been working perfectly for the last week or so since the update was made. The memory problem also appears to be what was causing our DNS server to drop out every couple of days, because that is fixed now too!

Thanks so much for all of your help, the support for this product is incredibly impressive even at the GPL level!

Best Regards,

Francis

Topic locked