No space left on device

9 posts / 0 new
Last post
#1 Wed, 12/24/2008 - 17:29
rebrain

No space left on device

Hallo.

Today I logged in to Webmin and tried creating a virtual server. But hten i got an error: - No space left on device

this appeared on multiple stages of creating the vurtual server.

Why is it happening? I have a 500GB HDD which is barely filled.

I ran df:

df

[code:1]Filesystem 1K-blocks Used Available Use% Mounted on /dev/md1 3020048 3019952 0 100% / tmpfs 1030524 0 1030524 0% /lib/init/rw udev 10240 44 10196 1% /dev tmpfs 1030524 0 1030524 0% /dev/shm /dev/md2 477178104 16606524 436332312 4% /home[/code:1]

I assume that 100% filled disk is not good, but what is it? Where can I find it and how to get some space there?

I tried to fix the problem every single way my knowledge lets me, but I did not succeed and I had to reboot because the Webmin interface started acting up and giving errors (i think connected with the "no space" problem). After a reboot the Webmin seems alright but the problem persists.

EDIT: I am having problems with using Webmin because it does not want to work without it's "free space":

[code:1]Failed to write to /etc/webmin/security-updates/current.cache : No space left on device[/code:1] (This appears when I click on the "Refresh System Information" link). Could it be a problem with permssions?

Any advice is appreciated!

Regards and Happy Holidays.

Post edited by: rebrain, at: 2008/12/24 17:34<br><br>Post edited by: rebrain, at: 2008/12/24 17:35

Wed, 12/24/2008 - 18:22
andreychek

I fear you're not going to like the answer, but you've got a real bugger of a problem :-)

As you said, your /home partition has a plenty of space, you'll be set for quite awhile with that.

Unfortunately, your / partition -- which houses all your system files, the log files (in /var/log), MySQL databases (in /var/lib/mysql), and everything else but what's in /home -- only has 3GB assigned to it.

That's just not really enough :-/

I mean, you can clear out some cache files, clean up /tmp, make sure all the old logs are purged, etc, but that's all just a bandaid... you're going to run out again. The first person to upload a big GIF into their database will cause the same problem you're seeing now.

To solve that, you'll unfortunately need to start over and re-partition, but I'd highly recommend allocating much more space to the / mount point.

You might even consider putting / and /home on the same partition so you don't have to guess as to how much space to make available to each. That will also make quotas easier.

There's differing philosophies on how to partition servers, and some folks dislike putting it all on one big partition.

Me -- I used the one partition scheme, and simply make sure that all the users on there have a quota, and I run monitoring software to warn me if the disk space is filling up too quickly.

Have a good one,
-Eric

Thu, 12/25/2008 - 03:57 (Reply to #2)
rebrain

I woke up today and checked the &quot;df&quot; again.
Now I have only 75% percent filled partition. This is a good sign since I have some time now to figure out how to allocate more space to the disk.

The space is used extensively by clamav. (Correct me please if I am wrong).

When installing the Debian I chose to put all files on one partition without &quot;cutting up&quot; the HDD. So should I be able to allocate more space to the &quot;/dev/md1&quot; by just editing quotas?
If not is there a way to &quot;repartition&quot; without formatting?

I am trying to read the /var/log/syslog.0 file (the syslog is empty)
but it is 250MB! How can I make Debian create logs with smaller sizes?

Regards, Rebrain

Fri, 12/26/2008 - 06:34 (Reply to #3)
andreychek

<div class='quote'>So should I be able to allocate more space to the &quot;/dev/md1&quot; by just editing quotas?
If not is there a way to &quot;repartition&quot; without formatting?</div>

Sorry, you can't resolve this issue by tweaking quotas.

There are some tools available out there for resizing Linux partitions. I have very little experience with them, and I'm not sure how they'll work on the RAID partitions you have setup, so you'd need to research that.

However, you'd boot off a boot CD, and run one of these tools on your drive to resize the partitions.

Just search on Google for &quot;resize linux partitions&quot; for some ideas.

If you haven't configured too much yet, the easier route may be to start from scratch by re-partitioning, but you'll have to decide that based on how much tweaking you've done so far :-)

<div class='quote'>I am trying to read the /var/log/syslog.0 file (the syslog is empty)
but it is 250MB! How can I make Debian create logs with smaller sizes?</div>

If you want the logs to be smaller, you need to poke through the logs to determine what service is generating lots of output, then edit the config for that service and turn down the amount it sends to the logs.

Also, make sure logrotate is running in the daily cron -- it should run each night and compress the previous days logs, as well as delete logs that are too old.
-Eric

Sat, 01/03/2009 - 16:28 (Reply to #4)
Joe
Joe's picture

Eric's right about reducing the size of logs...but I'll add that /var on a working virtual hosting system needs to be pretty big...like several GB big. So, if you're panicking about 250MB of usage, you're <i>definitely</i> going to run into problems when you really begin to work your box. 250MB is nothing on a modern server.

I'll also mention that I no longer split out my disk into a bunch of partitions, because the reasons for that sort of administrative overhead have long disappeared due to better filesystems, better backup tools, and faster disks...I use one big partition on all of my systems. This avoids all of those &quot;OMG, this partition is full, but I have so much free space on this other one...why didn't I make this one bigger to start with???&quot; moments. In such a layout, you don't run out of space until you actually run out of space.

--

Check out the forum guidelines!

Mon, 01/05/2009 - 01:43 (Reply to #5)
Maurizio1230

why don't you try to access in rescue mode and resize partitions?
What os are you using?

Mon, 01/05/2009 - 03:08 (Reply to #6)
Maurizio1230

I am sorry, maybe you don't have rescue mode like me :(

Thu, 11/20/2014 - 18:56
youngguru

Here is a very simply way to accomplish this for anyone else that's stumped!

This can be done on a live system as long as /home is not in use when you try to unmount it. You can also follow these steps to resize another logical volume if it has another name and isn’t in use. This may work with some adjustments on a RHEL/CentOS 5 system, I haven’t tried it though

cd /

cp -Ra /home /home.bak # Make a backup of home

umount /home

lvm lvremove /dev/vg_/lv_home # Remove the logical volume for home

lvm lvresize -l+100%FREE /dev/vg_/lv_root # Resize the root logical volume so it uses 100% of the now free space

Then edit /etc/fstab and comment out the old /dev/mapper/vg_hostname-lv_home that isn’t used anymore save

resize2fs /dev/vg_/lv_root # Resize the filesystem to use the whole logical volume mv /home.bak /hom

Depending upon your hard drive space this might take awhile. when done REBOOT.

CREDITS http://alexcline.net/2013/04/01/live-resize-of-logical-volume-in-rhelcen...

Fri, 11/11/2016 - 09:59
Masplus

I get the same problem today, found this post and I would post my solution:

Problem

failed to open /etc/webmin/apache/site for writing no space left on device

Tests

Checking disk space

df -h
/dev/vda1                  15G    12G  3,1G  79% /

So, we have space on the device.

Checking disk inodes

df -i
/dev/vda1                983040  983040   0   100% /

That's the problem ! But, we can't execute any instructions because /tmp is mounted on / that is full. Temporary i try move some files to other partitions to take some inode space.

Solution

After that, we can analyze the system searching what is generating that all little files.

ls / -AiR1U | sed -rn '/^[./]/{h;n;};G;  s|^ *([0-9][0-9]*)[^0-9][^/]*([~./].*):|\1:\2|p' | sort -t : -uk1.1,1n |cut -d: -f2 | sort -V |uniq -c |sort -rn | head -n10 > /tmp/inodes_list.txt

If you want all the list, remove "head -n10" and you'll get it.

After checking all system inodes, you can filter and solve the problem.

Topic locked