Failed to open /etc/webmin/security-updates/current.cache for writing : No space left on device

This is of course displayed in the system info panel of Virtualmin.

I have seen this issue before and it was typically due to old kernels taking up too much space in /boot

Looks like there should be plenty of room, and I have removed all unused kernels to free space.

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda2            5.0G  3.6G  1.2G  77% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/xvda1            107M   26M   76M  25% /mnt/boot
/dev/xvdf              50G  711M   47G   2% /var
/dev/xvdj              50G  572M   47G   2% /home
Status: 
Closed (fixed)

Comments

More info, I am seeing this on another EC2 instance in the same region, so it may possibly be related to AWS issues, like maybe Amazon's EBS is full?

This is in the second server in Edit Virtual Server, Enabled features: Failed to open /etc/webmin/apache/site for writing : No space left on device

Howdy -- yeah, it certainly appears that you have plenty of disk space available on your / partition there.

What does the command "mount" show?

And if you run "dmesg", do you see any relevant errors towards the end of the output?

mount

/dev/xvda2 on / type ext4 (rw,grpquota,usrquota)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/xvda1 on /mnt/boot type ext2 (rw,errors=remount-ro)
/dev/xvdf on /var type ext4 (rw,grpquota,usrquota)
/dev/xvdj on /home type ext4 (rw,grpquota,usrquota)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

dmesg

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
Command line: root=/dev/xvda2 ro console=hvc0 crashkernel=auto SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=de-latin1-nodeadkeys
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
ACPI in unprivileged domain disabled
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 00000000000a0000 (usable)
Xen: 00000000000a0000 - 0000000000100000 (reserved)
Xen: 0000000000100000 - 00000000f0000000 (usable)
DMI not present or invalid.
e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0xf0000 max_arch_pfn = 0x400000000
initial memory mapped : 0 - 20000000
init_memory_mapping: 0000000000000000-00000000f0000000
0000000000 - 00f0000000 page 4k
kernel direct mapping tables up to f0000000 @ 100000-885000
RAMDISK: 01cc1000 - 03d4a000
No NUMA configuration found
Faking a node at 0000000000000000-00000000f0000000
Bootmem setup node 0 0000000000000000-00000000f0000000
  NODE_DATA [0000000000008000 - 000000000003bfff]
  bootmap [000000000003c000 -  0000000000059fff] pages 1e
(7 early reservations) ==> bootmem [0000000000 - 00f0000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [00044cd000 - 00044f4000]   XEN PAGETABLES ==> [00044cd000 - 00044f4000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0001000000 - 0001ca0ff8]    TEXT DATA BSS ==> [0001000000 - 0001ca0ff8]
  #4 [0001cc1000 - 0003d4a000]          RAMDISK ==> [0001cc1000 - 0003d4a000]
  #5 [0003d4a000 - 00044cd000]   XEN START INFO ==> [0003d4a000 - 00044cd000]
  #6 [0000100000 - 000085b000]          PGTABLE ==> [0000100000 - 000085b000]
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x00000100 -> 0x000f0000
On node 0 totalpages: 982943
  DMA zone: 56 pages used for memmap
  DMA zone: 1886 pages reserved
  DMA zone: 2057 pages, LIFO batch:0
  DMA32 zone: 13384 pages used for memmap
  DMA32 zone: 965560 pages, LIFO batch:31
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 1 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
nr_irqs_gsi: 16
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
Allocating PCI resources starting at f0000000 (gap: f0000000:10000000)
Booting paravirtualized kernel on Xen
Xen version: 3.4.3.amazon (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff88000452a000 s95064 r8192 d23720 u126976
pcpu-alloc: s95064 r8192 d23720 u126976 alloc=31*4096
pcpu-alloc: [0] 0
trying to map vcpu_info 0 at ffff880004535020, mfn 74db28, offset 32
cpu 0 using vcpu_info at ffff880004535020
Xen: using vcpu_info placement
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 967617
Policy zone: DMA32
Kernel command line: root=/dev/xvda2 ro console=hvc0  SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=de-latin1-nodeadkeys
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
AMD-Vi disabled by default: pass amd_iommu=on to enable
Memory: 3815652k/3932160k available (4939k kernel code, 388k absent, 116120k reserved, 3931k data, 1220k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:256
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 39321600 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Xen: using vcpuop timer interface
installing Xen timer for CPU 0
Detected 1999.975 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 3999.95 BogoMIPS (lpj=1999975)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Unsupported number of siblings 32
Performance Events: unsupported p6 CPU model 44 no PMU driver, software events only.
alternatives: switching to unfair spinlock
SMP alternatives: switching to UP code
Freeing SMP alternatives: 32k freed
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20293 entries in 80 pages
Brought up 1 CPUs
sizeof(vma)=200 bytes
sizeof(page)=56 bytes
sizeof(inode)=592 bytes
sizeof(dentry)=192 bytes
sizeof(ext3inode)=800 bytes
sizeof(buffer_head)=104 bytes
sizeof(skbuff)=232 bytes
sizeof(task_struct)=2600 bytes
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 33316k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1346244557.409:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 7517
SELinux:  Registering netfilter hooks
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key 965331713A640F69
- User ID: CentOS (Kernel Module GPG key)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.0
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
XENBUS: Device with no driver: device/vbd/2064
XENBUS: Device with no driver: device/vbd/2048
XENBUS: Device with no driver: device/vbd/2192
XENBUS: Device with no driver: device/vbd/2128
XENBUS: Device with no driver: device/vif/0
XENBUS: Device with no driver: device/console/0
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 1220k freed
Write protecting the kernel read-only data: 7184k
dracut: dracut-004-33.2.el6_0
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.17.0-ioctl (2010-03-05) initialised: dm-devel@redhat.com
udev: starting version 147
dracut: Starting plymouth daemon
xlblk_init: register_blkdev major: 202
xvdb: unknown partition table
xvda: xvda1 xvda2 xvda3
xvdj: unknown partition table
xvdf: unknown partition table
EXT4-fs (xvda2): INFO: recovery required on readonly filesystem
EXT4-fs (xvda2): write access will be enabled during recovery
EXT4-fs (xvda2): orphan cleanup on readonly fs
EXT4-fs (xvda2): ext4_orphan_cleanup: deleting unreferenced inode 11691
EXT4-fs (xvda2): ext4_orphan_cleanup: deleting unreferenced inode 11327
EXT4-fs (xvda2): ext4_orphan_cleanup: deleting unreferenced inode 997
EXT4-fs (xvda2): ext4_orphan_cleanup: deleting unreferenced inode 837
EXT4-fs (xvda2): ext4_orphan_cleanup: deleting unreferenced inode 625
EXT4-fs (xvda2): 5 orphan inodes deleted
EXT4-fs (xvda2): recovery complete
JBD: barrier-based sync failed on xvda2-8 - disabling barriers
EXT4-fs (xvda2): mounted filesystem with ordered data mode
dracut: Remounting /dev/xvda2 with -o grpquota,usrquota,ro
EXT4-fs (xvda2): mounted filesystem with ordered data mode
dracut: Mounted root filesystem /dev/xvda2
dracut: Switching root
udev: starting version 147
Initialising Xen virtual ethernet driver.
EXT4-fs (xvdf): ext4_orphan_cleanup: deleting unreferenced inode 262213
EXT4-fs (xvdf): ext4_orphan_cleanup: deleting unreferenced inode 262207
EXT4-fs (xvdf): ext4_orphan_cleanup: deleting unreferenced inode 262204
EXT4-fs (xvdf): 3 orphan inodes deleted
EXT4-fs (xvdf): recovery complete
JBD: barrier-based sync failed on xvdf-8 - disabling barriers
EXT4-fs (xvdf): mounted filesystem with ordered data mode
EXT4-fs (xvdj): mounted filesystem with ordered data mode
JBD: barrier-based sync failed on xvda2-8 - disabling barriers
JBD: barrier-based sync failed on xvda2-8 - disabling barriers
JBD: barrier-based sync failed on xvdf-8 - disabling barriers
JBD: barrier-based sync failed on xvdj-8 - disabling barriers
Adding 3020212k swap on /dev/xvda3.  Priority:-1 extents:1 across:3020212k SS
ip_tables: (C) 2000-2006 Netfilter Core Team

i posted dmesg and mount output, but the message is queued for moderation...

OK, so it appears there are errors on the drives and they are remounted as read only.

Now I need to figure out why there are errors, any direction you can give me is appreciated.

Would the thing to do be:

  • create a new storage volume
  • copy contents of xvda1 to the new volume
  • reattach new volume as xvda1
  • reboot

??

Yeah, you certainly shouldn't be seeing any filesystem errors.

You may want to file a request with Amazon to have them look into that.

You may be able to create a new storage volume as you mentioned and copy your data there. However, that's going to be something you'd need to work out with Amazon, we don't know the specifics of how that would work.

I have done similar with AWS when I needed to create larger volumes and move /home and /var to the new volumes, so that would probably cure the issue.

Also these 2 instances are having the same issue, probably because I created an AMI from one instance to build the second... so probably the filesystem issues were probably a legacy from the parent instance.

Thanks for reminding me of the usefulness of dmesg one more time!!! /slaps head with palm/

Oh, and any admin-fu that I possess, I owe 75% to Virtualmin & Webmin AND your excellent support! (and the other 25% to Google Search)

Thanks!

I finally figured this one out: there were no inodes left, thus the rw issues.

A client had a php cron script that was triggering a PHP Warning, thus generating an email to root: there were 300,000+ email messages in /root/Maildir/new which were taking up all the inodes on that partition.

so, if you see a similar problem in the future, do df -i to show inodes in use.

then for i in ./* ; do echo $i; find $i -type f | wc -l; done to find the inode hogs, and drill down into the resulting hog dirs, and find the source of the problems.