Can not open /etc/webmin/site for write. No Space Left on disk Error

Submitted by roshanbudhathoki on Wed, 06/21/2017 - 09:13

My Disk is only 60% full which I confirmed with df ssh but when I open virtualmin it says. Could not open /etc/webmin/site for write . No space left on disk.

My Webmin Version : 1.844 Usermin : 1.711 virtualmin : 5.99 Ubuntu : 16.04

Please help out to fix the issue.

Status: 
Closed (fixed)

Comments

Submitted by andreychek on Wed, 06/21/2017 - 09:35

Howdy -- hmm, what is the output of these two commands:

mount
df -h

Submitted by roshanbudhathoki on Fri, 06/23/2017 - 00:09

Mount has Following output

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=2003036k,nr_inodes=500759,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=404644k,mode=755)
/dev/vda1 on / type ext4 (rw,relatime,quota,usrquota,grpquota,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/1019 type tmpfs (rw,nosuid,nodev,relatime,size=404644k,mode=700,uid=1019,gid=1012)
tmpfs on /run/user/1003 type tmpfs (rw,nosuid,nodev,relatime,size=404644k,mode=700,uid=1003,gid=1003)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=404644k,mode=700)

df -h has following

> df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           396M   41M  355M  11% /run
/dev/vda1        60G   34G   23G  59% /
tmpfs           2.0G  4.0K  2.0G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           396M     0  396M   0% /run/user/1019
tmpfs           396M     0  396M   0% /run/user/1003
tmpfs           396M     0  396M   0% /run/user/0

Submitted by andreychek on Fri, 06/23/2017 - 09:30

Hmm, that definitely does sound odd! Are you still seeing that issue now?

Also, what is the output of these commands:

echo foo > /etc/webmin/test.txt
ls -la /etc/webmin/
ls -la /etc/webmin/site
cat /proc/user_beancounters

Submitted by roshanbudhathoki on Sat, 06/24/2017 - 09:42

ls -la /etc/webmin/site ls: cannot access '/etc/webmin/site': No such file or directory

> ls -la /etc/webmin/site
ls: cannot access '/etc/webmin/site': No such file or directory
> cat /proc/user_beancounters
cat: /proc/user_beancounters: No such file or directory

This was the output.

Submitted by andreychek on Sat, 06/24/2017 - 11:11

What was the output of the first two commands though?

I didn't see the "echo" output above (if there was any, there may not have been), as well as "ls -la /etc/webmin".

And just to clarify, are you still seeing this issue now when you log in? If so, I don't imagine restarting Webmin with "service webmin restart" helps?

Submitted by roshanbudhathoki on Tue, 06/27/2017 - 02:42 

> echo foo > /etc/webmin/test.txt
> ls -la /etc/webmin/
total 728
drwxr-xr-x 123 root    root     4096 Jun 24 20:24 .
drwxr-xr-x 114 root    root    12288 Jun 23 19:24 ..
drwx--x--x   2 root    bin      4096 May 14 16:58 acl
drwx--x--x   2 root    bin      4096 May 14 16:58 adsl-client
drwx--x--x   2 root    bin      4096 May 14 16:58 ajaxterm
-rw-------   1 root    root      140 Jun  8 18:58 allsaphiya.acl
-rw-------   1 root    root      128 Jun  8 18:58 anurodh.acl
drwx--x--x   2 root    bin      4096 Jun 27 12:52 apache
drwx--x--x   2 root    bin      4096 Jun 23 19:24 at
drwxr-xr-x   2 root    root     4096 Jun 15 14:25 authentic-theme
drwx--x--x   2 root    bin      4096 May 14 16:58 backup-config
drwx--x--x   2 root    bin      4096 May 14 16:58 bacula-backup
drwx--x--x   2 root    bin      4096 May 14 16:58 bandwidth
-rw-------   1 root    root      148 Jun  8 18:58 bhaweshkafle.acl
drwx--x--x   2 root    bin      4096 Jun 23 19:24 bind8
-rw-------   1 root    root      160 Jun  8 18:58 bostoneducation.acl
-rw-------   1 root    root      176 Jun 23 19:24 bundesinternational.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 burner
-rw-------   1 root    root      132 Jun  8 18:58 canjhapa.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 change-user
-rw-------   1 root    root      152 Jun  8 18:58 clintonschool.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 cluster-copy
drwx--x--x   2 root    bin      4096 May 14 16:58 cluster-cron
drwx--x--x   2 root    bin      4096 May 14 16:58 cluster-passwd
drwx--x--x   2 root    bin      4096 May 14 16:58 cluster-shell
drwx--x--x   2 root    bin      4096 May 14 16:58 cluster-software
drwx--x--x   2 root    bin      4096 May 14 16:58 cluster-useradmin
drwx--x--x   2 root    bin      4096 May 14 16:58 cluster-usermin
drwx--x--x   2 root    bin      4096 May 14 16:58 cluster-webmin
-rw-r--r--   1 root    root     7486 Jun 23 19:24 config
drwx--x--x   2 root    bin      4096 Jun  4 17:09 cpan
drwx--x--x   2 root    bin      4096 Jun 23 19:24 cron
-rw-------   1 root    root      129 Jun  8 18:58 crystal.acl
-rw-------   1 root    root      160 Jun  8 18:58 crystalsolution.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 custom
drwx--x--x   2 root    bin      4096 May 14 16:58 dfsadmin
drwx--x--x   2 root    bin      4096 May 14 16:58 dhcpd
drwx--x--x   2 root    bin      4096 May 14 16:58 dovecot
-rw-------   1 root    root      136 Jun  8 18:58 evergreen.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 exim
drwx--x--x   2 root    bin      4096 May 14 16:58 exports
drwx--x--x   2 root    bin      4096 May 14 16:58 fail2ban
-rwxr-xr-x   1 root    root      462 Jun  4 17:09 failed.pl
drwx--x--x   2 root    bin      4096 May 14 16:58 fdisk
drwx--x--x   2 root    bin      4096 May 14 16:58 fetchmail
drwx--x--x   2 root    bin      4096 Jun 23 19:24 file
drwx--x--x   2 root    bin      4096 Jun 23 19:24 filemin
drwx--x--x   2 root    bin      4096 May 14 16:58 filter
drwx--x--x   2 root    bin      4096 May 14 16:58 firewall
drwx--x--x   2 root    bin      4096 May 14 16:58 firewall6
drwx--x--x   2 root    bin      4096 May 14 16:58 firewalld
-rw-r--r--   1 root    root       14 May 14 16:58 first-install
drwx--x--x   2 root    bin      4096 May 14 16:58 fsdump
-rw-------   1 root    root      160 Jun  8 18:58 gomendracollege.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 grub
drwx--x--x   2 root    bin      4096 May 14 16:58 heartbeat
drwx--x--x   2 root    bin      4096 Jun 23 19:24 htaccess-htpasswd
drwx--x--x   2 root    bin      4096 May 14 16:58 idmapd
drwx--x--x   2 root    bin      4096 May 14 16:58 inetd
drwx--x--x   2 root    bin      4096 Jun  4 17:09 init
drwx--x--x   2 root    bin      4096 May 14 16:58 inittab
-rw-r--r--   1 root    root     1269 Jun  4 17:09 installed.cache
drwx--x--x   2 root    bin      4096 May 14 16:58 ipfilter
drwx--x--x   2 root    bin      4096 May 14 16:58 ipfw
drwx--x--x   2 root    bin      4096 May 14 16:58 ipsec
drwx--x--x   2 root    bin      4096 May 14 16:58 iscsi-client
drwx--x--x   2 root    bin      4096 May 14 16:58 iscsi-server
drwx--x--x   2 root    bin      4096 May 14 16:58 iscsi-target
drwx--x--x   2 root    bin      4096 May 14 16:58 iscsi-tgtd
drwx--x--x   2 root    bin      4096 May 14 16:58 jabber
-rw-------   1 root    root      148 Jun  8 18:58 kanchanjunga.acl
-rw-------   1 root    root      148 Jun  8 18:58 kankaischool.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 krb5
drwx--x--x   2 root    bin      4096 May 14 16:58 ldap-client
drwx--x--x   2 root    bin      4096 May 14 16:58 ldap-server
drwx--x--x   2 root    bin      4096 May 14 16:58 ldap-useradmin
-rwxr-xr-x   1 root    root      460 Jun  4 17:09 login.pl
-rwxr-xr-x   1 root    root      462 Jun  4 17:09 logout.pl
drwx--x--x   2 root    bin      4096 May 14 16:58 logrotate
drwx--x--x   2 root    bin      4096 May 14 16:58 lpadmin
drwx--x--x   2 root    bin      4096 May 14 16:58 lvm
-rw-------   1 root    root      132 Jun  8 18:58 mahendra.acl
drwx--x--x   2 root    bin      4096 Jun 23 19:24 mailboxes
drwx--x--x   2 root    bin      4096 May 14 16:58 mailcap
drwx--x--x   2 root    bin      4096 May 14 16:58 man
-rw-------   1 root    root      140 Jun  8 18:58 merochords.acl
-rw-------   1 root    root      156 Jun 20 17:59 merochordsinfo.acl
-rw-------   1 root    bin      2378 Jun 23 19:24 miniserv.conf
-rw-------   1 root    bin      3046 May 14 16:58 miniserv.pem
-rw-------   1 root    bin       783 Jun 23 19:24 miniserv.users
drwx--x--x   2 root    bin      4096 May 14 16:58 mon
drwx--x--x   2 root    bin      4096 May 14 16:58 mount
drwx--x--x   2 root    bin      4096 Jun 23 19:24 mysql
-rw-------   1 root    root      132 Jun  8 18:58 nawodaya.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 net
-rw-------   1 root    root      136 Jun 13 12:48 netradham.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 nis
-rw-------   1 root    root      128 Jun  8 18:58 nkhabar.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 openslp
drwx--x--x   2 root    bin      4096 May 14 16:58 package-updates
drwx--x--x   2 root    bin      4096 May 14 16:58 pam
drwx--x--x   2 root    bin      4096 May 14 16:58 pap
drwx--x--x   2 root    bin      4096 Jun 23 19:24 passwd
-rw-------   1 root    root      140 Jun  8 18:58 pathibhara.acl
-rw-r--r--   1 root    root       14 May 14 16:58 perl-path
drwx--x--x   2 root    bin      4096 May 14 16:58 phpini
drwx--x--x   2 root    bin      4096 May 14 17:00 postfix
drwx--x--x   2 root    bin      4096 May 14 17:00 postgresql
drwx--x--x   2 root    bin      4096 May 14 16:58 ppp-client
drwx--x--x   2 root    bin      4096 May 14 16:58 pptp-client
drwx--x--x   2 root    bin      4096 May 14 16:58 pptp-server
-rw-------   1 root    root      144 Jun  8 18:58 pradeshinfo.acl
-rw-------   1 root    root      140 Jun  8 18:58 prajwalrai.acl
-rw-------   1 root    root      132 Jun  8 18:58 priyanka.acl
drwx--x--x   2 root    bin      4096 Jun 23 19:24 proc
drwx--x--x   2 root    bin      4096 May 14 16:58 procmail
drwx--x--x   2 root    bin      4096 May 14 16:58 proftpd
drwx--x--x   2 root    bin      4096 May 14 16:58 qmailadmin
drwx--x--x   2 root    bin      4096 May 14 16:58 quota
drwx--x--x   2 root    bin      4096 May 14 16:58 raid
-rwxr-xr-x   1 root    root      164 Jun  4 17:09 reload
-rwxr-xr-x   1 root    root       48 Jun  4 17:09 restart
-rw-------   1 root    root      124 Jun  8 18:58 roshan.acl
-rw-------   1 root    root      152 Jun  8 18:58 rotaractnepal.acl
-rw-------   1 root    root      140 Jun  8 18:58 sajhamanch.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 samba
drwx--x--x   2 root    bin      4096 May 14 16:58 sarg
drwxr-xr-x   2 root    root     4096 May 14 17:01 security-updates
drwx--x--x   2 root    bin      4096 May 14 16:58 sendmail
-rwx------   1 crystal crystal  1866 May 15 12:20 server.crystalsoft.com.np.cert
-rwx------   1 crystal crystal  1675 May 15 12:20 server.crystalsoft.com.np.key
drwx--x--x   2 root    bin      4096 May 14 16:58 servers
drwx--x--x   2 root    bin      4096 Jun 24 20:27 shell
drwx--x--x   2 root    bin      4096 May 14 16:58 shorewall
drwx--x--x   2 root    bin      4096 May 14 16:58 shorewall6
drwx--x--x   2 root    bin      4096 May 14 16:58 smart-status
-rw-------   1 root    root      148 Jun  8 18:58 smartconsult.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 smf
drwx--x--x   2 root    bin      4096 May 14 16:58 software
drwx--x--x   2 root    bin      4096 May 15 16:14 spam
-rw-------   1 root    root      140 Jun  8 18:58 springdale.acl
drwx--x--x   2 root    bin      4096 May 14 16:58 squid
drwx--x--x   2 root    bin      4096 May 31 13:22 sshd
-rwxr-xr-x   1 root    root      234 Jun  4 17:09 start
drwx--x--x   3 root    bin      4096 May 14 17:05 status
-rwxr-xr-x   1 root    root      157 Jun  4 17:09 stop
drwx--x--x   2 root    bin      4096 May 14 16:58 stunnel
drwx--x--x   2 root    bin      4096 Jun 23 19:24 syslog
drwx--x--x   2 root    bin      4096 May 14 16:58 syslog-ng
drwx--x--x   2 root    bin      4096 Jun  4 17:09 system-status
drwx--x--x   2 root    bin      4096 May 14 16:58 tcpwrappers
drwx--x--x   2 root    bin      4096 Jun 23 19:24 telnet
-rw-r--r--   1 root    root        4 Jun 27 13:22 test.txt
drwx--x--x   2 root    bin      4096 May 14 16:58 time
drwx--x--x   2 root    bin      4096 May 14 16:58 tunnel
-rwxr-xr-x   1 root    root      198 Jun  4 17:10 uninstall.sh
drwx--x--x   2 root    bin      4096 May 14 16:58 updown
drwx--x--x   2 root    bin      4096 May 14 17:00 useradmin
drwx--x--x   2 root    bin      4096 May 14 16:58 usermin
-rw-r--r--   1 root    root       12 May 14 16:58 var-path
-rw-r--r--   1 root    root        6 Jun  4 17:09 version
drwx--x--x   2 root    bin      4096 May 14 16:58 vgetty
drwx--x--x  11 root    bin      4096 Jun 27 12:52 virtual-server
drwxr-xr-x   2 root    root     4096 May 14 17:01 virtualmin-awstats
drwxr-xr-x   2 root    root     4096 May 14 17:01 virtualmin-htpasswd
drwx--x--x   2 root    bin     20480 Jun 23 19:24 webalizer
drwx--x--x   2 root    bin     12288 Jun 23 19:27 webmin
-rw-------   1 root    root     5664 Jun 23 19:24 webmin.acl
drwx--x--x   3 root    bin      4096 May 14 16:58 webmincron
drwx--x--x   2 root    bin      4096 May 14 16:58 webminlog
drwx--x--x   2 root    bin      4096 May 14 16:58 wuftpd
drwx--x--x   2 root    bin      4096 May 14 16:58 xinetd

above was the output of the commands and "service webmin restart" command does not help nor restarting the server helps :(

Submitted by andreychek on Tue, 06/27/2017 - 08:51

Yup, that all looks good!

Just to verify, if you run "service webmin restart" -- and then you log out of Virtualmin and back in, do you continue to see that error?

Submitted by roshanbudhathoki on Thu, 06/29/2017 - 23:54

Yeap, I still see the same error. :(

Submitted by andreychek on Fri, 06/30/2017 - 11:04

Jamie, how does Webmin decide there's a problem when it generates the error "Can not open /etc/webmin/site for write. No Space Left on disk"?

The disk doesn't appear to be out of space, and we were able to create a new temp file in /etc/webmin.

Submitted by JamieCameron on Fri, 06/30/2017 - 15:48

It happens when the write system call fails - there's no special logic in Webmin to detect an out of quota situation.

By the way, have you tried running df -i to show free inodes on the filesystem? If you have used them all up, you'll get the same error message.

Submitted by roshanbudhathoki on Fri, 06/30/2017 - 22:05

Jamie was right my inode limit was running out. But it was due to the fpm which creates tmp folder under user's home folder and uses for session storage but never cleans it up. And I found there was nearly 2 million session files created in tmp folder of a user. How can I configure php fpm to automatically clean such tmp folder?

Submitted by andreychek on Fri, 06/30/2017 - 23:14

Great, we're glad you were able to figure out the cause of that error.

I'll follow up in your other post regarding the session files.

Submitted by roshanbudhathoki on Sun, 07/09/2017 - 22:51

Status: Active » Closed (fixed)