Bug with NFS

Error displayed as example: In domain virtualmin.com Creating administration group virtualmin .. .. done Creating administration user virtualmin .. .. done

Creating aliases for administration user .. .. done

Adding administration user to groups .. .. done

Creating home directory .. .. done

Creating mailbox for administration user .. .. done

Adding new DNS zone .. .. done

Adding to email domains list .. .. done

Adding default mail aliases .. .. done

Adding new virtual website .. .. done

Adding webserver user apache to server's group .. .. done

Performing other Apache configuration .. .. configuration failed : Failed to copy /etc/php.ini to /home/virtualmin/etc/php5/php.ini : cp: accessing `/home/virtualmin/etc/php5/php.ini': Permission denied

This is not the issue Jamie. I have tested and replicated it several times. With the same NFS share, and with other NFS shares, you do not need to change anything on the share configuration, this includes "root_squash_" which is on already. The same share, with exactly the same configuration, fails in CentOS 6 but works without a single problem in CentOS 5. You can even run them in parallel and Virtualmin in CS5 creates his account, while CS6 fails.

The problem only happens in CentOS 6 as far as I see.

Share is CentOS 6.

Virtualmin running in CentOS 6 fails to create data. Virtualmin running in CentOS 5 creates data fine.

Using same share, same configuration, clean installation in both cases, default virtualmin installations. I cannot pin point this to be a OS bug because I tested other software which can write/read data fine. Its only Virtualmin here that is failing on the same and you can make CentOS 5 fail with the same error by:

1. Delete mounted share in CentOS 5.
2. Create the same mount using the option "nfs4" in Virtualmin.

Now, CentOS 5 can replicate the bug, it will fail with the exactly the same error as CentOS 6.

1. Delete the mounted shared again.
2. Create it again only using "nfs". Now Virtualmin creates data fine in the same machine without absolutely any other change.

.... In the case of the CentOS 6 systems, tested on several different clean installations, it will always fails. Based on this I concluded that this works in CentOS 5, and probably in every other OS, but the issue is specific to how Virtualmin runs in CentOS 6.

Its not only php.ini failing.

Virtualmin is actually not creating a single file inside the mount. This means the whole directory is completely empty as opposed with the same config in CentOS5 which creates all files fine, and also does not display any errors on the Virtualmin console while creating the account.

That PHP.ini was just one of the errors it displays while creating the account.

It also shows error on: Setting up scheduled Webalizer reporting .. .. Webalizer reporting failed! : Failed to open /home/example/public_html/stats/.htaccess.webmintmp.7458 : No such file or directory at ../web-lib-funcs.pl line 1360, line 1.

And

Adding DAV directives to website configuration .. .. DAV Login failed! : virtualmin-dav::feature_setup failed : Failed to open /home/example/etc/dav.digest.passwd : Permission denied at ../web-lib-funcs.pl line 1360, line 1.

And and...

Setting up AWstats reporting .. .. AWstats reporting failed! : virtualmin-awstats::feature_setup failed : Failed to open /home/example/cgi-bin/awstats.pl.webmintmp.7458 : Permission denied at ../web-lib-funcs.pl line 1360, line 1.

The command you gave above fails as well when running it directly from the server as root: cp /etc/php.ini /home/virtualmin/etc/php5/php.ini cp: cannot create regular file `/home/virtualmin/etc/php5/php.ini': No such file or directory

The file /etc/php.ini

Has permissions 644 on both the CentOS 5 environment and CentOS 6.

The reason your command will just fails is because there is no "etc" folder, or "php5" folder, like I said, Virtualmin does not create anything, its empty.

If I manually create this folder I can cp the file just fine, and of course the file is copied to that location.

So what does this means? Those errors are ok. They are just telling us that it can create the php.ini because this path is invalid, neither "etc", neither "php5" exists and I assume the same is trye for the other errors because "cgi-bin" does not exist either.

If Virtualmin was creating folders and files correctly the errors would not appear.

I have absolutely no problems copying files manually to the mount and neither I can see anything else which has, only virtualmin has. My guess is that something is running different in CentOS 6. Maybe Virtualmin is not using "root" privileges for all the operations it needs to do.

Virtualmin in the CentOS 6 machine also creates a folder in the mount when I delete the account (which if course only deletes 1 folder, as its the only think it created), called "virtualmin-autoreply" with permissions 1777. Behavior not true either when deleting an account from Virtualmin CentOS 5.

I think I may know the reason for this problem. When I monitor live inside the folder Virtualmin creates I can see there file: .stats-htpasswd.lock

This means, there is some locking going on which is not allowing Virtualmin in CentOS 6 to create the appropriate folders and configurations inside the account directory.

Why this locking is not happening in CentOS 5 is something I did not investigated but this seems to be the reason.

I access to the Virtualmin web GUI as root, and I assume Virtualmin is creating data as root as well.

I have no idea what Virtualmin uses when creating a virtual host.

The first one fails, as the directory does not exist. So lets create it it manually: [root@home]# mkdir example

Second already fails: su -s /bin/sh example su: user example does not exist

What is exactly the point of this?

Virtualmin creates the user and folders on account creation. It makes completely sense that it should create this folders and files with the user in question that will own this, otherwise we would have a huge security risk if they where created as root, not to mention which need this for suexec, etc.

But lets take something in consideration. The users does not exist in the NFS servers. I can create complete random accounts like: fwsrfwe33.com jkde9d33.com

They are all created in the share with CentOS 5 but not with CentOS 6. If you what you mean is that its failing because the user/group is not setup in the share, the same would be true for CentOS 5. They need to exists in the future to apache can read the files, user can modify their own files, etc. But this is the account creation and the users still does not exists, not in the share, and neither in Virtualmin, as its virtualmin that creates the users and groups on the virtual account creation. Also that was the first thing I thought, user permissions, so actually I did created the users/groups before the account creation in the NFS servers, just for testing.

Virtualmin was still failing with the same error and that does not explain either why in CentOS 5 its working without the user/group existing in the share either.

The only difference I know between how this systems mount is that CentOS 6 seems to force the NFS4 option and the client address while CentOS 5 does not.

If you look at the mount with "mount -l" besides of course that CentOS 5 is ext3 and CentOS 6 ext4, I can only see one difference.

CentOS 6 adds the option "vers=4" and "clientaddr=" when mounting which is not there in CentOS 5.

Here are the results.

Created example10.com with Virtualmin, fails with usual errors but folder "example10" is created, empty of course.

The commands: [root@server]# su -s /bin/sh example10 sh-4.1$ cd /home/example10 sh: cd: /home/example10: Permission denied sh-4.1$

Can´t access with the username.

Same procedure from CentOS5 but lets name it example11.com. No errors on Virtualmin. Files seems fine.

Same commands, and access is granted and files exist: [root@lab/]# su -s /bin/sh example11 sh-3.2$ cd /home/example11 sh-3.2$ mkdir public_html mkdir: cannot create directory `public_html': File exists sh-3.2$ dir cgi-bin etc fcgi-bin homes logs public_html tmp sh-3.2$

So this is the error. From CentOS 6 you cannot access the share and do not have access permissions. Why? Lets see the user/permissions.

I checked permissions on both example10 and example11, both are set to permissions 750, but here comes the reason why we can´t access example10 in our CentOS 6 machine and why Virtualmin also fails.

The folder is owned by user and group "root", as opposed to "example11" which is owned by both user and group "nobody".

[root@home]# ls -ld example10 drwxr-x--- 2 root root 4096 Nov 22 21:20 example10 [root@home]# ls -ld example11 drwxr-x--- 10 nobody nobody 4096 Nov 22 21:24 example11 [root@home]#

That output was from the CentOS 6 machine. By the way, the user "nobody" actually does exist in the share. But here comes the strange part. The same from the CentOS 5 shows something very different:

[root@lab home]# ls -ld example10 drwxr-x--- 2 root root 4096 Nov 22 21:20 example10 [root@lab home]# ls -ld example11 drwxr-x--- 10 example11 example11 4096 Nov 22 21:24 example11

So we know why the CentOS 6 machine is not creating any files and configs, because the folder is owned by root, which is wrong. In CentOS 5 we also see the correct user and group for the folder, but in CentOS 6 the same shows as nobody !!!! We know the correct one is what CentOS 5 displays, because just to be sure I unmounted the shared on both creating 2 more virtual account, in both cases the folder is owned by the user with the same name, which is correct. So what do we assume?

CentOS 6 is displaying the incorrect user/group for example11. Why? I don´t know. But we also know why its failing to create the data, because the folder is owned by "root" instead, this can be related to the above issues, it must be. Maybe this is why its being created as "root" in the first place.

Its my bed time, so I will dig tomorrow more. Im still thinking this is related to how its mounting the share, wrong. And I can replicate this. On CentOS 5 I delete the mount and remount it using the option in Virtualmin "nsf4" instead of "nsf", I told before that when mouting in CentOS 5 using "nfs4" we have the same error as in Centos 6.

So I just did that, remounted using the "nfs4" option and now he get incorrectly stats again: [root@lab home]# ls -ld example10 drwxr-x--- 2 nobody nobody 4096 Nov 22 21:20 example10 [root@lab home]# ls -ld example11 drwxr-x--- 10 nobody nobody 4096 Nov 22 21:24 example11 [root@lab home]#

But lets be sure, logging into the NFS server and checking which are the correct ones. I almost sure it will the results from CentOS 5:

[root@nfs home]# ls -ld example10 drwxr-x---. 2 root root 4096 Nov 22 21:20 example10 [root@nfs home]# ls -ld example11 drwxr-x---. 10 514 514 4096 Nov 22 21:24 example11

514 = example11

And I was right. This are the correct ones. And CentOS 5 shows the correctly when we use the mount option in Virtuamin "nfs", if we use the option "nfs4" we get incorrect details. And this is exactly what is happening in CentOS 6. I mentioned it has the option "nfs4" in it as I compared both servers to see which bits where different.

So where is the bug? In Virtualmin CentOS 6, we mount the share only as "nfs" and its still forcing the option as "nfs4", if we can have it mounted just like CentOS 5, I think this will work. Of course when mounting from the command line I got the same results, so its strange. I don´t know why CentOS 6 is mounting it differently than CentOS 5, but this is indeed the issue.

Even if create the users in the share with LDAP this would not work, as the users don´t exist before account creation. They are automatically created by Virtualmin. The behavior in CentOS 5 is correct if you ask me.

Indeed, forcing the option to NFS v3 works. Now CentOS 6 gets the correct permissions and usergroups from the share.

What is strange is that the NFS server is also CentOS 6 and nfs4 is default. So on the client side nfs3 talks nice to the NFS 4 server, but NFS 4 on the client gets the incorrect groups and permissions.

I think I may have read a bug about this on NFS4 reported to Red Hat but im not sure anymore. I guess for now I just force the option nsf3 then. At least we know this is not a bug in Virtualmin, but NFS.

I found this: http://serverfault.com/questions/364613/centos-6-ldap-nfs-file-ownership...

But that is not my issues, the NFS server is correctly configured with its domain and the idmap is not catched incorrectly, the errors persist while using just "nfs" which will use NFS4 by default.

Here is also more information for the casual reader which may or not be related: https://bugzilla.redhat.com/show_bug.cgi?id=823848

It seems the fix is indeed force ns3 on the client. I guess this can be closed, unless Virtualmin wants to force "nfs3" when selecting only "nfs" in the web GUI, and using "nfs4" option to actually use NFS4. Currently the option "nfs" will use what ever is default by the OS. Or an option to force "nfs3" can be added. In any case, I it just takes the value from the OS so it cannot be indeed considered a per Virtualmin bug.

Great, that would be nice.

Thanks.