Got a PRO LICENSE. - Problems with SFTP for admin user after creating an e-mail user for database only access

@Andreychek,

Thank you for the reply, This issue is happening on 2 separate servers, 1 a build that was only 6 weeks old that I did not have any issues with until the past 2-4 weeks (I don't know when, since I had not created any more new hosted domains outside of that time frame) and 1 that was just built this last weekend. The one that was 6 weeks old, I did not know the cause and I had to get this back up and running, only to run into the same issue, only I dug in a bit more and found the cause this time, that I am unable to fix. About rebooting SSH, yes I have, in fact I have rebooted both entire servers, no effect. Something coding wise had to have changed in the past month to cause this issue. Another piece of information, This issue does seem to set off the Fail2Ban as a flag as well. And another piece of information, all users on these installs of Virtualmin are Jailed. Here is the results of the syslog for that event.

Apr 24 04:30:37 host5 systemd[1]: Created slice User Slice of [affected user].

Apr 24 04:30:37 host5 systemd[1]: Starting User Manager for UID [affected ID]...

Apr 24 04:30:37 host5 systemd[1]: Started Session 639 of user [affected user].

Apr 24 04:30:37 host5 systemd[7590]: Reached target Timers.

Apr 24 04:30:37 host5 systemd[7590]: Reached target Paths.

Apr 24 04:30:37 host5 systemd[7590]: Reached target Sockets.

Apr 24 04:30:37 host5 systemd[7590]: Reached target Basic System.

Apr 24 04:30:37 host5 systemd[7590]: Reached target Default.

Apr 24 04:30:37 host5 systemd[7590]: Startup finished in 19ms.

Apr 24 04:30:37 host5 systemd[1]: Started User Manager for UID [affected ID].

Apr 24 04:30:37 host5 systemd[1]: Stopping User Manager for UID [affected ID]...

Apr 24 04:30:37 host5 systemd[7590]: Failed to enqueue exit.target job: Access denied

Apr 24 04:30:37 host5 systemd[1]: Stopped User Manager for UID [affected ID].

Apr 24 04:30:37 host5 systemd[1]: Removed slice User Slice of [affected user].

I intentionally have extra spaces between each because the posting was jumbling it all up.

I literally did a fresh install of Ubuntu 16.04.4 with all updates and went to create a Virtual Server (hosted domain), at this point the SFTP works great, but right after I create the extra user for database only access, this affects the admin user that I use, and have been using for years for SFTP access. I don't like using the domain admin account for the database for obvious reasons.

OK, now below is the only thing that shows up int he syslog for a SFTP login right after I create the Virtual Server and I am able to log in with out a problem. The above is what happens after I create the user for database access.

Apr 24 04:56:27 host5 systemd[1]: Started Session 648 of user [affected user].

Thank you so much, James

Has anyone been able to look into this, What I have had to do is, create a dummy virtual server just for databases, and I can create user names there with no problem. So if I had e-mail users, this would cause me more issues, which I do have a few domains that have e-mail users, that I am terrified of adding or changing anything because of this issue. I do not have any problems with my current setup, it is just new users cause me a big problem.

To help move this along, hopefully, I purchased a pro license for 100, I am not installing this on my production server since I really don't use the extra features, I only purchased this for the support right at the moment. I have a sandbox server that I will install this on to test this when I have a chance, but that will be a few weeks.

Yes, I am using Chroot jails for everything, for the purpose it was created for. I just tested this and there are no problems with Chroot Jails turned off. Even though that does work, this is not a fix, I need to use the chroot jails for the servers as a layer of security. Thank you

Please help, I will provide any logs needed, I am in a really difficult situation since I now cannot create an more e-mail users, nor modify them (don't want to take the chance). Please let me quickly say, I built a Virtualmin server back in the middle of March, the same way, with NO issues. The recent updates that were installed between then and the middle of April is when this issue started. So I figured it was a miss-configuration so I re-built the server (twice now). The second time was just a proof if there was an issue. The server from March is already decommissioned and re purposed now since the 1st rebuild showed the same issue anyway. Ultimately I can get someone access to the server itself if this can help, Please let me know.

Hello, Here are some outputs of auth.log. I will be breaking it into 2 sections, 1) the successful SFTP connection when I first create a virtual server. 2) the output after I try logging in after creating a mail user for database only access.

successful connection

successful disconnection

After creating the extra user

unsuccessful connection ending in SFTP connection error and when failtoban is turned on this trips that as well on the 2nd attempt

something else I discovered is I can't even log into the user any more in SSH SHELL session neither by username and password which worked in the past nor a su command by a sudo user or root. Here is the output of that, first a working one then the broken one.

successful

unsuccessful

when i log on via the console, I see the last time logon flash, then immediate get kicked back to the logon window for username and password. When I try to do a su command to that user, the prompt never changes for what user was logged in.

About the error in the unsuccessful attempt for SFTP connection above, I did log in as root to compare permissions between a user that I did not create an additional user name for that still works with SFTP and shell, and the one that is not working. I have looked at the passwd file and the fstab file as well. the permissions in each of the jailed directories look the same, with permissions and ownership of the /home/123abc directory and everything under it belonging to them.

As I have posted here and in another thread referring to this thread, we are willing to pay to get this fixed, We will get the 250 pro or unlimited pro yearly package. I also have everything setup and ready for someone to be able to get on the server and / or remote to my computer to watch what I am doing to see entirely what is going on, just in case I am not describing it well enough here.

What DOES still work, though I don't really use, is I can log in as the affected user into the Virtualmin admin panel still. AND I CAN use the affected user for straight FTP connection, but that does not help when I need those users to be able to use SHELL and SFTP

Thanks in advance for anyone that can help :)

I have just discovered something NEW, when I logged into the Virtualmin panel under the affected user and tried to use the "File Manager" and when I tried to click on public_html, I got the following error:

I know what is happening now, but I still don't know how to fix this NOR why creating a e-mail user would cause the mounting path to freak out as such.

update:

OK, the above only happens when you try to use the "Explorer" style navigation. If you click on the home button above and just look around, everything seems to work fine (within the File Manager, the main issue still exists). I have NOT tried to upload any files to a user affected like this since I will not be able to perform all the operations I need to.

edit again:

the above "may or may not" be contributing to the issue. I logged is as a virtual server admin user that I did not create any additional users for, that I am still able to use with SSH and SFTP. I get the same error as above when I log in the Virtualmin panel as them and try to use the File Manager.

Again, thanks in advance for any help.

Hello,

Can someone help me, help you, help me :)

I am still trying to gather more information, Can someone tell me exactly what happens when you create a user under a virtual server with e-mail only access. What changes are made and why? Where the changes are made. I ask because just deleting the extra user does not solve the issue. In order for the Virtual Server admin to log back in via SSH or SFTP, the Virtual Server has to be deleted and re-created (where my problem lies).

Thanks again, I look forward to your response

OK ok... I believe I found HOW this is happening. Can someone please help me fix it.

EDIT:

Please be aware when reading this, there is a main system passwd file, I am not referencing that right now. That one is fine, it is starting using the correct shell as /usr/sbin/jk_chrootsh

the passwd file I am showing below is the one inside one of the Jails themselves. I explain as best as I can, in later a later post but I do believe that Jamie will understand it.

Edit END:

the /etc/passwd file that is created in the /home/chroot/jailed/etc/ folder for the jail, for some reason when the extra user is created, the admin user, in this case 123abc ends up loosing the /bin/bash shell.

here is what I mean

before creating another user

after a user mysql was created

I tested this and adding the /bin/bash back to the passwd file, I am able to log back in SFTP, SSH and use the SU command once again.

Thanks

I did end up getting a PRO subscription for 100.

Look forward to hearing that you have a fix for this issue.

If someone needs access to this sandbox server I setup for this, I am more than willing to cooperate, if it will expedite a fix for this. While, yes I do know how to fix this, but I have to edit these passwd files in the jails every time I add or possibly edit a user (have not confirmed the editing yet). Which in a shared hosting environment, which is what I am trying to set up, is absolutely inexcusable. I would be having support requests out my ears, oh, you added an e-mail address, one second, you can now log back in SSH and SFTP. oh you added another one, ok it is fixed again. That would immediately loose me my customers.

you have 2 different passwd files, 1 "master" one out side the jails and the ones in each jail for only "That" user (virtual server, and all their users). the perticular passwd files I am talking about are the ones in each chrooted jail. when you create a user the main passwd file is fine with the correct shell. but the passwd file inside the jail, it is fine until you add another user then the shell just dissapears. I looked at what the passwd file looked like in the jail before adding the user and it was /bin/bash

I updated the versions of everything installed up top at the opening post.

thank you so much for replying.

Yes your right, but that is the "Master" passwd file that has the jk_chrootsh. The system passwd file is fine, no problems.

The problem is with the passwd file INSIDE each jail. the passwd file is fine until you create an additional user.

that is when it blanks out the shell for the admin user of the jail.

Make sense?

I have a lot of information in the post as a whole, but the part where I found the passwd file INSIDE each of the /home/chroot/domain/etc folders themselves (inside the jail[s]) is KEY. Essentially their local copy of their passwd file is for only their user names, e-mail address users, ftp users, ect, ect.

Since this is mounted as such that they would see anything under /home/chroot/domain as / when they logged in via SFTP or SSH. Another way to put it is /home/chroot/domain = / (when they log in / interact with the system)

If they were to go up to / they would see pretty much what the server has in root, but it is their own copy (which they STILL don't have ownership of except the contents in /home/domain/)(which, the other difference is in the /home/ they will only see their domain or username depending how you have your system setup).

and to try to clarify what Joe was confused about, yes they are jailed, but the passwd file inside the jail consists of what shell for them to use inside of the jail itself. Another way to think of it is as a separate partition that they can only see or even if this works as a better analogy (which would be bad since this is NOT how it works), each jail being it's own separate Virtual Machine so that would make sense for the shell to be /bin/bash at that point since they are working in a secluded area, which is what Jails are designed to do in the first place.

This took me a moment to grasp this as well, that is why it took me a moment longer to look for a passwd file inside the jail to look at that, instead of just looking at the "system" passwd file.

I was wondering if anyone could reply saying that they understand now since the last one seemed confused to what I was saying. Please, I really appreciate the reply, I just want to make sure I am being understood properly and I don't have to way another 2 months.

I have had very little time until just recently to be able to do exploratory surgery as such.

I do want to reiterate, that I am very pleased that someone replied and is working on this.

Thanks and eagerly waiting for further information,

Hello

I did notice 1 change when I built the server from the past. in the past, there was a character limit for MySql user names, so it would create 2 if it went over the limit. Could this change have affected anything along the way?

YES YES. please tell me where you want me to send logon details to and will do, I can even get you on my computer to show you exactly what I am doing as well so you can watch.

getting on my computer and showing you what exactly what I am doing will take approx 5-10 minutes once your on. that and I can / will still also let you use the sandbox as well :)

EDIT: pointing out all the customization I have done will take approx another 10-15 minutes. No hard file customization, just changes to the server templates, virtualmin config, ftp server for specific ports, php customization that works for us on a default level, everything I customize I don't see how it would affect anything. Besides the point I have tried this before I did any of these customization and I still get the same result. To be clear as well, this sandbox is a clean, fresh install from 2-4 days ago with all updates including the latest kernel for 16.04.5

Thank you :)

Hello,

I might have found another issue with the passwd file inside the jails. (when creating additional users for sub servers)

Jamie, Please tell me if sub servers inside of a jailed virtual server, any additional users are suppose to be added to the passwd file in that jail as well?

Basically anything with the same group should be encapsulated inside the virtual server

example: say have 123abc.com with a admin user name and group of 123abc:123abc and I have a subserver of that 345def.com. It has no separate username, is uses 123abc

but if you were to add an additional user for 345def.com say mysql.345def it should be added as such:

mysql.345def:123abc - correct?

the subserver extra user got added to the "system" passwd file, but I think it missed the Jailed passwd file.

it is not like the extra user has ssh or sftp or even ftp for that matter, even if it did, for ftp, it would still be locked in the domains folder as it's root. so... does this actually matter?

I am asking since the primary virtual server additional users ARE added to the jailed passwd file as well as the system passwd file.

hmmmm..

Thanks,

Just got up, I will be sending you this information shortly.

I sent the logon information right after I posted the previous post above. Can you confirm that you received it?

Thanks

Site was down earlier, I accidentally double posted, I edited the double post with this

Thank you so much. would it be alright if you could share the fix so I can fix this on my production server? Thank you

OK, the fix you are working on will work for new virtual servers as well?

I tried making 123456.com as a test and then created another user, and the shell still disappears from the passwd file in that jail

so I / you can fix existing virtual servers right now with a config fix on a per virtual server, but the complete fix for new virtual servers (as well) to not be affected with this is going to take a little while? Just a question for any kind of ETA you might have.

Thank you so much for finding the problem. I look forward to hearing back from you.

just created after your last post

Thanks,

Ok,

Correct me if I am wrong,

What you are doing to correct the Jails that are already created is just adding this:

to the affected jails under the following file:

/etc/webmin/virtual-server/domains/domain-id

I can wait for you to test a Long Term Solution if this is all you are doing.

does this fix the other issue with subdomains or subservers, their users not being added to the passwd file in the jail? If those users end up having FTP or SSH or other, they could be running outside the jail?

You are awesome Jamie, thanks

feel free to reboot the server or anything you feel necessary while you are working :) That is what it is for.

One thing to add, I applied this fix to all the users on my production server and while I was going through all the user config files, I noticed that only the new users were missing:

the restored users that I migrated from the working server, are fine.

Thanks,

yes, that is what you ended up finding that solved the domains that were suppose to be chrooted but when you add a user the /bin/bash would disappear, hence this whole thread.

Until you come up with a fix for Virtualmin or Webmin to add that unjailed_shell for jailed users like it should when jail's are first created, any new Virtual Servers that are suppose to be created with chroot enabled is not getting that unjailed_shell.

The only thing I was saying, is while this chroot was working correctly, I was able to restore a backup on a system that was not creating chroot users with the unjailed_shell in their config file, but the restored users still had the configuration correct. I was trying to let you know that this was only happening to NEW Virtualmin Servers with Chroot enabled.

Thanks for everything Jamie, do you have an ETA for the Long term fix for this issue?

Thanks Jamie