Security - Users able to view all files on server via SSH

I noticed that virtual server administrators can access *all* files on the server via SSH login/file transfer. Can we restrict them to their own home directory only?

I did these but they don't restrict SSH access:

> Usermin Configuration > Access Control Options > Root directory for file chooser > *tick* User's home directory

> Usermin Configuration > Usermin Module Configuration > File Manager > *tick* Allow access to home and directories below..

> System > Users & Groups > Module Configuration > Permissions on new home directories > 0750