Mail and the way it processed

How does it work to have few users for a domain? Where the other users' mail will be stored? How do you access it if the user under Linux is different? Is this possible to change homes directory to something else?

New users are located in /home/DOMAIN/users/USERNAME.

Emai is stored in Maildir format in /home/DOMAIN/users/USERNAME/Maildir.

To change where they're stored, you can go into System Settings -> Virtualmin Config -> Defaults for new domains, and there you can change "Subdirectory for mailbox user home directories" to your preferred Subdirectory.

Is there a scheduled script that could go through a specific folder in Maildir to learn spam messages that were manually moved to Spam folder?

No, the only way to do that is to send the spam message to the spamtrap@domain.tld address.

How to switch from using saslauthd/PLAIN to using md5_crypt with Postfix/Dovecot?

This isn't normally something you would need to change, as you could always just have your users login using TLS/SSL.

Reviewing the Postfix docs, it doesn't appear that there's a simple way to change that.

Thanks for the feature suggestion!

I'm not sure if we'll be able to implement that, though I'm planning to talk to Jamie about it.

There's a lot of challenges in making a change like that for everyone, since that's a big change from how things work now.

So if we did decide to add that, it would likely be a few Virtualmin versions before that could be properly planned and tested.

Part of me likes the idea, and part of me dislikes it.

The good: If we find a message with [SPAM] in inbox then assume it was moved out of Junk and mark as ham and remove [SPAM] from Subject. If we find a message without [SPAM] in Junk then assume it was manually moved into Junk and mark as spam and add [SPAM] to subject. It would have pretty reliable automation abilities this way and be very intuitive.

The bad #1: Dovecot and email clients cache folder indexes and mailfile lengths; in fact, each mail message in Maildir format contains the filesize as part of the filename, and editing the Subject-line in the raw message file breaks that, causing cache and file-size mismatches.

The bad #2: Email clients will have cached the old subject lines and will not see the [SPAM] tag get added/removed, since they are looking at locally cached email subjects.

The bad #3: Messages would be processed on a cron job so it would take hours for the user to see their [SPAM] tag get added/deleted from the subject,

The bad #4: Messages would be processed on a cron job so it would take hours before the messages are scanned. The user might delete them from Junk/Inbox way before that, assuming the job was immediately performed the second they moved the message to a different folder.

The bad #5: Scanning maildir folders for every user on every domain with potentially millions of emails, and opening each file and scanning the subject line on a cron job is a HEAVY, HEAVY job. Not a good idea AT ALL. Think of the server memory requirements and disk load.

The bad #6: The sa-learn would only run on messages manually moved into Junk; not on messages already in the folder and assumed to be spam. Normally you might want to sa-learn those too, to help SpamAssassin learn the traits of the junkmail to be even more certain about raising its junkscore next time.

The bad #7: Anyone can email you a message with a [SPAM] subject header into the inbox. This would cause the script to assume that it's a message that WE have moved from Junk to Inbox, and will ham-learn it, thereby removing their spamscore.

The only solution to ALL of these issues (if folder-based processing is to be used), is to NOT use Inbox/Junk at all, but to use two special folders: SASpam and SAHam. If any email is moved into them, the ham/spam learn is executed. In case of the SASpam folder, we delete the mailfiles after the cron job is done, and in case of the SAHam folder we remove the [SPAM] tag, update the filesizes in the filename, generate a new unique mailid for the filename to defeat cache (risky if the new ID causes a clash though) and move them to the Inbox maildir (this move+rename gets rid of the Dovecot/client cache-mismatch issue). The user just has to dump emails into either folder and wait until the folders empty themselves (meaning processing is done). However, it's kinda ugly to have two folders sitting there at all times, AND if you accidentally move messages into the wrong one then you are screwed.

There is of course a similar mechanism already in virtualmin: Forward the spam/ham contents to spamtrap/hamtrap@domain.com and it will be processed, if that feature has been enabled on the domain. This is less convenient than just moving files between folders, but much safer. It is also very fast, because the real user and these aliases exist on the same mail server, meaning that the forwarded messages are immediately passed between mailboxes on the same server without having to travel the internet (apart from the initial upload from the mail client).

Other solutions are to install the sa-learn plugin for Roundcube and do all marking from that GUI. Or to do it from Usermin.

I dunno... Part of me likes your idea but it has way too many risks and issues.

What's really needed is an extension to the IMAP protocol to tell the server "MARK/UNMARK AS SPAM", along with a new spam/not spam metadata field. This way the Dovecot/Courier server does all the work, immediately, and it won't matter what folder things are in. Anyone feel like writing an RFC, waiting 5 years for it to be standardized and another 5 years for clients/servers to appear? :)

There is actually a very long 2009 thread on the Internet Engineering Task Force mailing list, about extending IMAP:

www.ietf.org/mail-archive/web/imapext/current/msg00110.html

Needless to say, nothing came of it. It was deemed way too complex. "Should clients do a simple binary yes/no spam toggle? Should it be based on folders where the learning is immediately triggered upon moving messages in/out of a Junk folder? Should it be more gradual like server-spamscore of 2 + client-spamscore of 2 = total spamscore of 4 = message is spam on both client and server and should be learned as spam? Should it be bidirectional so that server and clients exchange bayesian filters? Should it be done as an extension or as a keyword or a trigger or what?" - etc etc etc.

One of the guys said it best: "If this proposal had come 10 years ago when we were standardizing extensions, it would have been easy to do. But now, the installed IMAP client base is too large for us to change things too much."

But I am sure something will happen in this arena within another 10-20 years. So If you still have spam then, you might wanna check if IMAP has been extended. ;-P

There is this dovecot plugin called antispam. We are using that and we have set several IMAP Folders that are Spam. If a message is moved to Spam it's learned to dspam (sa-learn would work too I think), and if it is moved away from Spam then it's ham and relearned.

http://wiki2.dovecot.org/Plugins/Antispam

There are SpamAssassin examples there too, this might be way easier then running a script with additional dependencies..

Hope this helps..

@jrhosting: That's pretty cool. I just worry that it triggers every time an incoming message is filed into the Junk folder automatically. In that case, any incoming email that SpamAssassin guesses is spam, is put in the Junk folder, triggering the Antispam plugin to do a sa-learn on it, thus boosting the spam score of what MAY have been misidentified ham.

But if it only triggers when an actual email client moves emails, then it sounds really useful.

The Dovecot antispam plugin is something we've been exploring. It is indeed pretty cool!

Actions are only performed during the "IMAP COPY" within Dovecot, so it's not called during the initial email delivery... only if the email client moves it into the Spam folder using IMAP. Or, another action is triggered by moving email out of the spam folder, which would allow messages to be tagged as ham.

What exactly occurs when those events are triggered is configurable -- we're working on a resource friendly way to handle that.

The Dovecot antispam plugin isn't likely to become "official" within Virtualmin installs in the near future, but we are hoping to write up some instructions on how to set it up.

Oh wow, that's excellent. Since it's only done on "IMAP COPY", I might actually implement this on all servers so that users can just move emails. Thanks for the tip andrey and jrhosting! I'll first have to make sure it doesn't act on Sieve "fileinto" events either, but I don't think it would since I believe fileinto is a literal move as opposed to IMAP's copy+delete original.

Dovecot-Antispam was a real bitch to get to build on the outdated Dovecot-2.0.9 version on CentOS 6, since the official Dovecot is up to 2.2. I figured out which commits introduced build errors and reverted them.

Here's my CentOS 6 installation guide:

Configuration is left up to you.

Turns out that getting the Antispam plugin to talk to spamc (the SpamAssassin spamd client) was non-trivial. Why? Because the plugin gives absolutely zero error-logging when the command fails, so I had to waste 3 hours trying about 10-15 different things until I figured out why the plugin wasn't calling the command properly.

The Antispam v2-plugin docs are very bad, because the current maintainer that has taken over the plugin and made it work with Dovecot 2 is clearly inexperienced and has stripped all kinds of useful information and examples that were there in the Antispam v1 plugin, making configuration a bit of a blind guess. Several things show his inexperience, such as the way he renamed the very logically named "pipe" backend (capable of passing the email to any program via STDIN pipes), to "sendmail", as if that was the only program it was capable of working with, clearly showing that he's a bit of a newbie. I also found indication that the original Antispam author dislikes this guy and didn't sanction the Dovecot 2 version. Anyway, to make matters worse, the plugin isn't very popular so it's very rare to find configuration examples anywhere online to compare mine against. I found one person using it with dspam and one with sa-learn but nobody successfully using it with spamc. I've now figured out how to do the latter.

Here are the two most important pieces of information that I have now learned:

• The Antispam plugin executes the external plugin as the UID/GID of the connected mail user, NOT as root (despite Dovecot itself running as root). Therefore, you do not need to use "sudo" in your external command to get it to execute as the permissions of the mail user; that's taken care of automatically.

• The spam/notspam parameter can only take ONE word. That's it. You CANNOT specify "antispam_mail_spam = "-L spam"" or similar. The workaround is to provide the -L flag as part of the generic argument list.

I've been on quite a journey, even going as far as talking directly to Dovecot's lead programmer to figure out some of how its internal parameter processing works. In the end, it came down to the two issues above.

Here is my finished configuration for a CentOS 6 system which runs spamc/spamd (far, far more efficient than sa-learn, since it doesn't have to load the full SpamAssassin engine every time it has to learn/unlearn a single email). This is for people using SpamAssassin with real home directories for each email user (NOT for virtual email users; for those, you probably have to use the spamc -u parameter which I mention below).

This configuration properly executes spamc as the exact email user, so that only the correct user's Bayesian filters are updated.

All of the comments are by me.

/etc/sysconfig/spamassassin

/etc/dovecot/conf.d/90-antispam.conf

/etc/dovecot/conf.d/20-imap.conf

It's been running for a few days now, and I've been double- and triple-checking that each action is correctly interpreted and trained as ham/spam as intended and that there are zero mistakes or errors.

It was a success. I can now recommend the above setup to any other admins that want a resource-efficient and very intuitive spamc/spamd training method.

Well our setup differs 'alot' from the standard Webmin/Virtualmin Pro kind of setup. We use dspam instead of spamassassin which is a succes but a pita to setup at first. Once it runs it's groovy though!

I would like to see a virtualmin friendly kind of configuration for that, possibly an option that can be toggled to be used instead (though with the various OS's around the admin should get the plugin going by himself).

Thanks Remko