Email Filters not working when received from admin@ or postmaster@ and other emails

I have also raised a forum thread here: https://www.virtualmin.com/node/38366#comment-156235

I found a very strange issue with Email Filters, when it received email sent to it from anyone sending from admin@ or postmaster@

If an email filter is created to forward ALL mail to another address, if you send it from admin@anything.co.uk (and by anything I mean anything in here, gobbledegook.co.uk), then it goes into the inbox and is not filtered, but if its not admin@anything (even admin1@) , it forwards OK. This also occurs for postmaster, but not hostmaster etc.

This is very strange and people are potentially not getting any emails when sent rom admin@.

Is this is a bug? Or a setting I can fix on the server?

I tried it on two separate mail servers, so for two different domains, and same issue.

Aliases forward OK, so its only email account filters.

This also happens when sending from administrator@ and anything-admin@ and anything-administrator@

Thanks

Status: 
Active

Comments

Howdy -- I'm not sure of anything that would cause something like that, unless there's some sort of pre-existing filter that's looking for an address with that name.

Can you paste in the contents of your /etc/procmailrc file, as well as the .procmailrc file from the home directory of a user who's experiencing this issue?

Ok. We created it in Usermin and not procmail direct, but as you can see the procmail rule is quite straightforward and simple, nothing crazy going on there:

:0
* !^FROM_MAILER
! myemail@amitywebsolutions.co.uk

We have a lot of users (like 50 or so) with rules setup exactly like this to send email to their personal emails. It happens for my own test account, and then many users are reporting this issue.

NOTE: I tried it on a completely different server I have and the issue is there too hence thinking its a server issue and not applicable to this specific account. So wondered if you have tried this? You need to send an email FROM admin@ (or other address like administrator@, postmaster@) to the user with the filter setup. It is an actual email account with filter in Usermin, and not an alias, and not a filter created in Virtulamin which I believe is different to the actual users Usermin filter?

Thanks

Sorry forgot the /etc/procmail file, here it is:

LOGFILE=/var/log/procmail.log
TRAP=/etc/webmin/virtual-server/procmail-logger.pl
:0wi
VIRTUALMIN=|/etc/webmin/virtual-server/lookup-domain.pl --exitcode 73 $LOGNAME
EXITCODE=$?
:0
* ?/usr/bin/test "$EXITCODE" = "73"
/dev/null
EXITCODE=0
:0
* ?/usr/bin/test "$VIRTUALMIN" != ""
{
INCLUDERC=/etc/webmin/virtual-server/procmail/$VIRTUALMIN
}
DEFAULT=$HOME/Maildir/
ORGMAIL=$HOME/Maildir/
DROPPRIVS=yes

I think that FROM_MAILER macro in the .procmailrc file is the problem - the admin@ address is being treated as a bounce message sender, and being forwarded to myemail@amitywebsolutions.co.uk

But this is added using Usermin (we didn't edit procmail direct), and its also on several servers. So is this a bug with the FROM_MAILER macro, or are you suggesting we need to take a look at this ourselves? Because we are just using the server default, I dont believe we have changed anything that would result in this behaviour. Thanks

I'd say it is a bug in the FROM_MAILER macro, which is provided by Procmail.

Sorry for going on again, but does this mean its something you would look into fixing in Virtualmin, or do I need to look into it? I need to fix it quickly thats all, customers are not receiving emails. Thanks

Unfortunately, Jamie is saying that's a problem with the way procmail works.

There isn't anything we would be able to do to fix that.

My best suggestion would be to not use the "FROM_MAILER" filter in your .procmailrc, if possible. Is removing that an option?

But we (and our customers with no technical knowledge) create the filter in Usermin. procmail and FROM_MAILER filter is not known to them.

Bascially a user needs to create an email account, go into Usermin, click Filters, create a filter to Forward All emails to another email address. Thats it. The procmail rule is automatically created from this.

Who is responsible for Usermin? Can Usermin be changed to create a different filter, if thats what is needed?

For info (and this issue is extremely important so please take note), we dont use Aliases for this because an Alias is not scanned for spam. So what happens with an Alias is all email including spam is forwarded. If the recipient uses Gmail, Outlook, Yahoo, or Apple (so a lot of people!), then these providers then blacklist the server for sending so much spam. If Aliases can go through the spam filter we would use those as its easier to setup. So because of this Aliases are a real problem and we advise no one ever use them.

Ok, I see the issue here - if a user sets up forwarding in the Usermin UI, by default the FROM_MAILER condition is added to prevent forwarding of bounce messages.

This is generally what you want to happen - however, procmail's FROM_MAILER macro matches admin@ and postmaster@ addresses, which seems overly broad to me.

The quick work-around is for users to click on "Email Filters" on the left menu, click on the forwarding rule, and un-check the box "Suppress forwarding of bounce messages".

The longer-term solution is less clear - Usermin could avoid using FROM_MAILER by default, but this could trigger bounce loops.

Ticking that box should be OK, I will try it.

Do you know what the negative impact would be? What bounce messages its referring to?

It works! Wonderful. Thanks a lot for that!! I will let our customer know about this to add to their mail filter process.

Could disabling suppressing bounce messages result in email loops? We have an account forwarded to two different email addresses. One does not exist anymore I dont think, the other is overquota. We have over 90,000 messages in the queue which mostly look like bounce messages informing the sender (which is forwarded on) that the email accounts it is forwarded to are overquote and dont exist. So everytime a message is sent to the full email box for example, and a bounce back returned, would that then send on to the user again, and the other user? Which in turn sends a bounce back etc etc? Thus causing a loop and a continuous sending and receiving of messages? Every message includes the previous message, which includes the previous message etc.

Yes, it could cause a loop - that's exactly why the bounce message suppression is setup by default.

Ok, so at the moment there is no suitable way to spam check emails and then forward on? You either use an alias which forwards spam, or use a filter which doesn't forward admin@ and other emails, or use a filter which forwards bounce messages which could result in an endless loop. I'm quite surprised there is no proper solution for this. I guess we'd have to use a third party service for spam filtering, or somehow change the way the system works to not use procmail and use something else that works (if there is something)?

Most people do spam filtering only at the final destination, rather than when forwarding, so this problem doesn't come up.

Actually its a massive problem. If the recipients are Gmail, Hotmail, Yahoo or Apple, then if they see quite a lot of spam being forwarded to them, they stop it even going to the users spam folder, and just block it and send a bounce back, and then the sending server is indicated as being a sender of spam and even more legitimate mail is blocked by these providers.

So this is a massive issue, its the whole reason we use filters and not aliases because using aliases the users were receiving less email, turns out Gmail etc were blocking email due to the high amount of spam forwarded.

For info, Google recommend spam forwarded to them is either marked as spam or that spam is not forwarded to them https://support.google.com/mail/answer/175365?hl=en.

Forgot to mention, you cant use Aliases either if using SPF records, as the sending server will look to be the forwarding server which may not be on the senders SPF, but using email filters it maintains the senders domain as the sender and so is on their SPF. We've investigated this due to people not receiving email.

Unchecking the suppress bounced messages is causing worse issues.... it results in email loops which are resulting in massive CPU of spamassassin, and the server slows down, sometimes crashes. We have to re-enable suppress bounce messages.

On this page http://linux.die.net/man/5/procmailrc I dounf the following:

If the regular expression contains '^FROM_MAILER' it will be substituted by '(^(((Resent-)?(From|Sender)|X-Envelope-From):|>?From )([^>][^(.%@a-z0-9])?(Post(ma(st(er)?|n)|office)|(send)?Mail(er)? |daemon|mmdf|n?uucp|ops|r(esponse|oot)|(bbs.)?smtp(error)?|s(erv(ices? |er)|ystem)|A(dmin(istrator)?|MMGR))(([^).!:a-z0-9][-_a-z0-9])?[%@>\t ][^<)]((.).*)?)?$([^>]|$))' (a stripped down version of '^FROM_DAEMON'), which should catch mails coming from most mailer-daemons.

So we can see the email addresses I reported... A(dmin(istrator) and Post(ma(st(er)

So I guess we should just edit this to remove the A(dmin(istrator) section. There are plenty of people in the world emailing people from Admin or Administrator email addresses, I cant believe this in there to be honest.

Do you know how to edit this? I cant find any reference to editing this procmail filter anywhere.

Thanks

The part "A(dmin(istrator)" will match "Admin" or "Administrator" .

Editing this to fix your specific problem with admin@ addresses seems like a reasonable idea. However, it looks like this is hard-coded into procmail and cannot be edited via any config file :-(

Is there an alternative to Procmail we can use? I'm not sure how much more I can re-iterate the importance of this. Suppressing Bounce backs is causing havoc, thousands of bounce backs being forwarded in an infinite loop. Emails sent from admin@ or administrator@ are not being forwarded so people are not getting important emails, Aliases do not filter spam.... its a completely ridiculous situation to be honest! The expected and normal behaviour is not possible. If I want a proper solution should I not use Virtualmin/Webmin, am I using thew wrong system to provide an email service? Or is it just because procmail is a bad choice and we should use something else? Thanks

Edit: I searched for FROM_MAILER on the server, and found /usr/bin/procmail. Opening this is gobbledygook. But the filter I mentioned above is clearly visible in that gobbledygook. If I edit this would it break procmail considering the rest of the file is gobbledygook?

Edit: I did just try editing /usr/bin/procmail to edit the rule, but no mail arrived at all, so I guess it broke it

Yeah, you can't just edit a binary like that.

Unfortunately this rule to detect bounce messages was presumably chosen by procmail because there are legitimately bounces that come from admin@ addresses. There is no standard other than looking at the sender to determine if a message is a bounce though, so as you saw this is a very imprecise solution :-(

I don't have any better solution unfortunately, and I can't see how any alternative system could solve this perfectly either. It's a real shame gmail blacklists the addresses of systems that are just alias forwarders, although I guess they have no way of detecting who the original sender of a message really is.

We will have to not send internally from admin@ or administrator@ but we cant control other organisations who send using this. Must be a very common issue around the world for people who forward using procmail.