Spam getting through

Submitted by RyanJohnson on Wed, 01/19/2011 - 20:48

Hello

I've had an influx of spam over the past week. A particular strain of messages are getting a rating of about 2.5 and thus slipping through.

But if I save the message on the server and run:

spamassassin < spam_test.txt

The score is 6.5! Which would not let it pass through.

Do you mind helping me understand what the differences are here?

Thanks, Ryan

Status: 
Active

Comments

Joe's picture
Submitted by Joe on Wed, 01/19/2011 - 21:51 Pro Licensee

We'd need to see the full headers of the message that made it through to know what's going on. SpamAssassin appends the tests that added up to the score in the headers.

Submitted by RyanJohnson on Wed, 01/19/2011 - 21:56

Here is a sample:

Return-Path: <JudeVangerbig@isoc.net>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on julius.neubreed.com
X-Spam-Level: **
X-Spam-Status: No, score=2.1 required=5.0 tests=BAYES_60,
HTML_FONT_LOW_CONTRAST,HTML_MESSAGE,RCVD_IN_PBL,RDNS_NONE autolearn=no
version=3.2.5
X-Original-To: accounts@neubreed.com.au
Delivered-To: accounts-neubreed.com@julius.neubreed.com
Received: by julius.neubreed.com (Postfix)
id 851F1238A0EB; Thu, 20 Jan 2011 14:41:10 +1100 (EST)
Delivered-To: accounts-neubreed.com.au@julius.neubreed.com
Received: from [190.87.92.187] (unknown [190.87.92.187])
by julius.neubreed.com (Postfix) with ESMTP id B5C4E238A0E9;
Thu, 20 Jan 2011 14:41:04 +1100 (EST)
Received: from (unknown [190.87.92.187]) by blade07 with smtp
id 4662_da5a_2cb0e054_2447_11e0_8010_00237d22fdf8;
Wed, 19 Jan 2011 21:41:39 -0600
Received: from snt0-omc1-s33.snt0.hotmail.com (snt0-omc1-s33.snt0.hotmail.com [65.55.90.136])
         by mailin.isoc.net;
        Wed, 19 Jan 2011 21:41:39 -0600
Received: from SNT101-W38 ([65.55.90.136]) by snt0-omc1-s33.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
Wed, 19 Jan 2011 21:41:39 -0600
Message-ID: <SNT101-W388BD3AE1DA75FFFFF8B675F8BD@phx.gbl>
Content-Type: multipart/alternative;
boundary="_bc81389b-6472-206e-3813-a555d9722206_"
X-Originating-IP: [190.87.92.187]
From: ELINORE HIRSCH <ELINOREHIRSCH@hotmail.com>
To: <accounts@neubreed.com.au>
Subject: The Hottest Job Offer you just can't miss!. accounts@neubreed.com.au
Date: Wed, 19 Jan 2011 21:41:39 -0600
Importance: Normal
MIME-Version: 1.0
X-Loop: support@neubreed.com

If I remove the spamassassin headers and run this from the command line I get a higher score.

Submitted by andreychek on Wed, 01/19/2011 - 22:05

Can you show the headers you see when running SpamAssassin against that email from the command line? That would allow us to compare the two, and we can get to the bottom of what's going on... thanks!

Submitted by RyanJohnson on Wed, 01/19/2011 - 22:24

Here is the output from the command line:

Return-Path: <JudeVangerbig@isoc.net>
X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on julius.neubreed.com
X-Spam-Level: ***********
X-Spam-Status: Yes, score=11.2 required=5.0 tests=HTML_FONT_LOW_CONTRAST,
        HTML_MESSAGE,RCVD_IN_PBL,RDNS_NONE,SPF_FAIL autolearn=no version=3.2.5
X-Spam-Report:
        *  0.5 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL
        *      [190.87.92.187 listed in zen.spamhaus.org]
        *   10 SPF_FAIL SPF: sender does not match SPF record (fail)
        *      [SPF failed: Please see http://www.openspf.org/Why?s=mfrom;id=judevangerbig%40isoc.net;ip=190.87.92.187;r=julius.neubreed.com]
        *  0.0 HTML_MESSAGE BODY: HTML included in message
        *  0.5 HTML_FONT_LOW_CONTRAST BODY: HTML font color similar to background
        *  0.1 RDNS_NONE Delivered to trusted network by a host with no rDNS
Delivered-To: accounts-neubreed.com@julius.neubreed.com
Received: by julius.neubreed.com (Postfix)
        id 851F1238A0EB; Thu, 20 Jan 2011 14:41:10 +1100 (EST)
Delivered-To: accounts-neubreed.com.au@julius.neubreed.com
Received: from [190.87.92.187] (unknown [190.87.92.187])
        by julius.neubreed.com (Postfix) with ESMTP id B5C4E238A0E9;
        Thu, 20 Jan 2011 14:41:04 +1100 (EST)
Received: from (unknown [190.87.92.187]) by blade07 with smtp
         id 4662_da5a_2cb0e054_2447_11e0_8010_00237d22fdf8;
        Wed, 19 Jan 2011 21:41:39 -0600
Received: from snt0-omc1-s33.snt0.hotmail.com (snt0-omc1-s33.snt0.hotmail.com [65.55.90.136])
         by mailin.isoc.net;
        Wed, 19 Jan 2011 21:41:39 -0600
Received: from SNT101-W38 ([65.55.90.136]) by snt0-omc1-s33.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
         Wed, 19 Jan 2011 21:41:39 -0600
Message-ID: <SNT101-W388BD3AE1DA75FFFFF8B675F8BD@phx.gbl>
Content-Type: multipart/alternative;
        boundary="_bc81389b-6472-206e-3813-a555d9722206_"
X-Originating-IP: [190.87.92.187]
From: ELINORE HIRSCH <ELINOREHIRSCH@hotmail.com>
To: <accounts@neubreed.com.au>
Subject: [SPAM] The Hottest Job Offer you just can't miss!. accounts@neubreed.com.au
Date: Wed, 19 Jan 2011 21:41:39 -0600
Importance: Normal
MIME-Version: 1.0
X-Loop: support@neubreed.com
X-Spam-Prev-Subject: The Hottest Job Offer you just can't miss!. accounts@neubreed.com.au

Submitted by RyanJohnson on Wed, 01/19/2011 - 22:44

Looks like the SPF is failing from the command line giving it a higher score.

Submitted by andreychek on Wed, 01/19/2011 - 22:58

Thanks -- as you mentioned, it looks when you run spamassassin from the command line, an SPF rule is being triggered.

That may be due to different settings being used with the two different ways of running SpamAssassin.

CentOS has a local.cf file for SpamAssassin configuration... I believe it's located in /etc/mail/spamassassin/local.cf. Can you attach that to this request? That may contain some clues as to what's going on.

Submitted by RyanJohnson on Wed, 01/19/2011 - 23:21

It wouldn't let me attach it but here's the contents of that file. I also noticed that RBL checks were on the command line test

# These values can be overridden by editing ~/.spamassassin/user_prefs.cf
# (see spamassassin(1) for details)

# These should be safe assumptions and allow for simple visual sifting
# without risking lost emails.

required_hits 5
report_safe 0
rewrite_header Subject [SPAM]
use_bayes 1
use_auto_whitelist 1

whitelist_from *@facebookmail.com

#score FH_DATE_PAST_20XX 0.0

skip_rbl_checks 0
use_razor2 1
use_pyzor 0

dns_available yes

## Optional Score Increases
score DCC_CHECK 4.000
score SPF_FAIL 10.000
score SPF_HELO_FAIL 10.000
score RAZOR2_CHECK 2.500
score BAYES_99 4.300
score BAYES_95 3.500
score BAYES_80 3.000

Submitted by andreychek on Fri, 01/21/2011 - 09:34

Hrm, none of those appear to be attempting to disable the SPF checks.

The RBL checks do seem to be running in both cases, as both emails had "RCVD_IN_PBL RBL" set, which is a Spamhaus RBL.

What about in your user's home directory -- the file "$HOME/.spamassassin/user_prefs.cf" may contain some preferences that were used when run from the command line. Can you paste those in?

Submitted by RyanJohnson on Fri, 01/21/2011 - 15:10

Thanks for getting back to me. The prefs file appears to be commented out:

# SpamAssassin user preferences file.  See 'perldoc Mail::SpamAssassin::Conf'
# for details of what can be tweaked.
###########################################################################

# How many points before a mail is considered spam.
# required_score                4

# Whitelist and blacklist addresses are now file-glob-style patterns, so
# "friend@somewhere.com", "*@isp.com", or "*.domain.net" will all work.
# whitelist_from        someone@somewhere.com

# Add your own customised scores for some tests below.  The default scores are
# read from the installed spamassassin rules files, but you can override them
# here.  To see the list of tests and their default scores, go to
# http://spamassassin.apache.org/tests.html .
#
# score SYMBOLIC_TEST_NAME n.nn

# Speakers of Asian languages, like Chinese, Japanese and Korean, will almost
# definitely want to uncomment the following lines.  They will switch off some
# rules that detect 8-bit characters, which commonly trigger on mails using CJK
# character sets, or that assume a western-style charset is in use.
#
# score HTML_COMMENT_8BITS      0
# score UPPERCASE_25_50         0
# score UPPERCASE_50_75         0
# score UPPERCASE_75_100        0

Submitted by andreychek on Fri, 01/21/2011 - 23:09

Hrm, as I ponder what might be going awry here -- I wanted to make sure that emails delivered to your account were indeed using the system-wide local.cf file.

If you look in System Settings -> Spam and Virus Scanning, which option is "SpamAssassin client program" set to?

Submitted by andreychek on Fri, 01/21/2011 - 23:12

Also, regarding your issue you mentioned here --

I've had an influx of spam over the past week

While we're working through some of these other issues, something you may want to look into in the meantime... Greylisting can make a big impact in how much spam is making it through to your users.

You can enable greylisting by going into Email Messages -> Email Greylisting.

Submitted by RyanJohnson on Sat, 01/22/2011 - 14:57

I can't seem to be able to find the two areas you've mentioned in Virtualmin. There's no item: System Settings -> Spam and Virus Scanning or Email Messages -> Email Greylisting

Can you provide me with more instructions to find these settings?

Thanks, Ryan

Submitted by andreychek on Sat, 01/22/2011 - 19:37

Sorry, you're right, "Spam and Virus Scanning" should be in the "Email Settings" menu, which is on the left-hand navigation bar.

However, there should be an "Email Greylisting" option within "Email Settings", you don't see that on your system?

Submitted by RyanJohnson on Sun, 01/23/2011 - 16:21

Hey,

Thanks for that, I found it in "Email Messages" and I have enabled Grey listing,

The Spamassassin client program is spamc

Submitted by RyanJohnson on Mon, 02/14/2011 - 20:23

Hi Andrey,

We are still having a problem with the email score differentiation. Have you been able to find out more?

Submitted by andreychek on Wed, 02/16/2011 - 09:50

I don't see anything in your config that immediately strikes me as a problem. I'm emailing Jamie to see if he has any thoughts as to what might be causing the symptoms you're seeing, maybe he'll see something that I missed :-)

Submitted by andreychek on Wed, 02/16/2011 - 22:03

Hi Ryan -- here's a thing to check.

Do you have the file "/etc/mail/spamassassin/init.pre"?

In it, you should see the following line:

loadplugin Mail::SpamAssassin::Plugin::SPF

It shouldn't be commented out though, it should look similar to the above.

Submitted by RyanJohnson on Wed, 02/16/2011 - 22:38

Hey there . yes that's enabled.

Here is a new strain that's just started this week. It has multiple sender addresses yet a score of only 1.9

Return-Path: <paulwynn@graytech.com.au>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on julius.neubreed.com
X-Spam-Level: *
X-Spam-Status: No, score=1.9 required=5.0 tests=AWL,BAYES_99,HTML_MESSAGE,
MIME_HTML_ONLY,RCVD_IN_PBL,RDNS_NONE,SPF_NEUTRAL autolearn=no version=3.2.5
X-Original-To: ryan@neubreed.com.au
Delivered-To: ryan-neubreed.com.au@julius.neubreed.com
Received: from [151.60.142.241] (unknown [151.60.142.241])
by julius.neubreed.com (Postfix) with ESMTP id B53572388FB0;
Wed, 16 Feb 2011 09:25:32 +1100 (EST)
From: <ashish@neubreed.com.au>,
<directly@neubreed.com.au>,
<ryan@neubreed.com.au>,
<ryand@neubreed.com.au>,
<ryanjohnson@neubreed.com.au>
Cc: <saavedra@neubreed.com.au>
To: <ashish@neubreed.com.au>,
<directly@neubreed.com.au>,
<ryan@neubreed.com.au>,
<ryand@neubreed.com.au>,
<ryanjohnson@neubreed.com.au>
Cc: <saavedra@neubreed.com.au>
Subject: A new job will bring you to success! ...and wealth.
Mime-Version: 1.0
Content-type: text/html; charset="utf-8"
Content-Transfer-Encoding: 7bit
Message-Id: <20110215222533.B53572388FB0@julius.neubreed.com>
Date: Wed, 16 Feb 2011 09:25:32 +1100 (EST)
X-Loop: support@neubreed.com

Submitted by andreychek on Thu, 02/17/2011 - 22:31

Yeah, sometimes you'll need to do some tweaking to get SpamAssassin to be more effective.

If the scores for a given spam message aren't as high as you'd like, there's always the option of adding in a custom rule based on text you're seeing in the email. You can setup custom rules by going into Webmin -> Servers -> SpamAssassin -> Header and Body Tests.

Now, in your case with this most recent email, it looks like the Auto-Whitelist (AWL) was tripped by one of the addresses it saw in there.

To prevent that from happening, you may want to disable the use of the Auto-Whitelist by adding this to your /etc/mail/spamassassin/local.cf file:

use_auto_whitelist 0

After doing that, you'd want to restart SpamAssassin, and that setting should then take effect.

Now, as far as the original problem with the SPF test not triggering when emails come through -- I honestly have no idea what might be causing that :-)

If you'd like, I can look into that further by logging into your system and running some tests against the sample email you have there

Submitted by RyanJohnson on Mon, 03/07/2011 - 22:13

Hi .. I'm having a problem with the Greylisting .. there seems to be a 14-20 minute delay in emails geting through to recipients. Looking at the logs everything gets rejected first time. Is there something I've set wrong?

Submitted by andreychek on Mon, 03/07/2011 - 22:22

Howdy -- that's actually how greylisting works. The first email your server receives from a given user will take roughly 20 extra minutes.

It takes longer to receive that first email, but in the process, much spam is prevented.

The idea is that all incoming emails are temporarily rejected. A typical mail server will resend the email in roughly 20 minutes. Whereas, a spam program will frequently not attempt to re-send the messages.

Once an email makes it through the greylisting, that sender will then be whitelisted.

You can read more about the details of how greylisting works here:

http://en.wikipedia.org/wiki/Greylisting

Submitted by RyanJohnson on Wed, 03/09/2011 - 16:28

Sometimes a client needs to send me an email immediately or if someone signs up for an account and needs to click a confirmation link ad 20minute delay is not at all practical.

How to I set postgrey to a different delay setting to say 5-10 minutes?

Submitted by JamieCameron on Wed, 03/09/2011 - 16:41

You can edit /etc/init.d/postgrey and change the postgrey command line, by adding --delay=60 to reduce the delay to 60 seconds.

Submitted by RyanJohnson on Wed, 03/09/2011 - 16:58

Thanks for that. Where in this file is that argument meant to be placed

Submitted by RyanJohnson on Wed, 03/09/2011 - 17:02

It's ok I found it next to OPRIONS=

Submitted by andreychek on Wed, 03/09/2011 - 17:02

To clarify though -- Jamie's suggestion will simply allow the emails to arrive quicker. It's up to the sending server as to how frequently it retries the email delivery.

Unless the defaults on the sending servers are changed, it would likely still take 20 minutes for the email to arrive, since most servers retry once every 20 minutes by default.

In general -- the only way to prevent delays is to not use greylisting. But that means more spam comes through :-)