Submitted by rjrsinc on Mon, 04/23/2012 - 09:04 Pro Licensee
We have a client that has about 130 email accounts and then tend to send a blast to all of them at various times during the week. Since we know the originating IP and or account, is there a way to have these broadcast emails bypass lookup-domain and/or spamassassin/clamav? We're trying to find a way to resolve this huge spike in load (as much at 60-80 on a 4 processor system - yikes!).
Status:
Closed (fixed)
Comments
Submitted by andreychek on Mon, 04/23/2012 - 09:35 Comment #1
Howdy -- SpamAssassin, ClamAV, and lookup-domain only run on incoming email, not outgoing email.
So any increase in resource usage you're seeing during that time from SpamAssassin/ClamAV/lookup-domain isn't because of the emails going out, it's due to incoming email -- perhaps bounce messages and replies.
The only way to whitelist them would be to disable SpamAssassin and ClamAV entirely for incoming email for that particular domain. But I suspect that'd be undesirable.
As for ways to improve performance, I might suggest the following (some of which you're doing already):
I would suggest asking users to send newsletters only after business hours
If a newsletter is causing problems, you may be able to configure the software that sends the news letter to use an alternate SMTP server for sending them. Whether or not you can do that depends on the newsletter software, but I know phpList can do that.
You may also want to look into making that newsletter send out emails in batches, rather than all at once. For example, you may be able to have it send 100 messages ever 5 minutes over the course of several hours, rather than 10,000 messages at once.
I know you're looking into this already, but you may want to put SpamAssassin and ClamAV onto another server, that should reduce the workload of your current server.
Submitted by rjrsinc on Mon, 04/23/2012 - 09:40 Pro Licensee Comment #2
I'm sorry. I wasn't clear with my question. The issue isn't with sending, it's with receiving. The 130 or so email accounts are on our mail server, so when the blast comes in to them, lookup-domain, spamd shoot up and basically lock out all other processes. So, I want to know if there is a way to bypass these processes when this blast comes into our server for all these accounts.
Submitted by andreychek on Mon, 04/23/2012 - 10:10 Comment #3
It's unfortunately not possible to selectively bypass SpamAssassin... it's either enabled for a Virtual Server, or disabled for that Virtual Server.
The suggestions I mentioned above may still assist here though.
The only other thing that could assist would be to create a new domain to use for this email blast, and within that particular domain, disable SpamAssassin for just it.
To disable SpamAssassin, you can go into Edit Virtual Server -> Enabled Features, and disable the Spam Filtering feature.
It's not possible to bypass the lookup-domain service, that's necessary to determine what Virtual Server the emails belong to.
Submitted by rjrsinc on Mon, 04/23/2012 - 15:01 Pro Licensee Comment #4
I like the idea of creating another virtual server for the use of the internal email blast. The question would be hot to route the email to their actual email account in the parent domain. Is there a way to so that? We are trying anything while we are waiting for a new server to allow us to set up separate mail and LAMP servers to handle the traffic better.
Submitted by andreychek on Mon, 04/23/2012 - 17:40 Comment #5
Well, one option would just be to have your users access this account separately from their current account.
Another option would be to setup an Email Alias in this new Virtual Server that is the return address for your email blast.
Then, when configuring the Email Alias -- set "Deliver locally" to "Yes", and list the name of the username (ie, the mailbox) that the email should be delivered to.
That should deliver directly to the mailbox, and not actually forward the email, which would cause it to be scanned by SpamAssassin.
Submitted by rjrsinc on Tue, 04/24/2012 - 12:51 Pro Licensee Comment #6
Here's a question for you regarding this topic, this client has 130 email/users in their domain on our server. Since we are looking to bypass spam and AV scannig for them to temporarily reduce the load on the server (we're deploying a separate server for spam/clam processing as permanent fix); I noticed under Server Configuration -> Spam and Virus Delivery there is a setting to "Always allow mail from mailboxes in domain?"
If the sender on this domain uses Outlook Express and has the SMTP set to this mail server and we select YES for the above option, would the email to users bypass being scanned by spamassassin, thus reducing load and processing? I'm trying anything to temporarily reduce load since until the new server is fully tested. Thanks
Submitted by andreychek on Tue, 04/24/2012 - 22:02 Comment #7
That option sets a SpamAssassin option, I believe adding the addresses to a whitelist.
It wouldn't actually prevent SpamAssassin from running though, and I believe it would even continue to run all of the available SpamAssassin tests. It's just that, with a particular address whitelisted, it would drastically reduce the score, making it always look like ham.
Submitted by rjrsinc on Tue, 04/24/2012 - 22:11 Pro Licensee Comment #8
Thanks. I'm running tests on our server and trying to reduce the overall load that lookup-domain and spamassasson. I already removed ClamAV for all virtual servers and then disabled form the install. server has Quad Core CPU, 10GB RAM, RAID 5 SATA II running under VMWare 3.51. I'm still battling with our server hitting 100+load and o% idle when 100 emails come into the system for any one of our virtual servers. Lookup-domain launches 100+ processes and the system basically becomes inoperable and I have to restart Postfix to have it try to get back to normal. I can easily reproduce the results. Is there any way to find out why this is happening? The server is not light weight.
That's weird. 100 emails shouldn't crush a system that size if the mail processing tools are daemonized; I've run thousands of mailboxes on a smaller server, including SpamAssassin and ClamAV. Is SpamAssassin running the spamc version, or do you have it configured to not run as a daemon?
Are you sure you've really disabled ClamAV and not just turned off the daemon? If you switched it to the non-daemon mode it would have to create a process for every email, and ClamAV is a massive resource hog.
I'm feeling like something is broken in this configuration for this to even be a noticeable spike in load on this big of a server, even with both clam and SA in the chain. Maybe something pathological is going on in our lookup-domain daemon that I'm not aware of. But, check to be sure you're configured to use the daemon version of SpamAssassin.
Submitted by rjrsinc on Wed, 04/25/2012 - 11:48 Pro Licensee Comment #10
I'm going to check through the system again to make sure ClamAV is not running, but I'm seeing both Spamd and Spamc in top when email comes in. Is it possible that there are duplicative processes hotting? Also, the process I ran to turn off ClamAV was from a different open issue ticket. It was a command prompt to disable it for all virtual domains and then to disable it in the settings area of Vm. That was the process.
Submitted by rjrsinc on Wed, 04/25/2012 - 12:00 Pro Licensee Comment #11
I see spamd come up and utilizing anywhere from 5% cpu to as much as 80% cpu. I did see some spamc pop up occasionally. So, is there a way to determine and make sure only the daemon is running?
Submitted by rjrsinc on Wed, 05/09/2012 - 08:55 Pro Licensee Comment #12
We are deploying a new server. Will issue a new post.