Best way to reduce load on box

Submitted by rjrsinc on Wed, 10/24/2012 - 09:09 Pro Licensee

In our eternal process of moving sites to a new powerful server, we are plagued with extremely high CPU usage on the old server which is primarily due to email. What measures can we take to help reduce the overall load. Since we are "splitting" the web site function to a new server, most of the processing is email (spamassassin, clamAV, lookup-domain, spamd, etc). Miniserv.pl also appears to consume a huge load especially if more than one person is connected.

Status: 
Closed (fixed)

Comments

Submitted by andreychek on Wed, 10/24/2012 - 09:47

Howdy -- yeah, depending on the volume of email you're receiving, receiving email really can use a lot of CPU.

You'd want to make sure you were using the spamd and ClamAV daemon processes, and not launching a new standalone process for each incoming email. That's configurable in Email Messages -> Spam and Virus Scanning.

You could also move some of the processing to another server.

It sounds like the server you have there is already primarily doing email processing, but if the load is still too high, you could always try splitting things up even further... maybe putting SpamAssassin on it's own system.

Also, how many emails are in the email queue now? You can determine that by running this command:

mailq | tail -1

Submitted by rjrsinc on Wed, 10/24/2012 - 10:01 Pro Licensee

The system actually does process the email and clears out the queue. So right now there are only about 9 email messages. I run ATOP all day and when we get a surge of email to a couple hundred users, the Idle goes to 0% and can stay there for about a minute or so before starting to alleviate. It then goes back up to ~300% - 360% idle. The big killer appears to be lookup-domain and spamd. I have monit running as well and that will email me that apache is timing our and postfix when there is high util for extended periods.

I really think the spam processing is causing the biggest load, especially when you have a couple hundred emails coming in at a time. Occasionally, I will also see users running usermin for email hogging multiple connections and high CPU, although that is not as consistent.

Is there a way to have the email pass through a separate scanning server before delivering to the main email server?

Submitted by andreychek on Wed, 10/24/2012 - 10:18

Well, as far as what Virtualmin can assist with -- Virtualmin supports having a separate server for SpamAssassin scanning. So when an email arrives on your server, it can be scanned on a remove server.

That can take a significant amount of the load from running SpamAssassin, and pass that to another server.

Details for setting that up are in the "Moving Spam and Virus Scanning to Another System" section of this document:

http://www.virtualmin.com/documentation/email/spam-av

Submitted by rjrsinc on Wed, 10/24/2012 - 13:13 Pro Licensee

Great. I'll reference that information. Also, I've been seeing multiple instances of lookup-domain and very high CPU util on a number of them (65%, 70%, etc). Is this normal when there is high email incoming?

Submitted by andreychek on Wed, 10/24/2012 - 14:21

Yeah, that's not abnormal.

Lookups normally happen fast enough that it's not noticeable... but as the system load goes up, the amount of time it takes lookup-domain to process a given request would go up, and the longer it would appear in the "top" output.

If you moved SpamAssassin to a different server, the amount of time you see lookup-domain running would likely go down significantly.

That said, you're still likely to see some load dealing with that sort of email volume... it would just be less with SpamAssassin processing on another system.

Submitted by rjrsinc on Thu, 10/25/2012 - 10:13 Pro Licensee

One more question on this before I dive into it. The current server is running Ubuntu 8.04 and the new one will be running Ubuntu 12.04. I also have a rather long deny and allow list on the originating server. 1) Will it work properly with different versions of Ubuntu? 2) Do I (and how) move the existing allow/deny lists to the remote server? 3) When needing to update the Deny/allow lists, is that then done on the remote server?

Submitted by andreychek on Thu, 10/25/2012 - 10:25

The version of Ubuntu won't matter.

However, if you're using SpamAssassin on a remote server, you'd no longer be able to use Webmin/Virtualmin to modify the allow/deny list (unless you were to install Webmin on the remote server, and access that Webmin installation to modify the config).

You should see the various changes you've made in this file here:

/etc/spamassassin/local.cf

You may want to verify that they are in there... and if so, all you should have to do is copy that file to the other server.

And then, for making future changes -- you could either edit that file manually, or you could install Webmin and make changes via the UI.

Submitted by rjrsinc on Thu, 10/25/2012 - 11:24 Pro Licensee

Fantastic. I will edit it the old fashioned way. I used to have to edit an access file on an old RedHat server many years ago and restart the daemon. Thank you. Once I start the process, I may have more questions. I'll leave this open until I have completed the process.

Submitted by rjrsinc on Sat, 10/27/2012 - 14:16 Pro Licensee

Question on the external spam/AV system. If I need to reboot the external Spam/AV server (or take it offline for a short period), what procedure would I need to do so that it does not have a negative impact on the primary mail server? I need to know the steps to ensure email flowing if that external server is down for any reason. Thanks

Submitted by JamieCameron on Sat, 10/27/2012 - 19:20

In that situation where the spamd server is down, no spam filtering will be done - however, email will still be delivered.

Submitted by rjrsinc on Mon, 10/29/2012 - 15:13 Pro Licensee

Fantastic. as long as email will still flow. Thanks.