Support for SpamAssassin user_prefs stored in LDAP

18 posts / 0 new
Last post
#1 Tue, 08/28/2007 - 18:59
TonyShadwick

Support for SpamAssassin user_prefs stored in LDAP

SpamAssassin supports reading user_prefs out of LDAP. Would be nice for the Usermin/Webmin modules to support it as well. :) Can you tell I'm pushing hard for LDAP support? ;)

Wed, 08/29/2007 - 20:41
Joe
Joe's picture

<div class='quote'>Can you tell I'm pushing hard for LDAP support?</div>

You do it so well, we hardly noticed. ;-)

Seriously, I think LDAP is great for large-scale hosting...and we're happy to keep pushing the support forward one piece at a time. This is a reasonable request--not even really Blue Skies. I'll file an RFE in the tracker, and we'll see what Jamie thinks.

--

Check out the forum guidelines!

Wed, 08/29/2007 - 21:00
Joe
Joe's picture

Hi Tony,
Do you have a pointer to some docs on getting spamassassin / LDAP integration setup?
Actually, I should also add support for using MySQL to store settings too while I am at it..

--

Check out the forum guidelines!

Thu, 08/30/2007 - 20:27
TonyShadwick

http://search.cpan.org/src/FELICITY/Mail-SpamAssassin-3.0.0/ldap/README

The short of it is that the admin needs to add the SpamAssassin schema to ldap, they provide you with connection details, and you use the SpamAssassin module to push out the info using those connection details. In fact, it's a safe bet that the user has LDAP Users and Groups configured properly prior to this step, so you could just do a cursory check that they have the SpamAssassin schema set up, if not, ask them if it's okay to do so, do it, and use those connection details for SA.

MySQL isn't so different.

http://search.cpan.org/src/FELICITY/Mail-SpamAssassin-3.0.0/sql/README

It wants you to load some .sql files (pretty standard), it wants connection details, and the module does teh rest for you.

Sun, 10/07/2007 - 12:32
Joe
Joe's picture

Hi Tony,

I've just started work on implementing MySQL and LDAP storage support in the SpamAssassin modules for Webmin and Usermin. If you are interested in getting a beta version to play with, let me know..

--

Check out the forum guidelines!

Mon, 10/08/2007 - 09:33 (Reply to #5)
TonyShadwick

I am. I am. :)

Tue, 10/09/2007 - 08:45
Joe
Joe's picture

Hi Tony,

I have just released Usermin development version 1.303, available from http://www.webmin.com/devel.html . Once you install it, you can configure the SpamAssassin module to use an LDAP database for showing and storing parameters (at Webmin -&gt; Usermin Configuration -&gt; Usermin Module Configuration -&gt; SpamAssassin Mail Filter).

Anyway, check it out and let me know how it goes for you. I'm not 100% sure that it uses the correct format for the 'spamassassin' LDAP attribute.

--

Check out the forum guidelines!

Wed, 10/10/2007 - 06:40 (Reply to #7)
TonyShadwick

I'll check it. I also need to make sure Virtualmin uses/respects the changes too. :)

Wed, 10/10/2007 - 09:00 (Reply to #8)
Joe
Joe's picture

There isn't really anything needed on the Virtualmin side - all it needs to do is call spamc, which is already possible. Let me know if you find anything I am missing though..

--

Check out the forum guidelines!

Sat, 06/07/2008 - 14:31
ctucker50

Along these same lines, I have an existing server using amavisd-new, spamd, and LDAP to store the custom spamassassin settings on a per-user basis. Amavisd knows how to read LDAP to retrieve the spam setting for a user. This setup handles mail many times faster than the procmail setup on Virtualmin Pro and I would like to move this configuration to the Virtualmin Pro box but don't see a way to do this with the current modules. Does anybody have something started or should create a custom module? I suppose I can leverage the existing spamassassin module into an amavisd/spamd module but would rather start further ahead that what is now there.

Chris

Sat, 06/07/2008 - 14:48 (Reply to #10)
Joe
Joe's picture

You can configure Usermin's spamassasin module to use LDAP as a data store for preferences, and virtualmin can store users in ldap. You could completely disable its spamassasin integration, and use your approach, which should work fine.

Let us know if you run into any problems though.

--

Check out the forum guidelines!

Sat, 06/07/2008 - 14:51 (Reply to #11)
Joe
Joe's picture

<div class='quote'>This setup handles mail many times faster than the procmail setup on Virtualmin Pro</div>

I very seriously doubt it. Maybe &quot;a few percent faster&quot;, but many times faster means you can't be running spamassassin or clamav, as that's where we all spend about 98% of our mail processing time. You can disable spamassassin and clamav in Virtualmin's procmail configuration, if you just want to deliver mail really freakin' fast. Arguing amavisd-new vs. procmail is like arguing over whether the red Ferrari is faster than the identically specced blue Ferrari. ;-)

In their daemonized form, spamassassin and clamav under procmail are going to be roughly identical in performance to calling them from amavisd-new.

So, you're almost certainly not gaining any performance at all by using amavisd-new over procmail. The slow parts are the spam and AV scanning, and lookup-domain.pl (which isn't all that slow, as it has a daemon of its own). LDAP lookups are possibly a little faster than lookup-domain.pl, depending on a few factors. But, if performance is the only benefit you see in sticking with your current configuration, I'll assure you that it's not worth the trouble. If there are other reasons, of course, I'm all for folks developing tools to manage it! (And at least you aren't trying to use MySQL for mail users, which is just silly. LDAP has some good and valid uses in a mail system.) ;-)

Also, if you're having trouble getting switched over to the daemonized form of SpamAssassin and ClamAV under Virtualmin Professional's procmail configuration...let us know. We'll be happy to help.

BTW-Virtualmin GPL 3.58, coming out probably Sunday or Monday, is getting the spam and AV configuration tools from Virtualmin Professional. So, hopefully, the proliferation of different (and often complex to migrate from) methods of processing email will converge down to one procmail solution. And the larger user base (GPL has tens of thousands of installations vs. 1500 or so Virtualmin Professional installations) should help us refine it so that it answers a wider range of user requirements.

--

Check out the forum guidelines!

Sat, 06/07/2008 - 15:26
ctucker50

Hmmm ... yes many times faster. I test with sending 100 emails to the Virtualmin Pro box (with attachments) and it takes about 3.5 minutes to process them all while the current amavisd server processes them in under 30 seconds. That is also with a normal mail load since it is a live mail server while the Virtualmin Pro is in testing and setup.

I am positive that the performance problem is with procmail and spamassassin/clamscan but I don't have them in daemonized form. I am new to procmail so I will take a look and see how to do that since it may solve my problem and I would rather just use the tools available rather than write new ones. Can you point to any quick doc on modifying procmail to use a daemonized form of spamassassin and clamav?

Sat, 06/07/2008 - 15:48 (Reply to #13)
Joe
Joe's picture

<div class='quote'>Hmmm ... yes many times faster. I test with sending 100 emails to the Virtualmin Pro box (with attachments) and it takes about 3.5 minutes to process them all while the current amavisd server processes them in under 30 seconds.</div>

Yes, the Virtualmin procmail rules are running spamassassin and clamav are running non-daemonized on your system--they have to start up for each message. (And clamav takes <i>forever</i> to start.)

You just need to switch to clamdscan and spamc. You don't have to know anything about procmail--pretend like procmail doesn't even exist for this exercise--just start the services and tell Virtualmin that's the way you want to process mail. ;-)

Do the following few steps and run your benchmark again:

<b>Start the daemons</b>

Start the spamassassin service on your system. Depending on your OS this can probably be done easily in the Webmin:System:Bootup and Shutdown module. Just find the spamassassin or spamd service and start it, and set it to start on boot.

If you're running Debian/Ubuntu, enable clamav-daemon while you're there. If you're on a Red Hat based system using our clamav packages, the service won't exist yet so continue on to the next step.

<b>Tell Virtualmin to use them</b>

Back in the Virtualmin menu, open Email Messages:Spam and Virus Scanning

If you're on a Red Hat based system, click the &quot;Enable ClamAV Server&quot; button at the bottom of the page. This will create a clamd.virtualmin service and start it. Debian/Ubuntu users continue on to the next step, doing nothing here.

Set &quot;SpamAssassin client program&quot; to &quot;spamc (client for spamd server&quot;

Set &quot;Virus scanning program&quot; to &quot;Server scanner (clamdscan)&quot;

Save it.

Done.

Now you should find that Virtualmin's procmail configuration takes about the same amount of time as your amavisd-new configuration.

I'll try to get this stuff turned on, by default, in the next version of the virtualmin-base script...it's a common misconception that procmail is slow, when in fact, procmail is faster than amavisd-new, <i>and</i>, far more importantly, it just doesn't matter because mail delivery is so little of the work required when processing mail--even if procmail were ten times slower than amavisd-new (which it isn't...it's faster) it would still only account for about a second of the mail delivery process. Performance of procmail and amavisd-new are simply not a factor you should consider in your plans, because the performance of either is dwarfed by the expense of processing mail for filtering viruses and spam.

If we want faster mail delivery, we have to figure out ways to make spamassassin and clamav faster (or make them work less...we plan to introduce a policy daemon soon that will bounce obvious spams earlier in the process and more cheaply, so they don't have to go through the more complex tests).

--

Check out the forum guidelines!

Sat, 06/07/2008 - 16:47
ctucker50

Well, if I had only known that I could do this with a few check boxes, I wouldn't have done most of it on the command line already ;-)

I am running CentOS5.1 but the option to create the clamd settings doesn't seem to show up. I already did this part anyway but I see that clamdscan cannot find the configuration file unless I symlink /etc/clamd.d/virtualmin.conf to /etc/clamd.conf (the option for --config-file is not in the Virtualmin screen).

After reconfiguring and testing the same email is processed in under 30 seconds so this does the trick, thank you.

Mon, 06/09/2008 - 09:25
ctucker50

The pieces are there and moving but it looks like lookup-domain.pl is not finding the users in LDAP. I am getting flooded with SPAM now and looking at the procmail.log, it appears that the user's procmail settings are not being run because they can't be found. I tried running lookup-domain.pl on the command line but it doesn't return anything. I must still be missing something.

procmail: Assigning &quot;TRAP=/etc/webmin/virtual-server/procmail-logger.pl&quot;
procmail: Executing &quot;/etc/webmin/virtual-server/lookup-domain.pl,user-domain.tld&quot;
procmail: Assigning &quot;VIRTUALMIN=&quot;
procmail: [16592] Mon Jun 9 11:09:58 2008
procmail: Executing &quot;/usr/bin/test,,!=,&quot;
procmail: [16592] Mon Jun 9 11:09:58 2008
procmail: Non-zero exitcode (1) from &quot;/usr/bin/test&quot;
procmail: No match on &quot;/usr/bin/test != &quot;

I changed the user.domain.tld but the actual user really exists in LDAP.

Mon, 06/09/2008 - 09:57 (Reply to #16)
Joe
Joe's picture

If you run the command :
/etc/webmin/virtual-server/lookup-domain.pl user-domain.tld &lt;/dev/null
does it output a domain ID?

If not, you may need to restart lookup-domain-daemon.pl, with the command :
/etc/init.d/lookup-domain restart

--

Check out the forum guidelines!

Mon, 06/09/2008 - 10:09
ctucker50

A restart of domain-lookup seems to have fixed this.

Topic locked