clamdscan instead of clamscan

16 posts / 0 new
Last post
#1 Mon, 06/26/2006 - 10:18
ScottAdministrator

clamdscan instead of clamscan

It appears that Virtualmin Pro launches a copy of clamav every time a message is scanned.

In launching "clamscan" instead of launching "clamdscan" creates a complete copy of the following for EVERY message that comes through:

-rw------- 1 user.group 17992 Jun 26 08:19 COPYING -rw------- 1 user.group 4745236 Jun 26 08:19 main.db -rw------- 1 user.group 1777 Jun 26 08:19 main.fp -rw------- 1 user.group 465075 Jun 26 08:19 main.hdb -rw------- 1 user.group 5627534 Jun 26 08:19 main.ndb -rw------- 1 user.group 293 Jun 26 08:19 main.zmd

du -h

11M

On a server with a heavy load, it doesn't make sense running "clamscan" instead of having the daemon "clamdscan" running.

Is there a way to reconfigure Virtualmin Pro to use clamdscan instead of clamscan?

Wed, 06/28/2006 - 23:54
Joe
Joe's picture

Hey Scott,

Yes, that is probably the direction we ought to go with it--it'll definitely have an improvement on performance (though SpamAssassin is still the biggest resource user on most Virtualmin systems, by far). It takes a bit more configuration to go there, but it's nothing major. Expect it to become the default in a very near version.

In the meantime, I believe you'd just need to fire up the clamd service (which might need a bit of configuration, I'm not sure):

service clamd start
chkconfig --level 345 clamd on

And then modify the procmail template that Virtualmin uses to call clamdscan rather than clamscan. Because command line options are mostly ignored by clamdscan, there may be some issues with our recent changes to where clam stores temp files, etc. I'm not sure off-hand. It'll definitely take some research to get it polished up for inclusion in the default installation.

I'll write up a FAQ with full docs when I actually get it working and tested.

--

Check out the forum guidelines!

Tue, 08/22/2006 - 15:54 (Reply to #2)
jpenix

Joe, you mention that SpamAssassin is still the biggest resource user, which is surely true... but it brings up a related question - why is Virtualmin not using spamc/spamd as well?

Thu, 06/29/2006 - 03:01
Joe
Joe's picture

Hey Scott,

While doing a bit of work on this issue, I noticed that there is an option in the Virtualmin Module Config named "Full path to clamscan command". So, assuming you have clamd running and configured, changing this to clamdscan is probably all that's needed. Give us a report if anything fails. ;-)

--

Check out the forum guidelines!

Tue, 08/22/2006 - 15:15 (Reply to #4)
ScottAdministrator

Well, that does work if you're NOT using FC4 and following their insturctions on how to use clamav-server package properly.

The "Full path to clamscan command" does not allow for configuration switches.

------

Here's the RH/FC4 approved method for setting up clamd set up per process (NOT FOR THE ENTIRE SERVER). This is relative for 2006-08-22 and you'll have to adjust some of the paths appropriately if you another version of clamav that's NOT 0.88.4.

To use "clamdscan", the command is "clamdscan --config-file /etc/clamd.d/CLAMSPAM.conf" and Virtualmin at this time does not allow the command switch.

Woud it be possible to fix the command that's checking for the presence of the command?

__clamdscan install with Virtualmin PRO on FC4__

useradd -c "CLAMSPAM" -d /var/empty -s /sbin/nologin clamspam

sed -e 's/^Example/#Example/'
-e 's/^#LogFile/LogFile/'
-e 's/^#LogTime/LogTime/'
-e 's/^#LogVerbose/LogVerbose/'
-e 's/^#TCPSocket/TCPSocket/'
-e 's/^#TCPAddr/TCPAddr/'
-e 's/^LocalSocket/#LocaLSocket/'
-e 's/^LogFileUnlock/#LogFileUnlock/'
-e 's/^LogFileMaxSize/#LogFileMaxSize/'
-e 's/<SERVICE>/CLAMSPAM/'
-e 's/<USER>/clamspam/'
/usr/share/doc/clamav-server-0.88.4/clamd.conf
> /etc/clamd.d/CLAMSPAM.conf

touch /var/log/clamd.CLAMSPAM
chgrp clamspam /var/log/clamd.CLAMSPAM
chmod 0620 /var/log/clamd.CLAMSPAM

sed -e 's/<SERVICE>/CLAMSPAM/'
/usr/share/doc/clamav-server-0.88.4/clamd.logrotate
> /etc/logrotate.d/clamd.logrotate

sed -e 's/<SERVICE>/CLAMSPAM/'
/usr/share/doc/clamav-server-0.88.4/clamd.sysconfig
> /etc/sysconfig/clamd.CLAMSPAM

sed -e 's/<SERVICE>/CLAMSPAM/'
/usr/share/doc/clamav-server-0.88.4/clamd.init
> /etc/init.d/clamd.CLAMSPAM

/sbin/chkconfig clamd.CLAMSPAM on

ln -s /usr/sbin/clamd /usr/sbin/clamd.CLAMSPAM

mkdir -p /var/run/clamd.CLAMSPAM
chown clamspam /var/run/clamd.CLAMSPAM

---------------------------------------

chmod a+x /etc/init.d/clamd.CLAMSPAM
chkconfig --add clamd.CLAMSPAM
chkconfig clamd.CLAMSPAM on

service clamd.CLAMSPAM start

---------------------------------------

Tue, 08/22/2006 - 19:37
Joe
Joe's picture

Hey guys,

Both good ideas, and it's kind of been simmering on the back burner for some time to make this switch for both clam and SpamAssassin. It's a reasonably easy change--we need to enable the daemons during install, make the changes to the defaults (I don't believe Virtualmin needs any changes--I don't think the config file option is required as long as it is in the default location). It's just something that hasn't bubbled to the top yet.

I also highly doubt we need a clamdscan user. There's already a clamav user.

So, Scott, have you tried just using "clamdscan" with no options, and using the clamav user rather than creating a new one? I suspect about 90% of the additional configuration you've specified is unnecessary. And if it is necessary, I'm strongly of the opinion that the Fedora Extras clam packages are poorly constructed. You should never have to do that much configuration, or create any users, for a service installed from RPM.

I'm in the midst of install.sh work now for Debian/Ubuntu, but I'll try to find some time to test spamc and clamdscan and get them added to the virtualmin-base configuration defaults (and extract out a little script to make the change to existing systems).

--

Check out the forum guidelines!

Wed, 08/23/2006 - 23:21 (Reply to #6)
jpenix

Thanks for the update Joe, sounds good. Know you're busy, but will look forward to the update. :) Happy to do any testing you need...

Tue, 08/29/2006 - 07:33
ScottAdministrator

Joe, from the 'rpm -qi clamav-server':
....
This package contains files which are needed to execute the clamd-daemon. This daemon does not provide a system-wide service. Instead of, an instance of this daemon should be started for each service requiring it.

See the README file how this can be done with a minimum of effort.

So the README says:
To create individual clamd-instance take the following files and
modify/copy them in the suggested way:

clamd.conf:
* set LogFile, PidFile, LocalSocket and User to suitable values
* place this file into /etc/clamd.d with an unique service-name;
e.g. as /etc/clamd.d/<SERVICE>.conf

To make logfile rotation work properly, the LogFile should be
writable for the assigned User. Recommended way to reach this, is
to:
* make it owned by the User's *group*
* assign at least 0620 (u+rw,g+w) permissions

A suitable command might be
| # touch[logfile>
| # chgrp[user>[logfile>
| # chmod 0620 [logfile>

NEVER use 'clamav' as the user since he can modify the database.
This is the user who is running the application; e.g. for mimedefang
(http://www.roaringpenguin.com/mimedefang), the user might be
'defang'.Theoretically, distinct users could be used, but it must be
made sure that the application-user can write into the socket-file,
and that the clamd-user can access the files asked by the
application to be checked.

clamd.logrotate:
* set the correct value for the logfile
* place it into /etc/logrotate.d

clamd.sysconfig:
* set the name of the config-file and the local socket
* copy it to /etc/sysconfig/clamd.<SERVICE>

clamd.init:
* set the service-name
* place it into /etc/init.d/ with an unique name and activate it
(e.g. with /sbin/chkconfig clamd.<SERVICE> on)

Additionally, a symlink must be set to clamd in a way like
| # ln -s clamd /usr/sbin/clamd.<SERVICE>
and the directory for the socket file must be created (see 'LocalSocket'
in clamd.conf)
| # mkdir -p /var/run/clamd.<SERVICE>

This directory must be writable by the 'User' chosen in the config-file.

[[Disclaimer:
this file and the script/configfiles are not part of the official
clamav package.

Please send complaints and comments to
mailto:enrico.scholz@informatik.tu-chemnitz.de!]]

.....
Which is exactly what my instructions do above -- they create a specific instance of clamav for a single purpose use instead of creating a system-wide clamav.

Is it me, or did I not follow the README and I thought I RTFM and understood the Fine-FM.

Tue, 08/29/2006 - 07:36
ScottAdministrator

What I'm really looking for, which it appears I may have to do it myself, is to extend the interface to allow for an extra field to append onto the command a configuration value. The current interface only allows for the executable without any additional configuration switches.

Would it be best that I send you the diffs for adding another field so that configuration switches can be added?

In that way, I won't have to mod the source after every update to Virtualmin Pro.

Sun, 06/07/2009 - 07:00
ScottAdministrator

Joe,

In /usr/libexec/webmin/virtual-server/virtual-server-lib.pl,
there is a test for the "clamscan_cmd" variable:

if ($config{'virus'}) {
# Make sure ClamAV is installed and working
&has_command($config{'clamscan_cmd'}) ||
return &amp;text('index_evirus', &quot;<i>$config{'clamscan_cmd'}</i>&quot;, $clink);
local $out = `$config{'clamscan_cmd'} -[/dev/null 2&gt;&amp;1`;
$? &amp;&amp; return &amp;text('index_evirusrun', &quot;<i>$config{'clamscan_cmd'}</i>&quot;, &quot;[pre]$out[/pre]&quot;, $clink);
&amp;$second_print($text{'check_virusok'});
}

Instead of sending 'clamscan_cmd' to &amp;has_command, couldn't it be split and only the first component of clamscan_cmd be sent to &amp;has_command?

like...

split(' ',$config{'clamscan_cmd'},2)
&amp;has_command($_[[0]]);

if ($config{'virus'}) {
# Make sure ClamAV is installed and working
split(' ',$config{'clamscan_cmd'},2)
&amp;has_command($_[[0]]) || [etc..etc..&gt;

----

sample test of code:

#!/usr/bin/perl
use strict;
use warnings;

my $string = 'clamdscan --config-file /etc/clamd.d/CLAMSPAM.conf';
@_ = split(' ',$string,2);
print $_[[0]];
print &quot;n&quot;;
exit 0;

Sun, 06/07/2009 - 07:00
ScottAdministrator

Joe,

In /usr/libexec/webmin/virtual-server/virtual-server-lib.pl,
there is a test for the &quot;clamscan_cmd&quot; variable:

if ($config{'virus'}) {
# Make sure ClamAV is installed and working
&amp;has_command($config{'clamscan_cmd'}) ||
return &amp;text('index_evirus', &quot;<i>$config{'clamscan_cmd'}</i>&quot;, $clink);
local $out = `$config{'clamscan_cmd'} -[/dev/null 2&gt;&amp;1`;
$? &amp;&amp; return &amp;text('index_evirusrun', &quot;<i>$config{'clamscan_cmd'}</i>&quot;, &quot;[pre]$out[/pre]&quot;, $clink);
&amp;$second_print($text{'check_virusok'});
}

Instead of sending 'clamscan_cmd' to &amp;has_command, couldn't it be split and only the first component of clamscan_cmd be sent to &amp;has_command?

like...

split(' ',$config{'clamscan_cmd'},2)
&amp;has_command($_[[0]]);

if ($config{'virus'}) {
# Make sure ClamAV is installed and working
split(' ',$config{'clamscan_cmd'},2)
&amp;has_command($_[[0]]) || [etc..etc..&gt;

----

sample test of code:

#!/usr/bin/perl
use strict;
use warnings;

my $string = 'clamdscan --config-file /etc/clamd.d/CLAMSPAM.conf';
@_ = split(' ',$string,2);
print $_[[0]];
print &quot;n&quot;;
exit 0;

Tue, 09/05/2006 - 12:06
Joe
Joe's picture

Hey Scott,

Excellent follow-up on this issue! ;-)

You've convinced me of the added complexity being necessary, and I'm working on an extra package that sets the clamd stuff up automagically. I'll ping Jamie on the code change to allow a more complex call to the clamscan command, and we can expect to see it, or something like it in version 3.22.

--

Check out the forum guidelines!

Tue, 09/05/2006 - 14:38
Joe
Joe's picture

I will allow extra arguments in that clamscan command field on the Module Config page in the next release of Virtualmin (which will be 3.22). That way you can have it call clamdscan, which should save some CPU.

It would be nice if Virtualmin could also use the daemonized version of SpamAssassin, but unfortunately there is no way to specify different configuration files when calling spamc .. which Virtualmin uses to implement different per-domain SpamAssassin settings.

--

Check out the forum guidelines!

Sat, 02/17/2007 - 06:08 (Reply to #13)
JohnFord

Hello Jamie,

I think there is a way and it might be pretty easy for you guys to figure out. It's MailScanner.

MailScanner somehow wraps Postfix into itself and the postfix daemon is not even running. it's part of MailScanner.

I had it installed before I did this fresh install with virtualmin pro. It's really cool... because MailScanner is setup to look for clamav and many other virus scanners and spam scanners[b&gt;automatically&lt;/b&gt;! It'll look for them, updates them regularly, and really appears to work great.

It takes a little studying but a chroot is setup and the mail gets dropped in a directory that MailScanner controls and checks it for virus and spam. Then it drops it into a chroot directory for Postfix to send to the right HOME directory for the user.

The issue with me was that the chroot for this process meant adding a /etc, /lib, - so that the proper TIME HOSTNAME et cetera were available to the chroot. That's not that big a deal, except the virtual domain part of it. It was impratical. But... further study showed that there is a way to extend these things and most especially the passwd file for the virtual domain user. It is the proxy:unix:passwd.byname construct. This was something that I could have done for one domain or two or more; but, I wanted it done for me automatically and decided to check out virtualmin because you guys already have been working on it.

I really like the features of MailScanner. They're great! And there's a lot of them that need no set up at all. There is a script the inventor of MailScanner wrote that installs MailScanner and clamav that needs no intervention, (I think it installs spamassasin too). It can be found on the MailScanner site. You could incorporate it into your install.sh, and then tweak the configuration to do the proxy construct for each virtual domain.

I had it going and I like it. There were only two issues that made me feel it was better to let it go and instead go with virtualmin. It used about 58mb of memory for each MailScanner child - i think this is because postfix is somehow meshed with it because the postfix daemon is not running but postfix is what is doing all the MTA. I told it to only use two childs which probably wasn't enough as the default is five. I was only using 500mb of ram but added another five so I have 1gb now.

The other thing was that I didn't want to set it up for every virtual domain. I was pretty sure I was going to take advantage of the power/feature of virtualmin anyway, so why not just go with virtualmin because you guys had spamassasin and clamav going anyways. Plus... maybe you might even get on the MailScanner bandwagon at some point.

I'm hoping after reading this post that you'll get on the MailScanner bandwagon. Again, it has a lot of this stuff done for you automatically. It knows virtually everything about all the spam reporting sites and virus scanner software. It will scan an email with about 20 different virus scanners if you had them all on your system. It works with bit-defender so you could have a virtual domain use MailScanner to check an email with both clamav and bit-defender - (which is what some of the experts that wrote up docs suggest- they actually suggest those two and one pay-for subscription one)

It's only setback is the constant virtual domain setup-stuff - but that is you guy's specialty! Also it uses a lot of memory. But, maybe I had something set up wrong. The docs say it should only use 20mb per child. But, it was an older doc so maybe it needs more now.

Anyway, if you looked into it I think you'd figure it out quickly and decide it does a bunch of your work for you and you can concentrate on just tying the virtual domain side of things together instead of the mechanics of calling from the command line, and watching queues, etc.

I hope this helps. I am thinking it will be a plus for all of us and save you some headache and masterminding.

I like the fact that it uses the chroot jail to decide if the mail is even going to get to stay on the server or not. Plus the guy is always making sure it knows how to talk to all the others in the industry that are trying to keep email clean and spam out.

Sat, 02/17/2007 - 06:27 (Reply to #14)
JohnFord

just to clarify, as i really didn't talk much about it.

I mentioned mailscanner somehow has postfix running without the postfix daemon ever being started at boot.

The whole post is about using the clam daemon and the spamd daemon - both are started and constantly running with mailscanner. so calls to both programs utilize the daemon method instead of the command line.

also... I mentioned that you guys were already working on this... what I meant was that virtualmin already used spamassasin and clamav and those two were important to me. so going with virtualmin made a lot of sense as virtualmin is all about making automatic setup/changes to virtual domains.

I do think MailScanner would be better than the current virtualmin setup, merely because it adds tons more features to the email process. If it can't easily be incorporated into it then that's just the way it is. spamassasin and clamav by themself provide abundant protection.

Sat, 02/17/2007 - 12:03 (Reply to #15)
Joe
Joe's picture

Hey John,

We won't be using MailScanner, but the issue is already solved in recent versions. This thread is pretty much dead. ;-)

You can configure Virtualmin to use spamc, instead of spamassassin, and I believe the default for clam is already to use the daemonized version (but I might be wrong...I'll look into it--but spamassassin is the real hog, so it is the more important one to get converted anyway, if performance is a problem).

We've also just daemonized lookup-domain.pl (our component in the mail processing chain), which will roll out in the next version of the virtual-server module.

Procmail is quite a bit more powerful and flexible than MailScanner. No need to downgrade capabilities just to get this functionality.

--

Check out the forum guidelines!

Topic locked