URGENT! - LibClamAV Error: mpool_malloc()

Submitted by Blueforce on Sat, 10/22/2016 - 11:17 Pro Licensee

Hi,

URGENT!

Can't start clamd, ClamAV is not running and mails are piling up and CPU load is maxing up!!!

I get this error Starting clamd.virtualmin: LibClamAV Error: mpool_malloc(): Attempt to allocate 8388608 bytes. Please report to http://bugs.clamav.net

I have tried do find a solution online but I can't find any, please help!

Best regards, Leffe (Blueforce)

Status: 
Active

Comments

Submitted by andreychek on Sat, 10/22/2016 - 12:04

Howdy -- in a pinch, you can always disable the ClamAV feature.

However, the error you're seeing suggests that there's some sort of memory issue preventing ClamAV from starting.

What is the output of these commands:

cat /etc/security/limits.conf
free -m

Submitted by Blueforce on Sat, 10/22/2016 - 12:11 Pro Licensee

Hi,

This is getting urgent, mails are piling up!!

> cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

# End of file
apache hard cpu -1
apache hard nproc -1
apache hard as -1
> free -m
             total       used       free     shared    buffers     cached
Mem:          3940       3506        433          0         29        334
-/+ buffers/cache:       3142        797
Swap:         5951          0       5951

Submitted by Blueforce on Sat, 10/22/2016 - 12:18 Pro Licensee

If I try to start clamd I get this error

Configuring and enabling the ClamAV scanning server ..

    Creating ClamAV configuration file /etc/clamd.d/virtualmin.conf ..
    .. done

    Fixing ClamAV bootup action /etc/rc.d/init.d/clamd-virtualmin ..
    .. already done

    Linking ClamAV server to /usr/sbin/clamd.virtualmin ..
    .. done

    Starting ClamAV server and enabling at boot ..
    .. failed to start : sh: /etc/rc.d/init.d/: is a directory

.. all done

And if I try starting clamd-virtualmin from Bootup and Shutdown i get a endless loop of this

LibClamAV Error: mpool_malloc(): Attempt to allocate 8388608 bytes. Please report to http://bugs.clamav.net
LibClamAV Error: mpool_malloc(): Attempt to allocate 8388608 bytes. Please report to http://bugs.clamav.net
LibClamAV Error: mpool_malloc(): Attempt to allocate 8388608 bytes. Please report to http://bugs.clamav.net
LibClamAV Error: mpool_malloc(): Attempt to allocate 8388608 bytes. Please report to http://bugs.clamav.net

Submitted by Blueforce on Sat, 10/22/2016 - 12:24 Pro Licensee

How do I disable ClamAV???

Mails don't get delivered and soon memory is out and CPU gets overloaded

CPU load averages 27.99 (1 min) 27.28 (5 mins) 25.04 (15 mins)
CPU usage 3% user, 15% kernel, 0% IO, 82% idle
Real memory 3.26 GB used, 3.85 GB total
Virtual memory 828 kB used, 5.81 GB total
Local disk space 45.37 GB used, 219.87 GB total

Submitted by Blueforce on Sat, 10/22/2016 - 12:29 Pro Licensee

I think I selected CentOS 6 when creating this issue - Sorry! It should be

CentOS Linux 5.11
Webmin version 1.821
Virtualmin version 5.04
Linux 2.6.18-412.el5 on x86_64
Intel(R) Xeon(R) CPU X3440 @ 2.53GHz, 8 cores

Submitted by andreychek on Sat, 10/22/2016 - 14:05

To disable ClamAV, you can disable the ClamAV feature for your various domains in Edit Virtual Server -> Enabled Features.

That will prevent the system from trying to use ClamAV when delivering email.

Submitted by Blueforce on Sat, 10/22/2016 - 14:49 Pro Licensee

Is it possible to do it server-wide, or do I have to do it one by one for all 50+ domains?

Do all the mails that is "on hold" by ClamAV get delivered if I disable ClamAV?

Is it possible to fix this issue, instead of disabling the virus scanner?

I really have to do something because the CPU load is getting higher all the time, CPU load averages is now 44.95 (1 min) and the the real memory and virtual memory is soon on it's max!

Best regards Leffe (Blueforce)

Submitted by andreychek on Sat, 10/22/2016 - 16:21

We'd certainly like to help resolve the issue, but since you're indicating that there is an urgent problem -- I'd suggest disabling ClamAV for now, and then we can try and sort out what's going on without an urgent email issue going on.

I'm unfortunately not sure what's going on, we've never run into that error before.

You can disable a feature for all domains by going into List Virtual Servers on the bottom-left of Virtualmin, select all your domains there, go into "Update Selected", then from there you can disable a given feature on all the domains.

Now, regarding the underlying ClamAV/email problem --

Something is preventing it from getting the RAM it needs... are you by chance using ClamAV, or CSF/LFD? Those two things could potentially cause such a problem.

Also, what is the output of the command "mailq | tail -1"?

Submitted by Blueforce on Sat, 10/22/2016 - 17:04 Pro Licensee

Hi,

And thank you for your help!

No, i'm not running CSF or LFD. Hmmm... yes, I have always used ClamAV, it has been default on Virtualmin Pro ever since we first installed 10+ years ago. I have not made any manual changes either on the server.

I get the output "Mail queue is empty".

I have disabled ClamAV on all domains but the CPU load just climbs up and more and more memory been used.

Submitted by Blueforce on Sat, 10/22/2016 - 17:39 Pro Licensee

Hi again,

I had to restart the server because CPU load was rapidly getting higher, and memory filling up! After the restart everything looks more normal again.

CPU load averages 0.06 (1 min)
Real memory 393.43 MB used, 3.85 GB total
Virtual memory 0 bytes used, 5.81 GB total

Now my stress factor is now down to "normal" again ;-)

Virus scanning with ClamAV (clamd) is now server-wide disabled. But it would be great to get ClamAV running again! Most of our customers are MS Windows users and the virus scanner drops rather many infected mails. But it's not that urgent anymore!

Regards, Leffe (Blueforce)

Submitted by andreychek on Sat, 10/22/2016 - 23:39

Now that you've rebooted -- are you still seeing the same problem trying to start the ClamAV service?

Also, what is the output of the command "cat /proc/user_beancounters"?

Not all systems have that file, but those that do, that might provide some additional insight.

Submitted by Blueforce on Sun, 10/23/2016 - 22:04 Pro Licensee

Hi,

Don't think I dare to try it right now, do not want to risk that I get the same problem as I did because of ClamAV. Customers will go mad, and we will lose customers if the server is down! Customer emails were heavily delayed the past days because they got stuck in ClamAV. And even when I turned off the virus scan and ClamAV the CPU load still increased to extremely high values, and all available memory was soon used up. In order to stop this before all the server's resources were exploited to the max, I chose to restart the server. The server rebooted but no services came up! I started them up manually using SSH and realized then that it was clamd-virtualmin that was the problem because it will start automatically at reboot. clamd-virtualmin in some kind of endless error loop. I disabled the start at boot time and tried to restart the server again, and this time it worked, and CPU load and memory utilization was now back to normal and the server worked well again except that the virus scanning was disabled for all domains.

I'll try to start ClamAV when I find time for it. But it would be nice if you could find the problem, because it seems like there are others who are experiencing the same thing.

I don't have user_beancounters file on our server.

Regards, Leffe (Blue Force)

Submitted by andreychek on Sun, 10/23/2016 - 23:18

We're seeing some other users on CentOS 5 experiencing this issue.

What we're looking to do is build a new ClamAV package for CentOS 5, hoping that the newer version resolves the problem.

Joe is working on that.

Submitted by caleidoskope on Mon, 10/24/2016 - 02:34 Pro Licensee

Hi, thank's for looking into that Problem. We are hit by this strange behaviour too.

clamd-virtualmin restart

throws this error:

LibClamAV Error: mpool_malloc(): Attempt to allocate 8388608 bytes. Please report to http://bugs.clamav.net

I would like to disable virus scanning for all domains until the new package is redy to be installed but I cannot find a decent method how to do it. Does anyone have a clue?

Regards

Submitted by Blueforce on Mon, 10/24/2016 - 04:06 Pro Licensee

Hi caleidoscope,

As andreychek wrote a few post up in this issue:

You can disable a feature for all domains by going into List Virtual Servers on the bottom-left of Virtualmin, select all your domains there, go into "Update Selected", then from there you can disable a given feature on all the domains.

I also had to restart the server before the CPU usage and memory usage dropped back to normal. In my case the restart failed at first, but when I disabled "clamd-virtualmin" from starting at boot the restart went fine. My suggestion is to disable virus scanning for all domains, disable start at boot for clamd-virtualmin before you try to restart the server. I don't know if there is some other way to get CPU and memory back to normal without restarting, maybe there are. I'm not that good at Linux and our server was down on it's knees so I tried the things I could think of.

Hope the Virtualmin team soon find a solution, as they always do! They provide the best service and support that you can find!

Anyways, Good luck and I hope you get your server back to "normal" again!

Regards, Leffe (Blueforce)

Submitted by andreychek on Mon, 10/24/2016 - 17:09

Yeah we'll be pushing out a new ClamAV package soon, we're hoping that resolves the issue.

Submitted by caleidoskope on Tue, 10/25/2016 - 05:04 Pro Licensee

Thank's guys. I disabled it for all domains. CPU back to normal.

Submitted by jeton on Tue, 10/25/2016 - 06:25

I have same problem now... how to stop this process so I can type anything on screen!!!

Submitted by andreychek on Tue, 10/25/2016 - 09:34

How to stop it from the command line depends on your distro, but you can always use Webmin -- log into Virtualmin/Webmin, click Webmin -> System -> Bootup and Shutdown, and there, you can disable any of the services that are running.

Submitted by caleidoskope on Tue, 10/25/2016 - 11:13 Pro Licensee

To stop all hanging clamscan processes on centos 5.x use

killall clamscan

this will stop all clamav scanning processes that have gone rogue

Submitted by Blueforce on Mon, 11/07/2016 - 19:59 Pro Licensee

I installed the new ClamAV package a few day ago and started things up again and everything has been working fine. And I guess the freshclam-sleep cron job is not needed anymore, or is it?

Thank you guys!

Regards, Leffe (Blueforce)

Submitted by andreychek on Mon, 11/07/2016 - 20:06

We're glad to hear that's working, thanks for the update!

The freshclam cron job grabs the updated virus data from the ClamAV website, so I'd leave that there unless you're seeing a problem of some kind.

Submitted by Blueforce on Mon, 11/07/2016 - 20:22 Pro Licensee

I got an error after I started the cron job again, and I got the error every time the cron job started. I also tried to kill running freshclam processes but it left me with a zombie process , but I found its parent process (the cron job for freshclam) and killing that got rid of the zombie. Then I tried to start the cron job again, ending up with the same error

ERROR: /var/log/freshclam.log is locked by another process

So I killed the freshclam cron job processes and then left it not active. And when I manually check freshclam its seems updated everytime, and btw, I could not run freshclam manually when I had the cron job active, i got the same error when running freshclam manually.

//Leffe

Submitted by andreychek on Mon, 11/07/2016 - 20:41

Hmm, that's all very odd!

We'll dig into that.

I'll offer that with CentOS 5 only being supported for a few more months (it reaches it's end of life in March of 2017), it might be reasonable to disable the freshclam cron job until that point.

However, we'll see if there's something that can be done about that in the meantime.

Submitted by Blueforce on Mon, 11/07/2016 - 21:01 Pro Licensee

Thanks,

But it's no problem for me, I'll leave the cron job off and manually update the ClamAv databases until our new servers are in place and the old ones have been migrated over. We should have bought the new servers some time ago... but finally I did order the new servers and hopefully they be up and running at the end of this year so I can say goodbye to our old servers! They have done a fantastic job for many years now!

//Leffe

Submitted by jeton on Tue, 11/08/2016 - 09:27

just run:

# yum install -y epel-release
# yum install -y clamav

and it's working fine...

Submitted by andreychek on Tue, 11/08/2016 - 10:17

The version of ClamAV that we're using is actually from EPEL. Joe just repackages it so that we can put it into our own repo's.

In theory things should work the same way :-)

Submitted by jeton on Fri, 11/11/2016 - 02:32

sorry... I don't know why these letters above appeared so big...