Fail2Ban

Hi,

After the latest update in mid January something happened with Fail2Ban, I did a restart on Fail2Ban and then everything went crazy! New conf files was written january 21 and that is when I installed the latest updates.

Now all kinds of "crap" is filling up the log file, in just 15 minutes almost 29000 rows was written, approximately a 8 MB log. Before the restart the Fail2Ban logs was around 1-1,5 MB on a weekly rotation.

I run Iptables Firewall and has selected banaction = iptables-multiport and also renamed the 00-firewalld.conf.

Our logwatch faild due to big file today and I to manually rotate the Fail2Ban log but that didn't work, actually the "Force Log Rotation" doesn't work at all, just throwing out an error

Forcing the immediate rotation of all log files ..

.. failed! See the output above for the reason why.

And there are no output! Exactly like when manually running a cron job, no output is shown.

What shall I do with Fail2Ban? Why has it suddenly stopped working!?

Operating system CentOS 7.7.1908, Webmin version 1.941, Usermin version 1.791, Virtualmin version 6.08 Pro, Authentic theme version 19.45

Best regards, Leffe

Status: 
Closed (duplicate)

Comments

Ilia's picture
Submitted by Ilia on Sun, 02/16/2020 - 04:32

Hi,

What do you have logged exactly, what's overflowing your logs? Share 10-15 lines of "crap", please.

Lines like this is filling up the log

2020-02-16 07:11:05,633 fail2ban.filtersystemd  [836]: HEAVY   [dovecot] Read systemd journal entry: 2020-02-16T07:11:05.632004 servertwo dovecot[2110]: pop3-login: Login: user=, method=PLAIN, rip=XXX.XXX.XXX.XXX, lip=XXX.XXX.XXX.XXX, mpid=1247, session=
2020-02-16 07:11:05,633 fail2ban.filter         [836]: HEAVY   Looking for match of [('', '2020-02-16T07:11:05.632004', 'servertwo dovecot[2110]: pop3-login: Login: user=, method=PLAIN, rip=XXX.XXX.XXX.XXX, lip=XXX.XXX.XXX.XXX, mpid=1247, session=')]
2020-02-16 07:11:05,634 fail2ban.filter         [836]: HEAVY     Looking for prefregex '^\\s*(?:\\S+\\s+)?(?:\\S*(?:\\[\\d+\\])?:?\\s+)?(?:kernel:\\s?\\[ *\\d+\\.\\d+\\]:?\\s+)?(?:(?:dovecot: )?auth(?:-worker)?(?:\\([^\\)]+\\))?: )?(?:pam_unix(?:\\(dovecot:auth\\))?: |(?:pop3|imap)-login: )?(?:Info: )?(?P.+)$'
2020-02-16 07:11:05,634 fail2ban.filter         [836]: TRACE     Pre-filter matched {'content': 'Login: user=, method=PLAIN, rip=XXX.XXX.XXX.XXX, lip=XXX.XXX.XXX.XXX, mpid=1247, session='}
2020-02-16 07:11:05,634 fail2ban.filter         [836]: HEAVY     Looking for failregex 0 - '^authentication failure; logname=(?P\\S*) uid=\\S* euid=\\S* tty=dovecot ruser=(?P\\S*) rhost=(?:\\[?(?:(?:::f{4,6}:)?(?P(?:\\d{1,3}\\.){3}\\d{1,3})|(?P(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?[\\w\\-.^_]*\\w))(?:\\s+user=(?P\\S*))?\\s*$'
2020-02-16 07:11:05,634 fail2ban.filter         [836]: HEAVY     Looking for failregex 1 - '^(?:Aborted login|Disconnected)(?::(?: [^ \\(]+)+)? \\((?:auth failed, \\d+ attempts(?: in \\d+ secs)?|tried to use (?:disabled|disallowed) \\S+ auth|proxy dest auth failed)\\):(?: user=[^>]*)>,)?(?: method=\\S+,)? rip=(?:\\[?(?:(?:::f{4,6}:)?(?P(?:\\d{1,3}\\.){3}\\d{1,3})|(?P(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?[\\w\\-.^_]*\\w))(?:[^>]*(?:, session=)?)\\s*$'
2020-02-16 07:11:05,634 fail2ban.filter         [836]: HEAVY     Looking for failregex 2 - '^pam\\(\\S+,(?:\\[?(?:(?:::f{4,6}:)?(?P(?:\\d{1,3}\\.){3}\\d{1,3})|(?P(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?[\\w\\-.^_]*\\w))(?:,\\S*)?\\): pam_authenticate\\(\\) failed: (?:User not known to the underlying authentication module: \\d+ Time\\(s\\)|Authentication failure \\(password mismatch\\?\\)|Permission denied)\\s*$'
2020-02-16 07:11:05,634 fail2ban.filter         [836]: HEAVY     Looking for failregex 3 - '^[a-z\\-]{3,15}\\(\\S*,(?:\\[?(?:(?:::f{4,6}:)?(?P(?:\\d{1,3}\\.){3}\\d{1,3})|(?P(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?[\\w\\-.^_]*\\w))(?:,\\S*)?\\): (?:unknown user|invalid credentials|Password mismatch)\\s*$'
2020-02-16 07:11:05,769 fail2ban.filtersystemd  [836]: HEAVY   [dovecot] Read systemd journal entry: 2020-02-16T07:11:05.767861 servertwo dovecot[2110]: pop3(XXXXXXXX): Disconnected: Logged out top=0/0, retr=0/0, del=0/156, size=4501601
2020-02-16 07:11:05,769 fail2ban.filter         [836]: HEAVY   Looking for match of [('', '2020-02-16T07:11:05.767861', 'servertwo dovecot[2110]: pop3(XXXXXXXX): Disconnected: Logged out top=0/0, retr=0/0, del=0/156, size=4501601')]
2020-02-16 07:11:05,770 fail2ban.filter         [836]: HEAVY     Looking for prefregex '^\\s*(?:\\S+\\s+)?(?:\\S*(?:\\[\\d+\\])?:?\\s+)?(?:kernel:\\s?\\[ *\\d+\\.\\d+\\]:?\\s+)?(?:(?:dovecot: )?auth(?:-worker)?(?:\\([^\\)]+\\))?: )?(?:pam_unix(?:\\(dovecot:auth\\))?: |(?:pop3|imap)-login: )?(?:Info: )?(?P.+)$'
2020-02-16 07:11:05,770 fail2ban.filter         [836]: TRACE     Pre-filter matched {'content': 'pop3(XXXXXXXX): Disconnected: Logged out top=0/0, retr=0/0, del=0/156, size=4501601'}
2020-02-16 07:11:05,770 fail2ban.filter         [836]: HEAVY     Looking for failregex 0 - '^authentication failure; logname=(?P\\S*) uid=\\S* euid=\\S* tty=dovecot ruser=(?P\\S*) rhost=(?:\\[?(?:(?:::f{4,6}:)?(?P(?:\\d{1,3}\\.){3}\\d{1,3})|(?P(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?[\\w\\-.^_]*\\w))(?:\\s+user=(?P\\S*))?\\s*$'
2020-02-16 07:11:05,770 fail2ban.filter         [836]: HEAVY     Looking for failregex 1 - '^(?:Aborted login|Disconnected)(?::(?: [^ \\(]+)+)? \\((?:auth failed, \\d+ attempts(?: in \\d+ secs)?|tried to use (?:disabled|disallowed) \\S+ auth|proxy dest auth failed)\\):(?: user=[^>]*)>,)?(?: method=\\S+,)? rip=(?:\\[?(?:(?:::f{4,6}:)?(?P(?:\\d{1,3}\\.){3}\\d{1,3})|(?P(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?[\\w\\-.^_]*\\w))(?:[^>]*(?:, session=)?)\\s*$'
2020-02-16 07:11:05,770 fail2ban.filter         [836]: HEAVY     Looking for failregex 2 - '^pam\\(\\S+,(?:\\[?(?:(?:::f{4,6}:)?(?P(?:\\d{1,3}\\.){3}\\d{1,3})|(?P(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?[\\w\\-.^_]*\\w))(?:,\\S*)?\\): pam_authenticate\\(\\) failed: (?:User not known to the underlying authentication module: \\d+ Time\\(s\\)|Authentication failure \\(password mismatch\\?\\)|Permission denied)\\s*$'
2020-02-16 07:11:05,771 fail2ban.filter         [836]: HEAVY     Looking for failregex 3 - '^[a-z\\-]{3,15}\\(\\S*,(?:\\[?(?:(?:::f{4,6}:)?(?P(?:\\d{1,3}\\.){3}\\d{1,3})|(?P(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?[\\w\\-.^_]*\\w))(?:,\\S*)?\\): (?:unknown user|invalid credentials|Password mismatch)\\s*$'

And before the Fail2Ban restart it look like this

2020-02-01 23:51:12,749 fail2ban.actions        [31074]: WARNING [postfix-sasl] 175.6.133.182 already banned
2020-02-01 23:55:36,843 fail2ban.filter         [31074]: INFO    [postfix-sasl] Found 193.56.29.10 - 2020-02-01 23:55:36
2020-02-01 23:56:12,080 fail2ban.filter         [31074]: INFO    [postfix-sasl] Found 193.56.29.10 - 2020-02-01 23:56:12
2020-02-01 23:56:51,608 fail2ban.filter         [31074]: INFO    [postfix-sasl] Found 193.56.28.68 - 2020-02-01 23:56:51
2020-02-01 23:56:52,930 fail2ban.filter         [31074]: INFO    [postfix-sasl] Found 193.56.29.10 - 2020-02-01 23:56:52
2020-02-01 23:57:07,094 fail2ban.filter         [31074]: INFO    [postfix-sasl] Found 193.56.28.68 - 2020-02-01 23:57:07
2020-02-02 00:01:13,480 fail2ban.actions        [31074]: NOTICE  [postfix-sasl] Unban 175.6.133.182
2020-02-02 00:03:42,408 fail2ban.filter         [31074]: INFO    [postfix-sasl] Found 80.66.81.143 - 2020-02-02 00:03:42

In a week long log I had around 8000 lines before the restart. And now I get like 29000 lines in 15 minutes!

//Leffe

Ilia's picture
Submitted by Ilia on Sun, 02/16/2020 - 06:59

Are you the only one who is managing the system?

In case you tried to add/modify regex for Fail2Ban - that might be the cause.

Either provide me with the diff of /etc/fail2ban since the last week (from the moment it stopped working) or tell me exactly, which rules were added/edited, which regex did you use to catch what string exactly?

Yes i'm the only one managing the server!

I have not made any changes at all to Fail2Ban, just did a restart and this happened. It was when I started to look for a reason for this I noticed that the new 00-firewalld.conf was written and many files/folders had the same date.

I can't do a diff since I dont have a backup to check, our backups run twice a day with R1Softs Server backup in the data center.

And as I said, I have not changed anything... I have run Fail2Ban "out of the box", and have not added, edited or removed any rules what so ever. The only thing I changed from start was using Iptables instead of Firewalld.

//Leffe

Ilia's picture
Submitted by Ilia on Sun, 02/16/2020 - 07:15

However, on the other post, you are discussing with Jamie, there is an issue with Postfix. Do you think that couldn't be related?

Just to be sure, tar /etc/fail2ban and attach it for me to have a look.

Regarding the other issue when making changes in network that also affected Postfix probably also affects other things! This dedicated server was set up almost a year ago but has had so many strange issues that we still hasn't dared to move our customers here yet... Our own domain is the only one at the moment! And the server was also totally reinstalled after the exploit just to make sure it wasn't affected, and I also hoped the new install would get rid of all the issues with network, Postfix and other stuff... but they all came back again.

I actually have a zip I made yestarday, should I upload it here or mail it to you?

//Leffe

Can i upload a zip here? it say only txt jpg png gif.

Ilia's picture
Submitted by Ilia on Sun, 02/16/2020 - 09:39

Can i upload a zip here? it say only txt jpg png gif.

May be renaming to .txt and uploading?

It worked...

There are mort issues with this Fail2Ban, also some Filter Action Jails are missing settings, like ssh-ddos

Ilia's picture
Submitted by Ilia on Sun, 02/16/2020 - 15:59

I actually have a zip I made yesterday, should I upload it here or mail it to you?

Does it mean that what you have sent is not actual configuration?

Meanwhile, I don't like the following file, delete it:

./jail.d/00-firewalld.confdisabled

Do you have FirewallD disabled, I suppose?

You should also then:

yum remove fail2ban-firewalld

You don't need really the following placed on the config, if you perform steps above, as it's already default:

banaction = iptables-multiport

I would reboot the whole server but I think restarting Fail2Ban should also just work fine.

Hi Ilia,

Yes the 00-firawalld.conf was "deleted"(moved) from the beginning, but I moved it back for testing, and then renamed it.

I know the banaction = is not needed, also put there during testing and I thought it would do no harm.

Are you sure yum remove fail2ban-firewalld doesn't remove other needed files as dependencies?

How can updates do this kind of changes... I doesn't feel safe to do updates any more!

//Leffe

Forgot to answer this,

Does it mean that what you have sent is not actual configuration?

Yes it's the actual config from the server, I downloaded the folder for a compare but I realized that my other servers did not have fal2ban installed.

Ilia's picture
Submitted by Ilia on Sun, 02/16/2020 - 16:35

Are you sure yum remove fail2ban-firewalld doesn't remove other needed files
as dependencies?

Yes, it is!

Example:

[root@debug-centos7 webmin]# repoquery -l fail2ban-firewalld
/etc/fail2ban/jail.d/00-firewalld.conf
[root@debug-centos7 webmin]# yum remove fail2ban-firewalld
Loaded plugins: fastestmirror
Resolving Dependencies
--> Running transaction check
---> Package fail2ban-firewalld.noarch 0:0.10.5-2.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================
 Package                                      Arch                             Version                                  Repository                       Size
==============================================================================================================================================================
Removing:
 fail2ban-firewalld                           noarch                           0.10.5-2.el7                             @epel                           356  

Transaction Summary
==============================================================================================================================================================
Remove  1 Package
Yes the 00-firawalld.conf was "deleted"(moved) from the beginning, but I moved it back for testing, and then renamed it.

You have actually two files there, first is 00-firewalld.confdisabled, created by user, and 00-firewalld, part of the package, that I suggested for removal.

Removed the file and removed fail2ban-firewalld and started fail2ban and the logs fill up in the same way as in post #2

I'll reboot the server and try starting fail2ban again.

Have restarted the server and have started fail2ban, and it is still the same...

Can I remove fail2ban totally and reinstall it and get a default working config?

Ilia's picture
Submitted by Ilia on Sun, 02/16/2020 - 17:15

Your config is fine. We have slightly modified config for Fail2Ban.

To make it our default, you just need to remove BlockTime and the other already mentioned option.

To reinstall it, while keeping the current config go with yum reinstall fail2ban. I don't think it's gonna help in anyway.

I did start fail2ban on our local testserver and that runs perfectly just renamed the 00-firewalld.conf and started it. This version is 0.9.7

And another thing, with the version on our live server no fail2ban log gets added in Server logs. With the old version it adds it automatically.

I'll try removing the BlockTime, end the other is already removed.

It didn't help, still the same filled up logfile, and no logfile in System Logs!

The fail2ban log WAS in "System/System Logs" until I restarted fail2ban.

Is it possible to get the 0.9.7 version or the latest workin verison back because this 0.10.5 version don't seems to work!

When I check yum list fail2ban it say there is a available package, it doesn't say it is installed!??

Shall I install fail2ban???

[root@servertwo ~]# yum reinstall fail2ban
Loaded plugins: fastestmirror
No Match for argument: fail2ban
Loading mirror speeds from cached hostfile
* base: mirrors.glesys.net
* centos-sclo-rh: mirrors.glesys.net
* centos-sclo-sclo: mirrors.glesys.net
* epel: mirrors.glesys.net
* extras: mirrors.glesys.net
* updates: mirrors.glesys.net
Package(s) fail2ban available, but not installed.
Error: Nothing to do

[root@servertwo ~]#

What packages shall I install?

[root@servertwo ~]# yum list fail2ban*
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.glesys.net
* centos-sclo-rh: mirrors.glesys.net
* centos-sclo-sclo: mirrors.glesys.net
* epel: mirrors.glesys.net
* extras: mirrors.glesys.net
* updates: mirrors.glesys.net
Installed Packages
fail2ban-server.noarch                      0.10.5-2.el7                   @epel
Available Packages
fail2ban.noarch                             0.10.5-2.el7                   epel
fail2ban-all.noarch                         0.10.5-2.el7                   epel
fail2ban-firewalld.noarch                   0.10.5-2.el7                   epel
fail2ban-hostsdeny.noarch                   0.10.5-2.el7                   epel
fail2ban-mail.noarch                        0.10.5-2.el7                   epel
fail2ban-sendmail.noarch                    0.10.5-2.el7                   epel
fail2ban-shorewall.noarch                   0.10.5-2.el7                   epel
fail2ban-systemd.noarch                     0.10.5-2.el7                   epel
fail2ban-tests.noarch                       0.10.5-2.el7                   epel

[root@servertwo ~]#
Ilia's picture
Submitted by Ilia on Sun, 02/16/2020 - 18:31

Those that are installed, they are marked with @epel.

What is the output of:

systemctl status fail2ban

It says [root@servertwo ~]#. What about server one? Does it work as expected? :)

ServerOne don't run fail2ban, it isn't installed on that server since it wasn't by default earlier.

And I don't want to try on that server since most of our customers is on that one. ServerTwo is our new were customers from ServerOne should been transferred long time ago, but since the new ServerTwo has so many issues I haven't dared to move customers there...

[root@servertwo ~]# systemctl status fail2ban
● fail2ban.service - Fail2Ban Service
   Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Mon 2020-02-17 00:34:48 CET; 1h 9min ago
     Docs: man:fail2ban(1)
  Process: 5244 ExecStop=/usr/bin/fail2ban-client stop (code=exited, status=0/SUCCESS)
  Process: 5107 ExecStart=/usr/bin/fail2ban-server -xf start (code=exited, status=0/SUCCESS)
  Process: 5104 ExecStartPre=/bin/mkdir -p /run/fail2ban (code=exited, status=0/SUCCESS)
Main PID: 5107 (code=exited, status=0/SUCCESS)

Feb 17 00:33:00 servertwo systemd[1]: Starting Fail2Ban Service...
Feb 17 00:33:00 servertwo systemd[1]: Started Fail2Ban Service.
Feb 17 00:33:00 servertwo fail2ban-server[5107]: Server ready
Feb 17 00:34:48 servertwo systemd[1]: Stopping Fail2Ban Service...
Feb 17 00:34:48 servertwo fail2ban-client[5244]: Shutdown successful
Feb 17 00:34:48 servertwo systemd[1]: Stopped Fail2Ban Service.

[root@servertwo ~]#

And.... Thanks Ilia for all your time trying to help me out here!!!

Ilia's picture
Submitted by Ilia on Mon, 02/17/2020 - 04:36

Dear Leffe, let's continue on your email, I'll recommend what I think would be best to do in your case to solve your problem, as what you are experiencing here is not directly related to Virtualmin itself.

And.... Thanks Ilia for all your time trying to help me out here!!!

Thank you for saying that.

Ilia,

It has probably something to do with some of these updates

Jan 16 23:46:54 Updated: usermin-1.791-1.noarch
Jan 16 23:47:16 Updated: webmin-1.941-1.noarch
Feb 06 23:36:21 Updated: fail2ban-server-0.10.5-2.el7.noarch

And I DO think this is related to Virtualmin since Virtualmin has a application for controlling and managing Fail2Ban and I think Virtualmin has to keep functions in working state after a update. I have not changed anything, the only thing I did was restart fail2ban and probably "activated" the updated fail2ban.

If Virtualmin no longer will support for example fail2ban I think we has to be informed about that in a very clear way, maybe a message in control panel. I don't know what would happened if wasn't around when fail2ban had restarted! In this case I did see the problems right away and could stop fail2ban. Just doing the testings for really short time periods did sum up to BIIIIG logwatch file for the last 24 hours, the mail size was 61MB and it contained almost 238000 rows of fail2ban log data!

But enough of that... do you think we can get this running or should I just drop fail2ban totally?

Best regards, Leffe

Ilia's picture
Submitted by Ilia on Mon, 02/17/2020 - 05:04

I have no idea!

Can I just log in and have a look?

I am replying to your email now..

Of course... but there is no a email from you!

I still haven't got any email, and I now has to go to a business meeting in a few minutes. If you have mailed me let me know, otherwise we have to take this a up again later today. If you have mailed me, what IP are you sending from?

You can also mark this thread as Private if you want!

Ilia's picture
Submitted by Ilia on Mon, 02/17/2020 - 05:57

Like you said, it takes a lot of time for preparing such replies!

I will reply very-very soon.

oh... sorry, I thought you were talking about a email regarding logging in on our server to take a look!

I misunderstood you - Sorry!

Ilia's picture
Submitted by Ilia on Mon, 02/17/2020 - 10:14

I have also sent you an email. Feel free to ask for help and let me know more, what you think about continuing using CentOS 7.

I have answered in the other, private issue/post. And it's okay to close this issue.

Regards, Leffe

Ilia's picture
Submitted by Ilia on Mon, 02/17/2020 - 14:52

Status: Active » Closed (duplicate)