reading current activity in log files

18 posts / 0 new
Last post
#1 Mon, 09/30/2013 - 17:18
edwardsmarkf

reading current activity in log files

hi all -

i am trying to put together a script to read all activity in my log files from the last 15 minutes. the best answer i came up with is to use the unix 'tac' (cat backwards!) command embedded into a php script. then compare the timestamps to a specially formatted timestamp called 'earliest date'.

the code below works, but somehow it seems like a pretty clumsy way to do this.

any other ideas?

<div class="codeblock"><pre><code><span style="color: #000000"><span style="color: #0000BB"><?php<br /></span><span style="color: #007700">&</span><span style="color: #FF8000">#10;&#10;$earliest_date =  date(&#039;Y-m-d H:i:s&#039; , strtotime(&#039;-15 minutes&#039;));&#10;&#10;$filename = &#039;/var/log/virtualmin/marksdomain.com_access_log&#039; ;&#10;&#10;$file = popen(&quot;/usr/bin/tac  $filename  2&gt;/dev/null;&quot;, &#039;r&#039;);&#10;&#10;while ($line = fgets($file)) {&#10;&#10;        ##example:  168.8.56.7 - - [26/Sep/2013:10:59:04 -0400] &quot;GET /wp-content/&#10;&#10;        $regex = &#039;|^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[(\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2})|&#039;;&#10;&#10;        preg_match($regex, $line, $matches )    ;&#10;       //$matches[1] is the IP number, while $matches[2] is the date.&#10;&#10;        $activityDateTime = date_create_from_format(&#039;d/M/Y:H:i:s&#039;, $matches[2])-&gt;format(&#039;Y-m-d H:i:s&#039;) ;&#10;        // reformat the date to a more php-friendly format&#10;&#10;        if  ( $activityDateTime &gt;= $earliest_date )     {&#10;                echo $line      ;&#10;        } else {&#10;                pclose($file);&#10;                break;&#10;        }&#10;}&#10;&#10;<br /></span><span style="color: #0000BB">?></span></span></code></pre></div>
Mon, 09/30/2013 - 19:23
edwardsmarkf

i am expanding it to include all log files. specifically i am looking for wordpress attacks:

<?php

//  example:      /var/log/virtualmin/marksdomain.com_access_log


$earliest_date =  date('Y-m-d H:i:s' , strtotime('-15 minutes'));

define  (       'DIRECTORY'     ,       '/var/log/virtualmin'   );

if  ( $handle = opendir(constant('DIRECTORY')) )        {

        while ( false !== ($filename = readdir($handle)))       {


                if  ( $filename != '.' && $filename != '..' )   {

                        $fullFileName = constant('DIRECTORY') . '/' . $filename ;

                        $path_parts = pathinfo($fullFileName);

                        if  ( $path_parts['extension'] === 'com_access_log' )   {

                                $ipArray = array();

                                $p_openHandle = popen("tac $fullFileName 2>/dev/null",'r');

                                while ($line = fgets($p_openHandle)) {


                                        ##example:  168.8.56.7 - - [26/Sep/2013:10:59:04 -0400] "GET /wp-content/themes/english2/images/back.jpg HTTP/1.1" 200 389 "htt

                                        $regex = '|^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[(\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2}) -\d{4}|';

                                        preg_match($regex, $line, $matches )    ;

                                        $activityDateTime = date_create_from_format('d/M/Y:H:i:s', $matches[2])->format('Y-m-d H:i:s') ;

                                        if  ( $activityDateTime >= $earliest_date )     {

                                                if  ( preg_match('/wp-login.php/', $line )      ) {

                                                        if  (!(array_key_exists($matches[1], $ipArray )))       {
                                                                $ipArray[$matches[1]] = 1 ;
                                                        } else {
                                                                $ipArray[$matches[1]]++;
                                                        }
                                                }

                                        } else {
                                                pclose($p_openHandle);
                                                break 1;
                                        }

                                }       // end while

                                if  ( $ipArray )        {
                                        echo $filename;
                                        echo "\n";
                                        arsort($ipArray);
                                        print_r($ipArray);
                                }

                        } // end if


                }       // end if

        } //end while
}


?>
Wed, 10/02/2013 - 12:25
edwardsmarkf

hello - my problem all along is people trying to break into my wordpress installations. although all the WP sites have captcha and limit-login-lockdown, i still wish to be able to block anybody who even tries.

this script below runs frequently, and puts any offending IP number into the csf blacklist.

i put in parameters passed from crontab, and now loop through the entire virtualmin log file. currently it runs every 10 minutes on my server. if there is a breakin attempt, i am notified via email.

<?php

/*      wordpressLoginTracker.php

        program to scan virtualmin log files looking for wordpress login attempts

        parameters:
                1)      max allowed count in time period
                2)      time period in minutes


        2013-10-01      - initial writing

*/

//      /var/log/virtualmin/englishwithoutaccent.com_access_log



define  (       'MAX_ALLOWED_ACCESSES'                  ,       $argv[1]                                );
define  (       'TIME_TO_CHECK'                         ,       '-' . $argv[2] . ' minutes'             );

define  (       'DIRECTORY'                             ,       '/var/log/virtualmin'                   );
define  (       'VIRTUALMIN_LOG_SUFFIX'                 ,       'com_access_log'                        );
define  (       'SEND_TO'                               ,       'mark@markyboy.com'                  );



$earliest_date =  date('Y-m-d H:i:s' , strtotime(constant('TIME_TO_CHECK')));

echo 'running on: ' . date("D M d, Y G:i a") . ' looking for ' . $earliest_date  . "\n" ;       // log this run

if  ( $handle = opendir(constant('DIRECTORY')) )        {

        while ( false !== ($filename = readdir($handle)))       {

                if  ( $filename != '.' && $filename != '..' )   {

                        $fullFileName = constant('DIRECTORY') . '/' . $filename ;

                        $path_parts = pathinfo($fullFileName);

                        if  ( $path_parts['extension'] === constant('VIRTUALMIN_LOG_SUFFIX') )  {

                                $ipArray = array();

                                $p_openHandle = popen("tac $fullFileName 2>/dev/null",'r');

                                while ($line = fgets($p_openHandle)) {


                                        ##example:  168.8.56.7 - - [26/Sep/2013:10:59:04 -0400] "GET /wp-content/themes/english2/images/back.jpg HTTP/1.1" 200 389 "htt

                                        $regex = '|^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[(\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2}) -\d{4}|';

                                        preg_match($regex, $line, $matches )    ;

                                        $activityDateTime = date_create_from_format('d/M/Y:H:i:s', $matches[2])->format('Y-m-d H:i:s') ;

                                        if  ( $activityDateTime >= $earliest_date )     {

                                                if  ( preg_match('/wp-login.php/', $line )      ) {

                                                        if  (!(array_key_exists($matches[1], $ipArray )))       {
                                                                $ipArray[$matches[1]] = 1 ;
                                                        } else {
                                                                $ipArray[$matches[1]]++;
                                                        }
                                                }

                                        } else {
                                                pclose($p_openHandle);
                                                break 1;
                                        }

                                }       // end while we still have stuff to read in the file

                                if  ( $ipArray )        {
                                        //echo $filename;
                                        arsort($ipArray);
                                        foreach ($ipArray as $ipNbr => $ipCount )       {
                                                if  ( $ipCount > constant('MAX_ALLOWED_ACCESSES')       )       {
                                                        exec("/usr/sbin/csf  --deny  $ipNbr")   ;

                                                        $to             = constant('SEND_TO')           ;
                                                        $subject        = 'WP attack attempt'           ;
                                                        $headers        = 'From: info@comptonpeslonline.com'            . "\r\n"
                                                                        . 'Reply-To: info@comptonpeslonline.com'        . "\r\n"
                                                                        . 'X-Mailer: PHP/' . phpversion()
                                                                        ;
                                                        $message        = $filename                                     . "\r\n"
                                                                        . 'http://whatismyipaddress.com/ip/' . $ipNbr   . "\r\n"
                                                                        . 'Nbr of attempts: ' . $ipCount                . "\r\n"
                                                                        ;
                                                        mail ($to, $subject, $message, $headers );
                                                        echo $message;
                                                } else {
                                                        break 1;
                                                }
                                        } // end foreach
                                } // end if $ipArray has anything in it or not
                        } // end if file is a virtualmin log file
                }       // end if file is anything besides a single dot or a double dot
        } //end while we still have files to process
}


?>
Wed, 10/02/2013 - 13:34
Locutus

You should probably use a command like logtail to only process logfile lines that have been added since last run. Going thru the whole logfile each time can be a performance killer.

Wed, 10/02/2013 - 14:46 (Reply to #4)
edwardsmarkf

true, but i didn't think the "tac" command reads through the entire file. does anybody know ?

i just did a 'tac' on a couple of the largest text files i could find and the command executed near instantly.

tac /home/marksDomain.com/public_html/DebugLogs/eval.log | head ;

Sat, 10/05/2013 - 12:01
edwardsmarkf

hi locutus -

i looked at logtail -- it looked very interesting, but it appears to be an ajax powered log reader. i guess i was looking for the easiest way to just look at the last 15 minutes of log activity, from inside a crontab bash script.

a month ago i was on the highway when i learned my server was under a horrible attack that was shutting our servers down! since then i have enacted synflood protection on CSF, and that seems to have slowed it down.

however, there is little one can do about it when you are driving in-between broadband-enabled cell towers!

Sat, 10/05/2013 - 12:06
Locutus

Ajax powered? Not that I know of... It's a Linux shell/command line tool that outputs those lines of a given logfile (or any file really) that have been added since last call.

http://linux.die.net/man/8/logtail

Sat, 10/05/2013 - 22:16
edwardsmarkf

ok i looked at the wrong one:

http://sourceforge.net/projects/logtail/

but you have to admit they are spelled similar ! maybe they are related somehow?

i still cant help but think there may be something to using the 'tac' command in conjunction with the 'head' command, but the command-line logtail command might be more fun.

Tue, 10/08/2013 - 14:51
edwardsmarkf

mystery solved!

the logtail you mention is a perl script:

http://www.fourmilab.ch/webtools/logtail/

while the one that google found is some ajax-webpage tool.

Wed, 10/09/2013 - 17:11
edwardsmarkf

does anybody have any example of using logtail? i cant seem to find anything out there.

i am concerned that logtail is pretty old. it does not look like it is actively maintained.

also, for fun i took a BIG log file and cat'ed it to a test log file in order to make it a two-gig file. then i issued a

tac temp.log | head -5 ;

and this command happened instantly ! this tells me that tac may be a suitable method to read off the bottom of a log file.

but again, if anybody has a working example of logtail i would appreciate it.

Thu, 10/10/2013 - 04:32
Locutus

What's complicated about logtail that you need a "working example"? It just outputs everything in the logfile that's been added since the last run (for that, it stores the "current position" in an additional file).

Of course, if you feed it an artificial 2 GB logfile, it will initially output ALL of it, which of course takes VERY long! So question is, what do you need/want? If you just want to always read the last X lines from a file, tail -n is the way to go. If you have to process ALL lines that have been added, so as not to miss any and also not process any multiple times, use "logtail".

Doesn't matter if it's being maintained or something. Software that just works and has a very simple purpose doesn't need to be updated all the time. I'm using it in several places in my scripts, but nothing more than just logtail FILENAME and piping that to other commands. Nothing fancy about it.

Sat, 10/12/2013 - 11:18
edwardsmarkf

hi locutus -

What's complicated about logtail that you need a "working example"?

you will recall the ferengi needed extra help navigating their starship, and Will Riker was more than happy to assist.

here is what i did so far:

this command works fine:

usr/lib/perl/logtail.pl   -u   ;   

Usage: logtail [ options ] logfile Version 1.1 -- 2002 August 7. blah blah blah

create a small test log and feed it into logtail:

tail  -100    edwardsmark.com_access_log   >locutus.log  ;   
usr/lib/perl/logtail.pl      locutus.log ;  

this sits there forever, no output.

perl  usr/lib/perl/logtail.pl      locutus.log ;   

this also sits there forever

perl  usr/lib/perl/logtail.pl      < locutus.log ;   

same

if i run it in debug, it goes into a function at 564 check-dns, and then sits there in an endless loop on line 886: recv(SOCKDNS, $msg, 65535, 0);

this suggests to me its going over the network, by default.

i am excited about using logtail, assuming i can get it working!

Sat, 10/12/2013 - 11:32
Locutus

I'm not sure that you're using the correct logtail. Mine is located in /usr/sbin/logtail, no ".pl" extension, and I also don't need to run it through "perl". Ubuntu 12.04 here. I installed logtail from an Ubuntu repository package.

It runs fine for me. The first call e.g. logtail /var/log/syslog outputs the whole file, no delays or anything. Creates /var/log/syslog.offset to store its last processing position. Subsequent calls only output what was added to syslog.

Sat, 10/12/2013 - 17:21
edwardsmarkf

locutus - up front, let me once again thank you for helping me.

could i please trouble you to get me your logtail version? here is what my logtail did (oddly, whatever i have needs a dash-u to show help)

 perl   /usr/lib/perl/logtail.pl      -u   ;
Usage: logtail [ options ] logfile ...
Version 1.1 -- 2002 August 7.
       Options:
             -d              Show full domain name for echo messages.
             -ehostname      Echo to named host.  Multiple -e options
                             echo to multiple hosts.
             -lport          Listen for echo on port (default 5741),
                             0 = don't listen.
             -n              No forking for -r DNS lookups
             -pinterval      Reopen log files every interval
                             minutes (default 15), 0 = never.
             -q              Quiet: don't print on local machine, just echo.
             -rtime          Resolve: attempt to find host names from IP
                             addresses in the log.  Cache timeout: time secs.
             -stime          Sleep time seconds between polls of files.
             -tport          Transmit echo on port (default 5741).
             -u              Print this message.
             -vlevel         Verbose: generate debug output.
             -wcols          Wrap lines at cols columns, 0 = no wrap.
The latest version of this program is available
from: http:/www.fourmilab.ch/webtools/logtail/
Report bugs to bugs@fourmilab.ch

thank you very much, locutus!

Sat, 10/12/2013 - 17:28
edwardsmarkf

locutus - by any possible chance did you mean logcheck rather than logtail ?

now i know how the "Pakleds" felt when the Enterprise had to rescue them !

Sat, 10/12/2013 - 18:21
Locutus

My logtail doesn't have the "-u" option. As I said, although it does seem to be a perl script as per the shebang, but it's named just "logtail", not "logtail.pl".

No, I didn't mean "logcheck" (which I also have in use), I did mean logtail.

Here's the package information from Ubuntu:

root@borealis:~# apt-cache show logtail
Package: logtail
Priority: optional
Section: admin
Installed-Size: 148
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian logcheck Team <logcheck-devel@lists.alioth.debian.org>
Architecture: all
Source: logcheck
Version: 1.3.14
Replaces: logcheck (<= 1.1.1-9)
Depends: perl (>= 5.8.0)
Filename: pool/main/l/logcheck/logtail_1.3.14_all.deb
Size: 16226
MD5sum: 6a018406d31f060ffb429767f62b5fe4
SHA1: 7d5f0ef70752f4403884cc84d77b85af2e892ef7
SHA256: 9188d8ac06e6e12c430095ca5feb049ee01746678b34cb8ac7326b72d1d874e0
Description-en: Print log file lines that have not been read (deprecated)
 This program will read in a standard text file and create an
 offset marker when it reads the end. The offset marker is read
 the next time logtail is run and the text file pointer is moved
 to the offset location. This allows logtail to read in the next
 lines of data following the marker. This is good for marking log
 files for automatic log file checkers to monitor system events.
 .
 The package also provides logtail2, which better deals with rotated log
 files: If logtail2 finds that the inode of the file was changed, it assumes
 that the log has been rotated, and tries to find the file it was rotated to
 using heuristic plugins. If it finds the file, it will print the remainder of
 the file starting at the offset saved to the offset file. If a file with the
 correct inode was not found, logtail2 will only print the new file in its
 entirety before writing a new offset file.
Homepage: http://www.logcheck.org/
Description-md5: a844125a20d0a40064a9bb3cc49d8c48
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 5y
Sun, 10/13/2013 - 01:20
edwardsmarkf
yum  --assumeyes  install logcheck ;

and that's ALL it took !

Locutus, i want to offer you my most sincere appreciation for your patience regaring this matter!

Sun, 10/13/2013 - 07:10
Locutus

You're welcome, and good luck with your coding endeavor! :)

Topic locked