I sometimes have to parse log files for different IP addresses and then group them by network owner. This becomes tedious If the number of IP addresses is rather long. The script below can help with automating this manual task.
It reads a log file and looks for a match based on keys in an iplist. Afterwards the result is summarized and grouped by a specified field. For example, say you have the log file
192.168.1.1 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.3 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.1 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.2 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.3 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.2 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.3 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
and you would like to have all the entries for IPs 192.168.1.2 and 192.168.1.3. Instead of grepping the content for every IP manually you can use the script below. Put all the IPs in an iplist similar to this
1234 | 192.168.1.1 | MyNet
4567 | 192.168.1.2 | MyNet
8901 | 192.168.1.3 | MyNet
2345 | 192.168.1.4 | MyNet
<?php
/**
*
* Parse a log file and group by entries from another file
*
* This script reads a log file and then groups the entries
* according to keys found in an iplist
* There's no input validation so make sure neither the
* log file or iplist contain malicious code
*
* This script is useful if you want to group log file entries
* based on AS number or network name.
*
* Koen Van Impe cudeso.be
* 20100525
*
**/
// Configuration array
$config = array( // file containing the IPs
"iplist" => "BE.txt",
// logfile with the individual entries
"logfile" => "Log_BE.txt",
// what field to use as a separator in iplist
"separator" => "|",
// position of the IP (0-based)
"ippos" => 1,
// position of the groupby field (0-based)
"groupby" => 0,
// newline after a logfile
"newline" => false
);
// Array for the resultset
$result = array();
$matchcount = 0;
// walk through the IP list
if (file_exists($config["iplist"])) {
$file_handle = fopen($config["iplist"], "r");
while (!feof($file_handle)) {
$fields = explode("|", fgets($file_handle));
$key = (string) trim($fields[$config["groupby"]]);
if (strlen($key) > 0) {
$data = trim($fields[$config["ippos"]]);
$result[$key][] = $data;
}
}
fclose($file_handle);
// read the log file
if ((file_exists($config["logfile"])) && count($result) > 0) {
$logfile = file($config["logfile"]);
echo "Parsing ".$config["logfile"]."\n".
"for matches in ".$config["iplist"]."\n".
"on field pos #".$config["ippos"]."\n".
"group by field pos #".$config["groupby"]."\n\n\n";
// walk through the resultset; scan the
// log file for every entry
// three foreachs ... optimization
foreach ($result as $key => $value) {
echo "\n******************\n$key\n******************\n";
foreach ($logfile as $line) {
foreach ($value as $match) {
// is position 0 and is not BOOLEAN
if ((strpos($line, $match) === 0) or
// position bigger than 0
(strpos($line, $match) > 0)) {
// we have a match
echo "$line";
if ($config["newline"]) echo "\n";
$matchcount++;
}
else $misscount++;
}
}
echo "\n\n\n\n";
}
echo "\n\n$matchcount relevant entries found in ".$config["logfile"];
}
}
?>
A couple of days back I received an e-mail from Deutsche Bank. I’m not a customer from DB. About a year ago I applied for some information and I guess my email addresses ended up in their mailinglist.
The mailing warns customers that there is a phishing attack ongoing. According to the mail, once infected, a virus on your computer lures you to a fake page where you are asked to enter your details.
So far so good. It seems like a good practice that banks try to warn their customers.
The mail contains a couple of links that should point you to sites that allow you to check if you are infected or not. Unfortunately the links point to another website. That website seems to have nothing to do with DB. It is a website for a “relationship marketing suite”. It is understandable that DB uses an external company to handle their mailings but I don’t get it … The message to their customers is “be on your guards” and then they ask you to click on a link that has nothing to do with DB?
There is a good writeup on XSS on Security Override.
I was recently in need for a parser that would convert Nmap XML output to a HTML file. As far as I could see there was no tool available so I wrote my own. Feel free to use or adjust it.
<!--
Parse nmap XML output
Koen Van Impe cudeso.be
20100303
-->
<html>
<head><title>nmap xml file to html</title></head>
<body>
<form method="POST" enctype="multipart/form-data" action="<?php echo $_SERVER["PHP_SELF"]; ?>">
XML file: <input type="file" name="xmlfile" <br />
<input type="checkbox" checked name="open"> Open <br />
<input type="checkbox" name="closed"> Closed <br />
<input type="checkbox" name="filtered"> Filtered <br />
<input type="submit" value="Press"> to upload the file!
</form>
<?php
if(isset($_FILES['xmlfile'])) {
// init
if (trim($_POST["open"]) == "on") $printOpen = true;
else $printOpen = false;
if (trim($_POST["closed"]) == "on") $printClosed = true;
else $printClosed = false;
if (trim($_POST["filtered"]) == "on") $printFiltered = true;
else $printFiltered = false;
$xmlObject = simplexml_load_file($_FILES['xmlfile']['tmp_name']);
// output the header
echo "<h1>".(string)$xmlObject["args"]."</h1>";
echo "<h2>Hosts up: ".(string) $xmlObject->runstats->hosts["up"]." / Hosts down: ".
(string) $xmlObject->runstats->hosts["down"]. " / Hosts total: ".(string) $xmlObject->runstats->hosts["total"]."</h2>";
// run through the xml and print hostinfo
foreach($xmlObject as $host => $value) {
// Only grab the data if it's host related info
if ((string) $host == "host") {
// declare portsarray
$nmap["ports"] = array();
// get the hostinfo
echo "<h2>".(string) $value->hostnames->hostname["name"].
" (".(string) $value->address["addr"]." / ".(string) $value->address["addrtype"].")</h2>";
echo "<table>";
// put the discovered ports in an array
foreach ($value->ports->port as $port) {
if ( ( ((string) $port->state["state"] == "filtered") and ($printFiltered)) or
( ((string) $port->state["state"] == "closed") and ($printClosed)) or
( ((string) $port->state["state"] == "open") and ($printOpen))
) {
echo "<tr><td>".(string)$port["portid"]."/".(string)$port["protocol"]."</td><td>".
(string)$port->state["state"]."(".(string)$port->state["reason"].")</td>
<td>".$port->service["name"]."(".(string)$port->service["product"].")
</tr>";
}
}
echo "</table>";
}
}
}
?>
</body>
</html>
An interesting post by Martin Roesch on the new architecture in the beta release of Snort.
It’s a sad day for IT security when even D. J. Bernstein has to admit that there’s a flaw in DJBDNS. Luckily the patch that needs to be applied is straightforward and has no known side-effects (at least, not on the DJBDNS servers that I patched).
I attended the last joint TF-CSIRT / FIRST meeting in Riga. One of the most interesting talks was by Wayne Routly from Dante. He talked on Quantitative Cross Comparative Analysis of Tools for Anomaly Detection. He gave some very good information on StealthWatch and Netreflex. If you’re moving Gbs of data then the set of tools you can use to analyze traffic are limited. According to the presentation both tools did quite well though.
SSL Blacklist is a nice addon for Firefox that has the ability to detect MD5 signed certificates and block access. If you don’t know what’s the problem with MD5 signed certificates then enjoy reading this.
Recently I needed to wipe a couple of disks because I wanted to sell them. I’ve played around with shred under Linux (like from a Knoppix live cd) but why go through all this hassle? I encountered Darik’s Boot and Nuke. Boot from the cd, select the drives and press F10. That’s all there is.
As incident handlers we always need to look out for sources that report possible malicious activity coming (or going to) our networks. We run a couple of honeypots and have a netflow monitor that alerts us when something is out of the ordinary. Extra sources however are always an extra bonus.
Recently I came across ATLAS from Arbor Networks.
They have an excellent service where you can easily sign up and if they approve your account you get access to alerts from their honeypots. According to their website they cover a large part of the Internet. There’s of course some commercial mumbo jumbo but at first their service seems to be very useful. A feed (RSS) allows you to get instant updates with a short description with the type of incident -scan, phish, …-, a timeframe and a link to their site with additional information.