I sometimes have to parse log files for different IP addresses and then group them by network owner. This becomes tedious If the number of IP addresses is rather long. The script below can help with automating this manual task.

It reads a log file and looks for a match based on keys in an iplist. Afterwards the result is summarized and grouped by a specified field. For example, say you have the log file

192.168.1.1 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.3 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.1 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.2 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.3 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.2 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"
192.168.1.3 - - [1/Apr/2010:1:1:39 +0200] "GET /favicon.ico HTTP/1.1"

and you would like to have all the entries for IPs 192.168.1.2 and 192.168.1.3. Instead of grepping the content for every IP manually you can use the script below. Put all the IPs in an iplist similar to this

1234 | 192.168.1.1 | MyNet
4567 | 192.168.1.2 | MyNet
8901 | 192.168.1.3 | MyNet
2345 | 192.168.1.4 | MyNet


<?php
/**
 *
 * Parse a log file and group by entries from another file
 *
 * This script reads a log file and then groups the entries
 * according to keys found in an iplist
 * There's no input validation so make sure neither the
 * log file or iplist contain malicious code
 *
 * This script is useful if you want to group log file entries
 * based on AS number or network name.
 *
 * 		Koen Van Impe				cudeso.be
 *		20100525
 *
 **/

// Configuration array
$config = array(	// file containing the IPs
					"iplist" => "BE.txt",
					// logfile with the individual entries
					"logfile" => "Log_BE.txt",
					// what field to use as a separator in iplist
					"separator" => "|",
					// position of the IP (0-based)
					"ippos" => 1,
					// position of the groupby field (0-based)
					"groupby" => 0,
					// newline after a logfile
					"newline" => false
				);

// Array for the resultset
$result = array();
$matchcount = 0;

// walk through the IP list
if (file_exists($config["iplist"])) {
	$file_handle = fopen($config["iplist"], "r");
	while (!feof($file_handle)) {
		$fields = explode("|", fgets($file_handle));
		$key = (string) trim($fields[$config["groupby"]]);
		if (strlen($key) > 0) {
			$data = trim($fields[$config["ippos"]]);
			$result[$key][] =  $data;
		}
	}
	fclose($file_handle);

	// read the log file
	if ((file_exists($config["logfile"])) && count($result) > 0) {
		$logfile = file($config["logfile"]);

		echo "Parsing ".$config["logfile"]."\n".
				"for matches in ".$config["iplist"]."\n".
				"on field pos #".$config["ippos"]."\n".
				"group by field pos #".$config["groupby"]."\n\n\n";
		// walk through the resultset; scan the
		// log file for every entry
		// three foreachs ... optimization
		foreach ($result as $key => $value) {
			echo "\n******************\n$key\n******************\n";
			foreach ($logfile as $line) {
				foreach ($value as $match) {
					// is position 0 and is not BOOLEAN
					if ((strpos($line, $match) === 0) or
					// position bigger than 0
						(strpos($line, $match) > 0)) {

							// we have a match
							echo "$line";
							if ($config["newline"]) echo "\n";
							$matchcount++;
					}
					else $misscount++;
				}
			}
			echo "\n\n\n\n";
		}

		echo "\n\n$matchcount relevant entries found in ".$config["logfile"];
	}
}

?>

A couple of days back I received an e-mail from Deutsche Bank. I’m not a customer from DB. About a year ago I applied for some information and I guess my email addresses ended up in their mailinglist.

The mailing warns customers that there is a phishing attack ongoing. According to the mail, once infected, a virus on your computer lures you to a fake page where you are asked to enter your details.

So far so good. It seems like a good practice that banks try to warn their customers.

The mail contains a couple of links that should point you to sites that allow you to check if you are infected or not. Unfortunately the links point to another website. That website seems to have nothing to do with DB. It is a website for a “relationship marketing suite”. It is understandable that DB uses an external company to handle their mailings but I don’t get it … The message to their customers is “be on your guards” and then they ask you to click on a link that has nothing to do with DB?

There is a good writeup on XSS on Security Override.

I was recently in need for a parser that would convert Nmap XML output to a HTML file. As far as I could see there was no tool available so I wrote my own. Feel free to use or adjust it.

<!--
Parse nmap XML output
	Koen Van Impe		cudeso.be
	20100303
-->
<html>
<head><title>nmap xml file to html</title></head>
<body>
	<form method="POST" enctype="multipart/form-data" action="<?php echo $_SERVER["PHP_SELF"]; ?>">
		XML file: <input type="file" name="xmlfile"	<br />
		<input type="checkbox" checked name="open"> Open <br />
		<input type="checkbox"  name="closed"> Closed <br />
		<input type="checkbox"  name="filtered"> Filtered <br />
		<input type="submit" value="Press"> to upload the file!
	</form>
<?php

if(isset($_FILES['xmlfile'])) {

	// init
	if (trim($_POST["open"]) == "on") 	$printOpen = true;
	else   								$printOpen = false;
	if (trim($_POST["closed"]) == "on") 	$printClosed = true;
	else   								$printClosed = false;
	if (trim($_POST["filtered"]) == "on") 	$printFiltered = true;
	else   								$printFiltered = false;
	$xmlObject = simplexml_load_file($_FILES['xmlfile']['tmp_name']);

	// output the header
	echo "<h1>".(string)$xmlObject["args"]."</h1>";
	echo "<h2>Hosts up: ".(string) $xmlObject->runstats->hosts["up"]." / Hosts down: ".
			(string) $xmlObject->runstats->hosts["down"]. " / Hosts total: ".(string) $xmlObject->runstats->hosts["total"]."</h2>";

	// run through the xml and print hostinfo
	foreach($xmlObject as $host => $value) {

		// Only grab the data if it's host related info
		if ((string) $host == "host") {

			// declare portsarray
			$nmap["ports"] = array();

			// get the hostinfo
			echo "<h2>".(string) $value->hostnames->hostname["name"].
						" (".(string) $value->address["addr"]." / ".(string) $value->address["addrtype"].")</h2>";
			echo "<table>";

			// put the discovered ports in an array
			foreach ($value->ports->port as $port) {

				if (  ( ((string) $port->state["state"] == "filtered")	and	($printFiltered)) or
					  ( ((string) $port->state["state"] == "closed")	and	($printClosed)) or
					  ( ((string) $port->state["state"] == "open")	and	($printOpen))
					) {
					echo "<tr><td>".(string)$port["portid"]."/".(string)$port["protocol"]."</td><td>".
							(string)$port->state["state"]."(".(string)$port->state["reason"].")</td>
							<td>".$port->service["name"]."(".(string)$port->service["product"].")
								</tr>";
				}
			}
			echo "</table>";
		}
	}

}
?>
</body>
</html>

I use Pidgin as my primary chat client. Up until now I was forced to update my Twitter feed through SMS or a Firefox plugin.

A new plugin for Pidgin, microblog-pidgin, allows you to update your Twitter feed within Pidgin. Installation is straightforward.

Now start Pidgin, enable the plugin and add a Twitgin account.

My upgrade to Ubuntu 9 resulted in quite some frustration (ipv6, cryptodisk, …) but there’s one nice though … installing Google Earth is very easy.

Then double click on the .deb file that’s the result of the above.

Found on http://ubuntuguide.org/wiki/Ubuntu:Jaunty#Google_Earth

An interesting post by Martin Roesch on the new architecture in the beta release of Snort.

A funny post on Niallh.com

It’s a sad day for IT security when even D. J. Bernstein has to admit that there’s a flaw in DJBDNS. Luckily the patch that needs to be applied is straightforward and has no known side-effects (at least, not on the DJBDNS servers that I patched).

I attended the last joint TF-CSIRT / FIRST meeting in Riga. One of the most interesting talks was by Wayne Routly from Dante. He talked on Quantitative Cross Comparative Analysis of Tools for Anomaly Detection. He gave some very good information on StealthWatch and Netreflex. If you’re moving Gbs of data then the set of tools you can use to analyze traffic are limited. According to the presentation both tools did quite well though.