Stay Up-to-Date on Security Trends

In 2015 I did a posting on the Security Intelligence blog on How to Stay Up-to-Date on Security Trends. The post describes how you can streamline the process of following different news and threat information channels, classify them and bring them to good use.

One of the tools that you can use is RSS feeds. I personally use a setup of fever to grab different RSS feeds and then have them delivered in one central RSS feed. This makes it easy for me to scroll through the different news items and mark the interesting articles for later reading.

OPML Security Feeds

I wanted to share my feeds with the community. The easiest way of sharing these feeds is through OPML files. OPML is an XML format for exchanging lists of Web feeds. You can import OPML feeds in most RSS readers.

You can find my OPML files in the GitHub repository : OPML-Security-Feeds. Feel free to contribute or mark older feeds as stale.

MISP

I love MISP, Malware Information Sharing Platform & Threat Sharing. I did three earlier posts on how to use and setup MISP. part 1, part 2 and part 3.

Open Source Information by MISP, OSINT

One of the nice new features by MISP is including feeds from different open source intelligence feed providers.

How does it work? Basically the feeds are provided as a JSON feed, you can browse them within MISP, import them individually or subscribe to the feed to get automatic updates.

Using OSINT feeds within MISP

First I assume that you upgraded MISP to the latest version. This is straightforward by pulling the latest version from Github.

git pull

Once this is done, log in to your MISP instance and go to Sync actions, List Feeds. You will then get a list of the available open source feeds.

Then you will need to enable or edit a feed. This will bring you to a screen where you can set the default distribution level and most importantly, the default tag.

For my setup I use the TLP:White tag. Remember that these tags are defined in the taxonomies that you have defined in MISP (see Event Actions -> List Taxonomies). I use TLP:White because OSINT information is by default TLP:White. Not sure about the TLP code? Use the post How to use the traffic light protocol – TLP.

Do not forget to tick the checkbox Enabled because otherwise the feed will not be enabled.

Once you have edited the feed you will return to the overview of available feeds.

You can browse the feed content. Do this for the feed that you enabled.

Basically that is all there is for including an open source feed to your MISP data.

• Enable a feed
• Set the distribution-level and tag it
• Sit back and enjoy

I want to contribute open source intelligence feeds

Using the OSINT feed is great but maybe you also want to contribute your OSINT feed to the community? You can. And it’s fairly easy. For this you need to use a Python script : PyMISP.

PyMISP

PyMISP is a Python library using the MISP Rest API. In essence this means that instead of crafting MISP API requests you can use Python request to interact with MISP.

Installing PyMISP is described on the GitHub page.

git clone https://github.com/CIRCL/PyMISP.git
cd PyMISP
python setup.py install

Basically it needs an API key and a URL. This can be set in any file you desire as long as you include it in your Python script that calls PyMISP. As an example you can use this in you Python scripts

from pymisp import PyMISP
from cudeso import misp_key
from cudeso import misp_url
from cudeso import misp_verifycert

This will import the API key and url from an external file.

Using PyMISP for OSINT

Once you have setup PyMISP you can use one of the example scripts to generate the OSINT feed. In the folder PyMISP/examples/feed-generator/ there are two files that you need to edit. Note that the OSINT generator script does not use the configuration script of PyMISP described above.

First you need to edit the file settings.py. The important settings to change are

url = '<MISP-URL>'
key = '<MISP-automation-key>'
outputdir = 'output-dir'

The MISP url (url) defines where the script can find you MISP instance. The automation key (key) is the key set by your automation user. Finally the outputdir sets where you want to output the files. Note that this directory will hold all the MISP events. You have to make sure that this directory

• is writable by the generate script
• is a separate directory
• accessible by the public if you want to publicize your feed (for example in your web directory).

Once you have set these settings you can run the feed generator script by issuing

./generate.py

This will generate all the MISP events in the output directory set by “outputdir”.

Make your feed known!

If you are happy with your feed you should announce it to CIRCL to have it included as one of the OSINT feeds in the next update of MISP. The easiest way for doing so is via the MISP GitHub repository.

Existing open source feeds

The current release of MISP contains these OSINT feeds

• https://www.circl.lu/doc/misp/feed-osint
• http://www.botvrij.eu/data/feed-osint

I maintain the feed for botvrij.eu. If you want to include a description of a threat info then you can send me a MISP XML file or a pointer to a public description.

Open source intelligence

Open source intelligence is collecting information from publicly available resources. If you are doing incident handling it’s one of the things that will use up a lot of your time. And it can quickly become very tedious. Imagine a list of IPs that you found hosts on your network connecting to. Query different public available resources (VirusTotal, Shodan, SANS, Cymon, XForce Exchange, …) for each and every IP and then converting that data into one uniform dataset is time-consuming.

I started developing my own tool (host-enrich on Github) for collecting this type of information but by coincidence I stumbled on a tool that had almost everything that I needed.

Enters SpiderFoot.

SpiderFoot

SpiderFoot automates open source intelligence. It’s a modular platform written in Python that gathers data from different sources and presents the data visually and allows for exporting data.

This post describes the use of SpiderFoot with a focus on gathering passive reconnaissance data and getting the data in an easy accessible (CSV) format.

It is important to note that this post is based on the changes that I commited to my fork of SpiderFoot. Not all changes have yet been committed as pull requests. Pending the pull requests I advice you to use the source in my Github fork at https://github.com/cudeso/spiderfoot.

The default source is available on Github. The installation and setup is described in detail in the online documentation.

Get started with SpiderFoot

Once downloaded and extracted you can start SpiderFoot easily with

./sf.py 192.168.168.1:5001

This commands has it running on port 5001 on the IP 192.168.168.1.

I you start SpiderFoot you should immediately head to the Settings tab. The left part of your screen will list the available modules. If you already have API keys then this is a good time to add them. For example the modules for Cymon, Shodan and VirusTotal require an API key. Add your keys and click Save Changes.

None of this data is sent outside of your network. Everything is stored locally in a sqlite database.

The next thing you want to do is start a New Scan. I advise you to use the Intelligence use case (note that this is one of the code changes that I added). Enter a descriptive scan name, a scan target and click Run Scan. For this post you can use the IP 211.202.2.97 as an example.

You’ll get visual feedback on the scan progress. Further details on the scan progress can be found in the log section.

Once the scan is finished you get an overview of all the elements that have been found.

You can also drill down on the data that was found and get for example the physical location, the passive dns data records or the URLs that were hosting malicious data

As a last step you can now export the data, either globally or per item found.

Passive reconnaissance with SpiderFoot

Passive reconnaissance is an attempt to gain information about targeted computers and networks without actively engaging with the system. When you are dealing with “malicious” IPs or domains this is very important. Contacting an IP (especially when you’re not using a proxied-connection, see below for proxy integration) that is involved with malicious activity on your network is basically giving away you found out about its presence. This is something that you should avoid.

SpiderFoot comes with use cases. I added a use case Intelligence that groups all modules that do passive reconnaissance, solely relying on third party public sources. Of course, nothing prevents the bad-guys from monitoring the use of these sources but that’s another issue.

You can easily add or remove modules from this use case by changing the sourcecode. A module is added to a use case via a comment setting in the class definition. For example the VirusTotal module this is set to

"""VirusTotal:Investigate,Intelligence:Obtain information from VirusTotal about identified IP addresses."""

This means that the VirusTotal module is used in the use cases Investigate and Intelligence. Finding out which modules are included in the Intelligence use case can be done via

fgrep -R Intelligence modules/*
modules/sfp_cymon.py:    """Cymon:Investigate,Intelligence:Obtain information from Cymon, a tracker of open-source security reports."""
modules/sfp_sansisc.py:    """SANS_ISC:Investigate,Intelligence:Obtain information from SANS ISC about identified IP addresses."""
...

Searching for specific intelligence

Next to performing a search by use case you can also search for a specific type of data with the “By Required Data” or “By Module” search. These become more useful if you know exactly what type of information you are looking for. Remember that some of the modules will interact with the search subject.

Command line interface

The web interface of SpiderFoot is great but sometimes a command line interface is easier when dealing with large sets of scans or if you want to use SpiderFoot for automation.

I included in my code fork a command line script to interact with SpiderFoot. It is very basic and primary focused on using the Intelligence use case.

You can start the script with

/sf_cli.py myScan 211.202.2.97

This will launch a scan named ‘myScan’ for intelligence data on the given IP. The output is a CSV-formatted set of data.

myScan,2015-12-30 21:18:13,URL_MALICIOUS,sfp_cymon,211.202.2.97,0,http://zenocompany.com/catalog/images/alexis.php
myScan,2015-12-30 21:18:13,URL_MALICIOUS,sfp_cymon,211.202.2.97,0,http://zenocompany.com/pshop/news_skin/file.php
myScan,2015-12-30 21:18:13,GEOINFO,sfp_sansisc,211.202.2.97,0,ip-adm@hanaro.com
myScan,2015-12-30 21:18:13,DNS_PASSIVE,sfp_cymon,211.202.2.97,0,jsoft21.co.kr
myScan,2015-12-30 21:18:13,DNS_PASSIVE,sfp_cymon,211.202.2.97,0,mdasj.co.kr
myScan,2015-12-30 21:18:13,DNS_PASSIVE,sfp_cymon,211.202.2.97,0,redbrick.co.kr

The configuration of the script is inline. Change the csf_host and sf_port for your configuration.

csf_host = "192.168.171.159"
sf_port = "5001"
sf_usecase = "Intelligence"
verbose = False

The script submits the scan requests, waits 10 seconds (set in the variable time_sleep) for the scan to finish and then returns a CSV output.

Visual representation

SpiderFoot also supports the visual representation of the data that has been found. I’ve not used this a lot but for some purposes this can give you an immediate overview of how the different data elements are inter-connected.

Proxy integration

If you plan on query your search subject directly I strongly advice you to use some sort of proxy that routes your traffic through an “anonymous” network. SpiderFoot has build-in integration for Tor (see the Settings page). Alternatively you can also use a proxy and export your session to use a designated proxy.

Future integration

SpiderFoot will see a lot of future development. One of my future ideas is to develop an interface with MISP, Malware Information Sharing Platform & Threat Sharing. I wrote about MISP before in three posts (Getting started with MISP, Malware Information Sharing Platform & Threat Sharing). The basic idea would be to

• Query a MISP instance for a subject, return the info found in MISP for that subject;
• Enrich a MISP event with data found for a given subject.

Next to the MISP integration it would also be great to export the SpiderFoot data immediately to Elasticsearch. That way I would be able to interconnect them with the data found in honeypots or enrich it with Twitter data (Tweetsniff).

Additionally, besides MISP, I’d also like to have SpiderFoot talk to a nfdump-nfsen sensor (alerts from nfsen, automatically investigate the remote side of the connection).

Conclusion

SpiderFoot is a tool that saves me a lot of time. Instead of manually having to query VirusTotal, Cymon, Shodan and other sources I can now do this with one command.

I primarily use the command line interface and then import the CSV in a central database.