Using open source intelligence feeds, OSINT, with MISP

MISP

I love MISP, Malware Information Sharing Platform & Threat Sharing. I did three earlier posts on how to use and setup MISP. part 1, part 2 and part 3.

Open Source Information by MISP, OSINT

One of the nice new features by MISP is including feeds from different open source intelligence feed providers.

How does it work? Basically the feeds are provided as a JSON feed, you can browse them within MISP, import them individually or subscribe to the feed to get automatic updates.

Using OSINT feeds within MISP

First I assume that you upgraded MISP to the latest version. This is straightforward by pulling the latest version from Github.

git pull

Once this is done, log in to your MISP instance and go to Sync actions, List Feeds. You will then get a list of the available open source feeds.


MISP Feeds

Then you will need to enable or edit a feed. This will bring you to a screen where you can set the default distribution level and most importantly, the default tag.

For my setup I use the TLP:White tag. Remember that these tags are defined in the taxonomies that you have defined in MISP (see Event Actions -> List Taxonomies). I use TLP:White because OSINT information is by default TLP:White. Not sure about the TLP code? Use the post How to use the traffic light protocol – TLP.

Do not forget to tick the checkbox Enabled because otherwise the feed will not be enabled.

MISP-Feed Edit

Once you have edited the feed you will return to the overview of available feeds.

You can browse the feed content. Do this for the feed that you enabled.


MISP feed browse

This will result in a an overview of the open source events from that feed. You can now select an invidividual event to be included in your MISP instance.

MISP Feed import

Basically that is all there is for including an open source feed to your MISP data.

  • Enable a feed
  • Set the distribution-level and tag it
  • Sit back and enjoy

I want to contribute open source intelligence feeds

Using the OSINT feed is great but maybe you also want to contribute your OSINT feed to the community? You can. And it’s fairly easy. For this you need to use a Python script : PyMISP.

PyMISP

PyMISP is a Python library using the MISP Rest API. In essence this means that instead of crafting MISP API requests you can use Python request to interact with MISP.

Installing PyMISP is described on the GitHub page.

git clone https://github.com/CIRCL/PyMISP.git
cd PyMISP
python setup.py install

Basically it needs an API key and a URL. This can be set in any file you desire as long as you include it in your Python script that calls PyMISP. As an example you can use this in you Python scripts

from pymisp import PyMISP
from cudeso import misp_key
from cudeso import misp_url
from cudeso import misp_verifycert

This will import the API key and url from an external file.

Using PyMISP for OSINT

Once you have setup PyMISP you can use one of the example scripts to generate the OSINT feed. In the folder PyMISP/examples/feed-generator/ there are two files that you need to edit. Note that the OSINT generator script does not use the configuration script of PyMISP described above.

First you need to edit the file settings.py. The important settings to change are

url = '<MISP-URL>'
key = '<MISP-automation-key>'
outputdir = 'output-dir'

The MISP url (url) defines where the script can find you MISP instance. The automation key (key) is the key set by your automation user. Finally the outputdir sets where you want to output the files. Note that this directory will hold all the MISP events. You have to make sure that this directory

  • is writable by the generate script
  • is a separate directory
  • accessible by the public if you want to publicize your feed (for example in your web directory).

Once you have set these settings you can run the feed generator script by issuing

./generate.py

This will generate all the MISP events in the output directory set by “outputdir”.

Make your feed known!

If you are happy with your feed you should announce it to CIRCL to have it included as one of the OSINT feeds in the next update of MISP. The easiest way for doing so is via the MISP GitHub repository.

Existing open source feeds

The current release of MISP contains these OSINT feeds

  • https://www.circl.lu/doc/misp/feed-osint
  • http://www.botvrij.eu/data/feed-osint

I maintain the feed for botvrij.eu. If you want to include a description of a threat info then you can send me a MISP XML file or a pointer to a public description.

5 thoughts on “Using open source intelligence feeds, OSINT, with MISP

    • You can do that for botvrij.eu ; the different files (NIDS, hashes, etc) are made available as download. For the MISP data, these are made available as JSON files, in theory you can just fetch them and postprocess the way you want.

Leave a Reply

Your email address will not be published. Required fields are marked *