MISP web scraper

MISP web scraper

I published an article on the MISP project website on the MISP web scraper.

There are a lot of websites that regularly publish reports on new threats, campaigns or actors with useful indicators, references and context information. Unfortunately only a few publish information in an easily accessible and structured format, such as a MISP-feed. As a result, we often find ourself manually scraping these sites, and then copy-pasting this information in new MISP events. These tedious tasks are time-consuming and certainly not the most interesting aspect of CTI-work.

MISP-scraper is a Python script that

  • Parses RSS feeds;
  • Extracts the URLs from these feeds;
  • Creates a MISP event for each URL. If the combination “event-URL” already exists then the event creation is skipped;
  • Adds a MISP report (with the content of the URL) to the MISP event;
  • And then uses the report feature to extract indicators and context from the web page;
  • It is also possible to manually add URLs and outdated events are automatically deleted.

You can find MISP scraper on Github via https://github.com/cudeso/misp-scraper.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.