MISP web scraper
I published an article on the MISP project website on the MISP web scraper.
There are a lot of websites that regularly publish reports on new threats, campaigns or actors with useful indicators, references and context information. Unfortunately only a few publish information in an easily accessible and structured format, such as a MISP-feed. As a result, we often find ourself manually scraping these sites, and then copy-pasting this information in new MISP events. These tedious tasks are time-consuming and certainly not the most interesting aspect of CTI-work.
MISP-scraper is a Python script that
- Parses RSS feeds;
- Extracts the URLs from these feeds;
- Creates a MISP event for each URL. If the combination “event-URL” already exists then the event creation is skipped;
- Adds a MISP report (with the content of the URL) to the MISP event;
- And then uses the report feature to extract indicators and context from the web page;
- It is also possible to manually add URLs and outdated events are automatically deleted.
You can find MISP scraper on Github via https://github.com/cudeso/misp-scraper.