The MISP API includes a couple of features that you can use to report on the type of data stored in the database. For example the User statistics or Attribute statistics give a pretty good overview. Unfortunately, as of now it’s not possible to limit the output of these functions to a specific timeframe. For my use case I’d like to report on the MISP data statistics for the last month. The information that I want to include is
- How many new or updated events?
- How many new or updated attributes?
- How many new or updated attributes with IDS flag?
- The category of the attributes
PyMISP example module
I wrote a PyMISP script that does all of the above and more. The script fetches the event and attribute data for a given timeframe and then reports the statistics. The report is send via e-mail and the data is attached as individual CSV files.
The script can be found in the PyMISP repository stats_report.py. The configuration of the script is inline in the Main module. If you want to receive the reports by e-mail you will have to change
smtp_from smtp_to smtp_server
The script should be run from cron and accepts these parameters
- -t : the timeframe, typically you’ll use ‘-t 30d’
- -e : include the MISP events titles in the output;
- -m : mail the report or only have the output to screen;
A typical use from cron would then be
*/5 * * * * mispuser /usr/bin/python3 /home/mispuser/PyMISP/examples/stats_report.py -t 30d -m -e
Refactor the output
Part (or in fact the quality) of the statistics rely on how the contributors have added the data. For instances that receive events from different sources this can result in a lack of consistency, or even quality of data. This script includes some basic logic to work with this but you might have to tune this to your environment.
Event and attribute statistics
The first part of the statistics should normally be usable by all environments. Note that if you use your reporting to people outside your organisation you should indicate that the data concerns new or updated events.
MISP Report 2019-07-12 23:53:12 for last 30d on https://XXXXX/ ------------------------------------------------------------------------------- New or updated events: 658 New or updated attributes: 24834 New or updated attributes with IDS flag: 14484 Total events: 60293 Total attributes: 7382714 Total users: 2519 Total orgs: 1208 Total correlation: 8521911 Total proposals: 77595
Items to include in your report based on the output of this script are
- Evolution of number of events and attributes over time
- Evolution of number attributes with IDS flag over time
The next part that’s interesting to report is the number of attributes per category. According to the MISP core format RFC, the category represents the intent of what the attribute is describing as selected by the attribute creator.
Network activity 9530 Payload delivery 4963 Antivirus detection 3914 Financial fraud 3114 External analysis 1828 Artifacts dropped 694 ...
If you report this information, then it’s useful to include an explanatory table for the different categories.
Antivirus detection: All the info about how the malware is detected by the antivirus products Artifacts dropped: Any artifact (files, registry keys etc.) dropped by the malware or other modifications to the system Attribution: Identification of the group, organisation, or country behind the attack External analysis: Any other result from additional analysis of the malware like tools output Financial fraud: Financial Fraud indicators Internal reference: Reference used by the publishing party (e.g. ticket number) Network activity: Information about network traffic generated by the malware Other: Attributes that are not part of any other category or are meant to be used as a component in MISP objects in the future Payload delivery: Information about how the malware is delivered Payload installation: Info on where the malware gets installed in the system Payload type: Information about the final payload(s) Persistence mechanism: Mechanisms used by the malware to start at boot Person: A human being - natural person Social network: Social networks and platforms Support Tool: Tools supporting analysis or detection of the event Targeting data: Internal Attack Targeting and Compromise Information
Reporting the attribute types might only be useful if you report to a more technical audience.
Reporting the TLP codes of the received events is useful to indicate if information was available for everyone or only for
restricted receivers. Note that the script tries to sanitise the different notations of the TLP codes by transforming everything to lower case and removing spaces. For example the notation of “TLP:White”, “TLP: White” and “tlp : white” should all result in “tlp:white”.
tlp:white 338 tlp:green 286 tlp:amber 7 tlp:red 0
The use of the MISP Galaxy really depends on your sector. The next sections of the report dive deep into some categories of the MISP Galaxies but items you can report are for example
- misp-galaxy:banker : Banker malware
- misp-galaxy:financial-fraud : Financial fraud
- misp-galaxy:tool : Threat actors tooling
MISP Galaxy MITRE
Events which are added to the clusters starting ‘misp-galaxy:mitre’ are reported individually. This is good data to report how your threat intel feed covers the Mitre ATT&CK framework. As mentioned before, the quality of this data depends on the contributors.
misp-galaxy:mitre-enterprise-attack-intrusion-set="APT28 - G0007" 12 misp-galaxy:mitre-enterprise-attack-intrusion-set="Lazarus Group - G0032" 5 misp-galaxy:mitre-intrusion-set="APT28" 4 misp-galaxy:mitre-enterprise-attack-intrusion-set="MuddyWater - G0069" 3 misp-galaxy:mitre-enterprise-attack-attack-pattern="Spearphishing Attachment - T1193" 3 misp-galaxy:mitre-attack-pattern="Standard Application Layer Protocol - T1071" 3 misp-galaxy:mitre-attack-pattern="Spearphishing Attachment - T1193" 3 misp-galaxy:mitre-pre-attack-intrusion-set="APT28" 2 misp-galaxy:mitre-malware="AutoIt" 2 misp-galaxy:mitre-enterprise-attack-attack-patt
MISP Galaxy Threat Actor
Similar to the Mitre ATT&CK framework, the script will also report on the threat actors, if they have been added by the event contributors. This is also a great resource to report.
misp-galaxy:threat-actor="Sofacy" 24 misp-galaxy:threat-actor="Lazarus Group" 9 misp-galaxy:threat-actor="OilRig" 5 misp-galaxy:threat-actor="MuddyWater" 5 misp-galaxy:threat-actor="INDRIK SPIDER" 5 misp-galaxy:threat-actor="APT37" 5
Do not get trapped into the “my instance has more indicators than yours”, eventually it’s the quality of the indicators that counts. Having recent and vouched (sightings isn’t included yet in the reporting) indicators is important. As a start, the MISP documentation provides you with a Feed overlap matrix.