MISP Statistics

The MISP API includes a couple of features that you can use to report on the type of data stored in the database. For example the User statistics or Attribute statistics give a pretty good overview. Unfortunately, as of now it’s not possible to limit the output of these functions to a specific timeframe. For my use case I’d like to report on the MISP data statistics for the last month. The information that I want to include is

• How many new or updated events?
• How many new or updated attributes?
• How many new or updated attributes with IDS flag?
• The category of the attributes
• TLP-coverage

PyMISP example module

I wrote a PyMISP script that does all of the above and more. The script fetches the event and attribute data for a given timeframe and then reports the statistics. The report is send via e-mail and the data is attached as individual CSV files.

The script can be found in the PyMISP repository stats_report.py. The configuration of the script is inline in the Main module. If you want to receive the reports by e-mail you will have to change

smtp_from
smtp_to
smtp_server

The script should be run from cron and accepts these parameters

• -t : the timeframe, typically you’ll use ‘-t 30d’
• -e : include the MISP events titles in the output;
• -m : mail the report or only have the output to screen;

A typical use from cron would then be

*/5 *    * * *   mispuser   /usr/bin/python3 /home/mispuser/PyMISP/examples/stats_report.py -t 30d -m -e

Refactor the output

Part (or in fact the quality) of the statistics rely on how the contributors have added the data. For instances that receive events from different sources this can result in a lack of consistency, or even quality of data. This script includes some basic logic to work with this but you might have to tune this to your environment.

Event and attribute statistics

The first part of the statistics should normally be usable by all environments. Note that if you use your reporting to people outside your organisation you should indicate that the data concerns new or updated events.

MISP Report 2019-07-12 23:53:12 for last 30d on https://XXXXX/
-------------------------------------------------------------------------------
New or updated events: 658
New or updated attributes: 24834
New or updated attributes with IDS flag: 14484

Total events: 60293
Total attributes: 7382714
Total users: 2519
Total orgs: 1208
Total correlation: 8521911
Total proposals: 77595

Items to include in your report based on the output of this script are

• Evolution of number of events and attributes over time
• Evolution of number attributes with IDS flag over time

Attribute category

The next part that’s interesting to report is the number of attributes per category. According to the MISP core format RFC, the category represents the intent of what the attribute is describing as selected by the attribute creator.

Network activity 	 9530
Payload delivery 	 4963
Antivirus detection 	 3914
Financial fraud 	 3114
External analysis 	 1828
Artifacts dropped 	 694
...

If you report this information, then it’s useful to include an explanatory table for the different categories.

    Antivirus detection: All the info about how the malware is detected by the antivirus products
    Artifacts dropped: Any artifact (files, registry keys etc.) dropped by the malware or other modifications to the system
    Attribution: Identification of the group, organisation, or country behind the attack
    External analysis: Any other result from additional analysis of the malware like tools output
    Financial fraud: Financial Fraud indicators
    Internal reference: Reference used by the publishing party (e.g. ticket number)
    Network activity: Information about network traffic generated by the malware
    Other: Attributes that are not part of any other category or are meant to be used as a component in MISP objects in the future
    Payload delivery: Information about how the malware is delivered
    Payload installation: Info on where the malware gets installed in the system
    Payload type: Information about the final payload(s)
    Persistence mechanism: Mechanisms used by the malware to start at boot
    Person: A human being - natural person
    Social network: Social networks and platforms
    Support Tool: Tools supporting analysis or detection of the event
    Targeting data: Internal Attack Targeting and Compromise Information

Reporting the attribute types might only be useful if you report to a more technical audience.

TLP-codes

Reporting the TLP codes of the received events is useful to indicate if information was available for everyone or only for
restricted receivers. Note that the script tries to sanitise the different notations of the TLP codes by transforming everything to lower case and removing spaces. For example the notation of “TLP:White”, “TLP: White” and “tlp : white” should all result in “tlp:white”.

tlp:white 	 338
tlp:green 	 286
tlp:amber 	 7
tlp:red 	 0

MISP Galaxy

The use of the MISP Galaxy really depends on your sector. The next sections of the report dive deep into some categories of the MISP Galaxies but items you can report are for example

• misp-galaxy:banker : Banker malware
• misp-galaxy:financial-fraud : Financial fraud
• misp-galaxy:tool : Threat actors tooling

MISP Galaxy MITRE

Events which are added to the clusters starting ‘misp-galaxy:mitre’ are reported individually. This is good data to report how your threat intel feed covers the Mitre ATT&CK framework. As mentioned before, the quality of this data depends on the contributors.

misp-galaxy:mitre-enterprise-attack-intrusion-set="APT28 - G0007" 	 12
misp-galaxy:mitre-enterprise-attack-intrusion-set="Lazarus Group - G0032" 	 5
misp-galaxy:mitre-intrusion-set="APT28" 	 4
misp-galaxy:mitre-enterprise-attack-intrusion-set="MuddyWater - G0069" 	 3
misp-galaxy:mitre-enterprise-attack-attack-pattern="Spearphishing Attachment - T1193" 	 3
misp-galaxy:mitre-attack-pattern="Standard Application Layer Protocol - T1071" 	 3
misp-galaxy:mitre-attack-pattern="Spearphishing Attachment - T1193" 	 3
misp-galaxy:mitre-pre-attack-intrusion-set="APT28" 	 2
misp-galaxy:mitre-malware="AutoIt" 	 2
misp-galaxy:mitre-enterprise-attack-attack-patt

MISP Galaxy Threat Actor

Similar to the Mitre ATT&CK framework, the script will also report on the threat actors, if they have been added by the event contributors. This is also a great resource to report.

 
misp-galaxy:threat-actor="Sofacy" 	 24
misp-galaxy:threat-actor="Lazarus Group" 	 9
misp-galaxy:threat-actor="OilRig" 	 5
misp-galaxy:threat-actor="MuddyWater" 	 5
misp-galaxy:threat-actor="INDRIK SPIDER" 	 5
misp-galaxy:threat-actor="APT37" 	 5

Reporting failures

Do not get trapped into the “my instance has more indicators than yours”, eventually it’s the quality of the indicators that counts. Having recent and vouched (sightings isn’t included yet in the reporting) indicators is important. As a start, the MISP documentation provides you with a Feed overlap matrix.