MISP correlations

MISP correlations are a way to find relationships between attributes and indicators from malware or attacks campaigns. Correlation support analysts in detecting clusters of similar activities and pivot from one event to another.

When the volume of data in your MISP instance grows, the number of correlations can however explode and make your system less responsive. I cover some approaches that you can use to stay in control.

What is correlation?

Correlation basically is a way for MISP to indicate that a certain value exists in more than one event. Typical examples include the same IP address observed during different attack campaigns or the same domain used by various phishing attacks. The correlation feature of MISP is not limited to basic technical indicators, you can also use it to correlate on for example (the same) YARA rule.

Data model

The correlation takes place on attribute level. But because attributes are enclosed in an event, they are also represented on event level.

There are in fact three ways how MISP informs you of correlations:

1. On the event index;
2. In the event detail page;
3. Next to the attributes that cause the correlation.

Options for disabling correlation

Reasons for disabling correlation

Some of the reasons why you would choose to disable correlation include

• The correlation is on attributes that have no real value for your organisation;
• The value is not very specific. An example is correlating on a destination port. Correlating on tcp/80 or tcp/443 is maybe not that useful, whereas correlating on a high network port (above 1024) can be useful;
• Your system has not sufficient resources to cope with all the correlations.

Per attribute

Correlation in MISP is done in the background and doesn’t require additional effort from an analyst.

That said, you can still prevent correlation from taking place. When you add attributes, either manually, in batch or via the freetext import you always have the choice to override the automatic correlation by checking Disable Correlation.

Important to realise is that disabling the correlation on attribute level only disables the correlation for the specific attribute you’re adding/editing. It does not disable the MISP correlation engine for other attributes or other events.

Per event

Instead of disabling correlation per attribute, you can also disable correlation on event level. By enabling MISP.allow_disabling_correlation you give event creators the option to disable all correlations for a specific event.

Exclude values for correlation

You can exclude correlation from taking place on specific values with correlation exclusion entries. Under Input Filters > List Correlation Exclusions you can add a list of attribute values for which you want no correlation to take place.

As you can notice, you can also cleanup the already existing correlations.

Completely disable correlation

You can disable the correlation index of MISP completely by enabling MISP.completely_disable_correlation. Enabling this setting will trigger a full recorrelation of all data which is an extremely long and costly procedure.

Only enable this if you know what you’re doing.

Correlation, performance and resources

Performance tuning

In some cases your system might have sufficient resources to cope with correlation but you still experience slowness when logging in. This is most likely because you are displaying the number of correlations on the event index page. Correlations aren’t cached, this means that they are requested (counted) every time when accessing the event index page. You can get a huge performance increase on the event index page by disabling MISP.showCorrelationsOnIndex.

This does not disable correlation. It just prevents correlations from being displayed on the event index page. You still have access to all correlations on the event details page and in the list of attributes.

“overhead” statistics

MISP helps you in diagnosing some of the resource issues related to correlation. Under Server Settings & Maintenance > Diagnostics you can use the SQL database status to get an overview of the volume of correlations and how much drive space is reclaimable. Reclaiming this drive space can give your server some extra (storage) breathing room.

The fact that you can “reclaim” drive space is the result of database manipulations taking place when adding, editing and deleting data.

MISP does not include tools to reclaim the drive space via its administration interface. You need to use the mysql administration tool for this. Once you are logged in into mysql, use the command optimize table correlations.

MariaDB [misp]> optimize table correlations;
+-----------------+----------+----------+-------------------------------------------------------------------+
| Table           | Op       | Msg_type | Msg_text                                                          |
+-----------------+----------+----------+-------------------------------------------------------------------+
| misp.correlations | optimize | note     | Table does not support optimize, doing recreate + analyze instead |
| misp.correlations | optimize | status   | OK                                                                |
+-----------------+----------+----------+-------------------------------------------------------------------+
2 rows in set (3 min 42.36 sec)

Important to note is that in order to reclaim space, you first need to have sufficient space (at least the size of the table) available to start the reclaiming. If there’s not sufficient space, you’ll get an error (the error message isn’t that informative).

MariaDB [misp]> optimize table correlations;
+-------------------+----------+----------+-------------------------------------------------------------------+
| Table             | Op       | Msg_type | Msg_text                                                          |
+-------------------+----------+----------+-------------------------------------------------------------------+
| misp.correlations | optimize | note     | Table does not support optimize, doing recreate + analyze instead |
| misp.correlations | optimize | error    | Incorrect key file for table 'correlations'; try to repair it     |
| misp.correlations | optimize | status   | Operation failed                                                  |
+-------------------+----------+----------+-------------------------------------------------------------------+
3 rows in set, 1 warning (1 min 13.59 sec)

Conclusions

MISP correlations provide an excellent way to pivot between different threat events. It can however be challenging to cope with the required system resources. It’s highly advised not to disable the correlation engine, but to exclude certain low values from correlation and run the SQL-optimize commands on a regular basis.

MISP and Elastic

In this post I go through the process of representing threat data from MISP in Elastic. The goal is to push attributes from MISP to Elastic and have a representation with a couple of pretty graphs. This is an alternative approach to using the MISP dashboard (and MISP-Dashboard, real-time visualization of MISP events).

Filebeat MISP

The Filebeat component of Elastic contains a MISP module. This module queries the MISP REST API for recently published event and attribute data and then stores the result in Elastic. Unfortunately it still has some sharp edges.

TL;DR : Representing MISP data in Elastic with the Filebeat module (via Docker) “kinda” works, but is not ready for production.

Workflow

Elastic and Filebeat

I used my Elastic dfir cluster setup to get an Elastic stack up-and-running and then started Filebeat from a Docker container. After starting the container I setup the build-in dashboard and vizualisations of the Filebeat plugins. The exact process is described in the text file filebeat-dfir.txt and in the article Analyse Linux (syslog, auditd, …) logs with Elastic. You need to change the Filebeat MISP configuration by adding the API key and setting the correct url.

filebeat.config.modules:
  enabled: true
  path: /modules.d/*.yml

filebeat.modules:
- module: misp
  threat:
    enabled: true
    # API key to access MISP
    var.api_key: ""

    # Array object in MISP response
    var.json_objects_array: "response.Attribute"

    # URL of the MISP REST API
    var.url: "https://misp/attributes/restSearch/last:15m"
    var.http_client_timeout: 60s
    var.interval: 15m
    var.ssl: |-
      {
        verification_mode: none
      }
output.elasticsearch:
  hosts: ["elasticdfir01:9200"]

setup.kibana:
  host: "kibana:5601"

Under the hood the MISP Filebeat module uses the httpjson fetcher of Filebeat.

Once you started the Filebeat container it is also useful to monitor the Filebeat logs. If you spot the below error then something between the container and MISP is wrong, either on a network level or the MISP server can be under a heavy load. Note that the container will not recover, or restart the process automatically. You’ll have to do this manually.

[httpjson]	httpjson/input.go:145	failed to execute http client.Do: Get "https://misp/attributes/restSearch/last:1d": GET https://misp/attributes/restSearch/last:1d giving up after 6 attempts

Frequency to query the data

As you can see in the configuration file, the data changed in the last 15 minutes (last:15m) is fetched every 15 minutes (var.interval). So why not fetch the data for the last day (last:1d) with an hourly interval (var.interval)? If you do this, then the returned data will be stored multiple times in the Elastic database. Yes, all the data returned from the REST API query is stored in Elastic. Needless to say that this duplication of data makes the approach via Elastic less useful.

There are two options to solve this, either delete the index first and then fetch all data again (for example with a daily interval) or align the interval timeframe with the timeframe for which you want to receive ‘updated’ data. I choose for the latter. The Elastic developers are aware of this issue, as described in the latest pull request to move from UUID to using the fingerprint processor on the ID field.

KQL and rule IDs

One of the interesting addition is that the Filebeat module immediately adds a KQL query based on the indicator data.

The MISP event IDs are added as the rule.id and rule.uuid.

Unfortunately these fields are not added as a hyperlink to MISP. This is something that you can achieve by using the string rewriting concept of Kibana, as described in Report sightings from Kibana to MISP.

Field conversions

The detailed field conversions from MISP to Elastic are covered in the Filebeat MISP pipeline. According to this file, it should also copy tag names from MISP event to tags field. In the Docker version of Filebeat (docker.elastic.co/beats/filebeat:7.9.2) this conversion was not included.

Kibana dashboard

The Kibana map visualisation Threat Indicator Geo Map [Filebeat MISP] from the Filebeat module does not correctly set the field for the geo location. By default it is set to source.geo.location but you need to change this to destination.geo.location.

Conclusion

Making available the MISP data via Elastic is a good alternative to grant (junior) SOC analysts access to threat data, without introducing some of the complexities of the MISP interface. Unfortunatley you loose some of the advantages such as correlation, context and galaxy/cluster relations.

This approach is not a replacement for using MISP for enriching (or querying) data in Elastic, it’s just an extra way of disseminating data to a larger audience.

The Filebeat module for MISP still requires some extra work, so go ahead and contribute to the code!