Metrics for simulation exercises

Tabletop simulation exercises have been used in various fields as a form of training for incident or crisis situations. In the field of incident response, specific objectives are sought, but the results can sometimes be subjective or diffuse and organizations are faced with the difficulty of not being able to clearly quantify their benefits and improvements.

In this post we will share a way to address this issue by sharing a proprietary model to quantify the effects of incident response simulation exercises in cybersecurity.

Simulation exercises

Tabletop simulation exercises (TTX), as we conduct them, are activities that allow us to simulate realistic situations in a controlled, low-risk environment, thus exposing people to fictitious scenarios aligned with an incident. These usually involve teams from different areas, the main objective is to assess the ability of teams to make decisions, coordinate in response to a scenario or crisis situation.

All this allows organizations to assess their ability to respond and prepare for potential crisis situations, identify areas for improvement and take preventive measures to minimize risks. They can also help meet regulatory or regulatory requirements, and find the best ways to maintain reputation and credibility in the event of a crisis situation that impacts these aspects.

Incident simulations as a point of improvement

This type of exercise is a key tool for improving response capabilities, although the scenarios must be aligned with the organization's operations, have a clear objective and a well-defined scope. In this way, participants can face events and situations that may include challenges and unexpected elements, similar to those that could appear in real life, acting according to their own role in the organization. During execution, participants discuss how they would address the situation presented, make decisions, and develop strategies based on the information provided, coordinating according to their roles to communicate and share information, collaborate on solutions and address the challenges posed.

From here on, we will focus only on exercises oriented to the practice of incident response, represented in many cases by a specialized response team (CSIRT).

ENISA and NIST for CSIRTs

In order to be able to assess the incident response capabilities of an organization, a specific methodology is required. ENISA (European Union Agency for Cybersecurity) has developed a methodology to assess the maturity in the capabilities of cybersecurity incident response teams (CSIRTs) that can also function as a suitable proxy for the determination of these capabilities, even in organizations that do not have a specialized cybersecurity incident response team. This makes a division in the form of "parameters" that include different evaluable aspects, Organization (O), Processes (P), Tools (T), and Human (H), a total of 44 parameters.

Diagram of the 3 maturity levels for each parameter (ENISA)

O-parameters address structural aspects of the team, defining management, authority, responsibilities and response services, as well as their integration into broader incident response frameworks, thus assessing their organizational and operational basis.

P-parameters refer to the procedures and workflows that a response team maintains in its day-to-day operation. They evaluate how the team manages incidents, from detection and prevention to resolution, as well as interaction and communication with other entities. They also provide a measure of the efficiency and effectiveness of the response processes themselves.

T-parameters refer to the technological tools and resources that a response team uses in its operations. They assess the availability, resilience and effectiveness of tools to detect, prevent and respond to security incidents, and provide a measure of the technological resources to deal with cybersecurity incidents.

H-parameters refer to aspects related to people and their skills within the team. They address issues such as code of conduct, staff resilience, skills description, internal and external training, and outward networking capability, and assess how the team manages and improves the competencies and behaviors of its members to respond to cybersecurity incidents.

ENISA's maturity assessment provides us with a set of parameters that need to be improved in order to achieve the expected maturity.

On the other hand, the NIST (National Institute of Standards and Technology) in its cybersecurity incident management guide (SP800-61r2) provides a guide for cybersecurity incident management. This guide is especially relevant for organizations that need to prepare to effectively respond to cybersecurity incidents. This guide contains recommendations on:

- Incident Preparedness
- Detection and Analysis
- Containment, Eradication and Recovery
- Post-Incident
- Coordination and Information Sharing

This guide will allow us to take the above parameters to be improved and prioritize them, so that we can select the most relevant ones to develop in relation to these frameworks.

Planning a TTX

The parameters to be improved will then be adopted as usable objectives for the design of a scenario to support their improvement. With this the complete process of a methodologically developed tabletop exercise allows us to make special focus on the improvements needed for the incident response team. After this execution the results will be worked on, plans will be developed to implement the improvements found.

Of course there are cases in which the complexity of the teams or the improvements required by the organizations will require a more exhaustive process, in which case, for example, a series of exercises with different scopes could be developed, instead of just one. Also, variants for each exercise can include specific instances for certain teams, instances between different hierarchies, instances involving third parties, and other possible combinations.

Proposed application model

Some conclusions

It is clear that tabletop simulation exercises are a useful tool for improving response and recovery capabilities in organizations, but their net effect is not obvious, and uncertainties about their actual usefulness are often raised.

Technical teams sometimes underestimate the revealing power of this type of exercise, especially when they have a great deal of technical knowledge and experience with technological tools, although this is also sometimes the case at the executive level, which minimizes the potential benefits derived from it.

The use of reference standards and methodologies in the industry gives greater legitimacy to the proposals, and allows achieving a validity with more rational elements. Beyond the fact that the application of each guide or framework can be done differently, this proposal proposes an approach in which elements of both references (ENISA and NIST) can be used, both being important in the industry.