The metrics killing your SOC, and what to use instead

Security operations centres risk being rendered entirely ineffective if organizations measure them using the wrong performance indicators, according to Dave Chismon, CTO for Architecture at UK’s National Cyber Security Centre.

Ticket-based metrics miss the point

Evaluating ones’ SOC using the same ticket-based metrics applied to IT service desks can actively work against its core purpose: detecting and responding to real attacks.

The problem, Chismon explains, is one of perverse incentives:

• When SOC analysts are measured on how quickly they close tickets, they are pushed to dismiss alerts as false positives rather than investigate them properly.
• Measuring the number of detection rules written tends to lead to a proliferation of low-quality rules
• Relying on log volume instead of log quality can create a false sense of coverage and may reduce the retention period for useful data.

The only metric that demonstrates a SOC is working, he argues, is whether it detects and responds to attacks in a timely manner, which is typically expressed as “time to detect” (TTD) or “time to respond” (TTR).

“However, this can be tricky to measure as an organization’s defence in depth means it should be a very rare event for an attack to make it into the organization’s environment,” he pointed out.

Thurefore, he recommends supplementing this with red teaming and purple teaming exercises to simulate realistic threats and test detection capability.

“The covert nature of red teaming can more accurately mimic a real attack, but purple teaming can often provide better value to a SOC (as the time saved through ‘not being covert’ can be put into greater coverage of attack paths,” he added.

The goal should be an analyst-focused SOC

The NCSC also advocates for SOC analysts to be treated as experts who know their organization’s systems intimately, understand the threats and tools they use, and have the data needed to proactively hunt for adversaries.

Of course, they also have to be given enough time to do it.

Chismon considers hypothesis-led threat hunting – where analysts form theories about plausible attacks and search for evidence in logs – as the most effective activity a SOC can undertake.

“In many cases the analyst won’t find anything, but the real output of a hypothesis-led threat hunt is the increased understanding of the techniques, and the alerts (or hardening suggestions) that the analyst proposes following the hunt,” he explained.

SOCs whould also regularly re-evaluate rules that generate too many false positives, to prevent time-consuming work that may distract from real attacks.

Finally, SOC analysts should also be evaluated on:

• How well they know the threats they’re facing (metrics: completeness of documentation; reports read and actioned)
• How proficient they are with various tools (metrics: specific training completed, certifications gained)
• How well they understand the organization they’re protecting (metrics: documentation of organizational systems, quality of relationship between the SOC analyst and IT admins)

The NCSC also says that analyst satisfaction matters, and that persistently low morale is usually a sign that something in the culture or management needs fixing.

Related:

• Building a stronger SOC through AI augmentation
• Essential metrics for effective security program assessment

Subscribe to our breaking news e-mail alert to never miss out on the latest breaches, vulnerabilities and cybersecurity threats. Subscribe here!