Do you have something cool to share? Some questions? Let us know:
This discussion explores the necessary transformation of Security Operations Center (SOC) metrics as organizations integrate Artificial Intelligence (AI) and advanced automation. The conversation moves beyond the traditional reliance on "Mean Time to Detect" (MTTD) and "Mean Time to Respond" (MTTR), which the participants argue are often gamed or fail to reflect true operational quality. Instead, the dialogue emphasizes a shift toward human-machine augmentation, quality-based outcomes, and risk-aligned storytelling for executive boards.
A central theme is the "Surface Maturity Triangle," a framework for measuring detection and response capabilities across infrastructure, applications, and data. As AI enters the SOC, the "goalposts" for these metrics are shifting from hours and minutes to seconds, transitioning the industry from a reactive "detect and respond" posture to a proactive "infer and interrupt" model.
Detailed Conversation Analysis
1. The Death and Evolution of Legacy Metrics
The panel addresses the "Caesar" of metrics: MTTD and MTTR. Michael Sinno posits that metrics must evolve rather than be discarded. The primary criticism of legacy time-based metrics is their tendency to create perverse incentives. When analysts are measured solely on speed, they may "teach to the test," clicking through tickets to satisfy a timer while sacrificing the quality of the investigation.
Alexander Pabst highlights the mathematical flaw in using "mean" (average) values. A million automated incidents resolved in seconds can hide a single catastrophic incident that took three days to resolve. The group suggests that while internal technical metrics remain necessary for operational hygiene, they must be paired with quality metrics—using AI to audit ticket summaries and reopen investigations that do not meet rigorous standards.
2. Shifting the Paradigm: Infer and Interrupt
The conversation pivots to the future of SOC operations, moving away from retrospective detection. Michael Sinno describes a shift toward "Infer and Interrupt." Instead of identifying a breach after it occurs, the goal is to use AI to identify suspicious patterns in real-time and issue challenges (such as identity verification) to mitigate threats before they manifest. This represents a fundamental change in the "game" of security, moving the field of play from historical logs to active session intervention.
3. The Surface Maturity Triangle and AI Integration
The guests revisit a maturity model—often referred to as the "Triangle of Doom"—which tracks security maturity across different layers (Infrastructure, OS, Application, Data). The integration of AI introduces a new management layer.
Automation Ratios: Google maintains a high ratio of tickets closed without human intervention, while highly regulated entities like Allianz maintain a lower ratio due to auditability requirements.
Goalpost Movement: AI doesn't just make processes faster; it changes the expected baseline. The "patch sound barrier"—a term for the physical and organizational limits of speed—is being challenged by agentic solutions that can write or tune detections autonomously.
4. Regulation, Auditability, and the "Non-Deterministic" Hurdle
A significant portion of the debate centers on the friction between AI and regulation (e.g., DORA in Europe).
The Auditor’s Dilemma: Regulators require determinism and clear audit trails. Explaining to an auditor that an incident was missed because a non-deterministic AI deemed it "uninteresting" remains a significant hurdle.
Observability: To counter this, SOC leaders must focus on "taking auditors on the journey," showing the "golden data" and the logic used to construct AI models, even if the specific output involves fuzzy logic.
5. Board-Level Reporting and Risk Economics
The guests contrast how they report to their respective boards:
Google’s Approach: Detailed metrics including pipeline latency and detection infrastructure health are reported, reflecting a high-tech-literate board.
Allianz’s Approach: Reporting focuses on Risk Appetite and Exposure. The board is less interested in "packets blocked" (which Pabst dismisses as a "fear-based" vanity metric) and more interested in how much risk was reduced per dollar/euro invested.
Unit Economics: AI is viewed not as a tool to reduce headcount, but as a way to achieve sub-line growth. As the volume of threats increases (30% year-over-year at Allianz), AI allows the existing team to manage the surge without a linear increase in hiring.
6. The Changing Role of the Security Analyst
The consensus is that AI will not replace humans but will fundamentally shift their job descriptions.
Toil Reduction: AI handles "triage toil" and duplicate tickets.
Critical Thinking: Humans remain essential for nuance, "long-tail" forensics, and high-stakes decision-making.
Augmentation: The future analyst will move from "human in the loop" to "human over the loop," acting as an orchestrator of agentic systems.
Topic Timeline
Introduction and Host Greetings
Welcome and introduction of the video format on YouTube.
Introduction of guests Alexander Pabst (Allianz) and Michael Sinno (Google).
The Relevance of Legacy Metrics (MTTD/MTTR)
Discussion on whether to "bury or praise" traditional time-based metrics.
The concept of "Evolving Caesar" and adjusting metrics for the AI era.
Incentives and the Flaws of "Mean" Averages
Analysis of how speed-based metrics can lead to poor quality outcomes.
The danger of "teaching to the test" in a SOC environment.
Quality Metrics and AI Auditing
Using AI to evaluate ticket quality and reopen incomplete investigations.
Balancing SLOs (Service Level Objectives) with investigative rigor.
The Surface Maturity Triangle
Measuring maturity across different layers of the technology stack.
How automation and agentic solutions change the maturity model.
From "Detect and Respond" to "Infer and Interrupt"
The shift toward real-time intervention and identity challenges.
Moving the "goalposts" and the "field" of security operations.
Human vs. Machine Metrics
Tracking operational reduction and the percentage of "untouched" tickets.
The "human in the loop" vs. "human over the loop" philosophy.
Regulatory Challenges and DORA
Navigating non-deterministic AI in highly regulated environments.
The difficulty of explaining AI decision-making to auditors.
Board-Level Reporting and Risk Storytelling
The difference between technical metrics and strategic risk exposure.
Why "packets blocked" is a poor metric for executive communication.
The Economics of AI in the SOC
Pursuing sub-line growth and managing increasing alert volumes.
AI as a "turbocharger" for labor rather than a replacement.
Closing Advice and Recommended Reading
Metrics as a living organism that must evolve.
The importance of storytelling and critical thinking.
References to Forrester reports and Google Threat Intelligence.
