A closed security case is an operational decision
I lead a team in the Oracle Cyber Security Operations Centre, and after more than two decades in cybersecurity, I have learnt to pay close attention to the evidence trail behind every case. Case records bring together alerts, analyst notes, telemetry, enrichment, containment actions, escalation history, and closure decisions. They explain what happened, what was known at the time, and why a particular action was taken.
A security case can look deceptively simple. An alert fires, an analyst reviews the evidence, a decision is made, and the case is closed, escalated, reworked, or marked no longer actionable. But in security operations, closing a case is more than the end of a workflow. It is an operational decision, and the case record should contain enough evidence to support it.
The challenge I wanted to solve was how to apply a consistent review standard across that work at scale. Cases vary in type, complexity, and the evidence they require. Human review takes time and can also be influenced by differences in reviewer experience or interpretation. A missing containment note, unclear analyst reasoning, or unsupported closure decision may look minor in isolation. Across a large case population, those gaps can become a wider quality problem.
That led me to a practical engineering question: How could I turn selective human review into a consistent process that could eventually operate at scale?
My approach developed in three stages. First, I created a repeatable manual review process. Next, I created an AI skill to apply that process across a broader case population. Then, longer-term, the goal is to embed the review into case management so that every eligible case can receive the same quality assessment.
Figure 1 shows that journey as a feedback loop. Case records enter an architecture-aware review process, which produces structured recommendations for human reviewers, feedback for analysts, and quality trends for leadership. Those findings can then improve the rubric, guidance, and workflows used for future cases.
Why selective review was not enough
Security operations reporting often leans on volume metrics: how many cases were opened, how many were closed, how quickly queues moved, and whether service levels were met.
Those metrics matter, but they do not show whether a closure decision was supported by evidence, whether containment was documented clearly, or whether another reviewer could understand the decision from the case record alone.
Selective human review can answer some of those questions. However, it introduces another challenge. When reviewers interpret evidence differently, the result can depend too much on individual judgement and style. That makes reviews harder to compare and limits their value as a consistent quality signal.
The goal was not to replace operational metrics. It was to add a complementary measure of case quality: where evidence was strong, where documentation was weak, which case types created ambiguity, and where analyst guidance or workflows needed improvement.
To produce that signal consistently, I first had to define what sufficient evidence looked like.
Building an architecture-aware review standard
Security cases rarely fit inside one system boundary. A single case can involve an alert, endpoint, identity service, cloud service, tenant, region, detection source, ownership context, and escalation path.
Reviewing all those cases with one generic checklist does not work well.
An endpoint alert should not be reviewed against the same evidence expectations as an identity-service alert. An endpoint case may need host context, process or command evidence, containment notes, and a clear explanation of why the final disposition follows from that evidence. An identity-service case may need account context, authentication evidence, and an assessment of access impact. Another service alert may require resource ownership, service context, and control-plane evidence.
That is what I mean by architecture-aware review. The process should understand what evidence ought to exist for the type of case being assessed. It should not ask only whether the analyst wrote enough notes. It should ask whether the record contains the right evidence to support the decision.
Figure 2 illustrates how evidence expectations vary by case type before each case enters the shared review framework.
I turned those expectations into a structured rubric.
| High-Level Rubric Assesses: |
| The available evidence and investigative context |
| Triage notes and analyst reasoning |
| Containment and other actions taken |
| Escalation and follow-up |
| The closure rationale and completeness of the case record |
The rubric provided reviewers with a shared framework for assessing case quality, but a score alone was not enough. Alongside the rubric, I defined a reopen criteria to distinguish between a documentation weakness and a material gap that could affect whether a case should remain closed.
Examples of issues that could lead a reviewer to consider reopening a case included:
| Reopen Criteria: |
| The recorded disposition was not supported by the available evidence |
| A required investigative step appeared not to have been completed |
| Containment was required, but the record did not show that it had taken place |
| The evidence indicated that escalation was needed, but no escalation was recorded |
| Contradictory evidence remained unresolved |
| A material follow-up action was still outstanding |
| The record did not contain enough information to determine whether the closure decision was appropriate |
These criteria did not mean that every incomplete record should be reopened. A case might contain an unclear note or a missing supporting reference while the underlying investigation, action, and closure decision remained supportable. In those situations, the case could remain closed with feedback and receive a lower rubric score.
This distinction made the process both consistent and proportionate. Reviewers could identify quality gaps while reserving the operational effort of reopening a case for issues that materially affected the outcome.
Figure 3 shows how the review process combines the rubric and reopen criteria to produce a score, evidence gaps, a recommended action, and aggregated quality signals.
At this point, I had created a repeatable manual process that had been tested and refined through reviewer feedback. Reviewers had architecture-aware evidence expectations, a shared rubric, and predefined reopen criteria. The remaining constraint was scale.
Scaling the manual process with AI
The manual process provided me a consistent standard. The next challenge was applying that standard across a broader case population without making the final decision an automated one. Creating an AI Skill gave me a way to support the first-pass assessment while keeping the review standard and final judgement under human control. The skill processes the case record, identifies missing or weak evidence, compares the closure decision with the documented facts, and produces structured findings. Those findings can include a score, rationale, evidence gaps, a recommended next action, and feedback themes.
To make the recommendations useful, I configured the review to generate them from the perspective of a senior SOC analyst. I also evaluated the model against cases that had already been reviewed manually, comparing its findings with the decisions and feedback recorded by human reviewers. The objective was not to train the skill to imitate every previous decision. It was to assess whether the skill could apply the same rubric and reopen criteria consistently and explain how it reached its recommendation.
The human reviewer remains responsible for the outcome. The reviewer decides whether a case should remain closed, remain closed with feedback, be reworked, or be considered for reopening. The skill also requires ongoing oversight. New case types, evidence patterns, and operational nuances may not fit the existing guidance. When reviewers identify those situations, their feedback can be used to refine the rubric, reopen criteria, or reviewed examples where appropriate.
Without the rubric, reopen criteria, reviewed examples, and human oversight, AI risks becoming little more than a case summariser. It may produce plausible observations without grounding them in the review standard or the available evidence. The case record therefore remains the evidence boundary. If the record does not show that a search, validation, or containment action occurred, the review should identify the missing evidence rather than assume that the work was completed.
What the broader review revealed
Applying the same standard across a broader case population made documentation and evidence patterns easier to identify.
In one reviewed case, the required containment action had been completed and recorded as confirmed. However, the case did not include a reference to the request or work item that authorised the action. The rubric identified a traceability gap. The operational work appeared complete, but another reviewer could not verify the full chain of evidence from the case record alone. Because the missing reference did not cast doubt on the containment action or the closure decision, the case did not meet the predefined criteria for reopening. The AI-assisted review recommended that the case remain closed with feedback, and the human reviewer agreed.
Although the gap was minor, it showed how an otherwise complete case could still be difficult to verify. Based on that finding, I clarified the guidance so that future case records include both confirmation that the action was completed and a reference to the supporting request or work item. This was intended to give analysts clearer evidence expectation and reviewers a more consistent basis for assessing similar cases. That is an important benefit of applying the process more broadly. An individual review can improve one case. Patterns across reviews can improve the rubric, the guidance, and the workflows that produce future case records.
Embedding review in the case-management workflow
Today, we apply the review process to a selected sample of cases outside the case-management system. This gives us a controlled way to test the rubric, refine the reopen criteria, and evaluate the recommendations. However, it limits coverage and keeps the findings separate from the analyst’s normal workflow. The ideal future state is to bring the review into the case-management system and apply it to every eligible case.
The system already contains the analyst notes, artefacts, status, disposition, escalation history, actions taken, containment activity, recovery steps, and closure record. Returning review findings there would place feedback directly in the analyst workflow.
I expect that transition to happen in stages. First, the model reviews a selected case population, and a human evaluates the output. Next, structured recommendations and feedback can be routed back into case management. Over time, the review could be triggered for every eligible case, moving the process from selective sampling to systematic quality control within the case lifecycle.
That integration could also give leadership a broader and more consistent view of case quality. Recurring findings could inform updates to the rubric, reopen criteria, analyst guidance, reviewed examples, and the workflows that produce the case record.
Figure 4 illustrates the ideal future state, in which review findings remain inside the operational workflow and support both immediate analyst feedback and broader quality reporting.
Why this matters in the Agent era
Security operations is moving into an agent era, where AI can support analysis, generate recommendations, and help execute defined workflows. That makes a clear review standard more important, not less.
If agents assist with triage, enrichment, case review, or follow-up recommendations, their determinations should be assessed against the same rubric, reopen criteria, and evidence requirements as human decisions.
The framework gives agents a standard to apply, analysts a clearer view of what a defensible case record looks like, and reviewers a consistent way to challenge recommendations.
I started with selective manual reviews and built a rubric and reopen criteria to make those reviews more consistent. I then used AI to apply the same first-pass assessment across a broader case population. Embedding that capability into case management could eventually make review part of every eligible case, including cases supported by agents.
That is the shift: from sampling quality after the fact to engineering quality into the workflow.
Acknowledgements
This case-review framework was shaped by the work of colleagues across Oracle security operations. I would particularly like to thank Athul Raju and Ryan Schilcher for reviewing this blog article, Brian Carroll, Carina Lins, Dean Duffy, Owen O’Connor, Peter Hanily and Romulo Freitas for manually reviewing cases, challenging and refining the rubric and reopen criteria, validating the model’s findings, and identifying new case patterns and operational nuances that required updates to the guidance.