Security tools change constantly, and the procedures your teams follow can fall behind without anyone noticing. An outdated runbook slows response and leads to inconsistent handling of alerts. This post explains how our Security Operations Center (SOC) keeps runbooks accurate and uses them to measure performance.

Why Runbooks Matter

Samtek supports the CMS Center for Clinical Standards and Quality (CCSQ) through the Cloud Operations and Maintenance (CCOM) contract, which provides cloud services for the CCSQ AWS cloud enclave. Our SOC monitors the environment, responds to security alerts, and recommends improvements to its security posture.

A runbook is a step-by-step procedure for triaging, investigating, and acting on a specific alert. You may also hear them called SOPs (standard operating procedures), canned responses, or operations manuals. A modern SOC receives alerts from dozens of tools, and the content and frequency of those alerts change often. When runbooks don’t keep pace, team members handle the same alert in different ways, and response times become harder to predict.

At Samtek, we use runbooks that are maintained continuously and tied to measurable targets instead of static documents. Well-maintained runbooks let the SOC:

• Triage alerts so the most critical receive priority.
• Handle each alert type the same way, regardless of who is responding.
• Confirm that the client agrees with how alerts are processed.
• Bring new team members up to speed quickly.
• Measure analyst output against defined targets.

Keeping Runbooks Current

Menu names, hierarchies, and item order can change between software versions, and even subversions. Our runbooks give click-by-click directions (Top Level > Next Level > Item Name), and we keep screen captures matched to the current software version.

Four practices support this:

• Version control. Runbooks live in Confluence, which provides version history, attribution, and comments. For more on structuring documentation, see our blog The Art of Writing Clear Documentation for Cloud Engineers.
• Rotating tool ownership. Each tool has a named reviewer for changes and updates, and assignments rotate.
• Log source reviews. On a schedule, we confirm that trigger data is still ingested and look for new fields that could improve alerting.
• Peer review. A senior team member or manager approves every updated runbook before it moves into production.

Testing & Improving Runbooks Over Time

Keeping runbooks current is critical. It’s also critical to ensure that the runbooks work and serve their purpose. Daily work tests frequently used runbooks. Less common ones need a scheduled review, with a cadence that fits the tool, the alert, and the situation it covers. For high-consequence procedures, such as a server infected with malware, we recommend a review at least once a quarter. Treat it as a tabletop exercise and dedicate real infrastructure resources to it. It’s also important to verify any runbook tied to a provider platform since cloud providers also add, change, and retire services.

Incidents and false positives show where a procedure falls short. After an incident, we hold a post-incident review. We also review false positives on a schedule and group recurring ones by alert type.

Each finding becomes a tracked action item with an owner. It goes through the normal peer review and is recorded in the version history. Runbooks refined this way are better positioned for automation because their steps have already been validated against real alerts.

Measuring Performance

A standard process for each alert type lets us compare the time spent in each stage across team members and alert types. Without one, those numbers mostly reflect individual working styles. With one, they show where the process is slow or unclear.

We track four measures:

• Mean time to respond: alert creation to first message action. It shows whether alert routing is working.
• Mean time to close: alert creation to a verified false positive, confirmed valid activity, or containment action.
• False-positive rate: the share of alerts closed as false positives.
• Incidents processed: a measure of overall analyst productivity.

On CCOM, we track each measure by alert type rather than as a SOC-wide average. A slow or noisy result points to a specific runbook and triggers the same peer review and version control described above. Working with CMS, we also have used PagerDuty and Splunk On-Call to alert high-priority events. The resulting metrics gave CMS near real-time data on the health and impact of each guardrail, which the agency didn’t have before.

Positive Runbook Results on CCOM

A maintained runbook library, combined with alert filtering and automation, improved how the SOC operates on CCOM in three ways:

• Right-sized staffing: Guardrails, automated alert filtering, and defined escalation processes optimized on-call team resources enabled more efficient operations while maintaining day-to-day service continuity. 
• Significant time savings: PagerDuty alerting removed approximately 660 hours of manual alert tracking each quarter.
• Positive customer feedback: CMS’s CPARS rated Quality and Management as Exceptional. A small survey of application development organizations scored support and quality outcomes 5 out of 5.

Takeaways

To keep runbooks effective, treat them as living documents that evolve with your organization’s needs.

Assign each tool a named owner, rotate that ownership, and require peer review before changes go live. Remember that regular testing is equally important. Conduct tabletop exercises at least once a quarter to validate high-consequence procedures, such as a malware-infected server. Finally, close the loop by turning lessons learned into action. Feed post-incident findings and recurring false positives back into the runbooks, and measure performance by alert type to see which runbook needs attention first. By establishing severity levels, escalation paths, and response targets before an incident occurs, organizations can build a more consistent, efficient, and reliable incident response process.

If your SOC runbooks haven’t been reviewed recently, start with your highest-risk alert and walk through it with your team. Samtek can help you assess your SOC processes and build a maintenance approach that fits your environment. Contact us to start the conversation.