How to Write a Production Runbook Engineers Will Actually Use

A production runbook is not supposed to prove how much the author knows about a system.

It is supposed to help someone else restore that system when it is failing.

That distinction matters.

I have seen operational documentation that was technically accurate but practically useless. It described the architecture, listed a few dashboards, and included vague instructions such as:

Check the logs and restart the service if necessary.

That is not a runbook.

A useful runbook helps an engineer answer five questions quickly:

1. What is this service responsible for?
2. How do I know what is failing?
3. What can I safely do about it?
4. How do I undo a bad change?
5. When should I stop troubleshooting and escalate?

The engineer reading it may not be the person who built the service. They may have been pulled into an incident because the primary owner is unavailable. They may be tired, under pressure, and working with incomplete information.

The runbook has to work under those conditions.

In this guide, I will build a fictionalized runbook for an e-commerce service called the Order Processing Service. The service receives validated orders, sends them to an ERP, records processing status, and publishes fulfillment events to downstream systems.

The exact technology is less important than the structure. The same approach works for APIs, background workers, queue consumers, internal platforms, scheduled jobs, and customer-facing applications.

What a Production Runbook Is Actually For

#

A production runbook is an operational recovery document.

It is not:

• A replacement for architecture documentation.
• A complete history of the service.
• An onboarding guide.
• A dumping ground for every command anyone has ever used.
• An excuse to leave recurring failures unresolved.

A good runbook exists to reduce the time between alert and safe recovery.

That means it should optimize for:

• Fast orientation.
• Clear diagnostics.
• Safe actions.
• Explicit rollback steps.
• Known limits.
• Escalation before the situation gets worse.

I think of the runbook as the operational interface to a service.

The application may expose an HTTP API to other systems, but the runbook is the interface exposed to the engineers responsible for keeping it alive.

If that interface is confusing, incomplete, or outdated, the service is harder to operate than it needs to be.

A Runbook Is a Temporary Bridge to Automation

#

I do not consider a runbook the final solution to a recurring production problem.

If an engineer has to execute the same recovery sequence repeatedly, that recovery path should eventually become automation.

The progression should usually look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Unknown failure
    ↓
Manual investigation
    ↓
Documented recovery procedure
    ↓
Safe script or controlled operational tool
    ↓
Automated remediation
    ↓
Failure prevented or self-healed

The runbook is still valuable throughout that process.

Before automation exists, it gives engineers a consistent recovery path.

After automation exists, it explains:

• What the automation does.
• When it should be used.
• What conditions prevent it from running.
• What to inspect when it fails.
• How to recover manually if necessary.

The mistake is allowing the runbook to become permanent justification for operational toil.

A section titled “Restart these six services every Tuesday” should not survive indefinitely. It should create an engineering task.

Every frequently used manual procedure is evidence of missing automation, insufficient resilience, or an unresolved design problem.

The Scenario: A Fictional Order Processing Service

#

For this example, the fictional service has the following responsibilities:

1. Receive order events from the commerce platform.
2. Validate required order data.
3. Publish valid orders to an Amazon SQS FIFO queue.
4. Send queued orders to an ERP.
5. Record the ERP response.
6. Publish shipment-ready events to downstream systems.

The simplified flow looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Commerce Platform
        |
        v
Order Processing API
        |
        v
Validation Layer
        |
        v
Amazon SQS FIFO Queue
        |
        v
Order Worker
        |
        v
ERP
        |
        v
Fulfillment and Notification Systems

The service runs in AWS and uses:

• Amazon ECS on Fargate.
• Application Load Balancer.
• Amazon SQS FIFO.
• Amazon RDS for PostgreSQL.
• AWS Secrets Manager.
• Amazon CloudWatch.
• AWS X-Ray.
• CloudWatch alarms.
• A dead-letter queue.
• A CI/CD deployment pipeline.

This gives us enough complexity to demonstrate a real runbook without tying the article to a specific production environment.

The Most Important Rule: Write for the Least-Informed Qualified Engineer

#

A runbook should assume the reader is technically capable but unfamiliar with the service.

That is different from writing for a beginner.

The engineer may understand AWS, Linux, HTTP, queues, containers, and databases. What they do not know is how this particular service behaves.

Do not assume they know:

• Which dashboard matters most.
• Which alarms are noisy.
• Which queue is safe to replay.
• Which database table records processing state.
• Whether restarting a worker creates duplicate orders.
• Whether a rollback also requires a schema rollback.
• Whether the ERP is allowed to be unavailable for an hour.
• Which team owns the upstream or downstream dependency.

The runbook should supply the service-specific knowledge that cannot be inferred from general engineering experience.

Start With a Fast Operational Summary

#

The first screen of the runbook should orient the engineer.

Do not begin with three pages of history.

Start with:

• What the service does.
• How critical it is.
• What customers experience when it fails.
• Where it runs.
• Who owns it.
• The first dashboard to open.
• The first safe action to consider.

A useful header might look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

service:
  name: Order Processing Service
  tier: 1
  owner: Commerce Platform Team
  escalation_channel: "#incident-commerce"
  repository: "git@example.com:commerce/order-processing.git"
  deployment_pipeline: "order-processing-production"

business_impact:
  summary: >
    Processes paid orders and sends them to the ERP.
    A full outage prevents new orders from entering fulfillment.
  customer_symptoms:
    - Orders remain in "Processing"
    - Confirmation emails may be delayed
    - Warehouse fulfillment does not begin
  data_risk:
    - Duplicate ERP submission if messages are replayed incorrectly
    - Delayed downstream shipment events

first_response:
  dashboard: "CloudWatch / Order Processing / Production Overview"
  logs: "/ecs/order-processing-production"
  safe_action: "Scale workers only after confirming ERP health"

This metadata can be displayed as Markdown rather than YAML, but the content should remain concise.

An engineer should understand the service within two minutes.

Service Overview

#

The service overview explains what the system does and where its boundaries are.

It should not try to document every implementation detail.

Include the Business Function

#

Start with the business responsibility:

The Order Processing Service receives paid orders from the commerce platform, validates required fulfillment data, and submits valid orders to the ERP. It tracks processing status and publishes downstream fulfillment events.

That sentence is more useful during an incident than:

The Order Processing Service is an event-driven microservice written in TypeScript.

The runtime and language matter, but the business function comes first.

During an outage, the engineer needs to understand what is at risk.

Define the Service Boundary

#

Document what the service does not own.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

Owned by this service:
- Order validation
- Queue publication
- ERP submission
- Processing status
- Retry and DLQ behavior

Not owned by this service:
- Checkout
- Payment authorization
- ERP availability
- Warehouse picking
- Customer email delivery

This prevents engineers from troubleshooting the wrong system.

It also makes escalation clearer.

Document Criticality

#

Define the operational tier.

For example:

The Order Processing Service would be Tier 1 because an extended outage stops fulfillment.

The runbook should also state how long the system can remain impaired before the impact becomes severe.

1
2
3
4
5

Expected recovery objective: 30 minutes

Maximum tolerable queue delay during normal operations: 10 minutes

Severe business impact threshold: Orders delayed longer than 60 minutes

These are operational expectations, not formal disaster recovery objectives unless the organization has defined them that way.

Architecture and Request Flow

#

An engineer should not need to reverse-engineer the system during an incident.

Provide a simplified architecture diagram and a short request flow.

Keep the Diagram Operational

#

The diagram should show:

• Entry point.
• Compute.
• Data store.
• Queues.
• External dependencies.
• Observability.
• Failure paths.
• Dead-letter behavior.

Avoid filling the diagram with implementation details that do not help recovery.

For this service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

Client Event
    |
    v
Application Load Balancer
    |
    v
Order Processing API
    |
    +----> PostgreSQL
    |
    v
SQS FIFO Queue
    |
    v
Order Worker
    |
    +----> ERP
    |
    v
Fulfillment Event Queue

Failed messages
    |
    v
Dead-Letter Queue

Explain the Normal Processing Sequence

#

A numbered sequence is often easier to use than a paragraph:

1. The commerce platform publishes an order event.
2. The API validates the payload.
3. Invalid orders are rejected and recorded with a validation reason.
4. Valid orders are stored with a pending status.
5. The service publishes the order to the FIFO queue.
6. A worker consumes the message.
7. The worker submits the order to the ERP.
8. The service records the ERP result.
9. The message is deleted after successful processing.
10. Permanent failures are moved to the DLQ after the configured retry limit.

This gives the on-call engineer a mental model for tracing failures.

Dependencies

#

Dependencies are often where incidents become confusing.

A service can be healthy while its upstream, downstream, or infrastructure dependencies are failing.

The runbook should identify each dependency and explain how its failure appears.

A dependency table works well:

Document Failure Characteristics

#

Not all dependency failures should trigger the same response.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

ERP unavailable:
- Do not purge the queue.
- Do not repeatedly restart workers.
- Confirm messages remain durable in SQS.
- Reduce worker concurrency if retry pressure is affecting the ERP.
- Escalate to the ERP team.
- Resume normal concurrency after ERP recovery.

Database unavailable:
- Stop deployment activity.
- Check RDS failover state.
- Confirm whether connections are exhausted or the database is unreachable.
- Do not restart every task simultaneously.
- Escalate to the database or platform owner.

The runbook should teach the engineer how the dependency fails, not merely name it.

Dashboards and Signals

#

“Check CloudWatch” is not sufficient.

A useful runbook tells the engineer exactly which dashboard to open and what normal looks like.

Create a Dashboard Hierarchy

#

I prefer three layers:

1. Service overview
2. Component detail
3. Raw logs and traces

The overview dashboard should answer:

• Is the service available?
• Is traffic entering?
• Is work being processed?
• Is the backlog growing?
• Are errors increasing?
• Is a dependency failing?

For the fictional service, the main dashboard should include:

• API request count.
• API 4xx and 5xx rate.
• API latency.
• ECS desired and running task count.
• ECS task restarts.
• Queue message count.
• Oldest message age.
• DLQ message count.
• Worker success rate.
• Worker failure rate.
• ERP response latency.
• Database connection count.
• Database CPU and storage.
• Deployment marker.

Document Normal Ranges

#

A dashboard is far more useful when the runbook explains expected behavior.

1
2
3
4
5
6
7
8
9

Normal production behavior:
- API 5xx rate: Below 0.5%
- API p95 latency: Below 500 ms
- Queue depth: Usually below 1,000 messages
- Oldest message age: Usually below 5 minutes
- DLQ depth: Zero
- Worker success rate: Above 99%
- ERP p95 response time: Below 2 seconds
- Database connections: Below 70% of configured maximum

The exact thresholds should reflect actual production behavior.

Do not invent arbitrary numbers because they look reasonable.

Document Correlated Signals

#

One metric rarely tells the whole story.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

Queue depth rising + oldest message age rising:
Workers are not keeping up or are unable to process messages.

Queue depth rising + worker success rate normal:
Traffic may have increased beyond current capacity.

Queue depth rising + ERP latency rising:
ERP performance is likely limiting throughput.

API 5xx rising + database connections near maximum:
Possible connection exhaustion.

ECS restarts rising + deployment marker present:
Possible bad deployment or startup failure.

This kind of signal correlation is one of the most valuable parts of a runbook.

It captures practical operational knowledge that otherwise lives in one engineer’s head.

Logs and Traces

#

The runbook should tell the engineer where logs live, how to filter them, and which fields matter.

Define Log Locations

#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

API log group:
/ecs/order-processing-api-production

Worker log group:
/ecs/order-processing-worker-production

Load balancer logs:
s3://company-production-alb-logs/order-processing/

CloudTrail:
Central log archive account

X-Ray service:
order-processing-production

Standardize Search Fields

#

Structured logs are significantly easier to use during incidents.

A production event should ideally include fields such as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

{
  "level": "error",
  "service": "order-processing-worker",
  "environment": "production",
  "order_id": "ORDER-PLACEHOLDER",
  "message_id": "SQS-MESSAGE-ID",
  "correlation_id": "TRACE-ID",
  "attempt": 3,
  "dependency": "erp",
  "error_code": "ERP_TIMEOUT",
  "duration_ms": 10002,
  "message": "ERP request timed out"
}

The runbook should explain which identifiers connect the workflow:

1
2
3
4
5
6
7

Primary correlation fields:
- order_id
- correlation_id
- message_id
- trace_id

Never paste real customer addresses, payment details, or sensitive order data into incident channels.

Include Useful Queries

#

For CloudWatch Logs Insights:

1
2
3
4
5

fields @timestamp, level, order_id, error_code, message
| filter service = "order-processing-worker"
| filter level = "error"
| sort @timestamp desc
| limit 100

To find ERP failures:

1
2
3
4
5

fields @timestamp, order_id, attempt, duration_ms, error_code, message
| filter dependency = "erp"
| filter level in ["error", "warn"]
| sort @timestamp desc
| limit 200

To find a specific order:

1
2
3

fields @timestamp, service, level, correlation_id, message
| filter order_id = "ORDER-PLACEHOLDER"
| sort @timestamp asc

The engineer should not need to remember field names during the incident.

Common Failure Scenarios

#

This is the core of the runbook.

Each scenario should follow the same structure:

1. Symptoms.
2. Likely causes.
3. Confirmation steps.
4. Safe remediation.
5. Rollback or recovery.
6. Escalation criteria.
7. Follow-up work.

Consistency makes the document easier to scan.

API Returns Elevated 5xx Responses

#

Symptoms

#

• API 5xx rate exceeds the alert threshold.
• ALB target errors increase.
• Customers or upstream services report failed requests.
• ECS task restarts may be elevated.

Likely Causes

#

• Bad application deployment.
• Database connection exhaustion.
• Missing or invalid secret.
• Dependency timeout.
• ECS tasks failing health checks.
• Application memory exhaustion.

Confirmation Steps

#

1. Check whether a deployment occurred near the start of the errors.
2. Review ECS desired, running, and pending task counts.
3. Inspect target health in the load balancer.
4. Check API logs for the dominant error code.
5. Review database connections and CPU.
6. Confirm Secrets Manager access if tasks fail during startup.

Useful AWS CLI checks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

aws ecs describe-services \
  --cluster production-commerce \
  --services order-processing-api \
  --query 'services[0].{
    desired:desiredCount,
    running:runningCount,
    pending:pendingCount,
    deployments:deployments[*].{
      status:status,
      taskDefinition:taskDefinition,
      running:runningCount,
      failed:failedTasks
    }
  }'

Check unhealthy load balancer targets:

1
2
3

aws elbv2 describe-target-health \
  --target-group-arn "$TARGET_GROUP_ARN" \
  --query 'TargetHealthDescriptions[?TargetHealth.State!=`healthy`]'

Safe Remediation

#

• If the failure began immediately after deployment, initiate the documented rollback.
• If database connections are exhausted, identify the source before scaling the API.
• If tasks cannot start because of a secret or permission failure, restore the previous task definition or credential configuration.
• If only a subset of tasks is unhealthy, allow ECS to replace them while watching capacity.

Do Not

#

• Do not restart every task without identifying the failure.
• Do not scale the service aggressively if the database is already saturated.
• Do not rotate credentials during the incident unless credential failure is confirmed.
• Do not deploy unrelated changes.

Escalate When

#

• The service remains impaired after rollback.
• Database health is degraded.
• A required secret or IAM permission changed unexpectedly.
• The upstream or downstream dependency is failing.
• Customer impact exceeds 15 minutes.

Queue Backlog Is Growing

#

Symptoms

#

• Queue depth rises continuously.
• Oldest message age exceeds the normal threshold.
• Order status remains processing.
• Workers may appear healthy while throughput declines.

Likely Causes

#

• Worker concurrency is too low.
• ERP latency has increased.
• Workers are repeatedly retrying poison messages.
• Database writes are slow.
• A deployment reduced processing throughput.
• FIFO message grouping is causing a hot partition.

Confirmation Steps

#

1. Compare queue ingress and deletion rates.
2. Check worker running task count.
3. Review worker success and failure rates.
4. Check ERP latency and error rate.
5. Inspect the most common worker error.
6. Check whether one MessageGroupId dominates processing.
7. Review recent deployment markers.

Safe Remediation

#

If workers are healthy and dependencies are stable:

1
2
3
4

aws ecs update-service \
  --cluster production-commerce \
  --service order-processing-worker \
  --desired-count 12

Only scale workers after confirming:

• The ERP can tolerate additional concurrency.
• The database has available connections.
• Messages are not failing repeatedly.
• The queue backlog is legitimate work rather than poison messages.

If ERP latency is elevated:

• Avoid increasing concurrency.
• Reduce retry pressure if necessary.
• Preserve messages in the queue.
• Escalate to the ERP team.

If a poison message is blocking a FIFO message group:

• Identify the affected message group.
• Confirm retry count.
• Allow the message to move to the DLQ according to policy.
• Do not manually delete it unless the business impact and replay implications are understood.

Escalate When

#

• Oldest message age exceeds 30 minutes.
• Throughput remains below ingress after safe scaling.
• ERP errors exceed the defined threshold.
• Multiple message groups are blocked.
• DLQ volume is increasing.

Database Connection Exhaustion

#

Symptoms

#

• API or worker errors mention connection timeouts.
• RDS connection count approaches the maximum.
• Application latency increases.
• Task restarts may increase.
• Scaling the application makes the incident worse.

Likely Causes

#

• Connection leak.
• Incorrect pool configuration.
• Sudden task count increase.
• Slow queries retaining connections.
• Database failover or degraded performance.
• A deployment changed connection behavior.

Confirmation Steps

#

1. Review RDS connection count.
2. Check application task count.
3. Compare the incident start time to recent deployments.
4. Inspect slow query metrics.
5. Check whether one service is consuming most connections.
6. Review connection pool errors in application logs.

Safe Remediation

#

• Stop further application scaling.
• Roll back a deployment that introduced the connection increase.
• Reduce worker concurrency if background processing is consuming the pool.
• Allow unhealthy tasks to drain rather than restarting everything simultaneously.
• Engage the database owner if active sessions need deeper inspection.

Do Not

#

• Do not increase the database connection limit as the first response.
• Do not restart every application task at once.
• Do not scale out the API while connections are exhausted.
• Do not terminate database sessions without identifying their owners.

Follow-Up

#

A recurring connection exhaustion incident should produce work such as:

• Fixing connection leaks.
• Correcting pool sizes.
• Introducing RDS Proxy.
• Adding per-service connection metrics.
• Adding load tests for deployment changes.
• Establishing task-count-aware pool limits.

ERP Is Unavailable

#

Symptoms

#

• ERP requests time out.
• Worker retry count increases.
• Queue depth and message age increase.
• DLQ messages may appear.
• API health may remain normal.

Immediate Objective

#

Preserve orders without creating duplicates or overwhelming the ERP.

Confirmation Steps

#

1. Check ERP health from the integration dashboard.
2. Confirm failures occur across multiple orders.
3. Review ERP response codes and latency.
4. Verify messages remain in the queue.
5. Confirm the queue retention period.
6. Contact the ERP owner.

Safe Remediation

#

• Allow SQS to retain unprocessed orders.
• Reduce worker concurrency if retries are creating unnecessary load.
• Do not purge the queue.
• Do not manually resubmit orders outside the normal idempotent path.
• Monitor queue age against business impact thresholds.
• Restore normal concurrency gradually after ERP recovery.

Escalate When

#

• ERP failure is confirmed across multiple requests.
• Oldest message age exceeds 15 minutes.
• Order fulfillment is materially delayed.
• The ERP team cannot provide a recovery estimate.
• Messages are approaching retention limits.

Dead-Letter Queue Is Growing

#

A DLQ is not a trash can.

Messages in the DLQ represent business events that failed beyond the normal retry policy.

Symptoms

#

• DLQ alarm fires.
• Orders remain unresolved.
• The same validation or dependency error appears repeatedly.

Confirmation Steps

#

1. Record the DLQ message count.
2. Sample a small number of messages.
3. Group failures by error code.
4. Determine whether the failure is transient or permanent.
5. Confirm whether the underlying issue has been corrected.
6. Verify replay is idempotent.

Safe Replay Requirements

#

Replay should only occur when:

• The root cause has been fixed.
• The target system is healthy.
• The message has not already been processed manually.
• The consumer is idempotent.
• The replay batch size is controlled.
• The results will be monitored.

Example controlled replay command:

1
2
3
4
5
6

./ops/replay-dlq.sh \
  --source-queue order-processing-dlq \
  --destination-queue order-processing.fifo \
  --max-messages 10 \
  --order-id ORDER-PLACEHOLDER \
  --dry-run

A real replay tool should default to dry-run mode.

The runbook should explain exactly how to verify that replay will not create duplicate orders.

Do Not

#

• Do not redrive the entire DLQ without sampling failures.
• Do not replay messages while the root cause is still active.
• Do not modify message contents manually unless there is an approved recovery procedure.
• Do not assume FIFO deduplication alone guarantees business idempotency.

Bad Deployment

#

A production runbook should make rollback easier than improvisation.

Symptoms

#

• Errors begin immediately after deployment.
• New tasks fail health checks.
• Latency increases after the release marker.
• A new error signature appears.
• The previous version was healthy.

Confirm the Deployment

#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

aws ecs describe-services \
  --cluster production-commerce \
  --services order-processing-api \
  --query 'services[0].deployments[*].{
    status:status,
    taskDefinition:taskDefinition,
    createdAt:createdAt,
    running:runningCount,
    failed:failedTasks
  }'

Rollback Criteria

#

Rollback should be the default when:

• Customer-impacting errors began immediately after deployment.
• The issue is not understood quickly.
• The previous version is known to be healthy.
• The rollback does not create a database compatibility problem.

Rollback Command

#

1
2
3
4
5

aws ecs update-service \
  --cluster production-commerce \
  --service order-processing-api \
  --task-definition order-processing-api:142 \
  --force-new-deployment

The runbook should not rely on an engineer guessing the previous task definition.

It should explain how to find it and how to verify it.

Verify the Rollback

#

1. Confirm the previous task definition is active.
2. Confirm running task count returns to desired count.
3. Check ALB target health.
4. Confirm API error rate declines.
5. Send a synthetic test request.
6. Verify queue and database behavior.
7. Watch the service for at least one normal processing cycle.

Database Compatibility Warning

#

A deployment rollback may be unsafe if the release includes a non-backward-compatible schema migration.

The runbook should explicitly classify migrations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

Safe for application rollback:
- Adding nullable columns
- Adding new tables
- Adding compatible indexes
- Expanding accepted values

Potentially unsafe:
- Dropping columns
- Renaming columns
- Changing data types
- Removing enum values
- Destructive data migrations

The safest deployment pattern is to make database changes backward-compatible across at least two application versions.

Rollback Must Be a First-Class Procedure

#

Many teams document deployment thoroughly and rollback poorly.

That is backwards.

The production runbook should answer:

• What is the rollback mechanism?
• Where is the last known-good version recorded?
• How long does rollback take?
• Does rollback affect data?
• Are schema changes compatible?
• Does configuration also need to be reverted?
• How is rollback verified?
• Who has permission to execute it?

A strong rollback section should include a decision tree:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

Did the incident begin after deployment?
    |
    +-- No --> Continue normal diagnosis
    |
    +-- Yes
         |
         v
Is the previous version compatible with the current schema?
         |
         +-- Yes --> Roll back application
         |
         +-- Unknown --> Escalate before rollback
         |
         +-- No --> Follow forward-fix or database recovery procedure

The engineer should not have to make a high-risk schema compatibility decision alone during an incident.

Escalation

#

Escalation is not failure.

Knowing when to escalate is part of operating the system safely.

A weak runbook says:

Contact the service owner if needed.

A useful runbook defines specific triggers.

Example Escalation Matrix

#

Include Contact Paths

#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

Primary team:
#commerce-platform

Incident coordination:
#incident-commerce

Platform escalation:
#platform-on-call

ERP escalation:
#erp-support

Business escalation:
Commerce Operations Manager

Severe incident bridge:
Use the incident management workflow in #incident-response

Avoid putting personal phone numbers directly in a repository unless there is a clear reason and access is controlled.

Use team schedules, incident tooling, or on-call systems wherever possible.

Known Gotchas

#

The known-gotchas section is where practical experience becomes extremely valuable.

These are the behaviors that repeatedly surprise engineers.

For the fictional service:

FIFO Ordering Can Hide a Localized Blockage

#

One failed message can block later messages that share the same MessageGroupId.

A healthy overall consumer count does not prove every order group is progressing.

Scaling Workers Can Make Dependency Failures Worse

#

More workers do not help when the ERP or database is the bottleneck.

Scaling concurrency during a dependency incident can increase retries, timeouts, and recovery time.

Restarting Workers Does Not Remove Poison Messages

#

A malformed message will still be present after the worker restarts.

Repeated restarts only delay diagnosis.

DLQ Replay Can Create Duplicate Business Events

#

Queue-level deduplication and application-level idempotency are not the same thing.

The service must verify whether the order has already been accepted by the ERP.

Oldest Message Age Is Often More Important Than Queue Depth

#

A queue containing 10,000 messages may be healthy during a traffic spike if messages are draining quickly.

A queue containing 100 messages may be unhealthy if the oldest message has been waiting for an hour.

A Healthy API Does Not Mean Orders Are Processing

#

The API and worker are separate components.

The customer-facing endpoint can be healthy while the background queue is stalled.

Deployment Rollback Does Not Automatically Revert Configuration

#

Task definitions, feature flags, secrets, parameter values, and database migrations may change independently.

The rollback procedure must identify which layers are included.

This section should grow over time.

Every incident that exposes unexpected system behavior should either:

• Add a gotcha.
• Improve automation.
• Add an alert.
• Fix the design.
• Remove the failure mode entirely.

A Practical Runbook Template

#

The following structure works well for most production services.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200

# Service Name Production Runbook

## 1. Service Summary

**Owner:**  
**Operational tier:**  
**Repository:**  
**Deployment pipeline:**  
**Primary dashboard:**  
**Logs:**  
**Escalation channel:**  

### Business Responsibility

Describe what the service does and why it matters.

### Customer Impact

Describe what users or internal teams experience when it fails.

### Recovery Expectations

- Expected recovery time:
- Maximum acceptable delay:
- Data-loss tolerance:

---

## 2. Architecture

Include a simplified production architecture diagram.

### Request or Event Flow

1. Step one
2. Step two
3. Step three

### Service Boundaries

**Owned by this service:**

- Item
- Item

**Not owned by this service:**

- Item
- Item

---

## 3. Dependencies

| Dependency | Purpose | Failure Symptom | Health Check | Owner |
|---|---|---|---|---|

---

## 4. Dashboards and Alerts

### Primary Dashboard

Link and explanation.

### Normal Operating Ranges

| Signal | Normal | Warning | Critical |
|---|---:|---:|---:|

### Alert Definitions

Document what each alert means and the first diagnostic step.

---

## 5. Logs and Traces

### Log Locations

- API:
- Worker:
- Load balancer:
- Audit:

### Correlation Fields

- request_id
- correlation_id
- message_id
- business_entity_id

### Common Queries

Include ready-to-run log queries.

---

## 6. Common Failures

### Failure: Name

**Symptoms**

- Symptom

**Likely causes**

- Cause

**Confirm**

1. Step
2. Step

**Safe remediation**

1. Step
2. Step

**Do not**

- Unsafe action

**Escalate when**

- Trigger

**Follow-up automation**

- Engineering improvement

---

## 7. Deployment and Rollback

### Current Deployment Process

Describe how production releases occur.

### Rollback Criteria

Describe when rollback is preferred.

### Rollback Procedure

Include exact commands or pipeline actions.

### Verification

1. Check health
2. Check errors
3. Check business flow

### Database Compatibility

Document schema rollback limitations.

---

## 8. Dead-Letter Queue and Replay

### DLQ Location

Provide the queue name and dashboard.

### Replay Preconditions

- Root cause fixed
- Idempotency confirmed
- Batch size approved
- Monitoring active

### Replay Command

Use a safe dry-run example.

---

## 9. Escalation

| Condition | Team | Time Threshold | Channel |
|---|---|---|---|

---

## 10. Known Gotchas

Document non-obvious production behavior.

---

## 11. Post-Incident Follow-Up

- Incident ticket:
- Root cause:
- Automation task:
- Alert improvements:
- Runbook changes:
- Owner:

This template is intentionally repetitive.

During an incident, predictable structure is more valuable than literary variety.

Make Every Command Safe to Copy

#

Commands in runbooks are dangerous because engineers tend to trust them.

Every command should be reviewed as though someone will paste it into production while tired.

Use Explicit Environments

#

Bad:

1

kubectl delete pod order-worker-123

Better:

1
2
3
4

kubectl \
  --context production-us-east-1 \
  --namespace order-processing \
  delete pod order-worker-123

Use Read-Only Commands First

#

Start diagnostic sections with inspection commands:

1
2
3
4
5
6

aws sqs get-queue-attributes \
  --queue-url "$QUEUE_URL" \
  --attribute-names \
    ApproximateNumberOfMessages \
    ApproximateNumberOfMessagesNotVisible \
    ApproximateAgeOfOldestMessage

Only then show commands that change production.

Add Preconditions

#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Preconditions:
# 1. Confirm ERP health is normal.
# 2. Confirm database connections are below 70%.
# 3. Confirm worker failures are not increasing.
# 4. Record the current desired count before changing it.

aws ecs update-service \
  --cluster production-commerce \
  --service order-processing-worker \
  --desired-count 12

Include a Recovery Command

#

If a command changes state, document how to reverse it.

1
2
3
4
5

# Restore the previous desired count after the backlog is cleared.
aws ecs update-service \
  --cluster production-commerce \
  --service order-processing-worker \
  --desired-count 6

Avoid Destructive Commands

#

A runbook should not casually include commands such as:

1

aws sqs purge-queue

If an operation is unusually destructive, it should require:

• Explicit approval.
• A separate controlled procedure.
• Clear auditability.
• Documented business impact.
• A recovery plan.

Link to Tools, Not Just Concepts

#

A runbook should point directly to the operational resources.

Useful links include:

• Primary dashboard.
• Component dashboards.
• Log groups.
• Trace explorer.
• Deployment pipeline.
• Repository.
• Infrastructure code.
• Feature flag system.
• Queue console.
• DLQ console.
• Database dashboard.
• Incident channel.
• Service ownership directory.
• External dependency status page.

Avoid vague directions such as:

Open the AWS console and find the queue.

The runbook should name the queue and link to it.

The goal is to eliminate unnecessary navigation during the incident.

Test the Runbook Before You Need It

#

A runbook is not complete because someone merged the Markdown file.

It is complete when another engineer can use it.

I would test it with someone who did not write the service.

Give them a fictional incident:

Order processing latency is increasing. The API is healthy, but the queue’s oldest message age has reached 25 minutes.

Then ask them to use only the runbook.

Watch for where they get stuck.

Common problems include:

• Missing permissions.
• Dead links.
• Commands that depend on undocumented environment variables.
• Dashboard names that no longer exist.
• Unclear rollback versions.
• Undocumented schema constraints.
• Escalation contacts that are outdated.
• Acronyms that only the original team understands.
• Procedures that rely on hidden knowledge.

A tabletop exercise will expose these problems before a real outage does.

Keep the Runbook Current

#

An outdated runbook can be more dangerous than no runbook.

The engineer assumes the documented procedure is safe because it exists.

Runbooks should have ownership and review dates.

1
2
3
4
5
6

runbook:
  owner: Commerce Platform Team
  last_reviewed: 2026-07-23
  review_interval_days: 90
  tested_by: Platform Engineering
  related_service_version: "2026.07"

Update the Runbook When

#

• The architecture changes.
• A dependency changes.
• A new dashboard is introduced.
• An alert changes.
• Deployment or rollback changes.
• A new failure mode appears.
• An incident reveals a missing step.
• A manual recovery step becomes automated.
• Ownership changes.

Make Runbook Updates Part of the Work

#

A service change is not operationally complete if it changes production behavior but leaves the runbook inaccurate.

The definition of done for a production change should include:

1
2
3
4
5
6

- Monitoring updated
- Alerts updated
- Dashboard updated
- Runbook updated
- Rollback verified
- Ownership confirmed

Documentation should change alongside the service, not months later.

Turn Repeated Procedures Into Automation

#

The runbook should identify which steps are temporary.

I like adding a small automation note to each failure scenario:

1
2
3
4
5
6

Automation status:
Manual today.

Planned improvement:
Create a controlled queue replay tool that validates order state,
defaults to dry-run, limits batch size, and records an audit event.

Or:

1
2
3
4
5
6
7
8
9

Automation status:
Partially automated.

Current behavior:
CloudWatch alarm detects backlog growth.

Missing behavior:
Automatically scale workers only when ERP latency and database
connections remain within safe limits.

This keeps the runbook from becoming a museum of manual procedures.

Good Automation Candidates

#

• Safe service restart.
• Worker scaling within limits.
• DLQ sampling.
• DLQ replay with idempotency checks.
• Deployment rollback.
• Credential validation.
• Synthetic health checks.
• Dependency status checks.
• Log collection.
• Incident timeline generation.
• Alert enrichment.
• Automated ticket creation.

Automation Should Be Safer Than the Manual Procedure

#

A recovery script should include:

• Environment validation.
• Dry-run mode.
• Input validation.
• Maximum batch sizes.
• Permission checks.
• Confirmation for destructive actions.
• Audit logs.
• Clear exit codes.
• Idempotency.
• Rollback or compensating actions.

Bad automation only executes the wrong action faster.

What Engineers Actually Need During an Incident

#

The engineer does not need a perfect explanation of the system.

They need a reliable path through uncertainty.

A useful runbook says:

1
2
3
4
5
6
7
8
9

If queue age is increasing:

1. Confirm worker task count.
2. Compare queue ingress and deletion rates.
3. Check worker failure rate.
4. Check ERP latency.
5. Check database connections.
6. Scale only if dependencies are healthy.
7. Escalate if age exceeds 30 minutes.

An unhelpful runbook says:

1
2

The queue may grow for several reasons.
Investigate the relevant service components.

The difference is operational specificity.

A Runbook Quality Checklist

#

Before publishing a runbook, I would verify the following.

Orientation

#

• Can an unfamiliar engineer explain what the service does?
• Is customer impact clear?
• Is the service owner obvious?
• Is the first dashboard easy to find?

Diagnosis

#

• Are normal ranges documented?
• Are alerts explained?
• Are dependencies listed?
• Are common signal combinations explained?
• Are useful log queries included?

Recovery

#

• Are procedures ordered?
• Are commands safe to copy?
• Are preconditions documented?
• Are destructive actions clearly marked?
• Can every state-changing action be reversed?

Rollback

#

• Is the last known-good version identifiable?
• Is schema compatibility explained?
• Is rollback verification included?
• Are configuration changes covered?

Escalation

#

• Are escalation conditions explicit?
• Are team channels current?
• Are severe data risks identified?
• Does the engineer know when to stop troubleshooting?

Maintenance

#

• Is there a named owner?
• Is there a review date?
• Has another engineer tested it?
• Does every recurring manual step have an automation candidate?

The Bigger Operational Lesson

#

A good runbook is evidence that a team understands its production system.

Writing one forces the team to answer uncomfortable questions:

• What does normal actually look like?
• Which metric tells us customers are affected?
• Can we roll back safely?
• Are our recovery commands tested?
• Do we know who owns every dependency?
• Can someone outside the core team operate the service?
• Which failures still depend on tribal knowledge?
• Which manual procedures should already be automated?

Those questions often reveal more than the document itself.

They expose weak observability, unclear ownership, unsafe deployments, missing idempotency, undocumented dependencies, and operational work that only one engineer knows how to perform.

That is why runbooks are useful beyond incident response.

They are a test of whether a production service is truly operable.

Final Thoughts

#

The best production runbook is not the longest one.

It is the one an engineer can trust.

It explains the service without burying the reader in architecture history. It connects symptoms to likely causes. It provides safe diagnostic steps before remediation. It treats rollback as a normal operational capability. It defines escalation clearly. It captures the gotchas that would otherwise remain tribal knowledge.

Most importantly, it does not normalize repeated manual work.

The runbook should stabilize the service today while giving the team a clear path toward automating the recovery tomorrow.

A production system is not mature because its original author knows how to fix it.

It is mature when another qualified engineer can understand the failure, recover the service safely, and improve the system so the same procedure becomes less necessary next time.