Runbooks and Playbooks

Overview

This document contains operational runbooks and playbooks for common scenarios in the Learnille platform. Runbooks provide step-by-step procedures for routine operations, while playbooks offer guidance for incident response and complex situations.

Incident Response Playbook

1. Service Outage Response

Detection Phase

When: Monitoring alerts indicate service degradation or outage

Immediate Actions:

Acknowledge Alert

# Check current system status
curl -f https://api.learnille.com/health || echo "API down"
curl -f https://app.learnille.com/health || echo "App down"

Assess Impact
- Check affected services and user impact
- Review error rates and latency metrics
- Identify affected user segments

Notify Stakeholders

# Send initial notification
curl -X POST https://api.pagerduty.com/incidents \
  -H "Authorization: Token token=$PAGERDUTY_TOKEN" \
  -d '{
    "incident": {
      "type": "incident",
      "title": "Learnille API Service Outage",
      "service": {"id": "SERVICE_ID"},
      "priority": {"id": "PRIORITY_ID"}
    }
  }'

Investigation Phase

Check Application Logs

# View recent application logs
aws logs tail /aws/ecs/learnille-api --since 10m

# Check for error patterns
aws logs filter-log-events \
  --log-group-name /aws/ecs/learnille-api \
  --filter-pattern "ERROR" \
  --start-time $(date -d '10 minutes ago' +%s)

Database Health Check

# Check PostgreSQL service status
pg_isready -h localhost -p 5432 -U learnille

# Query active database connections via psql
psql -h localhost -U learnille -d learnille_prod -c "SELECT count(*) FROM pg_stat_activity;"

# Query New Relic NRQL for database transaction duration
# NRQL: SELECT average(databaseDuration) FROM Transaction WHERE appName = 'Learnille API (Self-Hosted)' SINCE 1 hour ago

Infrastructure Status

# Check ECS service status
aws ecs describe-services \
  --cluster learnille-prod \
  --services learnille-api

# Check load balancer health
aws elbv2 describe-target-health \
  --target-group-arn $TARGET_GROUP_ARN

Resolution Phase

Common Quick Fixes

# Restart unhealthy tasks
aws ecs update-service \
  --cluster learnille-prod \
  --service learnille-api \
  --force-new-deployment

# Scale up service if overloaded
aws ecs update-service \
  --cluster learnille-prod \
  --service learnille-api \
  --desired-count 5

Database Issues

# Check for long-running queries
aws rds describe-db-instances --db-instance-identifier learnille-db

# Restart database if needed
aws rds reboot-db-instance --db-instance-identifier learnille-db

Rollback if Necessary

# Execute rollback procedure
kubectl set image deployment/learnille-api app=learnille/api:1.2.2
kubectl rollout status deployment/learnille-api

Recovery Phase

Verify Service Recovery

# Health checks
curl -f https://api.learnille.com/health
curl -f https://app.learnille.com/health

# Performance validation
ab -n 100 -c 10 https://api.learnille.com/api/v1/courses

Update Status

# Update incident status
curl -X PUT https://api.pagerduty.com/incidents/$INCIDENT_ID \
  -H "Authorization: Token token=$PAGERDUTY_TOKEN" \
  -d '{"incident": {"status": "resolved"}}'

Post-Mortem
- Document root cause
- Identify improvement actions
- Update runbooks if needed

2. Database Performance Issues

Symptoms

Slow query response times
High CPU utilization on database
Connection pool exhaustion
Increased error rates

Investigation Steps

Check Database Metrics

# CPU and memory usage on host
top -b -n 1 | head -n 20

# New Relic NRQL query for host CPU & Memory
# NRQL: SELECT average(cpuPercent), average(memoryUsedBytes) FROM SystemSample SINCE 1 hour ago

Identify Slow Queries

-- Find slow queries
SELECT
  query,
  calls,
  total_time,
  mean_time,
  rows
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;

-- Check active connections
SELECT
  pid,
  usename,
  client_addr,
  query_start,
  state,
  query
FROM pg_stat_activity
WHERE state != 'idle';

Check Index Usage

-- Unused indexes
SELECT
  schemaname,
  tablename,
  indexname,
  idx_scan
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY tablename;

-- Index hit rate
SELECT
  sum(idx_blks_hit) / (sum(idx_blks_hit) + sum(idx_blks_read)) AS hit_rate
FROM pg_statio_user_indexes;

Resolution Steps

Optimize Queries
- Add missing indexes
- Rewrite inefficient queries
- Implement query result caching

Scale Database

# Increase instance size
aws rds modify-db-instance \
  --db-instance-identifier learnille-db \
  --db-instance-class db.r5.large \
  --apply-immediately

# Add read replicas
aws rds create-db-instance-read-replica \
  --db-instance-identifier learnille-db-replica \
  --source-db-instance-identifier learnille-db

Connection Pool Optimization

# Update connection pool settings
aws rds modify-db-parameter-group \
  --db-parameter-group-name learnille-db-params \
  --parameters "ParameterName=max_connections,ParameterValue=200,ApplyMethod=immediate"

3. Security Incident Response

Detection

Unusual login patterns
Unexpected data access
Security monitoring alerts
User reports of suspicious activity

Containment

Isolate Affected Systems

# Block suspicious IP addresses
aws waf update-ip-set \
  --name suspicious-ips \
  --scope REGIONAL \
  --id $IP_SET_ID \
  --addresses $SUSPICIOUS_IP

# Disable compromised accounts
aws cognito-idp admin-disable-user \
  --user-pool-id $USER_POOL_ID \
  --username $COMPROMISED_USER

Preserve Evidence

# Collect logs
aws logs create-export-task \
  --log-group-name /aws/ecs/learnille-api \
  --from $(date -d '1 hour ago' +%s) \
  --to $(date +%s) \
  --destination $S3_BUCKET \
  --destination-prefix security-incident/$(date +%Y%m%d_%H%M%S)

# Take database snapshot
aws rds create-db-snapshot \
  --db-instance-identifier learnille-db \
  --db-snapshot-identifier security-incident-$(date +%Y%m%d)

Investigation

Log Analysis

# Search for suspicious patterns
aws logs filter-log-events \
  --log-group-name /aws/ecs/learnille-api \
  --filter-pattern "ERROR.*auth.*failed" \
  --start-time $(date -d '24 hours ago' +%s)

Access Review

# Check recent IAM activity
aws iam list-access-keys \
  --user-name $SUSPICIOUS_USER

# Review CloudTrail logs
aws cloudtrail lookup-events \
  --start-time $(date -d '24 hours ago' +%s) \
  --lookup-attributes AttributeKey=Username,AttributeValue=$SUSPICIOUS_USER

Recovery

Password Reset

# Force password reset for affected users
aws cognito-idp admin-set-user-password \
  --user-pool-id $USER_POOL_ID \
  --username $AFFECTED_USER \
  --password $TEMP_PASSWORD \
  --permanent

Security Updates

# Update security groups
aws ec2 revoke-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp \
  --port 80 \
  --cidr 0.0.0.0/0

# Rotate access keys
aws iam create-access-key --user-name $COMPROMISED_USER
aws iam delete-access-key --user-name $COMPROMISED_USER --access-key-id $OLD_KEY

Operational Runbooks

1. Deployment Runbook

Pre-Deployment Checklist

Deployment Steps

Prepare Release

# Create release branch
git checkout -b release/v1.2.3 main

# Update version
npm version 1.2.3 --no-git-tag-version

Build Artifacts

# Build application
npm run build

# Create Docker image
docker build -t learnille/api:1.2.3 .

# Push to registry
docker push learnille/api:1.2.3

Deploy to Staging

# Update staging environment
kubectl set image deployment/learnille-api app=learnille/api:1.2.3 -n staging
kubectl rollout status deployment/learnille-api -n staging

Validation

# Health checks
curl -f https://api-staging.learnille.com/health

# Smoke tests
npm run test:smoke -- --env staging

Production Deployment

# Blue-green deployment
kubectl set image deployment/learnille-api-blue app=learnille/api:1.2.3
kubectl rollout status deployment/learnille-api-blue

# Switch traffic
kubectl patch service learnille-api -p '{"spec":{"selector":{"version":"blue"}}}'

Post-Deployment

Monitor Performance

# Check metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name CPUUtilization \
  --start-time $(date -d '1 hour ago' +%s) \
  --end-time $(date +%s) \
  --period 300 \
  --statistics Average

Verify Functionality
- User login and registration
- Course creation and enrollment
- Payment processing
- Email notifications
Update Documentation
- Release notes published
- API documentation updated
- User guides updated

2. Backup and Recovery Runbook

Daily Backup Procedure

#!/bin/bash
# daily-backup.sh

DATE=$(date +%Y%m%d)
BACKUP_DIR="/backups/$DATE"

# Database backup
pg_dump learnille_prod > $BACKUP_DIR/database.sql

# File storage backup
aws s3 sync s3://learnille-uploads $BACKUP_DIR/uploads/

# Configuration backup
tar -czf $BACKUP_DIR/config.tar.gz /etc/learnille/

# Upload to S3
aws s3 cp $BACKUP_DIR s3://learnille-backups/daily/$DATE/ --recursive

# Cleanup old backups (keep 30 days)
find /backups -name "*" -type d -mtime +30 -exec rm -rf {} +

Database Recovery

Assess Damage

# Check database status
aws rds describe-db-instances --db-instance-identifier learnille-db

# Verify backup integrity
aws s3 ls s3://learnille-backups/daily/

Restore Database

# Create new instance from backup
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier learnille-db-restored \
  --db-snapshot-identifier learnille-backup-20231201 \
  --db-instance-class db.r5.large

# Update application configuration
kubectl set env deployment/learnille-api DATABASE_URL=$NEW_DB_URL

Data Validation

-- Verify data integrity
SELECT COUNT(*) FROM users;
SELECT COUNT(*) FROM courses;
SELECT COUNT(*) FROM enrollments;

-- Check for data corruption
SELECT * FROM users WHERE email IS NULL;

File Recovery

# Restore from S3 backup
aws s3 sync s3://learnille-backups/daily/2023-12-01/uploads/ s3://learnille-uploads/

# Verify file integrity
aws s3 ls s3://learnille-uploads/ --recursive | wc -l

3. Monitoring Setup Runbook

Application Monitoring

Install Monitoring Agent

# Install CloudWatch agent
wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
sudo rpm -U amazon-cloudwatch-agent.rpm

# Configure agent
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Configure Metrics

{
  "metrics": {
    "namespace": "Learnille/API",
    "metrics_collected": {
      "cpu": {
        "measurement": ["cpu_usage_idle", "cpu_usage_user", "cpu_usage_system"],
        "metrics_collection_interval": 60
      },
      "mem": {
        "measurement": ["mem_used_percent"],
        "metrics_collection_interval": 60
      },
      "disk": {
        "measurement": ["disk_used_percent"],
        "metrics_collection_interval": 300
      }
    }
  }
}

Set Up Alarms

# CPU utilization alarm
aws cloudwatch put-metric-alarm \
  --alarm-name "HighCPUUtilization" \
  --alarm-description "CPU utilization is high" \
  --metric-name CPUUtilization \
  --namespace AWS/ECS \
  --statistic Average \
  --period 300 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --alarm-actions $SNS_TOPIC_ARN

New Relic Alert Conditions Setup

Configure High Memory Alarm in New Relic
- Condition Type: NRQL Alert Condition
- NRQL Query: SELECT average(memoryUsedBytes / memoryTotalBytes * 100) FROM SystemSample
- Threshold: > 85% for 5 minutes
Configure Database Connection & Response Time Alarm
- Condition Type: NRQL Alert Condition
- NRQL Query: SELECT average(databaseDuration) FROM Transaction WHERE appName = 'Learnille API (Self-Hosted)'
- Threshold: > 0.1 seconds for 5 minutes

Database Monitoring

Enable Enhanced Monitoring

aws rds modify-db-instance \
  --db-instance-identifier learnille-db \
  --monitoring-interval 60 \
  --monitoring-role-arn $MONITORING_ROLE_ARN

Configure Database Metrics

# Database connections
aws cloudwatch put-metric-alarm \
  --alarm-name "HighDBConnections" \
  --metric-name DatabaseConnections \
  --namespace AWS/RDS \
  --statistic Maximum \
  --period 300 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold

# Read latency
aws cloudwatch put-metric-alarm \
  --alarm-name "HighReadLatency" \
  --metric-name ReadLatency \
  --namespace AWS/RDS \
  --statistic Average \
  --period 300 \
  --threshold 0.010 \
  --comparison-operator GreaterThanThreshold

4. Capacity Planning Runbook

Resource Usage Analysis

Current Usage Assessment

# CPU usage trends
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name CPUUtilization \
  --start-time $(date -d '30 days ago' +%s) \
  --end-time $(date +%s) \
  --period 3600 \
  --statistics Average

# Memory usage trends
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name MemoryUtilization \
  --start-time $(date -d '30 days ago' +%s) \
  --end-time $(date +%s) \
  --period 3600 \
  --statistics Average

Growth Projections
- Analyze user growth trends
- Project resource requirements
- Identify scaling thresholds
- Plan capacity upgrades

Scaling Procedures

Horizontal Scaling

# Scale ECS service
aws ecs update-service \
  --cluster learnille-prod \
  --service learnille-api \
  --desired-count 10

# Scale database read replicas
aws rds modify-db-instance \
  --db-instance-identifier learnille-db-replica-1 \
  --db-instance-class db.r5.large \
  --apply-immediately

Vertical Scaling

# Upgrade instance type
aws ecs update-service \
  --cluster learnille-prod \
  --service learnille-api \
  --task-definition learnille-api-v2 \
  --force-new-deployment

# Upgrade database
aws rds modify-db-instance \
  --db-instance-identifier learnille-db \
  --db-instance-class db.r5.xlarge \
  --apply-immediately

Maintenance Runbooks

1. Security Patching

#!/bin/bash
# security-patching.sh

# Update system packages
sudo yum update -y

# Update Docker images
docker pull learnille/api:latest

# Restart services
kubectl rollout restart deployment/learnille-api

# Verify updates
rpm -qa | grep -i security
docker images | grep learnille/api

2. Log Rotation

#!/bin/bash
# log-rotation.sh

# Rotate application logs
logrotate -f /etc/logrotate.d/learnille

# Archive old logs to S3
aws s3 sync /var/log/learnille/archive/ s3://learnille-logs/archive/

# Clean old archives (keep 90 days)
find /var/log/learnille/archive -name "*.gz" -mtime +90 -delete

# Verify log rotation
ls -la /var/log/learnille/
df -h /var/log

3. Certificate Renewal

#!/bin/bash
# certificate-renewal.sh

# Check certificate expiration
openssl x509 -in /etc/ssl/certs/learnille.crt -text -noout | grep "Not After"

# Request new certificate
aws acm request-certificate \
  --domain-name learnille.com \
  --validation-method DNS

# Update CloudFront distribution
aws cloudfront update-distribution \
  --id $DISTRIBUTION_ID \
  --distribution-config file://distribution-config.json

# Verify certificate
curl -I https://learnille.com

Communication Templates

Incident Notification

**INCIDENT ALERT**

**Service:** Learnille API
**Severity:** High
**Status:** Investigating
**Start Time:** 2024-01-15 14:30 UTC
**Description:** API service experiencing elevated error rates
**Impact:** Users may experience slow response times or temporary service unavailability
**Updates:** Investigating database performance issues
**ETA:** 15 minutes

Maintenance Notification

**MAINTENANCE NOTICE**

**Service:** Learnille Platform
**Date:** 2024-01-20
**Time:** 02:00 - 04:00 UTC
**Description:** Database maintenance and security patching
**Impact:** Service may be unavailable for up to 10 minutes
**Contact:** infrastructure@learnille.com

Status Update

**STATUS UPDATE**

**Incident:** API Service Outage
**Status:** Resolved
**Resolution:** Database connection pool optimized
**Timeline:**
- 14:30: Incident detected
- 14:35: Investigation started
- 14:45: Root cause identified
- 14:50: Fix deployed
- 15:00: Service fully recovered

**Next Steps:** Post-mortem analysis scheduled for tomorrow

Escalation Procedures

Level 1 Support

Monitor alerts and basic troubleshooting
Follow runbooks for common issues
Escalate to Level 2 if unresolved within 15 minutes

Level 2 Support

Advanced troubleshooting and diagnostics
Coordinate with development team
Implement fixes and workarounds
Escalate to Level 3 for critical issues

Level 3 Support

Executive decision making
External vendor coordination
Crisis management
Customer communication

Review and Updates

Monthly Review

Review incident response effectiveness
Update runbooks based on lessons learned
Validate monitoring and alerting
Test backup and recovery procedures
Update contact information

Continuous Improvement

Automate manual procedures where possible
Implement preventive measures
Enhance monitoring coverage
Improve communication processes
Update training materials

Overview​

Incident Response Playbook​

1. Service Outage Response​

Detection Phase​

Investigation Phase​

Resolution Phase​

Recovery Phase​

2. Database Performance Issues​

Symptoms​

Investigation Steps​

Resolution Steps​

3. Security Incident Response​

Detection​

Containment​

Investigation​

Recovery​

Operational Runbooks​

1. Deployment Runbook​

Pre-Deployment Checklist​

Deployment Steps​

Post-Deployment​

2. Backup and Recovery Runbook​

Daily Backup Procedure​

Database Recovery​

File Recovery​

3. Monitoring Setup Runbook​

Application Monitoring​

New Relic Alert Conditions Setup​

Database Monitoring​

4. Capacity Planning Runbook​

Resource Usage Analysis​

Scaling Procedures​

Maintenance Runbooks​

1. Security Patching​

2. Log Rotation​

3. Certificate Renewal​

Communication Templates​

Incident Notification​

Maintenance Notification​

Status Update​

Escalation Procedures​

Level 1 Support​

Level 2 Support​

Level 3 Support​

Review and Updates​

Monthly Review​

Continuous Improvement​

Contact Information​

Emergency Contacts​

Communication Channels​