Skip to main content

Rollback Instructions

Overview​

This document provides procedures for rolling back deployments in the Learnille platform. Rollbacks may be necessary due to application failures, performance issues, or other critical problems.

Rollback Types​

1. Automated Rollback​

  • Trigger: Health check failures, error thresholds exceeded
  • Execution: Immediate, no manual intervention required
  • Duration: < 5 minutes
  • Scope: Application container rollback

2. Manual Rollback​

  • Trigger: Detected issues requiring human judgment
  • Execution: Manual command execution
  • Duration: 10-30 minutes
  • Scope: Application, database, or infrastructure

3. Emergency Rollback​

  • Trigger: Critical system failure, security breach
  • Execution: Immediate action by on-call engineer
  • Duration: < 15 minutes
  • Scope: Complete environment restoration

Pre-Rollback Checklist​

Assessment​

  • Identify the issue and impact
  • Determine rollback scope (application, database, infrastructure)
  • Check current system status and monitoring
  • Notify stakeholders if needed
  • Gather evidence (logs, metrics, user reports)

Preparation​

  • Verify backup availability and integrity
  • Check previous version stability
  • Prepare rollback commands/scripts
  • Ensure access to deployment tools
  • Coordinate with team members if needed

Automated Rollback Procedures​

Application Rollback (Kubernetes)​

Immediate Rollback​

# Check current deployment status
kubectl get deployments -n learnille-prod

# Rollback to previous revision
kubectl rollout undo deployment/learnille-api -n learnille-prod

# Monitor rollback progress
kubectl rollout status deployment/learnille-api -n learnille-prod

Blue-Green Rollback​

# Check which environment is active
kubectl get service learnille-api -n learnille-prod -o yaml

# Switch traffic back to blue environment
kubectl patch service learnille-api -n learnille-prod -p '{"spec":{"selector":{"version":"blue"}}}'

# Verify traffic switch
kubectl get endpoints learnille-api -n learnille-prod

Database Rollback​

Migration Rollback​

# Check migration status
npm run migration:status -- --env production

# Rollback last migration
npm run migration:rollback -- --env production

# Verify database integrity
npm run migration:status -- --env production

Data Restore (if needed)​

# Restore from backup (if data corruption)
pg_restore -h production-db -U learnille -d learnille_prod backup_file.sql

# Verify data integrity
psql -h production-db -U learnille -d learnille_prod -c "SELECT COUNT(*) FROM users;"

Manual Rollback Procedures​

1. Application Rollback​

Self-Hosted PM2 / Git Rollback​

# SSH into self-hosted production server
ssh admin@server.learnille.dev

# Navigate to application root
cd /opt/learnille

# Checkout previous stable Git commit / tag
git checkout v1.2.0

# Rebuild and reload PM2 process
cd server && npm install && npm run build
pm2 reload learnille-api --update-env

--service learnille-api
--task-definition learnille-api:1.2.2
--force-new-deployment

Monitor deployment

aws ecs describe-services
--cluster learnille-prod
--services learnille-api
--query 'services[0].deployments'


#### Using Docker Compose
```bash
# Stop current containers
docker-compose -f production.yml down

# Update image version in compose file
sed -i 's/learnille\/api:1.2.3/learnille\/api:1.2.2/' production.yml

# Start previous version
docker-compose -f production.yml up -d

# Verify containers
docker-compose -f production.yml ps

2. Infrastructure Rollback​

Terraform Rollback​

# Check current state
terraform show

# Rollback to previous state
terraform plan -state=terraform.tfstate.backup
terraform apply -state=terraform.tfstate.backup

# Verify infrastructure
terraform show

CloudFormation Rollback​

# Check stack status
aws cloudformation describe-stacks --stack-name learnille-prod

# Rollback to previous version
aws cloudformation update-stack \
  --stack-name learnille-prod \
  --template-body file://template-v1.2.2.yaml \
  --parameters file://parameters.json

# Monitor stack update
aws cloudformation describe-stack-events --stack-name learnille-prod

3. Configuration Rollback​

Environment Variables​

# Backup current environment
cp .env.production .env.production.backup

# Restore previous environment
cp .env.production.v1.2.2 .env.production

# Restart application
kubectl rollout restart deployment/learnille-api -n learnille-prod

Configuration Files​

# Backup current config
cp config/production.yaml config/production.yaml.backup

# Restore previous config
cp config/production-v1.2.2.yaml config/production.yaml

# Apply configuration
kubectl apply -f config/production.yaml

Emergency Rollback Procedures​

Critical System Failure​

  1. Immediate Assessment

    # Check system status
    curl -f https://api.learnille.com/health || echo "API down"

    # Check database connectivity
    psql -h production-db -U learnille -d learnille_prod -c "SELECT 1;"

    # Check infrastructure status
    aws ecs describe-services --cluster learnille-prod --services learnille-api

  2. Execute Emergency Rollback

    # Force rollback to last known good state
    kubectl set image deployment/learnille-api app=learnille/api:1.2.2 -n learnille-prod
    kubectl rollout status deployment/learnille-api -n learnille-prod

    # Switch to backup database if needed
    aws rds failover-db-cluster --db-cluster-identifier learnille-prod

  3. Traffic Management

    # Enable maintenance mode
    kubectl patch configmap learnille-config -p '{"data":{"maintenance":"true"}}'

    # Switch to backup load balancer
    aws elbv2 set-ip-address-type \
      --load-balancer-arn $BACKUP_LB_ARN \
      --ip-address-type ipv4

Security Incident Response​

  1. Isolate Affected Systems

    # Block suspicious traffic
    aws waf update-ip-set \
      --name suspicious-ips \
      --scope REGIONAL \
      --id $IP_SET_ID \
      --addresses $SUSPICIOUS_IP

    # Disable compromised service
    kubectl scale deployment learnille-api --replicas=0 -n learnille-prod

  2. Data Recovery

    # Restore from clean backup
    aws s3 cp s3://learnille-backups/clean-backup.sql /tmp/
    psql -h production-db -U learnille -d learnille_prod < /tmp/clean-backup.sql

  3. Security Validation

    # Run security scan
    docker run --rm -v $(pwd):/src securecodebox/scanner

    # Update security groups
    aws ec2 revoke-security-group-ingress --group-id $SG_ID --protocol tcp --port 80 --cidr 0.0.0.0/0

Post-Rollback Procedures​

Validation Steps​

  • Application health checks passing
  • Database connectivity verified
  • User functionality tested
  • Performance metrics normal
  • Monitoring alerts cleared

Documentation​

# Document rollback in incident log
echo "$(date): Rollback executed - $(whoami)" >> rollback-log.txt
echo "Reason: $ROLLBACK_REASON" >> rollback-log.txt
echo "Impact: $ROLLBACK_IMPACT" >> rollback-log.txt

# Update status page
curl -X POST https://status.learnille.com/api/incidents \
  -H "Authorization: Bearer $STATUS_API_KEY" \
  -d '{"status": "resolved", "message": "Issue resolved via rollback"}'

Communication​

  • Notify internal team via Slack
  • Update external status page
  • Send customer communication if needed
  • Schedule post-mortem meeting

Investigation​

  • Analyze root cause of deployment failure
  • Review monitoring data during incident
  • Update deployment processes if needed
  • Document lessons learned

Monitoring During Rollback​

Key Metrics to Monitor​

  • Application response time
  • Error rates by endpoint
  • Database connection pool usage
  • Infrastructure resource utilization
  • User session success rate

Alerting During Rollback​

# Temporary rollback monitoring
groups:
  - name: rollback-monitoring
    rules:
      - alert: RollbackHealthCheck
        expr: up{job="learnille-api"} == 0
        for: 2m
        labels:
          severity: critical
          rollback: "true"
      - alert: RollbackErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
          rollback: "true"

Rollback Testing​

Regular Testing​

  • Monthly rollback drills
  • Automated rollback testing in staging
  • Documentation review and updates
  • Team training sessions

Test Scenarios​

  • Application deployment failure
  • Database migration failure
  • Infrastructure configuration error
  • Security vulnerability discovery
  • Performance degradation

Tools and Resources​

Rollback Scripts​

#!/bin/bash
# emergency-rollback.sh
set -e

echo "Starting emergency rollback..."

# Get last stable version
STABLE_VERSION=$(curl -s https://api.github.com/repos/learnille/learnille/releases/latest | jq -r .tag_name)

# Execute rollback
kubectl set image deployment/learnille-api app=learnille/api:$STABLE_VERSION
kubectl rollout status deployment/learnille-api

echo "Rollback completed successfully"

Monitoring Dashboard​

  • Rollback status dashboard
  • Real-time metrics during rollback
  • Historical rollback data
  • Success/failure analytics

Contact Information​

Emergency Contacts​

  • Primary On-call: +1-555-0123 (DevOps Engineer)
  • Secondary On-call: +1-555-0124 (Senior Engineer)
  • Management: +1-555-0125 (Engineering Manager)

Communication Channels​

Continuous Improvement​

Metrics Tracking​

  • Rollback frequency and success rate
  • Mean time to rollback
  • Customer impact assessment
  • Process efficiency improvements

Process Updates​

  • Regular review of rollback procedures
  • Incorporation of lessons learned
  • Tool and automation improvements
  • Team training and preparedness