# AEGIS SENTINEL: Operational Playbook
**Version:** 1.0 | **Last Updated:** January 2026 | **Classification:** INTERNAL - SRE TEAM

---

## Purpose

This playbook provides step-by-step procedures for the Site Reliability Engineering (SRE) team when critical alerts trigger. Every procedure has been tested during quarterly drills and is designed for execution under pressure.

**Golden Rule:** When in doubt, escalate. A false alarm is better than a missed breach.

---

## Table of Contents
1. [Alert Classification](#1-alert-classification)
2. [Sovereign Exit Switch Playbook](#2-sovereign-exit-switch-playbook)
3. [Dead-Man's Switch Playbook](#3-dead-mans-switch-playbook)
4. [Kill-Switch Activation Playbook](#4-kill-switch-activation-playbook)
5. [Mesh Network Failure Playbook](#5-mesh-network-failure-playbook)
6. [HSM Key Compromise Playbook](#6-hsm-key-compromise-playbook)
7. [DDoS Attack Playbook](#7-ddos-attack-playbook)
8. [Data Breach Playbook](#8-data-breach-playbook)
9. [Escalation Matrix](#9-escalation-matrix)
10. [Communication Templates](#10-communication-templates)

---

## 1. Alert Classification

### Severity Levels

| Level | Color | Response Time | Who Gets Paged |
|-------|-------|---------------|----------------|
| P0 - CRITICAL | RED | 5 minutes | CISO, CTO, CEO, Full SRE Team |
| P1 - HIGH | ORANGE | 15 minutes | CISO, SRE Lead, On-Call Engineer |
| P2 - MEDIUM | YELLOW | 1 hour | SRE Lead, On-Call Engineer |
| P3 - LOW | BLUE | 4 hours | On-Call Engineer |
| P4 - INFO | GREEN | Next business day | Ticket Queue |

### Alert Sources

| Source | Dashboard | Slack Channel |
|--------|-----------|---------------|
| Sovereign Readiness | /sovereign/status | #aegis-sovereign |
| Safety Rails | /safety/dashboard | #aegis-safety |
| Mesh Network | /mesh/status | #aegis-mesh |
| Threat Detection | /threats/live | #aegis-threats |
| HSM | /hsm/dashboard | #aegis-hsm |

---

## 2. Sovereign Exit Switch Playbook

### When This Triggers

The Exit Switch activates when:
- International cloud latency exceeds 500ms for >60 seconds
- Geopolitical sovereignty alarm is raised
- Manual CISO override is activated
- Regulatory order to localize data

### Alert Message Example
```
[P0-CRITICAL] SOVEREIGN EXIT TRIGGERED
Reason: AWS eu-west-1 latency exceeded 500ms (current: 2,340ms)
Target: NSSF-KLA (Kampala)
ETA: 45 minutes
Initiated by: AUTO-TRIGGER / [Username]
```

### Immediate Actions (First 5 Minutes)

| Step | Action | Owner | Verification |
|------|--------|-------|--------------|
| 1 | Join War Room call | All SRE | Zoom/Teams link in alert |
| 2 | Acknowledge alert in system | On-Call | Alert status changes to "Acknowledged" |
| 3 | Verify trigger is legitimate | SRE Lead | Check cloud status page, ping tests |
| 4 | Notify CISO | On-Call | Phone call + Slack DM |
| 5 | Open incident ticket | On-Call | Ticket ID noted in War Room |

### Phase 1: Pre-Flight (Minutes 5-10)

```bash
# Verify target datacenter readiness
curl -X GET https://aegis.local/api/cloud-exit/datacenters/NSSF-KLA/health \
  -H "Authorization: Bearer $TOKEN"

# Expected response: {"status": "READY", "capacity": "AVAILABLE", "lag": "0ms"}
```

**Checklist:**
- [ ] Target DC health: GREEN
- [ ] Network path verified: PASS
- [ ] Replication lag: < 10 seconds
- [ ] Security configs: SYNCED
- [ ] Firewall rules: APPLIED

### Phase 2: Data Synchronization (Minutes 10-25)

```bash
# Monitor sync progress
curl -X GET https://aegis.local/api/cloud-exit/progress \
  -H "Authorization: Bearer $TOKEN"

# Expected fields: dataSynced, totalData, percentComplete, estimatedTimeRemaining
```

**Monitoring Points:**
| Metric | Threshold | Action if Exceeded |
|--------|-----------|-------------------|
| Sync Rate | > 100 MB/s | Normal - continue |
| Sync Rate | < 50 MB/s | Investigate network |
| Errors | 0 | Normal - continue |
| Errors | > 10 | Pause and investigate |

### Phase 3: Traffic Cutover (Minutes 25-40)

```bash
# Initiate DNS cutover
curl -X POST https://aegis.local/api/cloud-exit/cutover \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"phase": "DNS_UPDATE"}'
```

**Cutover Sequence:**
1. DNS TTL reduction (already done in prep)
2. Update DNS records to local DC
3. Wait for propagation (2-5 minutes)
4. Drain connections from old DC
5. Redirect load balancers

**Verification:**
```bash
# Test DNS resolution
dig aegis.bank.ug

# Expected: Points to local DC IP (e.g., 102.x.x.x)
```

### Phase 4: Validation (Minutes 40-45)

**Smoke Tests:**
- [ ] Login to dashboard: PASS
- [ ] KYC verification test: PASS
- [ ] Threat detection ping: PASS
- [ ] SAR filing test: PASS
- [ ] Mesh sync test: PASS

```bash
# Run automated smoke tests
curl -X POST https://aegis.local/api/cloud-exit/smoke-tests \
  -H "Authorization: Bearer $TOKEN"

# All tests must return: {"status": "PASS"}
```

### Post-Exit Actions

| Time | Action | Owner |
|------|--------|-------|
| +1 hour | Verify all services stable | SRE Lead |
| +2 hours | Generate incident report | SRE Lead |
| +4 hours | Notify Bank of Uganda | Compliance |
| +24 hours | Post-incident review | Full team |
| +48 hours | Update runbooks if needed | SRE Lead |

### Rollback Procedure

If exit fails at any phase:

```bash
# Abort exit
curl -X POST https://aegis.local/api/cloud-exit/abort \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"reason": "Sync errors exceeded threshold"}'

# Verify rollback
curl -X GET https://aegis.local/api/cloud-exit/status \
  -H "Authorization: Bearer $TOKEN"

# Expected: {"status": "ABORTED", "currentDC": "AWS-EU-WEST-1"}
```

---

## 3. Dead-Man's Switch Playbook

### When This Triggers

Auto-lockdown activates when:
- Data volume exceeds 500 MB/hour outbound
- Record count exceeds 100,000/hour extracted
- Unusual bulk API access patterns detected
- Manual security officer override

### Alert Message Example
```
[P0-CRITICAL] DEAD-MAN'S SWITCH ACTIVATED
Reason: Data exfiltration detected
Volume: 847 MB in last hour (threshold: 500 MB)
Records: 234,567 (threshold: 100,000)
Source IP: 185.x.x.x (Unknown - GeoIP: Russia)
State: LOCKDOWN
```

### Immediate Actions (First 5 Minutes)

| Step | Action | Owner |
|------|--------|-------|
| 1 | DO NOT attempt to disable lockdown | ALL |
| 2 | Join War Room immediately | All SRE + Security |
| 3 | Preserve all logs | On-Call |
| 4 | Notify CISO via phone | SRE Lead |
| 5 | Alert legal/compliance | CISO |

### Lockdown States

| State | External Access | Internal Access | AI Operations |
|-------|-----------------|-----------------|---------------|
| NORMAL | Allowed | Allowed | Running |
| ELEVATED | Monitored | Allowed | Running |
| LOCKDOWN | Blocked | Allowed | Limited |
| FULL_ISOLATION | Blocked | Emergency Only | Stopped |

### Investigation Procedure

```bash
# Get exfiltration event details
curl -X GET https://aegis.local/api/sovereign/exfiltration/events \
  -H "Authorization: Bearer $TOKEN"
```

**Analysis Checklist:**
- [ ] Identify source IP/user
- [ ] Determine what data was accessed
- [ ] Check if legitimate (scheduled backup, audit)
- [ ] Review access logs for anomalies
- [ ] Correlate with Bio-Vault data

### If LEGITIMATE Activity

```bash
# Request lockdown downgrade (requires 2 authorized users)
curl -X POST https://aegis.local/api/sovereign/lockdown \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "state": "NORMAL",
    "reason": "False positive - scheduled backup",
    "authorizedBy": "CISO",
    "secondApproval": "HEAD_OF_RISK"
  }'
```

### If MALICIOUS Activity

1. **DO NOT downgrade lockdown**
2. Escalate to FULL_ISOLATION if needed:
```bash
curl -X POST https://aegis.local/api/sovereign/lockdown \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"state": "FULL_ISOLATION", "reason": "Confirmed breach"}'
```
3. Begin forensic evidence collection
4. Notify Financial Intelligence Authority (FIA)
5. Engage incident response team
6. Prepare regulatory notification (72-hour deadline per PDPO)

---

## 4. Kill-Switch Activation Playbook

### When This Triggers

The AI Kill-Switch activates when:
- Reward-hacking behavior detected in agents
- Logic loop detected (>5 iterations)
- Consensus validation failure between agents
- Model drift exceeds threshold
- Manual administrator activation

### Alert Message Example
```
[P0-CRITICAL] AI KILL-SWITCH ACTIVATED
Level: HARD_STOP
Reason: Logic loop detected in Validation Agent
Iterations: 23 (threshold: 5)
Affected: All Agentic KYC operations   <-- ILLUSTRATIVE ONLY. Agentic KYC is roadmap,
                                           not implemented (ENGINEERING_ROADMAP_2026.md
                                           Feature 5). No such operations exist today;
                                           this sample alert shows the FUTURE shape.
Fallback: Manual verification mode
```

### Immediate Actions

| Step | Action | Owner |
|------|--------|-------|
| 1 | Confirm all AI operations stopped | SRE Lead |
| 2 | Activate manual verification fallback | Ops Team |
| 3 | Notify affected bank departments | Comms |
| 4 | Preserve AI decision logs | ML Engineer |
| 5 | Begin root cause analysis | ML Team |

### Verify Kill-Switch Status

```bash
# Check current safety status
curl -X GET https://aegis.local/api/safety/status \
  -H "Authorization: Bearer $TOKEN"

# Expected: {"status": "HARD_STOP", "aiOperations": "HALTED", "fallback": "MANUAL"}
```

### Manual Fallback Procedures

While AI is stopped, banks must use manual verification:

1. **KYC Verification**: Route to human review queue
2. **Threat Detection**: Use static rule-based detection
3. **Transaction Monitoring**: Apply conservative thresholds

### Root Cause Analysis

```bash
# Get kill-switch incident details
curl -X GET https://aegis.local/api/safety/incidents \
  -H "Authorization: Bearer $TOKEN"
```

**Investigation Areas:**
- Agent decision logs
- Model input/output pairs
- Training data drift
- Threshold configuration
- Recent deployments

### Recovery Procedure

Only after root cause is identified and fixed:

```bash
# Reset safety system (requires CISO approval)
curl -X POST https://aegis.local/api/safety/reset \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "approvedBy": "CISO",
    "rootCause": "Memory leak in validation loop",
    "fix": "Deployed patch v2.3.1"
  }'
```

**Post-Recovery Verification:**
- [ ] Run AI test suite: PASS
- [ ] Verify thresholds correct: PASS
- [ ] Monitor for 1 hour: STABLE
- [ ] Gradual traffic increase: 10% → 50% → 100%

---

## 5. Mesh Network Failure Playbook

### When This Triggers

Mesh network alerts fire when:
- Node goes offline for >30 seconds
- Sync lag exceeds 60 seconds
- Multiple transport failures
- Threat intel propagation fails

### Alert Message Example
```
[P1-HIGH] MESH NODE OFFLINE
Node: GULU-003 (Mbarara)
Last Contact: 45 seconds ago
Transports Tried: fiber (FAILED), satellite (FAILED), radio (ATTEMPTING)
Impact: Regional threat sync delayed
```

### Immediate Actions

```bash
# Check mesh network status
curl -X GET https://aegis.local/api/mesh/status \
  -H "Authorization: Bearer $TOKEN"

# Check specific node
curl -X GET https://aegis.local/api/mesh/nodes/GULU-003 \
  -H "Authorization: Bearer $TOKEN"
```

### Troubleshooting by Transport

**Fiber Failure:**
1. Contact ISP NOC
2. Check for regional outages
3. Verify local router status

**Satellite Failure:**
1. Check satellite terminal power
2. Verify dish alignment
3. Check for weather interference

**Radio Failure:**
1. Verify radio tower power
2. Check frequency interference
3. Confirm line-of-sight

### Fallback Operations

If node remains offline:
- Regional branch operates in "Island Mode"
- Local KYC uses cached Merkle hashes
- Transactions queued for later sync
- Alert bank operations team

### Recovery

```bash
# Once node is back, force sync
curl -X POST https://aegis.local/api/mesh/sync \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"nodes": ["GULU-003"], "priority": "HIGH"}'
```

---

## 6. HSM Key Compromise Playbook

### When This Triggers

HSM alerts fire when:
- Failed attestation check
- Unauthorized key access attempt
- Key material tampering detected
- HSM hardware failure

### Alert Message Example
```
[P0-CRITICAL] HSM KEY COMPROMISE SUSPECTED
Key ID: MASTER-SIGNING-2026
Event: Failed attestation - hardware tampering detected
HSM Serial: HSM-UG-001
Location: NSSF Data Center, Rack 23
```

### Immediate Actions

| Step | Action | Owner |
|------|--------|-------|
| 1 | DO NOT access the HSM | ALL |
| 2 | Physical security to HSM location | Security Team |
| 3 | Preserve access logs | SRE |
| 4 | Notify CISO immediately | On-Call |
| 5 | Prepare for key ceremony | Security Officer |

### Key Rotation Procedure

If compromise is confirmed:

1. **Initiate Emergency Key Ceremony**
```bash
curl -X POST https://aegis.local/api/hsm/ceremony \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"type": "EMERGENCY_ROTATION", "keyId": "MASTER-SIGNING-2026"}'
```

2. **Requires M-of-N Custodians** (typically 3-of-5)
3. **Generate new key pair**
4. **Revoke compromised key**
5. **Update all dependent services**
6. **Notify regulatory bodies**

---

## 7. DDoS Attack Playbook

### When This Triggers

DDoS alerts fire when:
- Request rate exceeds 10,000/second
- Bandwidth exceeds provisioned capacity
- Geographic anomaly (unusual source countries)
- Application-layer attack patterns

### Alert Message Example
```
[P1-HIGH] DDOS ATTACK DETECTED
Type: Application Layer (L7)
Rate: 47,000 requests/second
Target: /api/kyc/verify
Source: 340 unique IPs (primarily CN, RU)
Status: Ghost Mesh absorbing
```

### Immediate Actions

```bash
# Check Ghost Mesh absorption
curl -X GET https://aegis.local/api/ghost/traps \
  -H "Authorization: Bearer $TOKEN"

# View captured traffic
curl -X GET https://aegis.local/api/ghost/captures \
  -H "Authorization: Bearer $TOKEN"
```

### Mitigation Steps

1. **Ghost Mesh is absorbing** - Monitor and let it work
2. **Rate limiting active** - Verify thresholds
3. **Geographic blocking** - Apply if needed:
```bash
curl -X POST https://aegis.local/api/firewall/geo-block \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"countries": ["CN", "RU"], "duration": "4h"}'
```
4. **Upstream scrubbing** - Escalate to ISP if needed
5. **Sovereign Exit** - Last resort if cloud is target

---

## 8. Data Breach Playbook

### When This Triggers

Breach alerts fire when:
- PII exposure detected
- Unauthorized database access
- Customer data in exfiltration event
- Third-party breach affecting AEGIS

### Regulatory Timeline (PDPO 2019)

| Deadline | Action | Owner |
|----------|--------|-------|
| T+0 | Detect and contain | SRE + Security |
| T+24h | Internal assessment complete | CISO |
| T+72h | Notify Data Protection Authority | Legal |
| T+72h | Notify affected individuals (if high risk) | Comms |
| T+30d | Full investigation report | Compliance |

### Immediate Actions

1. **Contain the breach** - Isolate affected systems
2. **Preserve evidence** - Do not delete logs
3. **Assess scope** - What data, how many affected
4. **Escalate** - CISO, Legal, CEO
5. **Document everything** - Timeline, actions, decisions

### Notification Template

```
Subject: Data Security Incident Notification

Dear [Customer Name],

We are writing to inform you of a data security incident that may have involved your personal information...

[See Section 10 for full template]
```

---

## 9. Escalation Matrix

### By Time of Day

| Time (EAT) | Primary On-Call | Backup |
|------------|-----------------|--------|
| 00:00-08:00 | Night SRE | SRE Lead (phone) |
| 08:00-18:00 | Day SRE | SRE Lead |
| 18:00-00:00 | Evening SRE | SRE Lead (phone) |

### By Severity

| Severity | Escalation Path |
|----------|-----------------|
| P4-INFO | Ticket → On-Call (next day) |
| P3-LOW | Ticket → On-Call (4 hours) |
| P2-MEDIUM | Slack → On-Call → SRE Lead |
| P1-HIGH | PagerDuty → On-Call → SRE Lead → CISO |
| P0-CRITICAL | PagerDuty (ALL) → War Room → CISO → CEO |

### Contact Information

| Role | Phone | Slack | Email |
|------|-------|-------|-------|
| SRE On-Call | [On-Call Rotation] | @sre-oncall | sre@aegis.io |
| SRE Lead | +256-XXX-XXXX | @sre-lead | sre-lead@aegis.io |
| CISO | +256-XXX-XXXX | @ciso | ciso@aegis.io |
| CTO | +256-XXX-XXXX | @cto | cto@aegis.io |
| CEO | +256-XXX-XXXX | @ceo | ceo@aegis.io |
| Legal | +256-XXX-XXXX | @legal | legal@aegis.io |

---

## 10. Communication Templates

### War Room Opening

```
@channel WAR ROOM ACTIVE

Incident: [Brief description]
Severity: P[X]
Incident Commander: [Name]
Zoom/Teams: [Link]

DO:
- Join the call immediately
- Mute unless speaking
- Use thread for updates

DON'T:
- Make changes without IC approval
- Communicate externally
- Speculate on cause
```

### Status Update (Internal)

```
[UPDATE] Incident #[ID] - [HH:MM EAT]

Status: [Investigating / Mitigating / Resolved]
Impact: [Description of customer impact]
Actions Taken: [What we've done]
Next Steps: [What we're doing next]
ETA: [When we expect resolution]

Posted by: [Name]
```

### Customer Communication (Outage)

```
Subject: Service Disruption - AEGIS Platform

We are currently experiencing [brief description of issue].

Impact: [What services are affected]
Status: Our team is actively working to resolve this issue.
Updates: We will provide updates every [30 minutes / 1 hour].

We apologize for any inconvenience.

AEGIS Operations Team
```

### Regulatory Notification (Breach)

```
To: Data Protection Authority Uganda
From: AEGIS Sentinel - Data Protection Officer
Date: [Date]
Re: Data Security Incident Notification

In accordance with Section 23 of the Data Protection and Privacy Act 2019, we hereby notify your office of a data security incident...

[Full details of incident]
[Scope of affected individuals]
[Measures taken]
[Contact information]
```

---

## Document Control

| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0 | January 2026 | SRE Team | Initial release |
| | | | |

---

## Quarterly Drill Schedule

| Quarter | Drill Type | Scheduled Date |
|---------|------------|----------------|
| Q1 2026 | Full Exit Switch | January 15 |
| Q2 2026 | Dead-Man's Switch | April 15 |
| Q3 2026 | Kill-Switch + Recovery | July 15 |
| Q4 2026 | Combined Multi-Scenario | October 15 |

---

**END OF OPERATIONAL PLAYBOOK**

*This playbook is mandatory reading for all SRE team members. Review and sign-off required quarterly.*