Recovery Planning: Getting Back to Normal — Safely

Prerequisites

• CIR-K005: Persistence Hunting & Eradication
• CIR-K004: Containment Decision-Making

Why This Lesson Matters

The most dangerous moment in an incident is recovery. The pressure to restore service is enormous. Business is losing money. Users are calling. Management is asking for ETAs. In this environment, the temptation to shortcut — restore from a backup that may be post-compromise, reconnect before patching, skip the monitoring uplift — is at its highest. This lesson gives you the framework to recover confidently without putting yourself back at square one.

1. Recovery Is Not "Put Everything Back"

Recovery means returning to a known-clean, hardened, monitored state. Three words, all of which matter.

Known-clean: You are confident the restored system contains no attacker artefacts. This requires knowing the date of initial compromise and restoring from a backup that predates it.

Hardened: The vulnerability that enabled the initial access has been closed. Restoring a system to its pre-compromise state without patching the vulnerability means restoring a compromised-waiting-to-happen system.

Monitored: Increased monitoring is in place for 72 hours post-recovery. If any residual attacker activity exists, you want to catch it in the first hours, not the first days.

2. Determining the Safe Backup Window

The hardest question in recovery: which backup is clean?

Timeline:
  2026-06-01  Last known-clean backup
  2026-06-05  Initial compromise (attacker gains access — may be uncertain)
  2026-06-08  Incident detected

Safe backup: any backup from before 2026-06-05
Unsafe backup: any backup from 2026-06-05 onwards (may contain malware)

Problem: You often do not know the exact date of initial compromise. You know when you detected it, not when it started.

Solution: Assume the worst. If the first confirmed suspicious event is June 5, the attacker may have been present since June 3 or earlier. Check backup contents (scan the backup image before restoring) and restore the oldest backup that satisfies business requirements if there is uncertainty.

# Before restoring: mount the backup image read-only and scan it
sudo mount -o ro,loop backup_20260601.img /mnt/backup_check
clamscan -r /mnt/backup_check 2>/dev/null   # quick malware scan
grep -r "suspicious_string" /mnt/backup_check/etc/cron.d/
# If the backup contains the malware → go to an earlier backup

3. The Recovery Checklist

A complete recovery checklist for a compromised Linux server:

RECOVERY CHECKLIST — IR-2026-0608-001
Host: SRV-001
Recovery lead: Alice Martin

PRE-RESTORE
[ ] Forensic image of compromised system taken and hashed (before any changes)
[ ] All evidence collected and stored per chain of custody
[ ] Eradication complete and validated (CIR-K005 checklist signed off)
[ ] Backup selected: 2026-06-01 00:00 UTC (predates earliest possible compromise)
[ ] Backup hash verified: matches record at time of backup

RESTORE
[ ] Restore from backup to isolated network segment (not production yet)
[ ] Validate restore integrity: sha256sum of critical files matches known-good
[ ] Apply all OS patches released since backup date
[ ] Apply application patches for the exploited vulnerability
[ ] Harden: disable the specific attack vector that was abused
    (e.g., if SSH brute force: enable key-only auth, rate limiting)

CREDENTIAL RESET
[ ] Reset ALL account passwords that existed on the compromised host
[ ] Revoke and reissue ALL SSH keys associated with this host
[ ] Rotate API keys and service account credentials stored on this host
[ ] Notify users of mandatory password reset

PRE-PRODUCTION VALIDATION
[ ] Run EDR/AV full scan on restored system (clean: 0 detections)
[ ] Run persistence check (CIR-K005 checklist) — confirm nothing found
[ ] Verify application functionality: application owner sign-off
[ ] Confirm logging is operational: test log shipping to SIEM

MONITORING UPLIFT
[ ] Enable enhanced logging for 72 hours (verbose process creation, all network connections)
[ ] Create a SIEM rule that alerts on ANY connection to the C2 IP from this host
[ ] Assign a dedicated analyst to monitor this host for 72 hours post-reconnect

RECONNECTION
[ ] Return host to production network
[ ] Record time and analyst name in the incident ticket
[ ] Notify business owner that service is restored

4. Credential Reset — A Frequent Gap

One of the most commonly skipped recovery steps is comprehensive credential reset. Attackers who dump credentials use them weeks or months later for secondary access.

Reset scope for a compromised host:

# Windows: all accounts that had local admin rights on the compromised host
Get-LocalGroupMember -Group "Administrators"
# For each listed account: force password reset

# Service accounts used by applications on this host
Get-WmiObject Win32_Service | Where-Object {$_.StartName -notmatch "LocalSystem|NetworkService|LocalService"} |
  Select-Object Name, StartName
# Each listed service account needs a credential rotation

# Domain admin accounts that logged in (Event 4672 — special privileges)
# Any account that appeared in Event 4672 during the compromise window

5. Monitoring Uplift Post-Recovery

The 72 hours after recovery are high-risk. If you missed any residual persistence, it will likely activate during this window.

# SIEM rule to add during recovery monitoring window
title: C2 Contact from Recovered Host
detection:
  selection:
    src_ip: 10.0.0.1    # SRV-001 (recovered host)
    dst_ip: 185.220.101.5   # confirmed C2 IP
  condition: selection
level: critical
# Alert within 1 minute; page the IR lead on-call

Also watch for: - New services or scheduled tasks created on the recovered host - Any outbound connection to unfamiliar external IPs - Authentication events from unusual source IPs - Process executions from writable directories (Temp, AppData, Public)

6. Common Mistakes

Mistake 1: Restoring from a post-compromise backup. If your backup from the day of the incident contains the malware, you are restoring the attacker's persistence mechanisms. Always verify the backup date against the earliest possible compromise date.

Mistake 2: Reconnecting before patching. A restored system with the same unpatched vulnerability will be compromised again, sometimes within minutes if the attacker is still actively scanning.

Mistake 3: No monitoring uplift after recovery. Recovery without enhanced monitoring is hope, not security. If there is residual persistence you missed, you need to catch it immediately — not when the next routine scan runs.

7. Practice Exercises

1. You discover that the earliest possible compromise date is June 3. Your backups are: June 1 (weekly), June 5 (daily), June 6 (daily), June 7 (daily), June 8 (daily). Which backup do you use? What do you do if the June 1 backup is too old to satisfy the application's data requirements?

2. After recovery, your SIEM fires: "New service installed on SRV-001, 6 hours after reconnect." What does this mean? What is your immediate action?

3. List four credential types that must be rotated after a server compromise, beyond the directly compromised user account.

8. Lab

Assessment mode: quiz

Scenario-based quiz: 5 questions on backup selection, recovery checklist gaps, credential rotation scope, and post-recovery monitoring. You will be given a timeline and asked to identify the safe backup window and the missing recovery steps in a described procedure.

9. Framework Alignment

10. Further Reading

• NIST SP 800-34 Rev.1 — Contingency Planning Guide — defines RTO, RPO, and backup strategy
• 3-2-1 Backup Rule — 3 copies, 2 different media, 1 offsite — the minimum viable backup policy
• Veeam Ransomware Recovery Guide — Practical guidance on clean-restore validation

["Identify the safe backup window for recovery given an incident timeline with an uncertain initial compromise date, and explain how to verify a backup image before restoring", "Produce a recovery checklist for a described Linux server compromise that covers pre-restore validation, patching, credential reset, and monitoring uplift steps", "Design a SIEM monitoring rule for a 72-hour post-recovery watch period that would detect residual C2 activity or new persistence installation"]

Prerequisites → Why this matters → Recovery ≠ "put it back" (3 required properties) → Determining safe backup window → Recovery checklist (full template with command examples) → Credential reset scope → Monitoring uplift post-recovery (SIEM rule example) → Common mistakes → Practice exercises → Quiz lab → Framework alignment → Further reading