Live Response Triage: Spotting Attacker Activity on a Running System

Prerequisites

• CIR-K002: Evidence Collection — Order of Volatility & Chain of Custody

Why This Lesson Matters

You have 20 minutes with a running compromised server before the business team restores it from backup. You need to know exactly where to look to answer: Who is still connected? What did they run? Where did they go next? This lesson is a field guide for that 20-minute window.

1. The Triage Mindset

Triage is not a full forensics investigation. It is rapid evidence gathering focused on three questions:

1. Is the attacker still present?      → Check active connections + processes
2. What did they do?                   → Check logs + process history
3. Where else did they go?             → Check lateral movement indicators

Answer these three questions and you have the information needed for containment and for the investigation that follows.

2. Is the Attacker Still Present?

2.1 Active Network Connections

# Show all TCP connections with the owning process
ss -antp

# Output to look for:
# ESTABLISHED → active connections; note external IPs
# LISTEN      → services listening; note unexpected ports

# Example suspicious output:
# ESTAB  0  0  10.0.0.42:49221  185.220.101.5:443  users:(("update",pid=1337))
#                                ↑ external IP        ↑ suspicious process name

Any ESTABLISHED connection to an external IP from a server that should not be making outbound connections is a red flag. Note the remote IP, the local port, and the process name and PID.

2.2 Suspicious Running Processes

# Full process tree — shows parent-child relationships
ps auxf

# What to look for:
# - Shell (bash, sh, cmd.exe) spawned by a web server (nginx, apache, w3wp)
# - PowerShell with suspicious arguments
# - Processes in /tmp, /dev/shm, C:UsersPublic, C:Temp
# - Processes with no associated binary on disk (deleted-while-running)

# Find processes with deleted binaries (Linux)
ls -la /proc/*/exe 2>/dev/null | grep deleted
# A running process whose binary has been deleted = malware hiding its tracks

Parent-process anomalies to watch:

3. What Did They Do?

3.1 Recent File Activity

# Files modified in the last 24 hours
find / -mmin -1440 -type f 2>/dev/null | grep -v /proc | grep -v /sys

# Files created in the last hour
find /tmp /var/www /home -mmin -60 -type f 2>/dev/null

# Recently executed binaries (accessed, not just modified)
find / -amin -60 -type f -executable 2>/dev/null | grep -v /proc

3.2 Bash History and Sudo Log

# Root command history
cat /root/.bash_history

# All users' histories
find /home -name ".bash_history" -exec cat {} ;

# Sudo usage log (who ran what as root)
grep sudo /var/log/auth.log | tail -50
# Or on RHEL/CentOS:
grep sudo /var/log/secure | tail -50

Attacker tradecraft note: Sophisticated attackers run history -c and unset HISTFILE to clear history before starting work. An empty bash history on a shared server is itself suspicious — document it.

3.3 Installed Persistence

Check all four common Linux persistence locations:

# 1. Cron jobs
crontab -l -u root
for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u $u 2>/dev/null; done
ls -la /etc/cron.d/ /etc/cron.daily/ /etc/cron.hourly/

# 2. Systemd services
systemctl list-units --type=service --state=running
ls -la /etc/systemd/system/ | sort -k6,7   # sort by modification time

# 3. SSH authorized_keys (backdoor access)
find /home /root -name "authorized_keys" -exec cat {} ;
# A key you do not recognise = backdoor

# 4. SUID binaries modified recently
find / -perm -4000 -mmin -1440 -type f 2>/dev/null

4. Where Did They Go? — Lateral Movement Indicators

Lateral movement leaves traces on the source host (what did it connect to?) and on the destination host (who connected to it?).

4.1 On the Source Host (where the attacker started)

# Recent SSH connections made FROM this host
grep "Connecting to|ssh " /root/.bash_history 2>/dev/null
grep "ssh" /var/log/auth.log | grep "Accepted|Connected"

# RDP/SMB connections made from this host (Linux smbclient)
grep "smbclient|rdesktop|xfreerdp" /root/.bash_history

# ARP table — who has this host recently communicated with?
arp -an
# Entries in the ARP cache = recently contacted hosts on the local network

4.2 On Destination Hosts — Auth Log Signatures

When an attacker moves laterally using SSH key theft or stolen credentials, the destination host logs the connection:

# Linux destination: successful SSH logins from internal IPs
grep "Accepted" /var/log/auth.log | grep -v "127.0.0.1|::1"

# Look for:
# - Admin account logging in from a workstation (not a jump server)
# - Login at unusual hours
# - Login from a host that is not in the approved source list

4.3 Windows Lateral Movement Evidence

On Windows, Event ID 4624 with Logon Type 3 (network logon) from an unexpected source is the key signal:

Event 4624 — Logon Type 3
  TargetUserName: Administrator
  IpAddress: 10.0.0.42        ← workstation, not jump server
  WorkstationName: WS-042
  LogonProcess: NtLmSsp       ← NTLM, not Kerberos = Pass-the-Hash indicator

5. Putting It Together: The 20-Minute Triage Checklist

Minute 0–2:   Record timestamp. Start logging your commands (script /media/usb/triage.log)
Minute 2–5:   ss -antp  →  ps auxf  →  Note suspicious connections and processes
Minute 5–8:   find /tmp /var/www -mmin -60 -type f  →  Recent file activity
Minute 8–11:  cat /root/.bash_history  →  crontab -l  →  systemctl list-units
Minute 11–14: grep "Accepted" /var/log/auth.log | grep -v localhost
              →  Lateral movement check
Minute 14–17: Hash all suspicious files:
              sha256sum /tmp/suspicious_binary > /media/usb/hashes.txt
Minute 17–20: Begin memory acquisition:
              sudo insmod lime.ko "path=/media/usb/memory.lime format=lime"

6. Common Mistakes

Mistake 1: Not recording the commands you run during triage. Start script /media/usb/triage_session.log before anything else. It records every command and every output. This becomes part of your evidence.

Mistake 2: Running triage tools from the local system. Use a trusted toolkit on a USB drive. Malware can replace system binaries (ps, ls, netstat) with rootkitted versions that hide the attacker's activity.

Mistake 3: Checking history and stopping when it is empty. An empty history is evidence of tampering — not evidence of innocence. Look harder.

7. Practice Exercises

1. ps auxf on a compromised system shows: nginx (pid 980) → bash (pid 3301). What does this tell you? What is the next command you run?

2. You find a file /tmp/.cache_update that was created 40 minutes ago, is executable, and has no corresponding package in dpkg -l. What do you do with it?

3. You check /var/log/auth.log and find: Accepted publickey for root from 10.0.0.87. The server should never accept direct root logins. What are three things you immediately investigate?

8. Lab

Assessment mode: flag

challenge_spec_id: 22 — Lateral movement trace

You are given a triage_bundle.zip containing auth.log, bash_history, and a netstat snapshot from a compromised Linux server.

Task: 1. Identify the source IP of the initial SSH compromise 2. Find the internal IP the attacker moved to laterally (appears in auth.log and bash_history) 3. Identify the persistence mechanism installed (cron or service name) 4. The flag is: PREFIX{source_ip:lateral_target_ip:persistence_name}

9. Framework Alignment

10. Further Reading

• SANS FOR508 Cheatsheet — Live response commands for Linux and Windows (free poster)
• Eric Zimmerman Tools — https://ericzimmerman.github.io — Windows triage tools (Timeline Explorer, MFT Parser)
• Sysinternals Suite — Microsoft; Autoruns, Process Explorer, TCPView for Windows live response
• "Incident Response & Computer Forensics" — Luttgens, Pepe, Mandia — Chapter 4 covers live response methodology in detail

["Execute a prioritised live response sequence on a Linux system that captures network state, process tree, recent files, persistence mechanisms, and SSH authorised keys in the correct order", "Identify two lateral movement indicators in an auth.log file — an NTLM logon from a workstation and an SSH login from an unexpected source — and explain what each indicates", "Distinguish a process with a deleted binary from a normal running process using /proc filesystem inspection, and explain why this is a malware indicator"]

Prerequisites → Why this matters (20-minute window analogy) → Triage mindset (3 questions) → Is the attacker still present? (network connections, suspicious processes, parent anomaly table) → What did they do? (file activity, bash history, persistence check) → Where did they go? (lateral movement on source and destination) → 20-minute triage checklist → Common mistakes → Practice exercises → Lab (flag, spec 22) → Framework alignment → Further reading