Log Analysis & First Investigations: From Raw Logs to a Timeline

Prerequisites

• FND-K003: Networking Fundamentals for Security Practitioners
• FND-K004: Operating Systems Fundamentals

Why This Lesson Matters

Logs are the primary evidence source in every security investigation. Before you can use a SIEM, before you can write a detection rule, before you can triage an alert — you need to be able to read raw log files and extract meaning from them.

This lesson teaches the skill of going from "I have some log files" to "I know what happened." It introduces log formats, parsing techniques, timeline construction, and the art of noticing anomalies — including the adversarial case where an attacker has tried to manipulate the logs themselves.

Every role in cybersecurity — SOC analyst, incident responder, forensics investigator, penetration tester, and even security engineer — needs this skill.

1. Why Logs Exist and What They Contain

Logs are the historical record of system and application behaviour. They exist for:

Well-designed logs contain: 1. Timestamp — when the event occurred (UTC, not local time) 2. Source — what system/application generated the log 3. Severity — DEBUG, INFO, WARNING, ERROR, CRITICAL 4. Actor — who performed the action (user, service, IP) 5. Action — what was done 6. Target — what was acted upon 7. Outcome — success, failure, partial

2. Log Formats

2.1 Syslog (RFC 5424)

The most widely used log format for Unix/Linux systems:

<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID STRUCTURED-DATA MSG

Simplified common form (RFC 3164 / traditional syslog):

Jun  8 14:32:17 server01 sshd[1234]: Failed password for alice from 10.0.0.5 port 51234 ssh2
├───────────────┤ ├──────┤ ├──┤ ├──┤  ├────────────────────────────────────────────────────┘
    timestamp     hostname  app pid       message

# Tail the syslog in real time
tail -f /var/log/syslog

# Filter for SSH events
grep "sshd" /var/log/auth.log

# Filter for a specific user
grep "alice" /var/log/auth.log

# Show failed logins
grep "Failed password" /var/log/auth.log
grep "authentication failure" /var/log/auth.log

# Show successful logins
grep "Accepted" /var/log/auth.log

2.2 Combined Log Format (HTTP Access Logs)

Used by Apache, Nginx, and most web servers:

203.0.113.5 - alice [08/Jun/2026:14:32:17 +0000] "GET /admin HTTP/1.1" 403 512 "https://example.com/" "Mozilla/5.0"
├──────────┘   ├───┘  ├──────────────────────────┘  ├──────────────────┘ ├─┘ ├─┘  ├─────────────────────┘  └──────────┘
  client IP   ident   timestamp                           request          code bytes   referer               user-agent

Fields: - Client IP: The IP address making the request (may be a proxy; check X-Forwarded-For) - Ident: - (almost always empty; deprecated RFC 1413 ident protocol) - Auth user: Authenticated username if HTTP auth was used (- if anonymous) - Timestamp: Local time of the server at request receipt - Request: METHOD URI PROTOCOL - Status code: HTTP response code - Bytes: Response body size - Referer: The page that linked here (note: misspelled in the standard) - User-Agent: Client browser/tool identification

# Parse access.log for 4xx and 5xx errors
awk '$9 >= 400' /var/log/nginx/access.log

# Top 10 most requested URIs
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head 10

# Top client IPs by request count
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head 10

# Find potential SQL injection in URIs
grep -E "UNION|SELECT|INSERT|DROP|'|--|%27|%22" /var/log/nginx/access.log

# Find directory traversal attempts
grep -E "../|.." /var/log/nginx/access.log

# Find scanner user agents
grep -iE "nikto|sqlmap|masscan|nmap|dirbuster|gobuster|wfuzz" /var/log/nginx/access.log

2.3 JSON Logs

Modern applications and platforms increasingly use structured JSON logging. Each event is a JSON object on a single line (JSON Lines / NDJSON format):

{"timestamp": "2026-06-08T14:32:17Z", "level": "WARN", "event": "failed_login", "user": "alice", "src_ip": "185.220.101.5", "attempt": 3}
{"timestamp": "2026-06-08T14:32:18Z", "level": "WARN", "event": "failed_login", "user": "alice", "src_ip": "185.220.101.5", "attempt": 4}
{"timestamp": "2026-06-08T14:32:19Z", "level": "INFO", "event": "account_locked", "user": "alice", "reason": "too_many_failures"}

# Parse JSON logs with jq
cat events.jsonl | jq '.event'
cat events.jsonl | jq 'select(.level == "WARN") | {ts: .timestamp, user: .user, ip: .src_ip}'

# Count events by type
cat events.jsonl | jq -r '.event' | sort | uniq -c | sort -rn

# Filter to a time window
cat events.jsonl | jq 'select(.timestamp >= "2026-06-08T14:00:00Z" and .timestamp < "2026-06-08T15:00:00Z")'

2.4 Windows Event Log (EVTX)

Windows logs are stored in binary EVTX format. They can be exported to XML or parsed with wevtutil or PowerShell:

# Export Security log to XML
wevtutil qe Security /f:XML /c:100 > security.xml

# Query specific event IDs in PowerShell
Get-WinEvent -LogName Security -FilterHashtable @{Id=4625} -MaxEvents 50 |
  Select-Object TimeCreated,
    @{n="Username"; e={$_.Properties[5].Value}},
    @{n="SrcIP"; e={$_.Properties[19].Value}}

3. Log Analysis Techniques

3.1 Timeline Construction

A timeline maps events across multiple log sources to a single chronological sequence. This is the most powerful investigation technique because: - Attackers operate in sequence — each action follows from the previous - Correlating events across sources reveals cause-and-effect chains - Timestamps expose gaps where logs may have been deleted

Timeline construction steps:

1. Collect all relevant log files for the investigation window
2. Convert all timestamps to UTC (source logs may be in local time)
3. Merge all events sorted by timestamp
4. Annotate events with their probable significance
5. Identify gaps (deleted logs?) and inconsistencies (time skew?)

# Merge and sort logs from multiple sources
cat auth.log access.log syslog 
  | sort -k1,3     # sort by date fields (depends on format)

# For JSON logs: merge and sort by timestamp
cat *.jsonl | jq -s 'sort_by(.timestamp) | .[]' | jq -c '.'

# Create a simple unified timeline
paste -d' ' <(awk '{print $1,$2,$3}' auth.log) <(awk '{print $NF}' auth.log) 
  | sort

3.2 Statistical Analysis

Statistical techniques reveal anomalies that manual review would miss:

# Frequency analysis — how many events per source IP?
awk '{print $1}' access.log | sort | uniq -c | sort -rn

# Time-of-day distribution — are there events at unusual hours?
awk '{print $4}' auth.log | grep -oP 'd{2}:d{2}' | cut -d: -f1 | sort | uniq -c

# New vs. recurring users — who appeared for the first time today?
awk '{print $9}' auth.log | sort -u

# Error rate over time — did errors spike?
awk '{print substr($4,2,11), $9}' access.log | grep "5[0-9][0-9]" | 
  awk '{print $1}' | uniq -c

3.3 Correlation Across Log Sources

The most valuable investigations combine evidence from multiple log sources:

This chain tells a story: alice's account was compromised, the attacker tried to access admin, uploaded a file (possible web shell), and installed persistence via cron.

4. Log Manipulation Attacks

4.1 Log Injection

Log injection occurs when user-controlled input is written to a log file without sanitisation. An attacker can inject fake log entries by embedding newline characters in their input.

Example — HTTP access log injection via User-Agent:

# Vulnerable Python logging (do not do this)
app.logger.info(f"Request from {request.user_agent.string}")

If the attacker sends:

User-Agent: Mozilla/5.0
10.0.0.1 - - [08/Jun/2026:14:32:17 +0000] "GET /secret HTTP/1.1" 200 4096

The log will contain:

2026-06-08 14:32:17 INFO Request from Mozilla/5.0
10.0.0.1 - - [08/Jun/2026:14:32:17 +0000] "GET /secret HTTP/1.1" 200 4096

The second line is a fake entry. An analyst reading the log might conclude that 10.0.0.1 accessed /secret with a 200 response — but that event never happened.

Detection: - Look for embedded newlines in HTTP headers (, %0a, %0d%0a) - Compare raw access log entries against application logs — entries without corresponding network traffic are injected - Verify that log entries follow the exact format — injected entries may have wrong spacing or field counts

Remediation: Sanitise all user-controlled input before logging. Replace or escape newline characters.

4.2 Log Deletion

The most direct approach: an attacker with sufficient privilege simply deletes or truncates log files.

# An attacker deletes the auth log
rm /var/log/auth.log
# or truncates it
> /var/log/auth.log
# or removes only their entries
grep -v "185.220.101.5" /var/log/auth.log > /tmp/clean.log && mv /tmp/clean.log /var/log/auth.log

Detection: - Gaps in log sequences (log rotation numbers skip) - File size suddenly zero - File modification timestamp newer than most recent entry - Forward to a remote syslog server (attacker cannot delete remote logs) - Log integrity with hash chains (see COA path, card COA-K004)

4.3 Timestamp Skew

An attacker may attempt to falsify event timestamps to create false alibis or confusion. This can also occur legitimately through NTP misconfiguration.

# Detect timestamp inconsistencies
# If logs should be in chronological order, find backward jumps
awk '{
  current = $1" "$2" "$3;
  if (current < previous) print "BACKWARD JUMP: was", previous, "now", current;
  previous = current
}' auth.log

# Detect timezone inconsistencies (different timezone offsets in same log)
grep -oP '+d{4}|-d{4}' access.log | sort -u
# Should show only one timezone offset

5. A Complete Investigation Workflow

Scenario: You receive an alert: "Potential brute force attack on SSH followed by successful authentication."

Step 1: Scope the investigation

Define the time window, source of the alert, and what questions you are trying to answer: - What IP was the source? - Was the brute force successful? - What did the user do after authentication? - Is the system still compromised?

Step 2: Collect relevant logs

# Collect all SSH-related events in the incident window
grep "sshd" /var/log/auth.log | grep "2026-06-08 14:3[0-5]"

# Collect web access logs from the same window
awk '$4 > "[08/Jun/2026:14:30:00" && $4 < "[08/Jun/2026:14:35:00"' /var/log/nginx/access.log

Step 3: Build the timeline

14:30:02  Failed password for alice from 185.220.101.5 (attempt 1)
14:30:03  Failed password for alice from 185.220.101.5 (attempt 2)
...
14:30:47  Failed password for alice from 185.220.101.5 (attempt 45)
14:30:48  Accepted password for alice from 185.220.101.5   ← SUCCESS
14:31:02  185.220.101.5  GET /admin HTTP/1.1 403          ← tried admin, blocked
14:31:15  185.220.101.5  GET /admin/users HTTP/1.1 200    ← found a working path!
14:31:33  185.220.101.5  POST /admin/upload HTTP/1.1 200  ← uploaded something
14:32:05  185.220.101.5  GET /uploads/sh3ll.php HTTP/1.1 200  ← executed web shell

Step 4: Assess and classify

• CIA impact: Confidentiality (admin data accessed), Integrity (web shell uploaded), Availability (potential ongoing access)
• Threat actor: Likely cybercriminal (automated brute force tool)
• Attack stages: Credential brute force → admin panel access → web shell upload → post-exploitation (C2)
• ATT&CK: T1110.001 (Password Guessing) → T1078 (Valid Accounts) → T1505.003 (Web Shell)

Step 5: Produce an investigation summary

Investigation Summary — Incident #2026-0608-001
Date: 2026-06-08 14:30–14:35 UTC
Analyst: [Name]

Summary: Successful SSH brute force attack on account alice from IP 185.220.101.5,
followed by admin panel access and web shell deployment.

Timeline:
- 14:30:02 — Brute force began (45 attempts over 46 seconds)
- 14:30:48 — Successful authentication (alice, SSH)
- 14:31:15 — Unauthorised admin panel access
- 14:31:33 — Web shell uploaded (filename: sh3ll.php)
- 14:32:05 — Web shell executed

Impact: Confidentiality (admin data), Integrity (web shell), potential ongoing access
Evidence: auth.log lines 14:30–14:31; access.log lines 14:31–14:32
Recommended actions: Block 185.220.101.5; disable alice account; remove sh3ll.php; 
  investigate scope of admin data accessed; implement SSH key-only authentication

6. Common Mistakes

Mistake 1: Using local timestamps without UTC conversion. Two log files from servers in different timezones will have events that appear out of order. Always convert to UTC before correlating.

Mistake 2: Only reading the last N lines. Attackers may delay their actions or plant persistence that fires much later. Collect a wide enough window.

Mistake 3: Not checking for log manipulation. If your investigation conclusion is "nothing happened" but there are log gaps, file modification anomalies, or unusually clean log files, consider that logs may have been tampered with.

Mistake 4: Ignoring the User-Agent field. The User-Agent often identifies the tool being used. sqlmap/1.7, Nikto, dirbuster are immediately recognisable attack signatures.

Mistake 5: Confusing correlation with causation. Two events happening close together in time does not mean one caused the other. Build a causal chain using logical inference, not just temporal proximity.

7. Practice Exercises

1. Given this auth.log excerpt, answer the questions below: Jun 8 14:30:02 server sshd[1001]: Failed password for alice from 185.220.101.5 Jun 8 14:30:03 server sshd[1001]: Failed password for alice from 185.220.101.5 Jun 8 14:30:48 server sshd[1001]: Accepted password for alice from 185.220.101.5 Jun 8 14:30:49 server sudo: alice : TTY=pts/0 ; PWD=/tmp ; USER=root ; COMMAND=/bin/bash
2. How long did the brute force take?
3. What happened immediately after authentication?

4. Is this evidence of compromise? What is the severity?

5. Write an awk one-liner that extracts all unique source IPs from an Nginx access log and counts their requests, sorted by count descending.

6. An access log contains this entry: 192.168.1.50 - - [08/Jun/2026:14:32:17 +0000] "GET /page HTTP/1.1" 200 512 "-" "Mozilla/5.0 ADMIN - - [08/Jun/2026:14:32:17 +0000] "GET /secret HTTP/1.1" 200 4096"

7. What attack is this?
8. What fake event is the attacker trying to inject?

9. How would you detect that this is fake?

10. A JSON log file contains events with "timestamp" fields. Write a jq command that filters events from a one-hour window between 2026-06-08T14:00:00Z and 2026-06-08T15:00:00Z.

8. Lab

Assessment mode: flag

challenge_spec_id: 202 — Log injection

You are given an access.log file. An attacker has injected fake log entries via an unsanitised HTTP header field. The injected entries contain encoded data.

Task: 1. Parse the access log and identify entries that do not conform to the standard Combined Log Format 2. The injected entry contains the flag encoded in Base64 within the fake log line 3. Extract the Base64 string and decode it to reveal the flag in PREFIX{...} format

Hint: Look for entries where the User-Agent or X-Forwarded-For field contains embedded newlines. The injected line will have a different structure than legitimate access log entries.

9. Framework Alignment

10. Further Reading

• The Practice of System and Network Administration (3rd ed.) — Limoncelli, Stross, Hogan — Chapter on logging architecture and operational practice
• SANS Reading Room: Log Management — Multiple practical papers on log format parsing and SIEM integration
• Logging Cheat Sheet — OWASP — https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html — What to log and how to log it securely
• Graylog / ELK Stack documentation — For understanding how real SIEMs ingest and normalise logs
• jq Manual — https://stedolan.github.io/jq/manual/ — Essential reference for JSON log parsing
• MITRE ATT&CK — Defense Evasion: Indicator Removal (T1070) — Comprehensive coverage of attacker log manipulation techniques

["Parse Combined Log Format HTTP access log entries to extract all six fields and identify the HTTP method, status code, and client IP for each request", "Use grep, awk, and sort/uniq to analyse an auth.log file and produce a count of failed login attempts per source IP, sorted by frequency", "Construct a chronological timeline by merging events from auth.log and access.log, converting timestamps to UTC, and annotating each event with its probable significance", "Identify a log injection attack in an HTTP access log by detecting an embedded newline in a header field and extracting the fake entry the attacker inserted"]

Prerequisites → Why this matters → Why logs exist and what they contain → Log formats (syslog/auth.log, Combined Log Format/HTTP, JSON lines, Windows EVTX) → Log analysis techniques (timeline construction, statistical analysis, cross-source correlation) → Log manipulation attacks (injection with newline technique, deletion, timestamp skew — all with detection methods) → Complete investigation workflow (5-step process with worked brute-force example) → Common mistakes → Practice exercises → Lab (flag, spec 202) → Framework alignment → Further reading