Reconstructing Event Timelines from Log Rotation Artifacts and Surviving Log Fragments

Why This Matters

Log rotation is a routine system administration operation: old log files are compressed or deleted on a schedule to prevent disk exhaustion. A sophisticated attacker who has studied the target environment can time a malicious action to fall precisely within the rotation window — after the previous log file is archived and before the new file begins capturing events — or can trigger an out-of-cycle rotation to force old logs into deletion. The resulting chronological gap is almost indistinguishable from normal rotation, and an analyst who does not think to correlate rotation metadata with the incident timeline may conclude that no evidence exists for the most critical hours of an intrusion.

Core Concept

Log rotation on Linux is managed by logrotate, which reads /etc/logrotate.conf and drop-in configs under /etc/logrotate.d/. The logrotate state file at /var/lib/logrotate/status (or /var/lib/logrotate.status depending on distribution) records the last rotation time for every managed log file.

After rotation, the previous log is renamed (e.g., auth.log.1) or compressed (auth.log.1.gz). With the rotate N directive, logrotate keeps N generations; older files are deleted. An attacker who triggers rotation manually (or waits for a nightly cron job) can arrange for the log file covering their activity to be exactly the one that gets deleted.

Timeline reconstruction after a rotation gap requires: 1. Reading logrotate.status to determine when each rotation occurred 2. Comparing rotation timestamps against the incident window 3. Examining compressed archives for earlier generations 4. Cross-referencing filesystem inode metadata (ctime, mtime) on surviving log files 5. Checking other evidence sources (network logs, SIEM-forwarded events, application logs in different rotation schedules) that may cover the gap

Technical Deep-Dive

# Read logrotate status file to find last rotation times
cat /var/lib/logrotate/status
# Example output:
# logrotate state -- version 2
# "/var/log/auth.log" 2024-03-15-6:25:1
# "/var/log/syslog" 2024-03-15-6:25:1

# Cross-reference: when did the incident occur vs last rotation?
INCIDENT_TIME="2024-03-15 03:47:00"
ROTATION_TIME=$(grep "auth.log" /var/lib/logrotate/status | awk '''{print $2}''')
echo "Incident: $INCIDENT_TIME"
echo "Rotation: $ROTATION_TIME"

# List surviving log generations with timestamps
ls -la /var/log/auth.log*
stat /var/log/auth.log.1 /var/log/auth.log.2.gz 2>/dev/null

# Check compressed archives (may cover the gap)
zcat /var/log/auth.log.2.gz | grep -E "Mar 15 0[2-4]:" | head -20

# Find chronological gap between current log and previous generation
CURRENT_FIRST=$(head -1 /var/log/auth.log | awk '''{print $1,$2,$3}''')
PREV_LAST=$(zcat /var/log/auth.log.1.gz 2>/dev/null || cat /var/log/auth.log.1 
            | tail -1 | awk '''{print $1,$2,$3}''')
echo "Previous generation ends : $PREV_LAST"
echo "Current generation starts: $CURRENT_FIRST"

# Python: parse logrotate status and compare to incident window
import re
from datetime import datetime

INCIDENT_START = datetime(2024, 3, 15, 3, 30)
INCIDENT_END   = datetime(2024, 3, 15, 4, 15)

with open("/var/lib/logrotate/status") as fh:
    for line in fh:
        m = re.match(r'"(.+?)" (d{4}-d{2}-d{2}-d+:d+:d+)', line)
        if not m:
            continue
        logfile, ts_str = m.groups()
        # Parse format: 2024-03-15-6:25:1
        ts = datetime.strptime(ts_str, "%Y-%m-%d-%H:%M:%S")
        if INCIDENT_START <= ts <= INCIDENT_END:
            print(f"ROTATION DURING INCIDENT: {logfile} rotated at {ts}")
        elif abs((ts - INCIDENT_START).total_seconds()) < 3600:
            print(f"ROTATION NEAR INCIDENT: {logfile} rotated at {ts}")

Analytical Methodology

1. Read /var/lib/logrotate/status. For each log file relevant to the investigation, note the last rotation timestamp. Compare against the incident window — any rotation occurring within 2 hours of suspected attacker activity warrants investigation.
2. List all surviving log generations: ls -la /var/log/<service>.log*. Note which generation numbers exist and which are absent (e.g., .1.gz exists but .2.gz is missing — one generation was deleted).
3. Determine the chronological coverage of surviving files. Parse the first and last timestamp of each generation. Identify any gap between the end of one generation and the start of the next.
4. Check compressed archives with zcat for events in the gap window. If the .2.gz file is missing but the .3.gz exists, the gap corresponds exactly to the deleted second generation.
5. Cross-reference SIEM: query for events from the affected host during the gap window. The SIEM receives a real-time forward and is not subject to local log rotation. Any events the SIEM captured during the gap prove what existed before rotation.
6. Check alternative logs on the same host that are managed on a different rotation schedule: application logs, database audit logs, or custom service logs may still cover the gap period.
7. Examine filesystem artefacts: inode numbers, ctime values, and directory entry timestamps for the log directory can confirm when files were created, renamed, or deleted.
8. Document the gap precisely: start timestamp (end of surviving pre-gap generation), end timestamp (start of surviving post-gap generation), duration, and which log rotation event corresponds to the gap.

Common Analytical Errors

• Confusing rotation with deletion: Not all rotations delete logs. If rotate 7 is configured, up to 7 generations are kept. Verify how many generations are configured and how many are present — the difference indicates how many were deleted.
• Ignoring the dateext option: With dateext, rotated files are named with dates (e.g., auth.log-20240315.gz) rather than sequential numbers. Sort by filename date to establish chronological order.
• Missing manual rotation triggers: Logrotate can be called manually (logrotate -f /etc/logrotate.conf) or via SIGHUP to a logging daemon. Check bash history and cron for manual rotation commands, especially if rotation occurred outside the scheduled window.
• Overlooking postrotate scripts: Logrotate postrotate hooks run arbitrary commands after rotation. Check for customised hooks that might delete, upload, or encrypt rotated logs before the analyst can examine them.

NICE Framework Alignment

Further Reading

• logrotate(8) man page — complete configuration option reference
• SANS DFIR: "Linux Live Response and Log Analysis" (FOR508 excerpt)
• Filesystem forensics: inode change times and directory entry modification (Carrier, Chapter 10)
• Eric Zimmerman: "KapeTriage" — triage tool that collects log rotation status files
• NIST SP 800-86: Guide to Integrating Forensic Techniques into Incident Response — log collection section