File Artefacts: Metadata, File Carving & Archive Analysis

Prerequisites

• FND-K001: CIA Triad, Security Goals & the Modern Threat Landscape
• FND-K004: Operating Systems Fundamentals (recommended)

Why This Lesson Matters

Digital files are not just the data you see on screen. Every file carries a shell of metadata — creation timestamps, GPS coordinates, author names, software versions, embedded thumbnails. Archives contain not just files but internal directory tables, comments, and access controls. Understanding what a file really contains is a foundational forensics skill.

Attackers hide flags in metadata fields because most users and even many security tools never look there. Defenders find critical evidence in the same fields. In CTF competitions, these artefacts are everywhere. In real investigations, a document metadata field has been the difference between identifying an anonymous attacker and losing the case.

1. File Magic Bytes and Format Identification

Every file format has a characteristic signature at the start of the file (and sometimes at the end). These "magic bytes" allow tools to identify a file by content rather than by extension.

Why this matters: An attacker who renames shell.php to shell.jpg has changed the extension but not the magic bytes. A server that validates file type by reading magic bytes (not by trusting the extension or Content-Type header) will catch this.

Common magic bytes:

# Identify a file by magic bytes
file mystery_file.bin
# or inspect directly:
xxd mystery_file.bin | head -4

2. EXIF Metadata

EXIF (Exchangeable Image File Format) is a standard that specifies metadata stored inside JPEG, TIFF, and some PNG images. It was designed for cameras to record technical shooting parameters, but the standard includes dozens of fields that can contain arbitrary text.

2.1 EXIF Fields and Security Implications

Automatically captured fields (by cameras and smartphones):

Freely editable fields (common hiding places in CTF and steganography):

2.2 Extracting EXIF with exiftool

exiftool is the universal metadata extraction tool. It supports 200+ file formats.

# Show all metadata
exiftool photo.jpg

# Show only GPS data
exiftool -gps:all photo.jpg

# Show field values in short form (tag name only, no description)
exiftool -s photo.jpg

# Show a specific field
exiftool -Comment photo.jpg
exiftool -Artist photo.jpg

# Extract metadata from all files in a directory
exiftool /path/to/images/

# Write metadata (to understand what attackers can do)
exiftool -Comment="FLAG{hidden_data}" photo.jpg

# Remove all metadata (privacy protection)
exiftool -all= photo.jpg

2.3 PNG Chunk Metadata

PNG files store metadata in named "chunks" inside the file structure. Unlike EXIF (which is a specific structured format), PNG chunks can hold arbitrary text:

• tEXt chunk: keyword and ISO-8859 encoded text value
• iTXt chunk: keyword and UTF-8 encoded text (supports compression and language tagging)
• zTXt chunk: compressed text

# exiftool reads PNG chunks automatically
exiftool image.png

# Python PIL inspection
python3 << 'PYEOF'
from PIL import Image
img = Image.open("image.png")
print(img.info)   # dict of tEXt/iTXt fields
# Example output: {'Title': 'Test image', 'secret': 'FLAG{hidden}'}
PYEOF

# pngcheck - validates PNG structure and lists chunks
pngcheck -v image.png

3. File Carving

File carving recovers files from raw binary data by identifying file format signatures, without relying on filesystem metadata. This is used in: - Forensic recovery of deleted files - Extracting hidden files appended to other files (common in CTF) - Recovering files from damaged or unformatted media

3.1 Polyglot Files

A polyglot file is valid as two or more different formats simultaneously. The most common CTF pattern: a JPEG or PNG with a ZIP archive appended after the legitimate image data.

[PNG data........IEND chunk][ZIP archive PK header and entries]
                 ↑                     ↑
             PNG ends here        ZIP starts here
Image viewers stop at IEND.
ZIP tools read from the PK signature.
The file is valid both as a PNG and as a ZIP.

# Step 1: Check for appended data
xxd image.png | grep -i "PK"
# or
xxd image.png | tail -20    # look for PK after IEND

# Step 2: Use binwalk to automatically detect and extract
binwalk image.png              # detection only
binwalk -e image.png           # extract to _image.png.extracted/

# Step 3: Manual extraction with dd
# If binwalk says ZIP starts at offset 0x1F40 (8000 decimal):
dd if=image.png bs=1 skip=8000 of=hidden.zip
unzip hidden.zip

3.2 Detecting Appended Data with Binwalk

# Basic scan
binwalk archive.zip

# Entropy analysis (high entropy = compressed or encrypted data)
binwalk -E image.png

# Extract all found artefacts
binwalk --extract --carve image.png

# Scan a directory of files
binwalk /path/to/files/*

Reading binwalk output:

DECIMAL     HEXADECIMAL   DESCRIPTION
-------------------------------------------
0           0x0           PNG image, 512 x 384, 8-bit/color RGB, non-interlaced
4096        0x1000        Zlib compressed data, default compression
8000        0x1F40        Zip archive data, at least v2.0 to extract, name: flag.txt

The ZIP starts at decimal offset 8000. Everything before that is the PNG.

4. Archive Analysis

4.1 ZIP File Structure

A ZIP file has three main structures: - Local file headers + file data: one per archived file, at start of archive - Central directory: at end of archive, indexes all files - End of central directory record (EOCD): tells ZIP tools where the central directory is

[Local header + data for file1]
[Local header + data for file2]
...
[Central directory entry for file1]
[Central directory entry for file2]
...
[End of central directory record]

# List contents without extracting
unzip -l archive.zip
zipinfo archive.zip

# Test archive integrity
unzip -t archive.zip

# Extract verbosely
unzip -v archive.zip

# Extract to specific directory
unzip archive.zip -d /tmp/extracted/

# Extract with password
unzip -P "password" protected.zip

# Inspect raw structure
python3 -c "import zipfile, sys; z=zipfile.ZipFile(sys.argv[1]); [print(i.filename, i.comment, i.create_system) for i in z.infolist()]" archive.zip

4.2 ZIP Hidden Data Locations

ZIP archives can hide data in several non-obvious places:

# Show archive comment
python3 -c "
import zipfile
z = zipfile.ZipFile('archive.zip')
print('Archive comment:', z.comment)
for info in z.infolist():
    print(f'  {info.filename}: comment={info.comment}, extra={info.extra.hex()}')
"

4.3 Corrupted Archive Repair

ZIP files are remarkably resilient because they store file data at the beginning and the index at the end. If only the central directory is damaged, tools can still extract files by reading local headers directly.

# Try extracting despite errors
unzip -F archive.zip       # fix archive
unzip -FF archive.zip      # more aggressive fix

# Use zip -F to recover
zip -F corrupted.zip --out fixed.zip

# Python - read despite errors
python3 -c "
import zipfile
try:
    z = zipfile.ZipFile('corrupted.zip', 'r')
    z.extractall('/tmp/recovered/')
except zipfile.BadZipFile as e:
    print(f'Error: {e}')
    # Try reading local headers directly
"

# Hex editor approach: look for PKx03x04 (local file header signature)
# and PKx01x02 (central directory signature)
grep -boa $'PKx03x04' corrupted.zip    # offset of local headers

5. Document Metadata (Office Files)

DOCX, XLSX, and PPTX files are ZIP archives containing XML. This means all the techniques above apply, plus specific document metadata structures.

# DOCX = ZIP; unzip and inspect XML
unzip -o document.docx -d /tmp/docx_extracted/
cat /tmp/docx_extracted/docProps/core.xml    # Author, title, subject, keywords
cat /tmp/docx_extracted/docProps/app.xml     # Company, Manager

# exiftool also reads DOCX metadata
exiftool document.docx

# Search all XML for hidden text
grep -r "FLAG|CTF|secret" /tmp/docx_extracted/

6. Common Mistakes

Mistake 1: Only running exiftool on JPEG files. exiftool supports PDF, DOCX, XLSX, MP4, MOV, and 200+ other formats. Always run it on any file that could contain metadata.

Mistake 2: Ignoring the end of a file. Appended data after the legitimate file end is a standard hiding technique. Always check xxd file | tail -30 for unexpected content.

Mistake 3: Not checking archive comments. The ZIP comment field is not shown by unzip -l — you need unzip -z or Python. It is a very common flag hiding location in CTF.

Mistake 4: Trusting the file extension. A file named image.jpg might be a ZIP, PDF, or ELF binary. Always verify with file command and magic byte inspection.

Mistake 5: Skipping binwalk on non-image files. Polyglot files are not limited to images. PDF+ZIP, MP3+ZIP, and ELF+ZIP are all valid constructions. Run binwalk on any file that seems unusual.

7. Guided Example — Multi-Layer Artefact Investigation

Scenario: You receive a challenge_image.png. The challenge text says "the flag is hiding where you least expect it."

Step 1: Initial triage

file challenge_image.png
# challenge_image.png: PNG image data, 800 x 600, 8-bit/color RGBA, non-interlaced

sha256sum challenge_image.png
# (record the hash for chain of custody)

Step 2: Check metadata

exiftool challenge_image.png
# Look for non-standard fields, long Comment values, base64-looking strings
# → Comment: "Nothing here. Really."   ← suspicious reassurance

Step 3: Run binwalk

binwalk challenge_image.png
# DECIMAL    HEX     DESCRIPTION
# 0          0x0     PNG image data
# 21504      0x5400  Zip archive data, name: secret.txt

Found a ZIP at offset 0x5400!

Step 4: Extract the ZIP

binwalk -e challenge_image.png
ls _challenge_image.png.extracted/
# 5400.zip  secret.txt
cat _challenge_image.png.extracted/secret.txt
# FLAG{metadata_carving_is_fundamental}

Step 5: Check the ZIP comment anyway

python3 -c "import zipfile; z=zipfile.ZipFile('_challenge_image.png.extracted/5400.zip'); print(z.comment)"
# b'     ← empty, nothing there

8. Practice Exercises

1. Download any JPEG photo from a camera (not a screenshot). Run exiftool -a -u on it. List every field that could reveal private information about the photographer.

2. A file named document.pdf has the following magic bytes at offset 0: 50 4B 03 04. What is the file actually? How do you extract its contents?

3. You run binwalk archive.zip and get: 0 0x0 Zip archive data, name: readme.txt 1024 0x400 Zip archive data, name: data.bin 45056 0xB000 Zlib compressed data Is the Zlib data inside the ZIP or appended after it? How do you extract it?

4. Write a Python one-liner that prints the archive comment of a ZIP file passed as a command-line argument.

9. Lab

Assessment mode: flag

challenge_spec_id: 1 — EXIF metadata extraction

You are given a photo.jpg file. The flag is hidden inside one of its EXIF fields.

Task: 1. Run exiftool photo.jpg to list all metadata fields 2. Identify the field containing the flag (it may be base64 or hex-encoded) 3. Decode if necessary and submit the flag in PREFIX{...} format

Additional practice: After completing the primary lab, try challenge_spec_id: 2 (Hidden ZIP in image) for file carving practice.

10. Framework Alignment

11. Further Reading

• exiftool documentation — https://exiftool.org/ — Complete field reference; Phil Harvey's tool is the industry standard
• binwalk documentation — https://github.com/ReFirmLabs/binwalk — Full reference for firmware and binary analysis
• ZIP File Format Specification — https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT — The authoritative ZIP format reference
• PNG Specification — http://libpng.org/pub/png/spec/ — Complete PNG chunk documentation
• File Magic Bytes Reference — https://www.filesignatures.net/ — Searchable database of file signatures
• Forensic Focus — Metadata Analysis — Community articles on practical metadata forensics

["Identify a file's true format by reading its magic bytes using xxd and the file command, independent of the file extension", "Use exiftool to extract all metadata from a JPEG file, identify non-standard fields, and decode a base64 or hex-encoded value found in a metadata field", "Use binwalk to detect and extract a hidden ZIP archive appended to a PNG image file", "Inspect a ZIP archive for hidden content in archive comments, file comments, and extra fields using Python's zipfile module"]

Prerequisites → Why this matters → Magic bytes and format identification (table + xxd examples) → EXIF metadata (fields, privacy concerns, hiding places, exiftool commands) → PNG chunk metadata (tEXt, iTXt, Python PIL) → File carving (polyglot files, binwalk, manual dd extraction) → Archive analysis (ZIP structure, hidden locations, corrupt archive repair) → Document metadata (DOCX as ZIP) → Common mistakes → Guided example (multi-layer investigation) → Practice exercises → Lab (flag, spec 1) → Framework alignment → Further reading