Encoding, Decoding & Classical Ciphers

Prerequisites

• FND-K001: CIA Triad, Security Goals & the Modern Threat Landscape

Why This Lesson Matters

Encoding and decoding is the first skill every CTF player needs, and the first skill that trips up people who have not studied it. The reason is a dangerous confusion: many people call Base64 "encryption." It is not. Encoding transforms data to a different representation — it is always reversible with no secret key, by anyone who knows the scheme.

Understanding this distinction is professionally important. Calling a Base64-encoded string "encrypted" in a security report is an error that will undermine your credibility. Recognising an encoded value in a log, a metadata field, or a network packet is a daily skill for SOC analysts and forensics investigators.

Beyond the technical skills, this lesson introduces classical ciphers. While Caesar and Vigenère are not used in modern cryptography, they are everywhere in CTF competitions, and understanding them builds intuition about the concept of a key and the difference between encoding and encryption.

1. Encoding vs. Encryption vs. Hashing

These three terms describe fundamentally different operations. Confusing them is one of the most common mistakes in security writing.

The critical insight: If there is no key involved, the operation is encoding, not encryption. Base64 is encoding. ROT13 is encoding. Even the Caesar cipher is more accurately called encoding than encryption (because the "key" is a single shift value that has only 25 possible values — trivially brute-forced).

2. Base64

2.1 How Base64 Works

Base64 represents arbitrary binary data using 64 printable ASCII characters: A–Z, a–z, 0–9, +, /. Every 3 bytes of input become 4 Base64 characters. If the input is not a multiple of 3 bytes, padding (=) is added.

Input bytes:  01001000 01100101 01101100
As 6-bit groups: 010010 000110 010101 101100
Base64 chars:       S      G      V      s
Result: "SGVs"

Input: "He"  (2 bytes)
Padded to 3: "Hex00"
Base64 groups: "SGU=" (padded to 4 chars)

2.2 Recognising Base64

Visual cues: - Characters: A-Z, a-z, 0-9, +, / - Ends with = or == (padding — rare but distinctive) - Length is always a multiple of 4 (padding ensures this) - Output length ≈ 133% of input length

Common variants: - Base64url: replaces + with - and / with _ (safe for URLs and filenames). Used in JWTs. - Base32: uses A-Z and 2-7; output is ~60% longer than Base64. Used in TOTP codes and DNS tunnelling. - Base58: removes visually ambiguous characters (0, O, I, l). Used in Bitcoin addresses.

2.3 Decoding Base64 in Practice

# Command line
echo "SGVsbG8gV29ybGQ=" | base64 -d
# Output: Hello World

# With newlines stripped (base64 sometimes wraps at 76 chars)
echo "SGVsbG8g
V29ybGQ=" | tr -d '
' | base64 -d

# base64url variant (replace - with + and _ with /)
echo "SGVs_G8g-29ybGQ=" | tr '_-' '/+' | base64 -d

# Python
python3 -c "import base64; print(base64.b64decode(b'SGVsbG8gV29ybGQ=').decode())"

# Decode a file
base64 -d encoded_file.b64 > original_file.bin

3. Hexadecimal

Hex represents each byte as two hexadecimal digits (0–9, a–f). Every byte becomes exactly two characters. Output is exactly 2× the input length.

Byte: 0x41 = decimal 65 = ASCII 'A'
Byte: 0x46 = decimal 70 = ASCII 'F'
String "AF" → hex: 4146

3.1 Recognising Hex

• Characters: 0-9, a-f (or A-F)
• Length is always even
• May have 0x prefix, or pairs separated by spaces or colons (41:46:4c:41:47)
• Exactly 2× as long as the original data

3.2 Decoding Hex

# Decode hex string to text
echo "48656c6c6f20576f726c64" | xxd -r -p
# Output: Hello World

# With spaces between bytes
echo "48 65 6c 6c 6f" | tr -d ' ' | xxd -r -p

# Python
python3 -c "print(bytes.fromhex('48656c6c6f20576f726c64').decode())"

# xxd to inspect a file
xxd file.bin | head
# 00000000: 4865 6c6c 6f20 576f 726c 6400 0000 0000  Hello World.....

4. URL Encoding (Percent Encoding)

URL encoding replaces unsafe or reserved characters with %XX where XX is the hex value of the character.

Common encoded characters:

# Decode URL-encoded string
python3 -c "from urllib.parse import unquote; print(unquote('%48%65%6c%6c%6f%20%57%6f%72%6c%64'))"
# Output: Hello World

# Double-encoded (common in WAF bypass)
# %2527 → decoded once → %27 → decoded again → '
python3 -c "from urllib.parse import unquote; print(unquote(unquote('%252527')))"

5. Classical Ciphers

Classical ciphers predate computers by centuries. They are not secure by modern standards but they teach the fundamental concept of a key and key-based reversibility.

5.1 Caesar Cipher (ROT-N)

Shifts each letter N positions in the alphabet. ROT13 (shift 13) is self-inverse — applying ROT13 twice returns the original.

Plaintext:   HELLO WORLD
Key:         3
Ciphertext:  KHOOR ZRUOG

ROT13:
Plaintext:  The flag is here
Ciphertext: Gur synt vf urer

# ROT13 on Linux
echo "Gur synt vf urer" | tr 'A-Za-z' 'N-ZA-Mn-za-m'
# or
python3 -c "import codecs; print(codecs.decode('Gur synt vf urer', 'rot_13'))"

# Brute-force all 25 Caesar shifts
python3 << 'PYEOF'
text = "KHOOR ZRUOG"
for shift in range(26):
    decoded = '.join(
        chr((ord(c) - ord('A') - shift) % 26 + ord('A')) if c.isupper()
        else chr((ord(c) - ord('a') - shift) % 26 + ord('a')) if c.islower()
        else c
        for c in text
    )
    print(f"ROT{shift:2d}: {decoded}")
PYEOF

5.2 Vigenère Cipher

Uses a keyword to apply different Caesar shifts to each letter of the plaintext. The keyword repeats to match the plaintext length.

Plaintext:  HELLO WORLD
Key:        KEYKE YKEYK
Shifts:     K=10, E=4, Y=24, K=10, E=4, ...
Ciphertext: RIJVS GQFVN

Breaking Vigenère without the key requires frequency analysis (Index of Coincidence to find key length, then Caesar-break each column). In CTF, the key is usually hinted or short enough to brute-force.

def vigenere_decode(ciphertext, key):
    key = key.upper()
    result = []
    key_idx = 0
    for c in ciphertext:
        if c.isalpha():
            shift = ord(key[key_idx % len(key)]) - ord('A')
            decoded = chr((ord(c.upper()) - ord('A') - shift) % 26 + ord('A'))
            result.append(decoded if c.isupper() else decoded.lower())
            key_idx += 1
        else:
            result.append(c)
    return '.join(result)

print(vigenere_decode("RIJVS GQFVN", "KEY"))
# HELLO WORLD

5.3 Morse Code

International Morse Code represents letters and digits as sequences of dots (·) and dashes (—), separated by spaces within a letter and longer gaps between letters/words.

A  ·—    B  —···   C  —·—·  D  —··   E  ·
F  ··—·  G  ——·    H  ····  I  ··    J  ·———
K  —·—   L  ·—··   M  ——    N  —·    O  ———
P  ·——·  Q  ——·—   R  ·—·   S  ···   T  —
U  ··—   V  ···—   W  ·——    X  —··—  Y  —·——
Z  ——··
0  ————— 1  ·———— 2  ··——— 3  ···—— 4  ····—
5  ····· 6  —···· 7  ——··· 8  ———·· 9  ————·

MORSE = {
    'A': '.-', 'B': '-...', 'C': '-.-.', 'D': '-..', 'E': '.',
    'F': '..-.', 'G': '--.', 'H': '....', 'I': '..', 'J': '.---',
    'K': '-.-', 'L': '.-..', 'M': '--', 'N': '-.', 'O': '---',
    'P': '.--.', 'Q': '--.-', 'R': '.-.', 'S': '...', 'T': '-',
    'U': '..-', 'V': '...-', 'W': '.--', 'X': '-..-', 'Y': '-.--',
    'Z': '--..', '0': '-----', '1': '.----', '2': '..---',
    '3': '...--', '4': '....-', '5': '.....', '6': '-....',
    '7': '--...', '8': '---..', '9': '----.'
}
REVERSE_MORSE = {v: k for k, v in MORSE.items()}

def decode_morse(text):
    words = text.strip().split('  ')  # double space = word boundary
    return ' '.join(
        '.join(REVERSE_MORSE.get(symbol, '?') for symbol in word.split())
        for word in words
    )

print(decode_morse(".... . .-.. .-.. ---  .-- --- .-. .-.. -.."))
# HELLO WORLD

5.4 NATO Phonetic Alphabet

Alpha, Bravo, Charlie... Each word represents one letter. Trivial to decode once you know the table.

Alpha=A  Bravo=B  Charlie=C  Delta=D  Echo=E  Foxtrot=F  Golf=G
Hotel=H  India=I  Juliet=J  Kilo=K  Lima=L  Mike=M  November=N
Oscar=O  Papa=P  Quebec=Q  Romeo=R  Sierra=S  Tango=T
Uniform=U  Victor=V  Whiskey=W  X-ray=X  Yankee=Y  Zulu=Z

5.5 XOR Encoding

XOR (exclusive OR) is the most common encoding primitive in malware. It is symmetric: applying XOR with the same key twice returns the original data.

Plaintext byte: 0x48 ('H')
Key byte:       0x1F
XOR result:     0x57 ('W')
XOR again:      0x57 XOR 0x1F = 0x48 ('H')   ← original recovered

# Decode XOR-encoded data with a known single-byte key
data = bytes.fromhex("577e736c7260207c68726c64")   # XOR key = 0x1F
key = 0x1F
decoded = bytes(b ^ key for b in data)
print(decoded.decode())   # Hello World

# Brute-force single-byte XOR key
for key in range(256):
    attempt = bytes(b ^ key for b in data)
    try:
        text = attempt.decode('ascii')
        if all(32 <= ord(c) < 127 or c in '
    ' for c in text):
            print(f"key=0x{key:02X}: {text}")
    except UnicodeDecodeError:
        pass

6. Multi-Layer Encoding Chains

Real CTF challenges and real malware often stack multiple encoding layers. The strategy is always the same: peel one layer at a time, inspect the output, and repeat.

Standard recognition heuristics:

# Decode a multi-layer chain step by step
echo "R0ZIe3Rlc3RfZmxhZ30=" | base64 -d
# GFI{test_flag}  → still encoded? looks like ROT13 of PREFIX{...}
echo "R0ZIe3Rlc3RfZmxhZ30=" | base64 -d | tr 'A-Za-z' 'N-ZA-Mn-za-m'
# TOV{grfg_synt}  → hmm, different shift

# Brute force all Caesar shifts on the Base64-decoded output
python3 << 'PYEOF'
decoded = "GFI{test_flag}"
for shift in range(26):
    attempt = '.join(
        chr((ord(c) - 65 - shift) % 26 + 65) if c.isupper()
        else chr((ord(c) - 97 - shift) % 26 + 97) if c.islower()
        else c
        for c in decoded
    )
    if attempt.startswith("CTF") or attempt.startswith("FLAG") or attempt.startswith("PREFIX"):
        print(f"shift {shift}: {attempt}")
PYEOF

7. Common Mistakes

Mistake 1: Calling Base64 "encrypted." Base64 is encoding. It has no key. Anyone can decode it. Writing "the password was encrypted with Base64" in a security report is incorrect and damages your credibility.

Mistake 2: Not trying Base64 without the trailing =. The padding = is added to make the string a multiple of 4 characters. Many implementations work without it. If base64 -d fails, try adding = or ==.

Mistake 3: Forgetting that hex and Base64 can represent binary data that is not text. Decoded hex or Base64 may be a ZIP file, PNG, ELF binary, or any other binary format — not necessarily printable text. Pipe to file - or xxd | head to identify the decoded content.

Mistake 4: Giving up after one decoding pass. Always inspect the decoded output and ask: "is this the final answer, or is there another layer?" Stack two or three layers of encoding before you check.

Mistake 5: Confusing ROT13 with ROT-N. ROT13 is only one specific Caesar cipher (shift = 13). Other rotations are also used. If ROT13 does not produce readable output, try all 25 shifts.

8. Guided Example — Multi-Layer Decode

Challenge: You find the following string in a file comment: LkouZ2NoWVdacFVYUnZjbk09

Step 1: Recognise the outer layer

Characters A-Za-z0-9+/= → looks like Base64. Decode:

echo "LkouZ2NoWVdacFVYUnZjbk09" | base64 -d
# output: .*.gnhYWZpUXRvcnM=

Still looks like Base64 (ends with =, mixed alphanumeric).

Step 2: Decode the second layer

echo ".*.gnhYWZpUXRvcnM=" | tr -d '.*' | base64 -d
# Note: strip non-base64 characters first
echo "gnhYWZpUXRvcnM=" | base64 -d
# output: zxYZiQtors

Not obvious. Try ROT13:

echo "zxYZiQtors" | tr 'A-Za-z' 'N-ZA-Mn-za-m'
# output: mxLMvQgbef   ← still not clear

Try hex check — is it hex?

echo "7A7859595169746F7273" | xxd -r -p 2>/dev/null
# or check if the string has only hex chars

The string zxYZiQtors does not look like hex. Try Base64 again:

echo "zxYZiQtors" | base64 -d 2>/dev/null | xxd | head

Got binary output → it's an encoded binary artefact. Use file -:

echo "zxYZiQtors" | base64 -d 2>/dev/null | file -
# data  ← need more bytes to identify

At this point in a real CTF: try all heuristics systematically. This is the analysis skill that matters.

9. Practice Exercises

1. Decode each of the following without using an online tool (command line or Python only):
2. Q1RGe2Jhc2U2NF9pc19ub3RfZW5jcnlwdGlvbn0=
3. 4354467b6865785f646563...6f64657d (complete the hex string yourself)

4. CTF%7Burl_encoding%7D

5. A SOC alert contains the following value in the User-Agent header: Y3VybCBodHRwczovL2V2aWwuY29tL21hbHdhcmUuc2g=. What is it? Should this alert be escalated?

6. You see the following in a PCAP: ·—· ·— —·— ·· — ·—·· ···. Decode it.

7. Write a Python function decode_all(s) that tries Base64, hex, URL decoding, and ROT13 in sequence, returning the first result that looks like printable ASCII.

10. Lab

Assessment mode: flag

challenge_spec_id: 237 — Multi-layer encoding chain

You are given a text file containing a string that has been encoded through 3–5 chained encoding steps.

Task: 1. Identify and peel each encoding layer in sequence 2. Each layer uses one of: Base64, Hex, ROT13, URL encoding, or Morse code 3. The final layer contains the flag in PREFIX{...} format

Hint: Use the recognition heuristics table from this lesson. Look at the character set and length of each intermediate output.

11. Framework Alignment

12. Further Reading

• CyberChef — https://gchq.github.io/CyberChef/ — The browser-based universal encoder/decoder; invaluable for identifying and peeling encoding layers interactively
• dCode — https://www.dcode.fr/en — Comprehensive cipher and encoding reference with online tools
• Applied Cryptography — Bruce Schneier — Chapter 1 gives the definitive distinction between encryption, encoding, and hashing
• CTF Field Guide (Trail of Bits) — https://trailofbits.github.io/ctf/ — Excellent coverage of encoding challenges in CTF
• RFC 4648 — Base64, Base32, and Base16 specification (authoritative)

["Distinguish encoding from encryption from hashing and correctly apply the term to Base64, XOR, and AES respectively", "Recognise Base64, hex, URL-encoded, and Morse-encoded strings on sight using character set and length heuristics, and decode each using command-line tools", "Brute-force all 25 Caesar cipher shifts in Python and identify the readable output", "Decode a three-layer encoding chain by identifying and peeling each layer in sequence using Python or CLI tools"]

Prerequisites → Why this matters → Encoding vs encryption vs hashing (table with examples) → Base64 (how it works, recognition, variants, CLI and Python decode) → Hexadecimal (recognition, decode) → URL encoding (percent encoding table, security relevance, double-encoding) → Classical ciphers (Caesar/ROT, Vigenère, Morse, NATO alphabet, XOR encoding) → Multi-layer chains (strategy, heuristics table, worked example) → Common mistakes → Guided example → Practice exercises → Lab (flag, spec 237) → Framework alignment → Further reading