Custom VM Obfuscation Reversal: Dispatcher Loop Analysis and Python Disassembler Construction

Reverse Engineering Methodology

Custom virtual machine (VM) obfuscation is one of the most technically demanding obfuscation techniques encountered in both commercial software protection (e.g., VMProtect, Themida) and CTF reverse engineering challenges. The protected binary implements a complete bytecode interpreter in native code: it reads an array of custom opcodes, dispatches each to a handler, and executes the protected logic inside this synthetic CPU. The analyst never sees the original machine instructions — only the VM engine and the bytecode it processes.

Anatomy of a custom VM. Every bytecode interpreter shares the same structural components regardless of complexity:

• Fetch: read the next opcode byte (and optional operand bytes) from the bytecode array, advancing the virtual program counter (vPC).
• Decode: extract opcode and operand fields. May involve masking or XOR to obscure the raw values.
• Dispatch: branch to the handler for this opcode. Implementation styles: switch statement (compiled to a jump table), computed goto (GCC extension, very fast), or a function pointer table (array of handler addresses indexed by opcode).
• Execute: the handler performs the operation — arithmetic, memory access, branch, I/O — on the VM's virtual registers or stack.
• Loop: return to Fetch.

Finding the dispatcher. In disassembly, the dispatcher appears as a large switch or an indirect jump: jmp [rax*8 + table_base] (x86-64 jump table) or br x8 (AArch64). The jump table typically contains 16–256 entries, one per opcode. In IDA/Ghidra, navigate to cross-references of the bytecode buffer pointer to find where bytes are fetched; that is the top of the fetch-decode-dispatch loop.

Mapping opcodes. For each jump table entry, examine the handler and assign a mnemonic. Document: opcode byte value, mnemonic, operand count, operand sizes, and semantic description. Build a Python dictionary or CSV mapping {opcode_byte: (mnemonic, operand_fmt)}.

Technical Deep-Dive

# Minimal custom VM disassembler skeleton
# Assumes: bytecode in file; opcode table reverse-engineered from IDA/Ghidra

OPCODE_TABLE = {
    0x01: ("PUSH",  "B"),   # 1 byte immediate operand
    0x02: ("POP",   ""),    # no operand
    0x03: ("ADD",   ""),
    0x04: ("XOR",   "B"),
    0x05: ("JMP",   "H"),   # 2-byte (short) absolute address
    0x06: ("JZ",    "H"),
    0x07: ("LOAD",  "B"),   # load from virtual memory address
    0x08: ("STORE", "B"),
    0x09: ("CALL",  "H"),
    0xFF: ("HALT",  ""),
}

import struct

def disassemble(bytecode: bytes, base_addr: int = 0) -> list[str]:
    lines = []
    i = 0
    while i < len(bytecode):
        addr = base_addr + i
        op = bytecode[i]; i += 1
        if op not in OPCODE_TABLE:
            lines.append(f"{addr:04x}:  db  0x{op:02x}   ; unknown opcode")
            continue
        mnem, fmt = OPCODE_TABLE[op]
        operands = []
        for ch in fmt:
            if ch == 'B':
                operands.append(f"0x{bytecode[i]:02x}"); i += 1
            elif ch == 'H':
                val, = struct.unpack_from("<H", bytecode, i); i += 2
                operands.append(f"0x{val:04x}")
        operand_str = ", ".join(operands)
        lines.append(f"{addr:04x}:  {mnem:<8s} {operand_str}")
        if mnem == 'HALT':
            break
    return lines

# Usage:
# bytecode = open("vm_bytecode.bin", "rb").read()
# for line in disassemble(bytecode):
#     print(line)

# Locate the VM dispatch loop in Ghidra using the Script Manager
# Script: FindJumpTables.java (built-in) highlights indirect jumps

# In IDA Pro: find the fetch loop by searching for array accesses
# Edit > Find > Text: "movzx" near a loop — common fetch pattern

# Extract embedded bytecode blob from binary (once offset is known from IDA):
dd if=target_binary bs=1 skip=$((0x4020)) count=512 of=vm_bytecode.bin

# Run the disassembler:
python3 vm_disasm.py vm_bytecode.bin

# Dynamic: use gdb to trace handler dispatch
gdb -batch -ex "break *0x$(nm target | grep dispatch | awk '{print $1}')" 
    -ex "commands 1" -ex "silent" -ex "x/1bx $rip" -ex "continue" -ex "end" 
    -ex "run" ./target 2>&1 | grep "^0x" | head -100

Common Reversing Errors

1. Reversing the VM engine instead of the bytecode. Analysts spend hours understanding how ADD is implemented in native code rather than simply noting "this opcode adds two values" and moving on. Treat the VM as a black box once each opcode's semantic is known; focus on the bytecode program it runs.

2. Mis-identifying opcode boundaries. If operand sizes are wrong in your table, the disassembler falls out of sync and every subsequent instruction is garbage. Validate by finding known patterns: sequences that push a constant, compare it, and branch — these appear in flag-checking logic and have a predictable structure.

3. Ignoring the virtual register file. Custom VMs often maintain a small set of virtual registers in a fixed memory region. Not tracking which register holds the flag comparison result means missing the critical branch that gates success.

4. Not dumping bytecode dynamically. The bytecode array is sometimes decrypted at runtime. Static extraction from the binary yields the encrypted form. Always attempt a memory dump at the point the interpreter begins fetching: use gdb dump memory or a Frida Memory.readByteArray hook at the fetch address.

5. Assuming a single dispatch loop. Complex VMs may have multiple interpreters (e.g., one for arithmetic, one for I/O) or a recursive dispatch for nested VM layers. Look for more than one indirect jump pattern.