NERV Center — CSAW CTF 2023 Finals Writeup

Category: Crypto + Pwn Author: Brendan Dolan-Gavitt (moyix) Points: 500 (dynamic scoring, minimum 50)

Get into the server, Shinji. Or Rei will have to do it again. nc {box} {port}

---

Overview

nervcenter is a 64-bit Linux ELF binary that implements a multi-threaded server with a sensor monitoring subsystem and an authenticated admin interface. The challenge requires chaining three bugs to forge an RSA-1024 public key and authenticate as an administrator, then decrypt the encrypted flag.

The bugs involved:

1. An fd_set overflow that lets an attacker control the top 8 bytes of an RSA modulus
2. A TCP out-of-band (OOB/urgent) data trick to precisely set those 8 bytes
3. An EMFILE (file descriptor exhaustion) issue that prevents fopen() from succeeding unless the attack is tuned carefully

---

Server Architecture

The server forks a child process for each control connection. Each child spawns two pthreads:

• Control thread — handles the menu on the control socket (port 2000)
• Sensor thread — accepts and monitors connections on a per-session sensor port (port 2001 + session offset)

On startup the child process has these file descriptors open:

fd	Description
0	/dev/null
1	stdout pipe (docker log)
2	stderr pipe (docker log)
3,4	/challenge/nervcenter (linker mapping)
5	/run/rosetta/rosetta (Rosetta JIT, Apple Silicon only)
6	sensor listen socket
7	control connection socket

Sensor connections begin at fd 8.

Control Menu (unauthenticated)

1. Authenticate
2. Print public key
3. Issue sensor system halt
4. Resume sensor operations
5. MAGI status
6. Help
7. Exit

Session RSA Key

Each session generates an RSA-1024 keypair stored in a session struct. The modulus N is stored at session+0x270 (128 bytes). The public key is exposed via menu option 2 as an SSH-format base64 blob.

---

Bug 1: fd_set Overflow → RSA Modulus Corruption

The Vulnerability

The sensor thread runs a tight loop:

// Pseudocode of sensor thread loop (FUN_00105030)
while (true) {
    // Clear fd_sets for fds 0..1023 (fds_bits[0..15])
    // NOTE: fds_bits[16] (fds 1024..1087) is NOT cleared
    memset(readfds,   0, 128);  // 16 longs × 8 bytes
    memset(writefds,  0, 128);
    memset(exceptfds, 0, 128);

    // Set bits for all active sensor connections
    for each active_fd in sensor_fds:
        BIT_SET(readfds,   active_fd);
        BIT_SET(writefds,  active_fd);
        BIT_SET(exceptfds, active_fd);  // ← writes to session+0x270 if fd >= 1024

    select(max_fd + 1, &readfds, &writefds, &exceptfds, &timeout);
    // After select(), exceptfds contains only fds with pending OOB data
}

The fd_set struct is a bitfield of 1024 bits (128 bytes = 16 × uint64). The server sets RLIMIT_NOFILE = 1088, allowing fds 0–1087. When sensor connections reach fd 1024, the code that ORs their bits into exceptfds writes into fds_bits[16] — which sits at session+0x270, exactly overlapping the first 8 bytes of the RSA modulus N.

select() and OOB Data

select() with nfds > 1024 causes the kernel to read 136 bytes from exceptfds (17 × 64-bit longs), including fds_bits[16] = N[0:8]. The kernel then clears any bits in exceptfds that do not have pending OOB (urgent) data, and writes the result back.

This gives us precise bit-level control: send MSG_OOB on the connections whose server-side fds correspond to the bits you want SET in N[0:8]. Since OOB data is never consumed by the server (the sensor thread only calls read(), never recv(MSG_OOB)), the bits remain set indefinitely.

After select(): N[0:8] = union of bits for fds 1024–1086 that have pending OOB data.

---

Bug 2: RSA Private Key Derivation

We can control N[0:8] (8 bytes), but the lower 120 bytes of N come from the server's RNG and are unknown to us... except we can read them via the "Print public key" menu option before doing anything else.

Attack Strategy

1. Read N from the server: we control N[0:8] = X_bytes, and N[8:128] is given.
2. Find a prime p such that N' = X_bytes || N[8:128] is divisible by p.
   • Solve: X * 2^960 ≡ -N[8:128] (mod p)
   • X = (-N[8:128] * inverse(2^960, p)) % p
3. Check: is q = N' / p also prime? If so, we have a factorization.
4. Compute the private key: d = inverse(65537, (p-1)(q-1))

We search primes p near 2^62. For a random N[8:128], a solution is typically found in a few hundred attempts. We additionally require:

• X_bytes[0] != 0 (the server rejects RSA keys with a zero leading byte)
• bit 63 of int.from_bytes(X_bytes, 'little') == 0 (see Bug 3 below)

---

Bug 3: File Descriptor Exhaustion

The Problem

The admin menu (option 1 when authenticated) sends the client a notice and "Sending flag...", then calls fopen("flag.txt", "r"). If fopen fails, the function returns 0, which triggers session cleanup.

With the attack as described:

• 8 system fds (0–7) + 1080 sensor connections (fds 8–1087) = 1088 fds total
• RLIMIT_NOFILE = 1088 → max fd = 1087
• fopen("flag.txt") needs fd 1088 → EMFILE → fopen returns NULL

The "NOTE:" and "Sending flag..." messages are sent before the fopen call, so the client sees them just before the connection is closed with no encrypted data.

The Fix

Use only 63 overflow connections (fds 1024–1086) instead of 64. This leaves fd 1087 free for fopen().

• N_FILLER = 1016 (fds 8–1023, to reach the overflow threshold)
• N_OVERFLOW = 63 (fds 1024–1086)
• N_TOTAL = 1079

The missing connection (fd 1087) corresponds to bit 63 of fds_bits[16] in little-endian interpretation, i.e., bit 7 of X_bytes[7]. We add the constraint X_bytes[7] & 0x80 == 0 to the factorization search. This roughly halves the candidate space; solutions are still found quickly.

---

Exploit Flow

1. Connect to control port (2000), read sensor port and session banner
2. Send "2" → read N[0:128] (server's public key)
   - We control N[0:8] later; read N[8:128] now (the uncontrollable part)
3. Search for prime p such that:
   - X = (-N[8:128] * inverse(2^960, p)) % p
   - X_bytes[0] != 0  (non-zero leading byte)
   - X_bytes[7] & 0x80 == 0  (bit 63 of fds_bits target must be 0)
   - q = (X * 2^960 + N[8:128]) / p is prime
   - gcd(65537, (p-1)(q-1)) == 1
4. Open 1079 sensor connections serially (server accepts 1 per select() cycle)
5. Send MSG_OOB on the 63 overflow connections (fds 1024-1086) where the
   corresponding bit of int.from_bytes(X_bytes, 'little') is set
6. Send "3" on control socket → halt sensor thread
   (The halt waits for the sensor thread to enter pthread_cond_wait;
    under Rosetta emulation this takes 5–10 minutes per select() call)
7. Verify N[0:8] == X_bytes by reading the public key again
8. Send "1" on control socket → authenticate
   - Server issues a 32-byte random challenge
   - Sign challenge with PKCS#1 v1.5 using our private key (d, N')
   - Server pauses sensor thread, reads N from session+0x270, verifies
     RSA_verify(challenge, signature, N') → success
9. Send "1" on admin menu → request encrypted flag
   - Server calls fopen("flag.txt") → succeeds (fd 1087 is free)
   - Server encrypts flag.txt with AES-256-GCM, wraps key with RSA-OAEP using N'
   - Receives PEM block: [ciphertext_len][ciphertext][tag][iv][RSA(aes_key)]
10. Decrypt:
    - RSA-OAEP decrypt wrapped_key using our private key (d, p, q, N')
    - AES-256-GCM decrypt ciphertext using recovered aes_key and iv
    - Print flag

---

Key Implementation Details

PKCS#1 v1.5 Signature

The server uses RSA_verify() from OpenSSL, which expects a PKCS#1 v1.5 DigestInfo-wrapped SHA-256 hash. We construct the signature manually:

0x00 || 0x01 || 0xFF...0xFF || 0x00 || DigestInfo_SHA256_header || challenge_hash

Then compute sig = pow(em_int, d, N').

Encrypted Message Format

[8 bytes LE]  ciphertext_length
[cl bytes]    AES-256-GCM ciphertext of flag
[16 bytes]    GCM authentication tag
[12 bytes]    GCM nonce / IV
[128 bytes]   RSA-OAEP encrypted AES key (using our N')

Sensor Thread Select() Details

The sensor thread's fd_set layout in memory (relative to session struct):

• exceptfds starts at session+0x1f0
• fds_bits[0..15] = bytes 0–127 = fds 0–1023 ← cleared each iteration
• fds_bits[16] = bytes 128–135 = fds 1024–1087 = N[0:8] ← NOT cleared

---

Solve Script

# Key parameters:
FIRST_SENSOR_FD = 8     # verified from /proc/<pid>/fd inspection
N_FILLER   = 1016       # fds 8..1023 (below overflow threshold)
N_OVERFLOW = 63         # fds 1024..1086 (leave fd 1087 free for fopen)
N_TOTAL    = 1079

# Factorization constraint:
# X_bytes[7] & 0x80 == 0  (bit 63 of fds_bits target must be 0)

See solve.py for the full implementation.

---

Timing Notes (Rosetta / QEMU)

Running the x86-64 binary under Apple Rosetta 2 (or QEMU) introduces significant overhead:

Step	Native	Emulated
Per select() iteration	~1ms	~185ms
Open 1079 sensor connections	~1s	~3 min
Halt sensor thread	~0.1s	~5–10 min

On native x86-64 hardware, the full exploit completes in under a minute.

---

Flag

flag{...}

(The actual challenge flag is set by the CTF infrastructure at deploy time.)