HackTheBox - vvm: ELF 64-bit LSB PIE Executable (x86_64) & Custom Stack-Based Virtual Machine (VM) with Self-Decrypting Handlers

Executive Summary

The vvm challenge from HackTheBox features a proprietary client-side password validation engine protected by a custom, stack-based Virtual Machine (VM). The challenge presents a typical scenario where proprietary logic is guarded against static inspection through layers of virtualization and dynamic self-decryption:

1. Dynamic Opcode Unpacking: 24 of the 28 VM opcode handlers are stored as encrypted blobs in .rodata/.data and decrypted at runtime via a 6-byte repeating XOR keystream into executable (RWX) memory pages allocated via mmap.
2. Virtual Machine Interpretation: A centralized interpreter loop processes a linear 32-bit dword bytecode array (0x5540), maintaining an explicit evaluation stack (0x6220) and instruction pointer.
3. Multi-Stage Validation Pipeline:
   • Enforces an exact 32-character password length constraint via non-linear arithmetic.
   • Permutes the 32 input bytes across 8 separate 32-bit dwords using a custom sub-interpreter function (sub_282).
   • Applies variable bitwise circular left rotations (ROL32) via another sub-routine (sub_456).
   • Verifies the rotated dwords against 8 hardcoded 32-bit integer targets.

Because every cryptographic transformation in the verification routine is strictly bijective and reversible, the entire algorithm can be algebraically inverted via ROR32 and inverse permutation, yielding the flag without requiring brute force or dynamic symbolic execution.

1. Challenge Overview & Scenario

The challenge description states:

"A new startup claims to have developed an unbreakable client-side password validation system. VCs have invested millions, but I'm a bit skeptical of their claim. Can you prove them wrong?"

Client-side validation systems claiming to be "unbreakable" almost universally employ virtualization obfuscation (VM protectors) combined with code packing and anti-analysis countermeasures. Our objective is to statically and dynamically dissect the binary, reconstruct the VM's operational semantics, decode its bytecode program, and extract the valid secret key.

2. Initial Binary Triage & Reconnaissance

We begin by inspecting the file headers, architecture, and sections of the provided binary:

$ file vvm
vvm: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=..., for GNU/Linux 3.2.0, stripped

$ checksec --file=vvm
RELRO           STACK CANARY      NX            PIE             RPATH      RUNPATH      Symbols
Full RELRO      Canary found      NX enabled    PIE enabled     No RPATH   No RUNPATH   No Symbols

Key Triage Observations

• Stripped Binary: Symbol tables (.symtab) have been removed; functions must be mapped via control flow tracing from _start.
• Full Protections: Full RELRO, Stack Canary, NX (No-Execute), and PIE are enabled. Note that despite NX being enabled on the ELF binary itself, we will see that the binary dynamically allocates RWX pages via mmap to execute decrypted JIT-like bytecode handlers.
• Dynamic Imports:
  • mmap: Allocates memory pages with custom protection flags (indicates dynamically generated or unpacked code).
  • malloc, free: Dynamic heap memory management.
  • ptrace: Classic Linux anti-debugging primitive (PTRACE_TRACEME).
  • getline, __printf_chk, puts: Standard I/O operations for reading user input and printing output.

Locating main via _start

Inspecting the entry point at offset 0x11b0:

0x11b0 <_start>:
    endbr64
    xor    ebp, ebp
    mov    r9, rdx
    pop    rsi
    mov    rdx, rsp
    and    rsp, 0xfffffffffffffff0
    push   rax
    push   rsp
    xor    r8d, r8d
    xor    ecx, ecx
    lea    rdi, [rip - 0x4f]        ; 0x11cf - 0x4f = 0x1180
    call   qword ptr [rip + 0x3e03] ; __libc_start_main@plt

Following the standard x86_64 System V ABI, the first argument in RDI passed to __libc_start_main points directly to the main function at 0x1180.

3. Decompiling main (0x1180)

The disassembly of main reveals an exceptionally clean and modular architecture:

0x1180 <main>:
    endbr64
    sub    rsp, 8
    xor    eax, eax
    call   0x2910      ; Phase 1: Print Challenge Banner
    xor    eax, eax
    call   0x1530      ; Phase 2: Decrypt & Populate VM Dispatch Table
    xor    eax, eax
    call   0x2870      ; Phase 3: Execute Virtual Machine Main Loop
    xor    eax, eax
    add    rsp, 8
    ret

In high-level pseudocode:

int main(void) {
    print_banner();        // 0x2910: Displays "vvm v0.0.3" ASCII header
    vm_init_handlers();    // 0x1530: Allocates table & decrypts 28 opcode handlers
    vm_execute();          // 0x2870: Interprets bytecode from 0x5540
    return 0;
}

The binary's entire logic is encapsulated in these three distinct phases. Next, we analyze how the VM handlers are materialized in memory.

4. Unpacking the Opcode Decryption Engine (0x1530)

When examining function 0x1530, we see that it acts as a dynamic unpacker for the VM's instruction set:

0x1530 <vm_init_handlers>:
    endbr64
    push   r15
    push   r14
    push   r13
    push   r12
    push   rbp
    push   rbx
    sub    rsp, 0x18
    mov    edi, 0xe0      ; 0xe0 = 224 bytes = 28 * 8 bytes (28 function pointers)
    call   malloc@plt
    mov    qword ptr [rip + 0x5f7a], rax ; Stored in global table at 0x74e0

4.1. Memory Allocation & Dispatch Table

The function allocates 224 bytes (0xe0), which corresponds exactly to an array of 28 64-bit function pointers (28 * sizeof(void*) = 224). The returned pointer is stored in the global variable dispatch_table at offset 0x74e0.

4.2. Decryption Routine

Immediately following the table allocation, the function initializes all 28 entries:

• 24 Handlers are dynamically decrypted: For each dynamic opcode:

  1. Calls mmap(NULL, code_size, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) to allocate executable memory.
  2. Reads an encrypted byte buffer from .rodata/.data.
  3. Decrypts the payload using a repeating 6-byte XOR keystream: \text{Key} = [0\text{x}2a, 0\text{x}0b, 0\text{x}21, 0\text{x}21, 0\text{x}4d, 0\text{x}2a]
  4. Copies the decrypted bytes into the mmap allocation.
  5. Assigns the allocated function pointer to dispatch_table[opcode].

• 4 Handlers are static subroutines: Four opcodes bypass decryption and point directly to pre-compiled functions within .text:

  • dispatch_table[3] (0x18): Points to 0x1320 (GETLINE handler).
  • dispatch_table[7] (0x38): Points to 0x13e0 (ptrace anti-debugging check).
  • dispatch_table[10] (0x50): Points to 0x13a0 (PRINTF / FREE handler).
  • dispatch_table[15] (0x78): Points to 0x12a0 (String reconstruction handler).

4.3. Automated Opcode Recovery

By extracting the ciphertext arrays and applying the repeating XOR key, we can disassemble all 28 handlers cleanly:

KEY = bytes([0x2a, 0x0b, 0x21, 0x21, 0x4d, 0x2a])

def decrypt_handler(ciphertext):
    # In the binary, the first byte is often a length header or skipped
    return bytes(b ^ KEY[i % len(KEY)] for i, b in enumerate(ciphertext))

Every decrypted stub follows standard System V x86_64 calling conventions and operates directly on the VM's state structures.

5. Virtual Machine Architecture & Runtime Model

The core execution engine is located at 0x2870. Examining the interpreter loop provides the complete VM runtime specification:

0x2870 <vm_execute>:
    ...
    lea    rax, [rip + 0x2cb1]           ; rax = 0x5544 (pointer to second dword)
    mov    qword ptr [rsp], rax          ; [rsp] = IP (Instruction Pointer)
    mov    eax, dword ptr [rip + 0x2ca3]  ; eax = dword at 0x5540 (Initial Opcode)
    cmp    eax, 0x1c                     ; Check if opcode == 28 (0x1c = HALT)
    je     0x28ef                        ; Exit if HALT
loc_28a2:
    mov    r13, rsp                      ; r13 = &IP
    lea    r12, [rip + 0x3954]           ; r12 = &SP (Stack Pointer at 0x6200)
    lea    rbp, [rip + 0x396d]           ; rbp = stack buffer at 0x6220
    lea    rbx, [rip + 0x2c86]           ; rbx = bytecode base at 0x5540
    mov    r8, qword ptr [rip + 0x4c19]  ; r8 = dispatch_table (0x74e0)
    movsxd r9, eax                       ; r9 = opcode
    mov    rsi, r13                      ; arg2: &IP
    mov    rcx, r12                      ; arg4: &SP
    mov    rdx, rbp                      ; arg3: stack buffer
    mov    rdi, rbx                      ; arg1: bytecode base
    call   qword ptr [r8 + r9*8]         ; Indirect call to opcode handler!

5.1. Architectural Layout & State Machine

                   +-----------------------------------------------+
                   |             Bytecode Array (0x5540)           |
                   |   [Opcode 0][Operand 0][Opcode 1] ...         |
                   +-----------------------------------------------+
                                      ^
                                      | *IP (Instruction Pointer)
                                      |
+-------------------+      +-------------------+      +-------------------+
|  Dispatch Table   | ---> |  Opcode Handler   | <--- | Evaluation Stack  |
|  (28 Pointers)    |      |  (JIT / Native)   |      |  (0x6220, SP)     |
+-------------------+      +-------------------+      +-------------------+

• Bytecode Format: 32-bit signed integers (dwords). Opcodes and their immediate operands are stored sequentially as 4-byte values starting at address 0x5540.
• Registers & State Context:
  • RDI (bytecode_base): Base address of the bytecode segment (0x5540).
  • RSI (ip_ptr): Pointer to the current Instruction Pointer (int32_t **ip_ptr). Handlers consume operands by reading **ip_ptr and advancing *ip_ptr += 1.
  • RDX (stack): Pointer to an array of 64-bit values (int64_t stack[0..1023]) located at 0x6220.
  • RCX (sp_ptr): Pointer to a 32-bit signed integer tracking the current top of the stack (int32_t *sp_ptr) located at 0x6200.
  • R8: Base address of the 28-entry function pointer dispatch table (0x74e0).
  • R9: Current opcode index (0x00 through 0x1c).

6. Complete VM Instruction Set Architecture (ISA)

By reverse-engineering all 28 handlers (both decrypted stubs and static routines), we reconstruct the complete ISA specification:

7. Bytecode Decompilation & Execution Flow Analysis

The program bytecode consists of 808 32-bit dwords located at offset 0x5540. Disassembling the entire bytecode reveals a linear execution pipeline divided into logical stages:

[0x000] Setup & Anti-Debugging Check (Opcode 7)
   │
[0x002] Construct & Print Banner: "Enter password: " (BUILD_STR 22, PRINT_STR)
   │
[0x147] Read Input from stdin (GETLINE 32, STRIP_NL)
   │
[0x153] Password Length Check: (24*len - 8) // 10 == 76 (Requires len == 32)
   │
[0x185] Pack 32 Chars into 8 Dwords (Calls sub_282 via permutation indices)
   │
[0x475] Bitwise Circular Left Rotations (Calls sub_456 with rotation counts)
   │
[0x501] Linear Equality Verification (Subtracts 8 targets, branches to FAIL if != 0)
   │
[0x780] SUCCESS: Print "Correct password!" / FAIL: Print "Incorrect password!"

7.1. Console Prompt & Input Acquisition

The bytecode begins by computing ASCII values on the stack using mathematical instructions (PUSH, MATH_2, ADD, MUL), assembling 22 characters into a string via BUILD_STR 22, and outputting it via PRINT_STR:

Enter password: 

Next, it executes:

[ 147] GETLINE 32       ; Allocates buffer and reads input
[ 149] STRIP_NL         ; Strips trailing '\n' (0x0a)

At this point, the stack contains a pointer to the user's input string.

7.2. Password Length Validation Equation

Immediately following input collection, the bytecode evaluates the length of the string:

[ 153] DUP              ; Duplicate input pointer
[ 154] STRLEN           ; Top of stack = len
[ 155] MATH_2           ; len = 6 * len - 12
[ 156] PUSH 4
[ 158] ADD              ; len = 6 * len - 8
[ 159] PUSH 2
[ 161] DIV              ; len = (6 * len - 8) // 2 = 3 * len - 4
[ 162] MATH_3           ; len = 8 * (3 * len - 4) + 24 = 24 * len - 8
[ 163] PUSH 5
[ 165] PUSH 2
[ 167] MUL              ; 5 * 2 = 10
[ 168] DIV              ; (24 * len - 8) // 10
[ 169] PUSH 22
[ 171] PUSH 7
[ 173] PUSH 4
[ 175] MOD              ; 7 % 4 = 3
[ 176] SUB              ; 22 - 3 = 19
[ 177] PUSH 4
[ 179] MUL              ; 19 * 4 = 76
[ 180] EQ               ; Compare result with 76
[ 181] JNZ 97           ; If equal, proceed to packing; else abort

Algebraic Derivation:

The input password must be strictly 32 characters long.

7.3. Sub-routine sub_282: Little-Endian Dword Packing

At bytecode index 282 (0x59a8), we find a reusable sub-function invoked with 4 stack arguments:

[ 282] PEEK 3           ; Load argument c0
[ 284] PEEK 3           ; Load argument c1
[ 286] PUSH 8
[ 288] SHL              ; c1 << 8
[ 289] OR               ; c0 | (c1 << 8)
[ 290] PEEK 2           ; Load argument c2
[ 292] PUSH 16
[ 294] SHL              ; c2 << 16
[ 295] OR               ; c0 | (c1 << 8) | (c2 << 16)
[ 296] PEEK 1           ; Load argument c3
[ 298] PUSH 24
[ 300] SHL              ; c3 << 24
[ 301] OR               ; c0 | (c1 << 8) | (c2 << 16) | (c3 << 24)
[ 302] HALT             ; Return 32-bit dword to caller

This routine packs 4 distinct bytes [c_0, c_1, c_2, c_3] into a single 32-bit unsigned little-endian word: \text{dword} = c_0 \mid (c_1 \ll 8) \mid (c_2 \ll 16) \mid (c_3 \ll 24)

7.4. Permutation Matrix: 32 Characters to 8 Dwords

The main routine extracts characters from the 32-byte input string using GET_CHAR <index> and passes them into CALL 282, 4 exactly 8 times. Tracing the indices extracted for each dword gives the following permutation table:

[!NOTE] Every character index in the range [0, 31] appears exactly once, confirming that this is an invertible full-rank permutation: \{0, 1, 2, \dots, 31\} = \bigcup_{i=0}^7 \text{Indices}(d_i).

7.5. Sub-routine sub_456: 32-bit Left Circular Shift (ROL32)

At bytecode offset 456 (0x5c60), we observe the second reusable sub-function:

[ 456] PEEK 1           ; Load value x
[ 458] PEEK 1           ; Load shift amount k
[ 460] SHL              ; x << k
[ 461] PEEK 2           ; Reload x
[ 463] PUSH 32
[ 465] PEEK 3           ; Reload k
[ 467] SUB              ; 32 - k
[ 468] SHR              ; x >> (32 - k)
[ 469] OR               ; (x << k) | (x >> (32 - k))
[ 470] HALT             ; Return result

This is the standard implementation of 32-bit Circular Left Rotation: \text{ROL32}(x, k) = (x \ll k) \mid (x \gg (32 - k))

The rotation amounts k_i applied to each dword are calculated on the fly using VM arithmetic:

• d_7: PUSH 8; PUSH 7; ADD \implies k_7 = 8 + 7 = \mathbf{15}
• d_6: PUSH 24; PUSH 5; SUB \implies k_6 = 24 - 5 = \mathbf{19}
• d_5: PUSH 4; PUSH 3; ADD \implies k_5 = 4 + 3 = \mathbf{7}
• d_4: PUSH 11; PUSH 7; ADD \implies k_4 = 11 + 7 = \mathbf{18}
• d_3: PUSH 24; PUSH 2; DIV \implies k_3 = 24 / 2 = \mathbf{12}
• d_2: PUSH 5; PUSH 4; MUL \implies k_2 = 5 \times 4 = \mathbf{20}
• d_1: PUSH 17; PUSH 3; SUB \implies k_1 = 17 - 3 = \mathbf{14}
• d_0: PUSH 51; PUSH 3; DIV; PUSH 10; SUB \implies k_0 = (51 / 3) - 10 = \mathbf{7}

7.6. Target Verification System & Validation Equations

Following the rotations, the VM performs an equality check on each transformed dword by pushing a target constant, executing SUB, and branching to the failure block via JNZ if the result is non-zero:

[ 501] CALL 456, 2      ; ROL32(d7, 15)
[ 504] PUSH 706975780   ; Target for d7
[ 506] SUB
[ 507] JNZ <fail>
...

The 8 equations enforced by the VM are:

8. Algorithm Inversion & Flag Recovery

Because bitwise circular rotation is an orthogonal bijection over \mathbb{Z}_{2^{32}}, the inverse of \text{ROL32}(x, k) is simply \text{ROR32}(x, k):

Where: \text{ROR32}(v, k) = (v \gg k) \mid (v \ll (32 - k)) \pmod{2^{32}}

Step 1: Recovering the 8 Dwords (d_0 \dots d_7)

Applying \text{ROR32} to each target constant:

Step 2: Unscrambling the Permuted Characters

Now we place each byte back into its original input index S_j using the permutation table:

From d7:
  flag[29] = 0x47 ('G')
  flag[1]  = 0x54 ('T')
  flag[0]  = 0x48 ('H')
  flag[17] = 0x30 ('0')

From d6:
  flag[25] = 0x4e ('N')
  flag[5]  = 0x31 ('1')
  flag[10] = 0x4c ('L')
  flag[31] = 0x7d ('}')

From d5:
  flag[3]  = 0x7b ('{')
  flag[30] = 0x79 ('y')
  flag[21] = 0x74 ('t')
  flag[22] = 0x33 ('3')

From d4:
  flag[28] = 0x6f ('o')
  flag[16] = 0x77 ('w')
  flag[23] = 0x43 ('C')
  flag[8]  = 0x75 ('u')

From d3:
  flag[19] = 0x44 ('D')
  flag[13] = 0x34 ('4')
  flag[20] = 0x5f ('_')
  flag[26] = 0x30 ('0')

From d2:
  flag[6]  = 0x72 ('r')
  flag[9]  = 0x34 ('4')
  flag[2]  = 0x42 ('B')
  flag[18] = 0x72 ('r')

From d1:
  flag[11] = 0x5f ('_')
  flag[7]  = 0x54 ('T')
  flag[12] = 0x70 ('p')
  flag[4]  = 0x76 ('v')

From d0:
  flag[24] = 0x68 ('h')
  flag[14] = 0x73 ('s')
  flag[27] = 0x6c ('l')
  flag[15] = 0x53 ('S')

Step 3: Character Array Reconstruction

Mapping all positions 0 \dots 31:

Index:   00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15
Char:     H  T  B  {  v  1  r  T  u  4  L  _  p  4  s  S

Index:   16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Char:     w  0  r  D  _  t  3  C  h  N  0  l  o  G  y  }

Combining the characters reconstructs the full flag: \mathbf{HTB\{v1rTu4L\_p4sSw0rD\_t3ChN0loGy\}}

9. Complete Python Solver & Proof of Execution

The full, standalone solve script is located at solve.py:

#!/usr/bin/env python3
"""
HackTheBox: vvm
Category: Reverse Engineering
Author: @uzii2208
Description: Inverts the custom VM bitwise rotation and character permutation.
"""

def ror32(val: int, k: int) -> int:
    """Performs a 32-bit bitwise circular right rotation."""
    val &= 0xFFFFFFFF
    k %= 32
    return ((val >> k) | (val << (32 - k))) & 0xFFFFFFFF

def solve() -> str:
    # 1. Verification constraints extracted from VM bytecode
    # Format: dword_index: (target_signed_int32, left_rotation_amount)
    checks = {
        7: ( 706975780, 15),
        6: (-1972114847, 19),
        5: (-1170424423,  7),
        4: (-574761715, 18),
        3: (-213630203, 12),
        2: (1193747491, 20),
        1: ( 353885596, 14),
        0: (-1237732311,  7),
    }

    # 2. Invert ROL32 via ROR32
    d = {}
    for i in range(8):
        signed_val, shift = checks[i]
        unsigned_val = signed_val & 0xFFFFFFFF
        d[i] = ror32(unsigned_val, shift)

    # 3. Permutation mapping: maps each byte of d[i] to its original input index
    indices = [
        [24, 14, 27, 15],  # d0
        [11,  7, 12,  4],  # d1
        [ 6,  9,  2, 18],  # d2
        [19, 13, 20, 26],  # d3
        [28, 16, 23,  8],  # d4
        [ 3, 30, 21, 22],  # d5
        [25,  5, 10, 31],  # d6
        [29,  1,  0, 17],  # d7
    ]

    # 4. Reconstruct the 32-character string
    flag = [0] * 32
    for i in range(8):
        val = d[i]
        bytes_unpacked = [
            (val >>  0) & 0xFF,
            (val >>  8) & 0xFF,
            (val >> 16) & 0xFF,
            (val >> 24) & 0xFF,
        ]
        for byte_pos, char_idx in enumerate(indices[i]):
            flag[char_idx] = bytes_unpacked[byte_pos]

    return "".join(chr(c) for c in flag)

if __name__ == "__main__":
    recovered_flag = solve()
    print(f"[+] Successfully recovered flag: {recovered_flag}")

Verifying Against the Challenge Binary

Running the solver and piping the result directly into the target binary:

$ python3 solve.py
[+] Successfully recovered flag: HTB{v1rTu4L_p4sSw0rD_t3ChN0loGy}

$ ./vvm
Enter password: HTB{v1rTu4L_p4sSw0rD_t3ChN0loGy}
Correct password!

10. Key Takeaways & Anti-Reversing Insights

1. Identifying VM Obfuscation:

   • Look for the classic interpreter triad: an Instruction Pointer (IP), an Evaluation Stack (SP), and an indirect jump/call through a table (call [dispatch_table + opcode*8]).
   • Binaries that load a single array of dwords in .data and loop until a specific terminator (0x1c / HALT) are almost guaranteed to be custom VMs.

2. Defeating Self-Decrypting Handlers:

   • When handlers are located in RWX memory allocated by mmap, inspect the initialization function preceding the execution loop (0x1530).
   • Extract the XOR keystream (0x2a, 0x0b, 0x21, 0x21, 0x4d, 0x2a) and write a simple script to unpack the bytecode handlers statically, eliminating the need to attach a dynamic debugger under anti-debugging scrutiny.

3. Sub-Function Identification in Bytecode:

   • Modern VM challenges frequently implement modular sub-routines inside the bytecode (CALL 282, CALL 456).
   • Rather than manually analyzing hundreds of low-level stack instructions, identify functions that are called repeatedly. Recognizing sub_282 as dword packing and sub_456 as ROL32 collapses an otherwise intimidating bytecode program into basic modular algebra.

4. Exploiting Bijective Operations:

   • Bitwise rotations and index permutations do not destroy information. Whenever a verification algorithm relies strictly on reversible primitives, inverse operations (ROR32 and inverted permutations) provide an instant, deterministic solve without requiring symbolic execution (angr/Z3).