So far we’ve seen how to hide syscalls: searching for them in ntdll (Hell’s Gate), in neighboring functions (Tartarus’ Gate), or redirecting through kernel32 (Hell’s Hall).
VEH Syscalls does something different. Vectored Exception Handlers, legitimate Windows mechanisms, allow us to execute syscalls that appear to originate inside ntdll. We’ll explore how this two-layer architecture works and why it’s so evasive against kernel-level detection.
Introduction
- Syscall Series #1 - Hell’s Gate
- Syscall Series #2 - Tartarus’ Gate
- Syscall Series #3 - Hell’s Hall
- Syscall Series #4 - VEH Syscalls
What are VEH Syscalls and why basic implementations are not enough
Hell’s Gate and Hell’s Hall leave anomalous stacks. When you execute a syscall from userland code, the stack trace shows your application’s address space calling the kernel directly. An EDR inspecting the CONTEXT structure or unwinding the stack sees the jump: our code to the kernel. That is suspicious and, if the EDR is well configured, can raise alarm bells.
Basic VEH Syscalls are better, but still detectable. Registering a Vectored Exception Handler is legitimate, but a handler that intercepts execution at specific addresses and modifies the CONTEXT to execute syscalls is a recognizable pattern. A well configured EDR can hook AddVectoredExceptionHandler, monitor handler execution, or inspect the handlers registered on a process.
LayeredSyscall solves this by making the stack look legitimate. Instead of our code calling the kernel, the syscall appears to originate from within ntdll, from a function that Windows itself owns. The two-layer handler architecture (detection + execution) ensures the exception is triggered and intercepted inside trusted system code. This way, the stack frame, the RIP, and the entire CONTEXT structure appear to come from Windows internals, not from our application.
Why it’s called Layered
This mechanism uses a two-layer handler architecture:
- Layer 1 (Detection Handler): the initial exception triggers. We scan ntdll, find the syscall gadget, and set Dr0/Dr1 breakpoints on critical addresses.
- Layer 2 (Execution Handler): the breakpoints trigger inside ntdll. We enable the Trap Flag, monitor execution, inject our syscall via CONTEXT modification, and spoof the stack to make it look like ntdll called itself.
Two handlers separate setup from execution, allowing us to hijack syscalls from within legitimate ntdll stack frames.
LayeredSyscall Core Mechanism
Hardware Breakpoints
Hardware breakpoints live in the CPU’s debug registers (Dr0 through Dr7). Unlike software breakpoints, which modify the instruction stream by replacing bytes with INT 3, hardware breakpoints are set at the CPU level and do not touch code in memory. An EDR scanning the executable or doing static analysis will not see them.
The breakpoint is purely a CPU state: when execution reaches address X, trigger an exception. When we set Dr0 to an address inside ntdll, every time RIP lands on that address a EXCEPTION_DEBUG_EVENT triggers. We are not modifying ntdll, we are telling the CPU to watch it.
VEH Handler Interception
When the CPU detects that RIP has hit a breakpoint address, it generates an exception and Windows delivers it to our Vectored Exception Handlers, functions registered with AddVectoredExceptionHandler. Our handler receives a PEXCEPTION_POINTERS structure containing the CONTEXT (register state) and EXCEPTION_RECORD (what triggered it).
At this point we are executing inside our handler, which runs at the privilege level of the thread that hit the breakpoint. We can inspect and modify the CONTEXT, change registers, modify RIP, alter the stack. Then we return EXCEPTION_CONTINUE_EXECUTION and the thread resumes from the modified CONTEXT.
Call Stack Spoofing
Call stack spoofing means constructing a fake stack frame that makes it appear as though the syscall originated from within ntdll, not from our application code. Normally, when a function calls another function, the return address is pushed onto the stack. If our code at address 0x00400000 calls a syscall, the stack contains addresses in that range. An EDR unwinding the stack sees “this syscall came from application code,” which is anomalous.
In LayeredSyscall, we intercept execution inside ntdll and manipulate the stack to make it look like a series of legitimate ntdll function calls led to that point. We write fake return addresses pointing to other ntdll functions, set up a fake saved RBP, and arrange register values as though the current execution came from a legitimate call chain. When an EDR inspects the stack by unwinding via CONTEXT.RBP, it sees only ntdll addresses.
Trap Flag
The Trap Flag (TF) is bit 8 of the EFLAGS register. When set, it causes the CPU to trigger a SINGLE_STEP exception after every instruction executes, without a debugger running. Inside our VEH handler, we set CONTEXT.EFlags |= 0x100. The thread resumes, executes one instruction, and immediately triggers another exception. Our handler runs again, inspects the new CONTEXT, and decides whether to continue stepping or disable TF and resume normally.
LayeredSyscall uses this to verify that execution is flowing through ntdll as expected, that RIP stays within ntdll’s boundaries and that certain instructions executed in order. It is a way to instrument execution without a debugger attached.
Choosing the Trigger
Before going into the execution path, we must choose the trigger, since it determines how many bytes we must skip when returning from the handler.
- NULL dereference: write to a protected address. Triggers immediately. We advance RIP by 2 to 8 bytes depending on the instruction.
- INT 3: explicit breakpoint instruction. One byte (0xCC). We advance RIP by 1.
- UD2: undefined instruction. Two bytes (0x0F 0x0B). Triggers ILLEGAL_INSTRUCTION. We advance RIP by 2.
- Division by zero: variable size depending on the div/idiv instruction. Less predictable.
- Hardware breakpoint: set via Dr0-Dr7. No bytecode injected. The CPU triggers the exception when RIP lands on the monitored address. No RIP advancement needed, we control execution flow directly via CONTEXT modification.
Hardware breakpoints are cleaner because they do not require injecting bytecode into ntdll.
Execution Path
Stage 1: Access Violation Triggers VEH
We trigger the first exception, for example a NULL dereference or INT 3. This breaks into our VEH handler while the thread is frozen at the point of exception.
1
2
3
4
PVOID handler_handle = AddVectoredExceptionHandler(1, veh_handler);
volatile int *ptr = NULL;
*ptr = 0; // Access violation, VEH fires
1
2
3
4
5
6
LONG WINAPI veh_handler(PEXCEPTION_POINTERS pExceptionInfo) {
PCONTEXT ctx = pExceptionInfo->ContextRecord;
// Thread is suspended at the exception point.
// ctx->Rip, ctx->Rsp, all registers are frozen.
return EXCEPTION_CONTINUE_EXECUTION;
}
Stage 2: Handler Places Breakpoints on Syscall and Return
Inside the handler, we set Dr0 and Dr1 on two critical addresses within ntdll: the syscall instruction (0x0F 0x05) and the return instruction of the function we are about to call. Then we return EXCEPTION_CONTINUE_EXECUTION.
1
2
3
4
5
6
7
ctx->Dr0 = (DWORD64)syscall_gadget_address; // 0x0F 0x05
ctx->Dr1 = (DWORD64)ntdll_function_ret_address;
// Dr7 = 0x00000401 enables Dr0 and Dr1 (local, execute breakpoints)
ctx->Dr7 = 0x00000401;
return EXCEPTION_CONTINUE_EXECUTION;
Stage 3: Redirection to Legitimate API
The thread resumes and calls a legitimate Windows API, for example MessageBoxA. The stack looks legitimate: our app calls MessageBoxA in kernel32, which calls into ntdll internally.
1
2
MessageBoxA(NULL, "Hello", "Title", MB_OK);
// Stack: our app → MessageBoxA → kernel32 → ntdll
Stage 4: Trap Flag Traces Inside ntdll
When a breakpoint inside ntdll triggers, our handler fires again. We enable the Trap Flag in CONTEXT.EFlags. The thread resumes, executes one instruction, and immediately triggers a SINGLE_STEP exception. We are now tracing instruction by instruction through ntdll.
1
2
3
4
5
6
7
8
9
10
11
12
LONG WINAPI veh_handler(PEXCEPTION_POINTERS pExceptionInfo) {
PCONTEXT ctx = pExceptionInfo->ContextRecord;
if (pExceptionInfo->ExceptionRecord->ExceptionCode == EXCEPTION_SINGLE_STEP) {
if (is_rip_at_return_instruction(ctx->Rip)) {
// About to return from ntdll function. Inject syscall now.
}
ctx->EFlags |= 0x100; // Keep TF enabled
}
return EXCEPTION_CONTINUE_EXECUTION;
}
Stage 5: Inject Syscall Between Real Frames
As we trace, we monitor for the RET instruction of the current ntdll function. Before it executes, we modify CONTEXT.RIP to point to the syscall gadget (0x0F 0x05). The thread executes our syscall, transitions to kernel mode, and returns to our handler.
1
2
3
4
5
6
7
8
9
10
11
12
ctx->Rip = (DWORD64)syscall_gadget_address;
ctx->Rcx = arg1;
ctx->Rdx = arg2;
ctx->R8 = arg3;
ctx->R9 = arg4;
ctx->EFlags &= ~0x100; // Disable Trap Flag
ctx->Dr7 = 0; // Disable all debug registers
return EXCEPTION_CONTINUE_EXECUTION;
// Thread executes syscall from ntdll context
Result: Stack Appears as Process → API → ntdll
When an EDR unwinds the stack, it sees: our application → MessageBoxA → ntdll internals. The syscall appears to originate from within ntdll, not from application code. The entire call chain is legitimate Windows code.
Evasion and Limitations
The main detection this technique evades is stack analysis. When an EDR unwinds the stack after our syscall, it sees only ntdll addresses as the immediate caller. The call chain matches what the OS does during normal API execution.
This technique is not without limitations. We are restricted to two hardware breakpoints (Dr0 and Dr1). If multiple syscalls trigger simultaneously, we cannot monitor them all. Trap Flag stepping introduces timing overhead that, if too slow, may cause EDRs to detect performance anomalies. Finally, ntdll varies across Windows versions, so syscall gadget locations and function stack layouts must be identified per version.
Closing Remarks
The evolution from Hell’s Gate to LayeredSyscall represents a shift in how we think about syscall evasion. We have moved from simple SSN extraction to sophisticated stack spoofing, from direct syscall execution to exception-driven hijacking. Each technique built upon the limitations of its predecessor, finding new blindspots in detection mechanisms.
As EDRs evolve to monitor hardware breakpoints, Trap Flag abuse, and VEH handler registration patterns, new techniques will emerge. Understanding these techniques in depth, not just how to implement them but why they work and where they fail, gives the foundation to recognize and adapt to future evasion mechanisms.
I hope you enjoyed the series as much as I enjoyed studying it.