NtVdmControl

Prototype

NTSTATUS NtVdmControl(
  VDMSERVICECLASS ServiceClass,
  PVOID           ServiceData
);

Arguments

Syscall IDs by Windows version

Syscall stub

4C 8B D1            mov r10, rcx
B8 E2 01 00 00      mov eax, 0x1E2
F6 04 25 08 03 FE 7F 01   test byte ptr [0x7FFE0308], 1
75 03               jne short +3
0F 05               syscall
C3                  ret
CD 2E               int 2Eh
C3                  ret

Undocumented notes

NtVdmControl is the kernel back end for the NT Virtual DOS Machine (NTVDM) — the Windows NT subsystem that ran 16-bit MS-DOS and Win16 binaries on 32-bit Windows via ntvdm.exe. NTVDM was *removed in its entirety* on 64-bit editions of Windows, because long-mode CPUs cannot execute the v8086-mode that NTVDM relied on. The syscall stub nevertheless survives in 64-bit ntoskrnl.exe and dispatches into a stub that returns STATUS_NOT_IMPLEMENTED for every ServiceClass on x64. On 32-bit Windows 10/11 (and on WoW64-hosted ntvdm.exe up through 32-bit editions) the call services interrupt queueing, BIOS data area maintenance, and v8086 state transitions for the trapped DOS process. There is no public Win32 wrapper; everything that uses it is internal to ntvdm.exe and the WoW16/NTVDM subsystem.

Common malware usage

Pure historical interest on modern x64. On 32-bit XP and 32-bit Windows 7 NTVDM was a recurring Trojan vector — 16-bit DOS components of mass-mailer families (Bagle, Sober, Bugbear variants) shipped a small DOS stub that, when launched, started ntvdm.exe which then called NtVdmControl as part of its normal initialization. The actual abuse vector was the 16-bit code, not the syscall itself, but the syscall is the choke point. NTVDM has also been the source of multiple privilege escalations: CVE-2010-0232 (Tavis Ormandy's #GP-handler bug, exploited on 32-bit XP/Vista/7) and CVE-2018-8897/8453 (kernel bugs reachable via legacy paths). On 64-bit Windows the syscall is unreachable as a meaningful primitive and there are no in-the-wild abuse reports.

Detection opportunities

On 64-bit Windows, *any* call to NtVdmControl is anomalous — the legitimate caller (ntvdm.exe) does not exist and the syscall returns STATUS_NOT_IMPLEMENTED unconditionally. ETW provider Microsoft-Windows-Kernel-Audit-API-Calls and EDR syscall-tracing hooks can flag invocations directly; the false-positive rate on a modern Windows 11 endpoint is essentially zero. On 32-bit hosts, scope detection to processes other than %SystemRoot%\System32\ntvdm.exe and look for ntvdm.exe parent chains that did not originate from a user double-clicking a .com/.exe MZ-only binary. Sysmon Event ID 1 (process creation) with parent image ntvdm.exe + child of an unusual DOS binary is the practical hunt on legacy systems.

Direct syscall examples

; Direct syscall stub for NtVdmControl (SSN 0x1E2, Win11 24H2)
; On x64 always returns 0xC0000002 STATUS_NOT_IMPLEMENTED.
NtVdmControl PROC
    mov  r10, rcx          ; syscall convention
    mov  eax, 1E2h         ; SSN
    syscall
    ret
NtVdmControl ENDP

// Historical sketch — does not work on any 64-bit Windows.
// 16-bit MZ-only DOS components of early-2000s mass-mailers (Bagle, Sober)
// relied on the Windows shell auto-launching ntvdm.exe when the user
// double-clicked the 16-bit dropper. Inside, ntvdm.exe issued a long
// sequence of NtVdmControl calls to set up interrupt queues and the
// BIOS data area for the trapped DOS task.
//
// On modern x64 Windows the shell refuses to run 16-bit MZ binaries
// ("unsupported 16-bit application") and this whole vector is dead.
//
// Reference: VX-Underground archives, Symantec write-ups on Bagle.A
// and Sober.D from 2003-2004.
typedef NTSTATUS (NTAPI *PFN_NtVdmControl)(ULONG ServiceClass, PVOID ServiceData);
HMODULE hNtdll = GetModuleHandleW(L"ntdll.dll");
PFN_NtVdmControl p = (PFN_NtVdmControl)GetProcAddress(hNtdll, "NtVdmControl");
NTSTATUS s = p(/*VdmInitialize*/ 0, NULL); // STATUS_NOT_IMPLEMENTED on x64

// Detection helper: on any x64 Windows this returns STATUS_NOT_IMPLEMENTED.
// Useful inside an EDR self-test or a defensive sanity check.
use std::ffi::CString;
use std::ptr;

type NtVdmControlFn = unsafe extern "system" fn(u32, *mut core::ffi::c_void) -> i32;

fn ntvdm_is_dead() -> bool {
    unsafe {
        let ntdll = libloading::Library::new("ntdll.dll").unwrap();
        let f: libloading::Symbol<NtVdmControlFn> =
            ntdll.get(b"NtVdmControl").unwrap();
        let status = f(0, ptr::null_mut());
        status == 0xC0000002u32 as i32 // STATUS_NOT_IMPLEMENTED
    }
}

MITRE ATT&CK mappings

• T1068Exploitation for Privilege Escalationmitre ↗
• T1106Native APImitre ↗