Glide Render Pipeline: From Black Screen to First Pixels — Work in Progress 3

Range: c96fef2 through 9718bf8 (v0.0.35 → v0.0.77)

The previous post was about how fast execution runs — introducing a native AOT dynamic translator to cut single-step trap overhead.

This range is about how far execution runs. Because execution got faster, it began reaching points it had never reached within the time budget before, and defects that had been hidden behind that ceiling surfaced one after another. Removing them let PIU.EXE pass boot and asset parsing, settle into a frame loop, and finally put real pixels on the window.

Major Changes

1. Removing defects that silently corrupted guest state

The first half of this range proceeded by chasing one frontier (the next crash point) at a time. Each frontier initially looked like a bug in the guest code, but without exception the root cause was the HLE layer failing to reproduce a contract of the original DOS/LE environment.

flowchart TD
    A["frontier: guest stack slot corrupted<br/>0x035D6B14 = 0xDD1523B1"] -->|root cause| A1["LE cross-page fixup<br/>source_offset 0xFFFF not sign-extended<br/>→ guest instruction corrupted"]
    A1 -->|fixed| B["frontier: base pointer corrupted<br/>slot 0x031A66FC → 0x4041"]
    B -->|root cause| B1["HLE never recycled DOS handle numbers<br/>handle 20 overflowed a 20-entry table"]
    B1 -->|fixed| C["frontier: null-string stricmp<br/>0x030F4A98, read of addr 0"]
    C -->|root cause| C1["DOS/4GW maps low memory, so the read is harmless<br/>Win32 null-page faults = HLE gap"]
    C1 -->|fixed| D["frontier: arena-end overflow<br/>0x045D3EB0"]
    D -->|root cause| D1["32-bit EBX resize requests ignored<br/>measured ~83 MiB heap demand unmet"]
    D1 -->|fixed| E["asset parsing passes → frame loop reached"]

• LE cross-page fixup sign extension (c4c2aad, v0.0.52). A fixup record's source_offset value of 0xFFFF means -1, denoting a fixup that straddles a page boundary; using it unsigned patched the wrong location and overwrote a guest instruction. Fixed with int16_t sign extension. Several watchpoint investigations failed before this; the decisive observation technique turned out to be full single-stepping on the trap backend.
• DOS file handle recycling (c1ffbc0, v0.0.54). OpenDosFile incremented handle numbers monotonically and CloseDosFile never reclaimed them. Opening and closing 16 files in sequence (never more than two open at once) made the 16th open return handle 20 — but the guest Watcom clib's handle-flag table has exactly 20 entries, and the address of table[20] is that table's base-pointer slot. Writing table[20] corrupted the base to 0x4041. Real DOS returns the lowest free handle and reclaims it on close, so numbers never leave the 5–6 range. Fixed with lowest-free allocation.
• DOS/4GW low-memory read tolerance (e55644e, v0.0.58). Parsing an empty texture descriptor reads address 0. Under DOS/4GW low memory is mapped and this is harmless; on Win32 it is a null-page fault. Reclassified from data bug to HLE gap and emulated the read.
• Arena sizing (b2c817f, v0.0.47). Honored 32-bit EBX resize requests and sized the arena to the game's measured ~83 MiB heap demand, eliminating the arena-end overflow.

The recurring lesson is clear: an observation that "the guest is behaving strangely" was almost always a signal that the HLE had broken a contract of the original environment, and every attempt to work around it on the guest side came back as a regression.

2. Hardening the execution infrastructure

• AOT return inline cache widening (fd2f906, v0.0.46). Live telemetry traced a post-Glide decode freeze to return-target thrashing. Widening the single-entry cache to a four-entry chain resolved it.
• execution_trampoline decomposition (d1673e2, v0.0.60). Split the accumulating monolithic execution trampoline into per-responsibility subsystem modules — a belated application of the AGENTS.md rule that independently nameable subsystems get their own files.
• AOT/Glide gate leak (b1d80ad, v0.0.66). The root cause of zero-EIP crashes was an unhandled gate leaking its stdcall frame. After adding stack-scan recovery, aot-dynamic survived 180 seconds and entered the main render frame loop.

3. Glide ABI alignment

• Restoring stdcall for grTexMinAddress / grTexMaxAddress (74482d8, v0.0.62). fxTMInit(gc, tmu) crashed with EAX (gc) = 0. The first diagnosis — an unallocated Mesa context, to be repaired by dynamic null-register patching — was wrong. The game allocates gc itself via malloc(0x1C88). The real cause: fxTMInit stores gc at [esp], calls both gates assuming stdcall (callee pops the argument), then reloads gc with mov eax,[esp]. A prior change had switched those gates to cdecl, leaving two arguments on the stack, so the reload read a leftover 0 instead of gc. An xref confirmed fxTMInit is the sole caller of both thunks, and stdcall was restored.

This episode is a reminder that "the error message went away" is not evidence of progress. The cdecl change merely made fxTMInit crash earlier, so execution never reached the later fxTMGetTMBlock error.

4. Glide render pipeline R0–R4: from black screen to first pixels

A full 600-second run pinned the black screen's root cause. The window is a real WGL window, not a dummy — but all three layers of the render path were ABI-preserving no-ops. That inventory produced a phased R0–R5 plan, of which R0 through R4 are now implemented.

flowchart LR
    subgraph guest["Guest PIU.EXE"]
        G1["grBufferClear"]
        G2["grDrawTriangle<br/>60-byte 2-TMU GrVertex"]
        G3["grTexDownloadMipMapLevel<br/>grTexSource"]
        G4["grColorCombine<br/>grAlphaBlendFunction"]
        G5["grBufferSwap"]
    end
    subgraph gate["LINEXE Glide gate (R0)"]
        GA["97 decorated names cataloged<br/>default handler = stdcall cleanup + status"]
    end
    subgraph backend["Win32 OpenGL backend"]
        B0["glOrtho(0,w,h,0,-1,1)<br/>y-flipped screen-space projection"]
        B1["glClear (R1)"]
        B2["glColor4f = GrVertex r/g/b/a (R2)"]
        B3["texture cache + GLSL sampler2D (R3)"]
        B4["glEnable(GL_BLEND) + glBlendFunc (R4)"]
        B5["SwapBuffers (R1)"]
    end
    G1 --> GA --> B1
    G2 --> GA --> B0 --> B2
    G3 --> GA --> B3
    G4 --> GA --> B4
    G5 --> GA --> B5
    B5 --> DIAG["glReadPixels non-black pixel count<br/>(REPIU_GLIDE_PIXEL_DIAG)"]

• R0 gate safety net + R1 frame presentation (ef89335, v0.0.69). PIU.EXE references 97 decorated Glide names, but only 44 were in the signature catalog; an unregistered name is rejected as signature-mismatch on call, which leads to an unhandled-gate crash. Cataloging all 97 with a default handler (stdcall cleanup plus a status return) eliminated that class of risk, and _GRBUFFERCLEAR@12 / _GRBUFFERSWAP@4 were wired to real glClear / SwapBuffers.

  Profiling at this point revealed something important: the game had settled into a stable 60 FPS frame loop but issued zero grDrawTriangle-family calls. The game was not dead — its main logic was skipping rendering and swapping empty frames while waiting on a non-Glide subsystem (I/O, EEPROM, sound).

• Screen-space orthographic projection (4c92428, v0.0.73). Even once draw calls appeared, the window stayed black. Runtime vertex capture confirmed the game passes 640×480 screen-pixel coordinates, but the backend set no glOrtho, so ftransform() applied an identity projection and pixel coordinates (x≈288, y≈330) landed outside NDC [-1,1] — every triangle was clipped. Added a y-flipped glOrtho(0, w, h, 0, -1, 1) matching the observed grSstWinOpen origin=1 (GR_ORIGIN_UPPER_LEFT). Non-black pixels went from 0 to 18,176.

• R2 vertex color + R3 texture path (ad7631c, v0.0.74). Wired the confirmed 60-byte 2-TMU GrVertex color fields through glColor4f (removing the hardcoded white). Then confirmed that content draws emit texture color via grColorCombine function 3 = SCALE_OTHER = TEXTURE, and implemented a platform-neutral decode module (src/hle/glide_texture_decode.{h,cpp}), a backend texture cache, and GLSL sampler2D sampling.

• R4 alpha blending (4b713ca, v0.0.75). R3 rendered transparent textures as opaque black. Two blend functions were observed — (4,0,4,0) = ONE,ZERO (opaque) and (1,5,4,0) = SRC_ALPHA/ONE_MINUS_SRC_ALPHA (standard transparency). Generalized SetAlphaBlend to map Glide blend factors onto GL factors.

Headless verification technique. This session cannot screenshot the GL window because of desktop/window-station isolation. Instead, an env-gated (REPIU_GLIDE_PIXEL_DIAG) glReadPixels non-black-pixel count in BufferSwap measures rasterization directly. Before the projection fix everything was clipped, so the count had to be 0 — which makes any change in that count decisive evidence that geometry now rasterizes.

5. Peripheral HLE and diagnostic infrastructure

• 93C46 EEPROM HLE state machine and timer interrupt injection (391e198, v0.0.70) — satisfying one of the hardware conditions the game was waiting on.
• Indirect LINEXE calls and INT 8 boundaries (d218b43).
• Texture BMP dumping and format validation (5864bff). With REPIU_DUMP_TEXTURE_BMP=1, decoded RGBA8 is written as 32-bit BGRA BMP to build/texture_dumps/ at grTexDownloadMipMapLevel time. A 100-second run dumped exactly two 1×1 textures (tex_0x0_fmt10_1x1_1.bmp, tex_0x8_fmt12_1x1_2.bmp). Alongside it, IsGlideTextureFormatAcceptable rejects unsupported formats before any decode.
• JAMMA I/O keyboard mapping (9718bf8, v0.0.77). Active-low bitmasks matching the MAME xtom3d.cpp specification map P1 pad (0x02A8), system (0x02A9: coin/service/test), and P2 pad (0x02AA), polled via GetAsyncKeyState. HandlePortIoInstruction was refactored into a dynamic byte-read loop so 8-, 16-, and 32-bit IN widths all work.

Execution log — current progress point

The diagnostic line emitted by BufferSwap in glide_opengl_backend.cpp has this form:

[repiu-live-debug] Glide swap #<n> non-black pixels=<count>/307200 avg-rgb=<r>,<g>,<b>

The counts recorded during the projection-fix run (v0.0.73):

Before the projection fix every triangle was clipped, so this count would have stayed at 0 — which is what makes the progression decisive evidence of rasterization.

After the R3 texture path and R4 alpha blending, the current state (v0.0.75) holds a stable 17,280 / 307,200 non-black at avg-rgb 255,255,0 across content swaps in a 135-second aot-dynamic pumpit1 run, with zero rejected gates, zero unhandled gates, zero GL errors, and no crash.

The drop from R2 (24,704) to R3/R4 (17,280) is a fidelity improvement, not a regression: R2 painted vertex color even where the texture is transparent, while R3 onward treats transparent texels as the game intends. The current screen is a yellow-on-black attract screen.

Current blockers

The render pipeline is alive, but this is not yet a game screen. The confirmed remaining work:

• Content textures are 1×1. Only 17,280 of 307,200 pixels are non-black and the average color is a flat 255,255,0 — meaning the game has not yet reached the texture path that references real art assets. largeLod=0, aspect=3 together with the 8-byte spacing confirms 1×1.
• The LFB (linear frame buffer) path is unimplemented. The seven LFB-family names among the 97 are safely absorbed by the R0 default handler but do nothing.
• R5 fidelity is unimplemented — depth, culling, mipmapping, filtering.
• Re-observation is needed after JAMMA I/O. The input path landed only in v0.0.77, so how far the game state machine advances past attract must be measured again.

Sample test results

The OpenWatcom sample suite is the regression indicator for DOS/4GW console sample compatibility. The baseline was refreshed at v0.0.59 in this range (fa97643): of 819 samples, 793 build, 26 are excluded from the build, 793 are run-eligible, and 529 pass. The run pass rate is 66.7% and the overall pass rate is 64.6%.

xychart-beta
    title "OpenWatcom Sample Cumulative Results"
    x-axis ["0.0.1", "0.0.5", "0.0.9", "0.0.15", "0.0.34", "0.0.59"]
    y-axis "Samples" 0 --> 850
    line "Total" [819, 819, 819, 819, 819, 819]
    line "Build Passed" [788, 793, 793, 793, 793, 793]
    line "Run Passed" [419, 473, 473, 522, 523, 529]
    line "Build Skipped" [0, 26, 26, 26, 26, 26]

Compared with the previous post's point (v0.0.34, 523), run passes are up by six. Most work in this range targeted execution paths specific to PIU.EXE — Glide, textures, JAMMA I/O — so the console-sample metric moves little. This indicator is best read as regression surveillance, not as a growth curve.

Technology Stack Used

LE cross-page fixups. An LE executable's fixup records store the relocation source offset in 16 bits; a value beyond the page size or negative means the fixup straddles a page boundary. 0xFFFF is -1: a four-byte relocation starting at the last byte of the previous page and continuing into the next. Reading it as unsigned applies the patch at the wrong address, and if that address holds code, a guest instruction is corrupted. LE field definitions are scattered across historical sources, so they were cross-checked against the Open Watcom linker's loader implementation. The project's own notes live in docs/kb/le-format-and-relocation.md.

The 3dfx Glide API and GrVertex layout. Glide is a thin abstraction over Voodoo hardware: vertices are handed over already in screen pixel coordinates, with no transform pipeline. The layout confirmed here is a 60-byte GrVertex in a 2-TMU configuration, with dwords 3/4/5/7 holding r/g/b/a in [0..255]. That "screen coordinates passed through directly" property is exactly why a missing glOrtho produced 100% clipping.

sequenceDiagram
    participant G as Guest PIU.EXE
    participant B as LINEXE Glide gate
    participant O as OpenGL backend
    G->>B: grTexDownloadMipMapLevel(info, data)
    B->>O: StoreTexture(format, lod, aspect, bytes)
    Note over O: IsGlideTextureFormatAcceptable check<br/>RGB565 / ARGB4444 → RGBA8 decode
    G->>B: grTexSource(tmu, startAddress, info)
    B->>O: BindTexture(startAddress)
    G->>B: grColorCombine(function=3 SCALE_OTHER, other=TEXTURE)
    B->>O: enable texture combine uniform
    G->>B: grAlphaBlendFunction(1,5,4,0)
    B->>O: glEnable(GL_BLEND) + glBlendFunc(SRC_ALPHA, ONE_MINUS_SRC_ALPHA)
    G->>B: grDrawTriangle(a, b, c)
    O->>O: glColor4f + sampler2D sampling → rasterize

Glide color/alpha combine. The function codes passed to grColorCombine and grAlphaCombine decide where fragment color comes from: function 1 = LOCAL emits the iterated vertex color, function 3 = SCALE_OTHER emits the other source (the texture). Separating the combine configuration used at initialization from the one used for content draws was the starting point for R3. Specifications from the Glide Programming Guide 2.4 and the Glide Reference Manual 2.4.

93C46 EEPROM. A 3-wire serial EEPROM (1024 bits as 64 sixteen-bit words) driven bit-by-bit over CS/CLK/DI/DO with READ/WRITE/EWEN/EWDS commands. Arcade boards use it to persist settings and counters. The HLE reproduces that bit protocol as a state machine and persists the contents to eeprom.dat.

JAMMA I/O and cross-checking against MAME. JAMMA harness inputs are active-low (unpressed reads as 1). Port assignments were confirmed against MAME's xtom3d.cpp driver specification: 0x02A8 (P1 pad), 0x02A9 (system: coin/service/test), 0x02AA (P2 pad). MAME source is not integrated into this project — the same principle as AGENTS.md's "do not integrate DOSBox source," and the licensing policy points the same way. It serves only as a reference for cross-checking hardware specifications.

Win32 window-station isolation and headless verification. Automated sessions run on a window station separated from the interactive desktop, so a GL window cannot be screenshotted. To work around this, render results are reduced to pixel statistics (glReadPixels non-black count plus average RGB). The observation that non-black pixels moved from 0 to 18,176 proves the clipping was resolved without any screenshot. All such diagnostics are environment-variable gated and do not affect the default execution path.