Wolfenstein Modloader Buy me a coffeeKaffee spendieren

Wiki / Latest research roundsNeueste Forschungsrunden

Texture Database Codec 2: Tile Chains for Large BC4/BC5 Mips

Partly solvedTeilweise gelöst Original page in the GitHub wikiOriginalseite im GitHub-Wiki

In plain wordsIn einfachen Worten

Large normal maps and similar textures in Wolfenstein II's texture database are often stored as chains of compressed tiles ('codec 2'). The outer container is now fully understood and rebuilds byte-exact across hundreds of blocks, but the pixel arrangement inside a tile is still not decoded, and three natural guesses were ruled out. This does not block modding: the simpler, untiled 'codec 1' is shown to be accepted at any size, so a replacement never needs codec 2. That replacement path still needs a BC4/BC5 encoder and has not been tested in game.

  • Each tile has a 24-byte header plus an Oodle stream; BC4 tiles are always 512x512, BC5 tiles 512x256.
  • The container rebuilds byte-exact across hundreds of blocks, including DLC.
  • Three layouts for the tile's pixels (raw planes, row-order blocks, Z-order blocks) were tested and rejected.
  • The theory that a GPU hardware feature explains codec 2 was refuted; the switch is an ordinary I/O cache setting.
  • 2% of images ship their largest mip as codec 1 (one even at 4096x4096), so codec 1 works at any size; an encoder is still missing.

Große Normal-Maps und ähnliche Texturen in der Texturdatenbank von Wolfenstein II liegen oft als Ketten komprimierter Kacheln vor („Codec 2“). Der äußere Container ist jetzt vollständig verstanden und lässt sich über Hunderte Blöcke byte-genau neu bauen, aber die Pixelanordnung innerhalb einer Kachel ist noch nicht entschlüsselt; drei naheliegende Vermutungen wurden widerlegt. Das blockiert das Modden nicht: Der einfachere, ungekachelte „Codec 1“ wird nachweislich in jeder Größe angenommen, ein Ersatz braucht Codec 2 also nie. Dieser Ersatzweg braucht aber noch einen BC4/BC5-Encoder und wurde nicht im Spiel getestet.

  • Jede Kachel hat einen 24-Byte-Kopf plus einen Oodle-Datenstrom; BC4-Kacheln sind immer 512x512, BC5-Kacheln 512x256.
  • Der Container lässt sich über Hunderte Blöcke byte-genau neu bauen, auch in DLCs.
  • Drei Anordnungen für die Kachelpixel (rohe Ebenen, Blöcke in Zeilenfolge, Blöcke in Z-Reihenfolge) wurden getestet und verworfen.
  • Die Theorie, ein GPU-Hardwarefeature erkläre Codec 2, ist widerlegt; der Schalter ist eine gewöhnliche I/O-Cache-Einstellung.
  • 2 % der Bilder liefern ihre größte Stufe als Codec 1 aus (eines sogar mit 4096x4096), Codec 1 geht also in jeder Größe; ein Encoder fehlt noch.

Technical detailsTechnische Details

Der technische Teil ist auf Englisch – so, wie er im Wiki steht. Unterstrichene Begriffe haben eine Erklärung auf Deutsch.

Most large (≥512 px) BC4/BC5 texture mips — normal maps, roughness, terrain — ship under a tiled "codec 2" container that was previously left undecoded. The outer container format is now fully solved and round-trips byte-exact at scale. The inner pixel layout of a decompressed tile is still not decoded: three natural hypotheses (raw 8-bit planes, standard raster-order BC-compressed blocks, Morton/Z-order BC blocks) were all tested with real position-derangement negative controls and correlation metrics, and all three are rejected. A plausible "GPU sparse-residency hardware feature" explanation for why codec 2 resists decoding was independently disassembled and refuted — the dispatcher's gating flag is an ordinary I/O-cache cvar, not a GPU capability switch. None of this blocks modding in practice: codec 1 (a plain, untiled BC stream) is proven accepted by the engine at any mip size, so a replacement never needs to produce a codec-2 tile at all.

Container format (solved, PROVEN at scale)

Per tile, a 24-byte header followed by an Oodle/Kraken stream:

u32 comp     -- compressed length of this tile
u32 raw      -- decompressed length of this tile
u32 where    -- (row << 24) | x, both in texels
u16 w, u16 h -- tile size: BC4 tiles are always 512x512/274,624 B raw,
                BC5 tiles always 512x256/280,960 B raw — a fixed
                per-format invariant, zero exceptions across thousands
                of sampled tiles
u8[8] unknown -- constant within one mip's chain; byte 0 is a format
                 tag (0x18 = BC4, 0x19 = BC5)
u8[comp] body -- Oodle/Kraken stream

A chain ends exactly at the end of its database block (no trailing bytes, no gap). Round-tripped byte-exact by reserializing every parsed field (not copying bytes verbatim) across hundreds of blocks spanning most archives in the game, including DLC. The raw length field is load-bearing — corrupting it reliably breaks decompression.

Inner tile layout — still unsolved, three hypotheses rejected

Each rejection used a genuine derangement negative control (swapping which tile's/block's decoded bytes land in which position, same content, only position scrambled) plus a correlation metric against an independently-decoded reference mip of the same texture, not just mean-error (which cannot distinguish "spatially correct" from "both landed near the same flat average" — an early false-positive on a nearly-flat normal map demonstrated exactly this trap).

  1. Two raw 8-bit planes (X, then Y), row-major, split at tile_w*tile_h bytes. Rejected: correct-vs-deranged placement produces statistically indistinguishable error on a 24-image corpus with real spatial detail.
  2. Standard row-major BC4/BC5-compressed blocks. Rejected: mean correlation ≈ 0 for both correct and deranged block placement (n=24).
  3. The same blocks in Morton/Z-order (the standard GPU tiling swizzle). Rejected the same way (n=20).

Why codec 2 resists decoding — the GPU-sparse-residency lead is refuted, not just unconfirmed. The dispatcher that picks between two bookkeeping-record layouts for a per-mip-batch gather is gated by the engine's ordinary is_cachedIO cvar (an I/O streaming-cache toggle), not a GPU capability flag; neither of the two consumer functions contains anything Vulkan/driver-specific — both are the same container/array bookkeeping pattern, differing only in record size. A string that looked like corroborating evidence for a GPU-sparse explanation has zero code cross-references anywhere in the binary.

Practical recipe for writing a replacement (does not need codec 2)

Retail ships plenty of large BC4/BC5 mips as codec 1 — a single Oodle stream of plain, un-tiled BC-compressed blocks — the identical technique already proven for the BC1 weapon-skin pipeline:

  1. Encode the new pixels as plain BC4 (single-channel) or BC5 (dual-channel) blocks. Not yet implemented in the project's own BC encoder (which currently only has BC1); extending it is mechanical, since the BC4/BC5 decoders already exist and the quantization math has no remaining unknowns.
  2. Oodle/Kraken-compress the plain block stream (already implemented).
  3. Write the mip-table record with codec = 1, raw_size = block count × 8 (BC4) or × 16 (BC5), and the compressed length.
  4. Write the texture-database row at the already-solved key formula (hash-derived key, shifted and OR'd with the mip level) — the same mechanism already proven for the weapon-skin pipeline.

No tile header, no 24-byte records, no sparse-residency guesswork anywhere in this path — reading a codec-2 tile is still unsolved, but producing a replacement never needs to.

Codec choice is size-tiered, not a free per-asset pick: across 200 mixed-codec images, 98% follow a strict "largest mip = codec 2, middle band = codec 1, smallest = codec 0" rule by level. The 2% that invert this — shipping their largest mip as codec 1 — are the proof codec 1 is accepted at any size; one named example ships its full 4096×4096 BC5 mip as codec 1 with codec 2 only in the middle of the chain.

Open questions

  • The exact inner byte layout of a codec-2 tile — three natural hypotheses rejected, no further lead from static analysis of the shipped executable. The remaining path is a dynamic GPU/Vulkan trace of an actual texture upload while the game runs.
  • What the unexplained ~143–150 KB "tail" per BC4 tile (beyond the block-sized region tested) actually is — not yet checked for its own internal structure.
  • Why retail chooses codec 2 for most large mips but codec 1 for a consistent ~2% minority — not needed for the practical recipe, but would explain the original asset pipeline's own logic.

In-game test (not yet run)

Once a BC4/BC5 encoder exists: replace a normal map's largest mip (any model texture above 512 px) using codec = 1 as above, get close enough in-game that this mip level loads, and confirm it renders correctly — the same already-verified path as the BC1 weapon skin, extended to BC4/BC5.