Wolfenstein Modloader Buy me a coffeeKaffee spendieren

Wiki / Research reportsForschungsberichte

matpipe: material → image → texdb, end-to-end

Partly solvedTeilweise gelöst Original page in the GitHub wikiOriginalseite im GitHub-Wiki

In plain wordsIn einfachen Worten

UpdateAktualisierung Newer than this page: the wiki's start page now records custom weapon skins as working and measured in The New Colossus – a custom BC1 pistol skin shows on the first-person model in the running game.Neuer als diese Seite: Die Startseite des Wikis führt eigene Waffen-Skins in The New Colossus jetzt als funktionierend und gemessen – ein eigener BC1-Pistolen-Skin erscheint im laufenden Spiel am Modell in der Ich-Perspektive.

This report follows the texture chain in Wolfenstein II: The New Colossus: a material names an image, and the image's detail levels are stored in a texture database. The formats were read and rewritten byte-for-byte across the whole game, and a self-made flat magenta texture passes all offline checks and an independent second reader. A test file that turns 218 textures in the Gestapo Office DLC map magenta is ready for the player to try. Nothing has been confirmed in the running game yet, and arbitrary pictures cannot be written because no BC texture encoder exists.

  • All 11,469 retail materials and 35,842 images match the format rules; 300 of 300 sampled images rewrite byte-for-byte.
  • Single-color textures can be written exactly: 22 of 22 retail single-color images were regenerated identically.
  • Arbitrary pixels are not possible yet because there is no BC compressor.
  • A test file replaces 218 map textures with magenta and leaves 6,767 other entries unchanged; it is ready but not run in the game.
  • 10 of 10 deliberately broken test cases are caught by the checker.

Dieser Bericht verfolgt die Texturkette in Wolfenstein II: The New Colossus: Ein Material nennt ein Bild, und dessen Detailstufen liegen in einer Texturdatenbank. Die Formate wurden im ganzen Spiel gelesen und byte-genau neu geschrieben, und eine selbst erstellte einfarbige Magenta-Textur besteht alle Offline-Prüfungen und einen unabhängigen zweiten Leser. Eine Testdatei, die 218 Texturen der DLC-Karte Gestapo-Büro magenta färbt, liegt zum Ausprobieren bereit. Im laufenden Spiel ist noch nichts bestätigt, und beliebige Bilder lassen sich nicht schreiben, weil kein BC-Textur-Encoder existiert.

  • Alle 11.469 Materialien und 35.842 Bilder des Spiels folgen den Formatregeln; 300 von 300 Stichproben-Bildern werden byte-genau neu geschrieben.
  • Einfarbige Texturen lassen sich exakt schreiben: 22 von 22 einfarbigen Bildern des Spiels wurden identisch neu erzeugt.
  • Beliebige Pixel gehen noch nicht, weil es keinen BC-Kompressor gibt.
  • Eine Testdatei ersetzt 218 Kartentexturen durch Magenta und lässt 6.767 andere Einträge unverändert; sie ist fertig, aber nicht im Spiel getestet.
  • 10 von 10 absichtlich kaputten Testfällen werden vom Prüfer erkannt.

Technical detailsTechnische Details

Der technische Teil ist auf Englisch – so, wie er im Wiki steht. Unterstrichene Begriffe haben eine Erklärung auf Deutsch.

Worker w-matpipe-a2, 2026-09-23. This is the second run of a2 (the first terminated due to usage limits; a1 was paused by the user). All numbers below originate from this run; nothing has been carried over from the aborted state. Tools: tools/probe_matpipe.py. Artifacts: re_probes/matpipe/.

Result: The chain is structurally valid after these checks; whether it renders remains unconfirmed at runtime. A material binds via loosealbedo to a self-written image, whose streaming mips reside in its own, sorted .texdb. Both hash validators—the validator itself and an independently written reader—accept the pair and read Magenta. New in this run: a runtime test that the user can try out in a window (§5). Flat colors are exactly encodable. For arbitrary pixels, a BC-Encoder is missing (§2.3).

Check Command Result
Material Round-Trip, 3 Archives --material 300 300/300 (chunk_1 197, chunk_7 69, chunk_8 34)
… entire corpus --formats 11469/11469 in 27 Retail Archives
loose* value → Retail image --material 300 2024/2024
loosealbedo → image with mtlkind 1 --formats 7937/7938; the one remainder is a Retail reference to nothing
Image Round-Trip, 3 Archives --image 300 300/300 byte-exact, 300/300 contradiction-free
Mip stored == .texdb block length --image 300 1576 own + 31 foreign .texdb, 0 false
BIM-v14 census --formats 35 842 images, 382 114 mip sets, 0 violations
Retail single-color images regenerated --author 22/22 byte-identical
Demo: Header/Mip table vs. Retail twins --demo 10/10 and 10/10
Demo: second reader (wolfsdk/tncimage.py) --demo 87 376 + 336 pixels, all Magenta
Demo: probe_datahash / probe_idclhash --demo 2/2, +0x08 1/1, +0x20 1/1 (both variants)
Runtime test chunk c2v1 --recolor … 218/218 replaced, 6767/6767 unchanged byte-identical, datahash 6984/6984, idclhash OK
Negative controls --negative 10/10 caught, 2 anchors clean

python tools/probe_matpipe.py --selftest summarizes everything except --recolor (rc 0, output in re_probes/matpipe/selftest.txt).

1. material: Text, loosealbedo is the Albedo Slot

The payload is UTF-8: {, newline, per property key\tvalue, then }, newline. All 11 469 Retail materials follow this byte-exactly. Two types:

  • 7 938 "loose" materials (Standard PBR) carry all eight slots loosealbedo/-normal/-spec/-smoothness/-cover/-sssmask/-colormask/-bloommask. Their value is literally the name of an image entry. The material parser constructs it with "%s$mtlkind=albedo$streamed" (.text 0x747099).
  • 3 531 program materials (stageprogram, ambientprogram, …) refer to images under other keys (spare1map, transmap). The demo does not use them.

Albedo is loosealbedo: 7 937 of the 7 938 values point to an image whose BIM header carries mtlkind 1, in every copy. The one remainder, models/characters/boss/boss_head.tga$mtlkind=albedo$streamed (Material in gameresources), exists in no archive. Retail thus provides a dangling reference itself. In the sample, additionally loosealbedo == lower(diffusemap) + "$mtlkind=albedo$streamed" holds (253/253).

2. image (BIM\x0e)

2.1 Header

From the loader 0x75FC60 and the writer 0x75EC60, all 0x32 bytes covered (BIM_FIELDS):

0x04 texture_type (2 = Cube)   0x08 mtlkind   0x0C w   0x10 h   0x14 depth
0x18 numLevels                 0x1D format (u32, unaligned)
0x2E u32 number of streaming sets   from 0x32: 32-byte sets
     level, face, w, h, raw, codec (0 raw / 1 Oodle / 2 tilechain), stored, offset
  • 0x18 is numLevels. The slot in .texdb equals numLevels − level. The census checks that for each of the 35,842 images, exactly numLevels × Faces lines with edges w>>level follow, and the inline chain ends at the file end. There are 0 deviations.
  • 0x2E is a u32: Writer 0x75F19E–0x75F1BC, Lines − Inline Lines, written with length 4. A line goes into .texdb if max(w,h) > 32, compared at 0x75EFC8 with the int value of the CVar image_streamingSeparatelyCompMip (0x20).
  • The remaining header bytes (0x1C, 0x21 … 0x2D) are uninterpreted. All 8,075 Retail-BC1_SRGB Albedos carry the same tuple, and exactly that is written by author_flat().

2.2 Formats: read from the engine, not copied

engine_formats() reads the names (jump table 0x784C54) and bits per pixel (BitsForInternalFormat, 0x783BA0) directly from module.bin. The raw size of all 382,114 Mip lines matches these bits.

BIM v14 in the 27 Retail Archives:

BC1_SRGB 15466   BC4 8302   BC5 7480   BC6H 2058   BC1 1382
BC3 876          BC7 256    R8 15      RGBA8 7

This includes 600 older BIM v10–v13 and 6 entries in mode 4, which are not decoded.

2.3 What we can write, and what we cannot

  • Flat color in BC1/BC1_SRGB: yes, exactly, without an encoder. Retail stores them the same way (c0 = c1, indices 0). author_flat() generates 22/22 Retail single-color images byte-identically.
  • Arbitrary pixels: no. We do not have a BC1/BC7 encoder and have not written one. Uncompressed only RGBA8 occurs (7 images, all mtlkind 0, one of which streams), plus R8 (15). No Albedo is uncompressed, RGBA8_SRGB never occurs. Whether the engine displays an RGBA8 Albedo correctly, including Gamma, is not evidenced.

3. The Lookup (0x7A18A0) and what Retail reveals about it

The order is: Memory databases, then all mounted .texdb via binary search, then as fallback GetEmbeddedStreamingFile (0x7A1BCC). If none finds anything, follow FatalError("Couldn't find mip %d for texture '%s'") (0x7A1E5D). I read this in this run from the Disassembly.

New measurements:

  • Sorting is unsigned. The search jumps with jae, and all 28 installed tables rise as u64 unsigned. Signed would be only 2 of 28 sorted, the two trivial ones with 0 and 4 lines. write_texdb sorts Python Ints, so unsigned.
  • Duplicate keys are legal. 24 of 28 tables contain them. Sample chunk_8, chunk_14, patch_1: 39 of 39 repeated keys point to byte-identical blocks. This was a one-time check and is not in the tool. The file branch (0x7A1AB0–0x7A1B77) runs over all equal keys and picks one. check_bundle allows repetitions but forbids descents.
  • Not every line is requested. In the 300-sample, 132 Streaming lines are missing in every installed .texdb (level 0: 101, level 1: 31). It is always an upper prefix, with no gap for any image. Since no Retail Archive contains a single rs_emb_sfile entry (one-time check over all tables), the fallback cannot deliver them either. Some upper limit prevents these peak Mips from being requested. Where this limit sits, I have not found it.
  • rs_emb_sfile: 32,502 of 32,502 streaming Retail images carry exactly this one dependency record, 3,238 pure inline images none, 102 inline images nevertheless one. It is generated in the build branch of resourcestorage.cpp (0xAC71E0: name "%s#%s#%x"). Afterwards there [res+0x28] (= +0x60) is linked per xor with a value of the new record; that is presumably part of the origin of +0x60, I did not pursue it further. In the Lookup only the fallback path reads it. After other readers I have not searched. Our demo image carries no Record.

4. Demo: re_probes/matpipe/demo_{streamed,inline}/base/chunk_31.*

Each variant contains two entries:

  • image matpipe/flat_magenta.tga$mtlkind=albedo$streamed: BC1_SRGB, (255,0,255), +0x60 = 0x0BADC0DEFACE0001 (upper nibble 0, no installed texture ID).
  • material matpipe/flat_magenta: a copy of the Retail material textures/tiles/wood/fine_wood_03_blend_01 (chunk_10), read at build time. Only diffusemap and loosealbedo are replaced. The other seven slots point to constant images in gameresources.
Variant Image .texdb
demo_streamed 256², numLevels 7, 3 Streaming sets 3 lines, Slots 5/6/7, 41/42/42 B, sorted
demo_inline 16², numLevels 3, all inline empty (32 B, like Retail patch_4_chunkbase_6.texdb)

The checks are deliberately separated:

  • "The lookup finds our block": check_bundle computes every key from the .resources and verifies sorting, length against stored, and collisions with all 28 installed .texdb. It also checks whether the loose* slots resolve.
  • "The block decodes to our pixels": decode_check (custom BC1 decoder, calibrated against Retail: red, green, blue, white, black, and yellow yield 6/6) and additionally wolfsdk/tncimage.py, the texture viewer of another worker. It has its own parser, its own key function, its own binary search, and its own BC1 decoder. Both find 87,376 and 336 pixels respectively, all Magenta. The preview from the viewer is located in demo_*/preview.png: a flat magenta square, viewed.
  • Retail Twins: Ten Retail images of identical form (format, mtlkind, size, numLevels, 0x2E) have 10/10 the same 0x32 header bytes as ours. The Mip table also matches, except for codec/stored/offset of the Streaming sets, which depend on the compressor. The 256² case has 288 Retail predecessors with exactly 7 levels and 3 Streaming sets.
  • The entry fields lie within the Retail distribution for image and material (desc −1, flags 0, version 14/0, nstr 2). This was a one-time check and is not part of the tool.
  • Both containers are deterministic. sha256 .resources: streamed e2caa496…, inline 68f65ca1….

Boundary: chunk_31 is mounted by nothing, and matpipe/flat_magenta is used by no surface. The demo proves the format. It becomes visible only when a model (meshfmt) or an entity (entitypipe) names the material and the map mounts the chunk (mapmanifest).

5. Runtime test for the user: re_probes/matpipe/recolor_c2v1/

To allow one to see something without the open parts of the other workers: --recolor takes a map chunk and replaces the payload of every map-local BC1_SRGB albedo with the 16² magenta image from demo_inline. Map-local means: The name is not in any base/*.resources. Entry, Name, and +0x60 remain; Materials and .texdb remain untouched. Because the new image is entirely inline, the engine reads no .texdb line for it. This exact form has Retail itself, e.g., textures/system/constant_color/yellow$mtlkind=albedo$streamed (16², 3 levels, 0x2E = 0, BC1_SRGB, mtlkind 1).

python tools/probe_matpipe.py --recolor D:/…/dlc/dlc_2/base/chunk_22.resources re_probes/matpipe/recolor_c2v1/base/chunk_22.resources
   replaced               218 map-local BC1_SRGB albedo images (not in base/*.resources)
   new payloads           218/218 parse, pass check_bimage and decode to (255, 0, 255)
   untouched payloads     6767/6767 byte-identical to the source
   finalize               data_hash 218 changed, hash_0x60 0 changed
   +0x50 6984/6984 (1 mode-4 entry skipped)   +0x08 1/1   +0x20 1/1
recolor clean

The build is deterministic: two runs produce the same sha256 1383bc5e….
The list of 218 images is in recolor_c2v1/recolored.txt, including office requisites, lamps, files, flags, and the Blitzmensch figure.
dlc_2/base/chunk_22.resources and dlc_4/base/chunk_22.resources are byte-identical (sha256 eccee9ec…). The file therefore fits into both folders.

Suggested test window (the user decides, the tool never writes under Steam): backup dlc_2/base/chunk_22.resources, copy the file from recolor_c2v1/base/ in there, start a DLC-2 card (c2v1/c2v2/c2v3 use this chunk), then restore the original.

  • If requisites are magenta, it loads and displays our BIM engine without .texdb.
  • If the game crashes or they appear black or with standard texture, that indicates the engine rejects our BIM; this is also a finding; the cause must be searched in the log.
  • demo_streamed additionally checks the .texdb path. For this, the chunk must be mounted and the material referenced (§4).

6. Negative controls (--negative, cases remain for --check)

Case (re_probes/matpipe/negative/<fall>/) Error --check
unsorted .texdb descending rc 1, "table not sorted"
top_nibble +0x60 with nibble 0xA, key (h & ~0xF) \| slot rc 1, "is not in"
slot_is_level Slot = level instead of numLevels − level rc 1, "is not in"
levels_plus1 Header 0x18 +1 rc 1, "record 7 is level …"
levels_minus1 Header 0x18 −1 rc 1, Inline chain breaks
streamed_minus1 Header 0x2E −1 rc 1, "0x2E says 2 …"
short_block Block 1 byte shorter than stored rc 1, "holds 41 bytes, … announces 42"
retail_id +0x60 = Retail texture ID rc 1, "collides with a retail .texdb id"
dangling_albedo loosealbedo to empty rc 1, "which is no image entry"
badpixel an inline block green --check rc 0 intended; only decode_check catches it
anchor_hinibble (counter-anchor) upper nibble set, key correctly masked rc 0

An untrimmed (h << 4) \| slot does not fit into the 64-bit field. write_texdb rejects it. Outside my tool: a flipped byte in the demo container causes probe_datahash to fail with 1/2 and rc 1.

7. Corrections compared to the abandoned state

  • dlc_4 is not Retail. It is the user's copy of dlc_2/chunk_22 (sha256 identical). The old census counted it as (37,304 images); the correct number is 35,842 in 27 archives. For collisions, all 28 installed .texdb files still count, because the engine searches every mounted one.
  • The length cross-check ran only against its own .texdb. Now it runs against all. Of the previously 163 "missing" sentences, 31 lie with correct length in another .texdb; the remaining 132 in none (§3).
  • Materials are now checked across the entire corpus (11,469), and the one dangling Retail reference is named.
  • New: Sorting order and duplicates of .texdb, Retail twins, the second reader, thumbnails, --check, --recolor.

8. Open

  • No runtime proof, for nothing here. §5 is the prepared test.
  • Which surface matpipe/flat_magenta uses belongs to meshfmt or entitypipe. How a new chunk mounts belongs to mapmanifest (packagemapspec.json). Whether a second container overwrites a same-named Retail entry is not documented. --recolor bypasses the question by replacing entries in place.
  • The upper limit that prevents requesting never-delivered mip levels is not localized. For 256² images it is presumably irrelevant, but this is not documented.
  • No BC encoder, so no arbitrary pixels (§2.3).
  • Header fields 0x1C, 0x21–0x2D are unexplained; they are Retail constants.

9. Reproduction (from repo root directory)

python tools/probe_matpipe.py --material 300   # 300/300, 2024/2024, 253/253
python tools/probe_matpipe.py --image 300      # 300/300, 1576 + 31 ok, 0 wrong, 28/28 unsigned
python tools/probe_matpipe.py --formats        # 35842 v14 0 problems; materials 11469/11469
python tools/probe_matpipe.py --author         # 22/22, 6/6
python tools/probe_matpipe.py --demo           # beide Varianten clean, Zwillinge 10/10
python tools/probe_matpipe.py --negative       # caught 10/10, anchors ok
python tools/probe_matpipe.py --check re_probes/matpipe/demo_streamed/base/chunk_31.resources
python tools/probe_datahash.py re_probes/matpipe/demo_streamed/base/chunk_31.resources
python tools/probe_idclhash.py re_probes/matpipe/demo_streamed/base/chunk_31.resources
python tools/probe_idclhash.py re_probes/matpipe/recolor_c2v1/base/chunk_22.resources