IDCL-Writer: Write the four hashes instead of copying

Worker w-idclwriter-a1, 2026-09-23. New: wolfsdk/idclwrite.py, tools/probe_idclwriter.py, re_probes/idclwriter/.

Result in one sentence

The Writer writes three Retail containers byte-for-byte identical and recalculates all four hashes during this process; the defective Bundle container of the other Master session is repaired and drops from 7123/7127 to 7127/7127 without a single Payload byte changing.

1. The Defect, Reproduced Before Every Change

python tools/probe_datahash.py re_probes/map_alias_v3/bundle_headerfixed/base/chunk_22.resources
   BAD compfile:maps/game/dlc/c02/c2v1_custom.entities            comp=0 usize=256319
   BAD extkisclule:.../sacramento_gameplay_kiscule_combat_sim_screen_fx_893872923.decl
                                                                  comp=0 usize=1643
   BAD json:maps/game/dlc/c02/c2v1_custom/devmapinfo.json         comp=0 usize=304
   BAD file:generated/buildgame/game/dlc/c02/c2v1_custom.mapresources
                                                                  comp=2 usize=2150169
chunk_22.resources  7123/7127  (skipped 1 of 7128 entries, mode 4)

Exactly the four Payloads generated by the Bundle itself, exactly as described in the order. They carry the +0x50 of their template. With resourceStorage_checkDataCheckSum set to the default 0, no one reads it; with 1, it is a FatalError -- a latent crash, not a cosmetic error.

re_probes/map_alias_v3/ was only read. The repaired container lies in re_probes/idclwriter/chunk_22.resources.

2. The Writer

wolfsdk/idclwrite.py, stdlib plus the existing Oodle integration. No compressor added -- it is not needed for this repair either, see Section 5.

The order is the only one that works, and it is documented in the module:

1. Entry +0x50 = MurmurHash64B(decompressed Payload, Seed 0xDEADBEEF), length from usize at +0x48.
2. Entry +0x60 according to the rule from Section 4, then follow dep+0x18.
3. Header +0x20 = FarmHash64 over the metadata +0x78 up to and including the IDCL marker -- contains entry and dependency tables, so must wait until (1) and (2) are done.
4. Header +0x08 over the finished header. +0x20 lies within header[0x10:0x78], therefore last.

The hash implementations are imported from tools/probe_datahash.py and tools/probe_idclhash.py, not rewritten: an implementation of a carrying hash, not two that can diverge.

3. The Round-Trip -- the Carrying Control

python tools/probe_idclwriter.py --roundtrip. Read Retail, write unchanged back, compare bytes. This can fail in two independent ways: the layout can deviate, and every newly calculated hash must land on the value that the delivered file already carries.

3/3 byte-identical, with 4,453 of 4,454 entry data hashes plus 3x +0x20 and 3x +0x08 recalculated and exactly matched. SHA256 of the sources (and thus the outputs):

3937cfe5c1f9ec03a7a68d90a0491f05257add72f3b2c9acd212544765396fab  patch_4_chunkbase_6
2f01439bf3854b544079c31415fadad3ebfeef2dc9dcec25560cfb8b4480aa35  chunk_8
2dac9300fdacefa1c7308c5a06e7dfabade431b2d3eccaacf2e7da5bdc862325  chunk_16

The driver additionally checks that finalize() moves nothing in an unchanged file (data_hash/hash_0x60/deps all 0) -- otherwise an identical file with secretly shifted fields would be visible. The copies are deleted after comparison; a byte-identical second build of 125 MB Retail adds nothing that the digest does not already state (--keep preserves them).

4. Two Measurement Corrections in the Docstring of wolfsdk/idcl.py

Both concern assumptions on which the Writer must act, so they are measured here rather than copied.

(a) +0x60 also deviates for type cm. The docstring lists image/extkisclule/cfile/compfile. Across four archives (chunk_8, chunk_16, gameresources, Retail-chunk_22, 62,293 entries), an additional 21 cm entries deviate (gameresources 16, chunk_22 5). The Writer therefore ignores the type entirely: +0x60 follows +0x50 only where it was already equal to it beforehand, and otherwise stops. This is a No-Op by construction on an unchanged container -- precisely what causes the Round-Trip to fail.

(b) dep+0x18 applies only to the 64-bit Records of the +0x60 mirror. The statement "the same value must be in dep+0x18 of every Record that references the entry" holds for Records that carry a 64-bit value at all -- all 233 locally resolvable skeleton references in gameresources match it. It does not hold generally:

The 3,508 are material/renderParm/sound and carry 32-bit values unrelated to +0x60 and never had them. All dep_type=5 Records (11,545 in gameresources, 1,214 in chunk_22) are rs_emb_sfile with names of the form image#path.tga$mtl and resolve to no entry of the archive. Propagating solely over (Type, Name) would thus destroy these Records. The Writer updates only Records that actually carried the old value. Changes to +0x60 on a (Type, Name) shared by multiple entries (860 renderProgResource pairs per archive) are not propagated at all: which entry is meant cannot be determined, and guessing writes an incorrect hash into foreign Records. In chunk_22 there are 0 such pairs.

5. The Repair

python tools/probe_idclwriter.py --repair

stale +0x50 in the source: 4
   [7079] compfile:maps/game/dlc/c02/c2v1_custom.entities
          44391c8b57bc24be -> 7eb399dc78f13f40
   [7113] extkisclule:.../sacramento_gameplay_kiscule_combat_sim_screen_fx_893872923.decl
          f27e3773c8834680 -> bfd0ad86e7e503de
   [7116] json:maps/game/dlc/c02/c2v1_custom/devmapinfo.json
          c443cc8e7041f11b -> 315df24b4cd12bec
   [7125] file:generated/buildgame/game/dlc/c02/c2v1_custom.mapresources
          43b978e93afe35df -> 01e4556138289e61
{'entries': 7128, 'skipped': 1, 'data_hash': 4, 'hash_0x60': 2, 'deps': 0}
stale +0x50 afterwards: 0     header +0x08 OK   +0x20 OK     stale dep+0x18: 0

hash_0x60: 2 are the json and file entries whose +0x60 was equal to +0x50; for compfile and extkisclule it deviates and stops (Section 4a). deps: 0, because none of the 1,943 dependency records referenced one of the four entries -- measured, not assumed.

Header: +0x08 c729da551d3803c1 -> 96ce3ba05494deae, +0x20 500bb10c8f8f27f2 -> 98564e06dfa20da7.

sha256 Source 95236c3ebcaffb73a0f367bedbec71b720537677c3ab2aa3d87bd8cafd5fe2c2 sha256 repaired b41f25135bd3374e594a7c37fe9896d9f66966c75351340b15bd76891bb945f4 151 702 019 Bytes, unchanged

No compressor is involved. The four payloads are already what they should be -- the only error was the hash above them. The repair copies the file and rewrites exclusively the header and table entries. The question "can the original codec be reproduced" does not arise for this container; idcl.pack_payload (Oodle-DLL) is used only when a caller passes changes, which here happens only in Control 3 on a disposable copy.

6. Evidence instead of Assertions

Payload Identity (--payloads), Source vs repaired, all 7 128 entries individually:

payloads identical 7128/7128
entry fields changed: {'0x50': 4, '0x60': 2}
dependency fields changed: none
header fields changed: ['0x8', '0x20']

This is more than required: not only are the payloads identical, but also every other field of the entry table, the complete dependency block, and the rest of the header.

Independent Verifiers on the repaired container:

python tools/probe_datahash.py re_probes/idclwriter/chunk_22.resources
   -> 7127/7127   (1 entry Mode 4, no decoder)
python tools/probe_idclhash.py re_probes/idclwriter/chunk_22.resources
   -> +0x08 1/1   +0x20 1/1

Negative Controls (--negative), each with Positive Anchor, 6/6 caught:

Control 3 is real, not simulated: the payload of skeleton:models/mapobjects/c04p2/c04p2_elevator_piston/… in chunk_8 is actually changed, recompressed, and rehashed; dep#8 carries its +0x60.

7. What Remains Open

• One entry per container with compression mode 4 remains unchecked. In chunk_22 this is staticShadowGeom:maps/game/dlc/c02/c2v2/_combo/_world.bshadows (usize 29 227 487). For the 12-byte prefix there is no decoder, so its +0x50 cannot be recomputed or verified. It comes unchanged from the Retail Archive and is therefore presumably correct, but that is a conclusion, not a measurement. With the set Cvar it is the only entry of the container for which this work makes no statement. finalize() leaves it intentionally in place, rather than guessing.
• +0x60 for image/cm/cfile/compfile/extkisclule remains underived. The Writer conserves the value. For a newly invented entry of these types there is thus no correct +0x60 -- it is checked by no read code path, but whoever chooses it freely must keep dep+0x18 consistent themselves.
• re_probes/idclwriter/ is excluded via .gitignore (re_probes/*). Intentionally not released: the content is a 151-MB binary artifact that is reproducible at any time from source plus tools.
• The Round-Trip ran on 3 of 24 Retail Archives (401 KB to 97 MB). gameresources.resources (702 MB, 50 839 entries) was not round-tripped due to runtime reasons; the +0x60 and dep+0x18 collections from Section 4 include it instead.

8. Reproduction

python tools/probe_idclwriter.py              # all: Round-Trip, Repair,
                                              # Payload Identity, Controls
python tools/probe_idclwriter.py --roundtrip --keep
python tools/probe_idclwriter.py --repair
python tools/probe_idclwriter.py --payloads
python tools/probe_idclwriter.py --negative