Wolfenstein Modloader Buy me a coffeeKaffee spendieren

Wiki / Research reportsForschungsberichte

entry+0x50 (data_hash) and entry+0x60 (hash_0x60) — from the code

SolvedGelöst Original page in the GitHub wikiOriginalseite im GitHub-Wiki

In plain wordsIn einfachen Worten

Every file entry in a Wolfenstein II: The New Colossus archive carries two 64-bit values. This report reads the game code and shows that the first one is a MurmurHash64B hash of the unpacked content with a fixed seed, which a mod tool can now compute. The game only checks it when a hidden setting is on (off by default), and then a wrong value causes a crash. The second value is never recalculated from the content by the game, so builders just need to keep it consistent, meaning self-built archives with changed or new files are possible.

  • Formula confirmed on 4,499 of 4,499 sampled entries from five archives.
  • The check runs only when resourceStorage_checkDataCheckSum is on; its default is 0 (off). With it on, a wrong value is a hard crash.
  • 7 of 7 deliberately wrong variants fail as expected.
  • The second value equals the first for almost all normal entries (59,630 of 59,662); for images and a few other types it is a separate build value whose origin is unknown.
  • An earlier guess that the second value is the texture database key was measured and is wrong (only about 5% match).

Jeder Dateieintrag in einem Archiv von Wolfenstein II: The New Colossus trägt zwei 64-Bit-Werte. Dieser Bericht liest den Spielcode und zeigt, dass der erste ein MurmurHash64B-Hash über den entpackten Inhalt mit festem Startwert ist, den ein Mod-Werkzeug jetzt berechnen kann. Das Spiel prüft ihn nur, wenn eine versteckte Einstellung an ist (standardmäßig aus), und dann führt ein falscher Wert zum Absturz. Den zweiten Wert berechnet das Spiel nie aus dem Inhalt nach; ein Werkzeug muss ihn nur stimmig setzen. Eigene Archive mit geänderten oder neuen Dateien sind damit möglich.

  • Formel bestätigt an 4.499 von 4.499 Stichproben-Einträgen aus fünf Archiven.
  • Die Prüfung läuft nur, wenn resourceStorage_checkDataCheckSum an ist; Standardwert ist 0 (aus). Ist sie an, führt ein falscher Wert zum harten Absturz.
  • 7 von 7 absichtlich falschen Varianten scheitern wie erwartet.
  • Der zweite Wert ist bei fast allen normalen Einträgen gleich dem ersten (59.630 von 59.662); bei Bildern und wenigen anderen Typen ist er ein eigener Build-Wert unbekannter Herkunft.
  • Die frühere Annahme, der zweite Wert sei der Schlüssel der Texturdatenbank, wurde gemessen und ist falsch (nur etwa 5 % Treffer).

Technical detailsTechnische Details

Der technische Teil ist auf Englisch – so, wie er im Wiki steht. Unterstrichene Begriffe haben eine Erklärung auf Deutsch.

Short answer to Question 4: Yes. A self-built Chunk can contain MODIFIED and NEW Payloads and still mount and load — provided the Builder recalculates entry+0x50 (formula below, 4499/4499 verified) and sets entry+0x60 consistently. No single verification path compares +0x60 with newly calculated content.

A limitation that is not a mount-blocker but should be documented: for image entries with $streamed, +0x60 does not provide the TexDB key — the previous assumption hash_0x60 << 4 | level is measurably false (Section 5). Textures therefore remain a separate, unsolved problem, but not because of +0x50/+0x60.


1. entry+0x50 — solved

data_hash = murmur64b(uncompressed_payload, seed=0xDEADBEEF)
  • Algorithm: MurmurHash64B (the 32-bit variant with two accumulators), m = 0x5BD1E995, r = 24, h1 = seed ^ len, h2 = 0, result (h1 << 32) | h2. Function .text 0x12BB970–0x12BBA77 (.pdata boundaries), reconstructed instruction-by-instruction in tools/probe_datahash.py::murmur64b().
  • Which stream: over the decompressed Payload.
  • Length: usize (entry+0x48), specifically only the lower 32 bits — mov edx, [rbx+0x48].
  • What else flows in: nothing. No name, no type, no offset, no metadata.

The evidence is in the Reader, not the Writer. WriteContainer (0xAC9BD0–0xACADD4) fills the 144-byte entry in the loop at 0xACA340 with two 16-byte moves from a generator structure:

aca3a2  movups xmm0, [r13+0x10]      ; -> entry+0x48 (usize) and entry+0x50
aca3a7  movups [r12+0x48], xmm0
aca3ad  movups xmm1, [r13+0x20]      ; -> entry+0x58 (time) and entry+0x60
aca3b2  movups [r12+0x58], xmm1

The Writer copies both fields and proves nothing via the algorithm. The derivation comes from idResourceStorage::ReadResourceData, .text 0xACD7C0–0xACDCFD:

acda76  movzx eax,[r8+0x70]          ; compression
acda7b  test  al,al
acda7d  je    0xacdb19               ; mode 0: raw buffer is already the payload
        ... Allocation of entry+0x48 bytes, call 0xAC64F0 (Oodle) ...
acdb16  mov   r15, rdi               ; r15 = DECOMPRESSED buffer
acdb19  test  r12b, r12b             ; 3. Argument "verify"
acdb1c  je    0xacdb64               ; ... otherwise skipped
acdb1e  mov   rbx,[rsi+0x1a8]        ; Entry table
acdb25  add   rbx, r14               ; + 144*index
acdb28  mov   r8d, 0xdeadbeef        ; Seed
acdb2e  mov   edx,[rbx+0x48]         ; Length = usize
acdb31  mov   rcx, r15
acdb34  call  0x12bb970              ; MurmurHash64B
acdb39  mov   rcx,[rbx+0x50]         ; stored data_hash
acdb3d  cmp   rax, rcx
acdb40  je    0xacdb64               ; ok
        -> Warning "resource %s:%s data integrity checksum failure,
                    has: 0x%llx, wants 0x%llx" and return NULL

2. Is +0x50 checked on load? — Yes, but only with CVar enabled

The check block is hooked to the third argument. There are exactly three callers of 0xACD7C0:

Caller 3rd Argument Meaning
0xACD761 (Wrapper 0xACD730) cvar != 0 the normal load path
0xACE430 1 Verify pass 0xACE3F0–0xACE6C0
0xACE5D8 1 same Verify pass
acd754  cmp   dword [rip+0x2485dc5], 0    ; -> .data 0x2F53520
acd75d  setne r8b
acd761  call  0xacd7c0

.data 0x2F53520 is offset +0x60 of the CVar object at .data 0x2F534C0 (object size 0x90). Its pointers at +0x78/+0x80/+0x88 read in runtime capture:

+0x78 -> .rdata 0x26D02F8  "resourceStorage_checkDataCheckSum"
+0x80 -> .rdata 0x25FC350  "0"                       <- Default value
+0x88 -> .rdata 0x26D02C8  "check resource data checksums on resource load"

In the running process shown in the capture, it is 0. The static constructor 0x146B8E also writes 0 into it.

When checked: not at mount, but on every access to the payload of an entry — precisely in the function that retrieves and unpacks the bytes from disk. An entry whose payload is never read will also never be checked.

What happens on mismatch — and this is sharper than a warning: ReadResourceData returns NULL. The wrapper 0xACD730 catches this:

acd766  test rax, rax
acd769  jne  ok
        ... retrieve names from the entry table ...
acd7af  lea  rcx,[rip+0x1c03a2a]     ; "Get entry of %s:%s from %s failed"
acd7b6  call 0x12386b0               ; FatalError
acd7bb  int3

So: CVar out (Retail-Default) → no test. CVar in → a false +0x50 is a hard crash, not a warning. Both statements hold true; they do not contradict each other.

3. Positive and Negative Controls

re_probes/datahash/derive_retail.txt:

chunk_8.resources                    900/900    (0 of 1562 skipped)
gameresources.resources              900/900    (0 of 50839)
patch_4.resources                    900/900    (0 of 2649)
chunk_20.resources                   899/899    (1 of 13366)
chunk_22.resources                   900/900    (0 of 6985)

+0x50  4499/4499

4499/4499 across five archives from three build stages: Retail base chunk, gameresources, a patch, and two DLC chunks. Sample size of 900 entries per archive, random selection with a fixed seed (reproducible). The one skipped entry is compression mode 4, for which we have no decoder.

re_probes/datahash/negative_controls.txt — every line must fail:

flipped byte 0 of the payload                     0/20 failed as expected
flipped last byte of the payload                  0/20 failed as expected
compressed stream instead of uncompressed         0/20 failed as expected
truncated to csize instead of usize               0/20 failed as expected
seed 0 instead of 0xDEADBEEF                      0/20 failed as expected
seed 0xDEADBEEF but payload read at offset+16     0/20 failed as expected
data hash used for entry+0x60 (image)             0/5  failed as expected

The two controls required by the order (a flipped byte, false stream) are listed below; python tools/probe_datahash.py --negative yields an exit code ≠ 0 if either of them somehow passes.

4. entry+0x60 — "default hash"

The name is not guessed; it appears in the binary. Two format strings in the container comparison function 0xACE750–0xACE8E9 name both fields:

.rdata 0x26D1978  "entries has different data hash: %llx - %llx"      <- +0x50
.rdata 0x26D18F0  "entries has different default hash: %llx - %llx"   <- +0x60

Empirically (re_probes/datahash/hash60_survey.txt, all entries, not just a sample):

chunk_8          1562 entries   +0x60==+0x50: 1384   differing: 178
gameresources   50839            +0x60==+0x50: 39220  differing: 11619
patch_4          2649            +0x60==+0x50: 2635   differing: 14
chunk_20        13366            +0x60==+0x50: 12093  differing: 1273
chunk_22         6985            +0x60==+0x50: 5689   differing: 1296

The differences are almost entirely restricted to four types: image (11545 of 11619 in gameresources), extkisclule, cfile, compfile. Across all five archives combined, excluding these four types:

59662 entries   +0x60 == +0x50: 59630   differing: 32

The 32 exceptions are exclusively cm entries with empty payloads (usize == csize == 0, version 21); they carry the sentinel value 1 at +0x60, while +0x50 correctly holds murmur64b(b"", 0xDEADBEEF) == 0x42F683B123F895E6 — which incidentally confirms the derivation on the edge case of length 0.

Origin of the Value: It is assigned by the build tool, not derived. For types where +0x60 == +0x50, it is simply the data hash. For image entries, it is a distinct value that is not present in the payload (byte search for LE-u64 in unpacked .bimage data: 0 hits in 6 of 6 samples) and cannot be reproduced as a hash of the name (sweep over name, name+NUL, name before $, type:name, lowercase × MurmurHash64B/FarmHash64/CityHash64 × raw/>>4/<<4: 0 of 40 hits).

Who reads +0x60 — complete list, generated by a scan over .text that tracks every register chain mov/add R, [X+0x1a8] (entry table) plus 144-indexing and logs all field accesses:

RVA What happens there
0xAB6DF7, 0xAB6FEC Copy to resource+0x28 (runtime object)
0xAB6A8D Getter, writes to [rdi]
0xAB8402, 0xAB841F Resource_GetEmbeddedStreamingFile: cmp r9, entry+0x60, where r9 = res+0x28, so the same previously copied value
0xABC2B9 Output
0xAC94AE, 0xAC9563, 0xAC95A6 Generator: writes +0x60 as hash into Dependency Records (call 0xAC78A0)
0xACD18C, 0xACD665 Stale-Check 0xACCC70: cmp entry+0x60, dep+0x18
0xACE3A8 Same, Dependency-Iteration 0xACE215
0xACE836 Container-Diff (Dev tool)

This is the critical point: every single comparison location compares entry+0x60 against a different stored field — against the hash of a Dependency Record (dep+0x18), against the previously copied runtime value from the same field, or against +0x60 of a second container. No location calculates +0x60 from content afterwards. A builder that enters the same value at both ends passes through.

For +0x50, the same scan yields exactly seven read locations, four of which are false positives in foreign structures; the real ones are 0xACDB39 (the test from Section 1) and 0xACE7D2/0xACE7D6 (Container-Diff, Dev tool).

5. Correction: +0x60 is not the TexDB Key

The task specifies as preliminary finding TexDB-Key = hash_0x60 << 4 | level. This does not hold up to measurement. Across all 24 base .texdb files, there are 24,713 different texture IDs (key >> 4); the comparison against the image entries of the same chunks:

chunk_8        138 images   hash_0x60 as TexDB-ID:    4
gameresources 12194 images  hash_0x60 as TexDB-ID:  650
chunk_1         732 images  hash_0x60 as TexDB-ID:   38
chunk_16        357 images  hash_0x60 as TexDB-ID:   20

Approximately 5%, not 100%. Notable is that chunk_8.texdb has exactly 106 different IDs and chunk_8.resources has exactly 107 $streamed images — the 1:1 mapping exists, but +0x60 is not their key. Control: +0x50 appears as a TexDB-ID in 0 cases, so the hit was not generated by counting. Reproducible with python tools/probe_datahash.py --hash60.

This leaves the origin of the TexDB key open; it does not belong to this task, and re_probes/agent_reports/texdb_rewrite.md had already marked it as unsolved. This measurement confirms that assessment and refutes the temporarily assumed formula.

6. What was not found (honestly labeled)

  • No Reader for +0x50 at Mount. Searched in idResourceStorage::OpenContainer 0xACB780–0xACC441 (fully disassembled; the function copies only the 0x78 header into the Container object at +0x140 and holds the metadata block, it does not iterate the entries) and via the described .text-wide field scan. Therefore: no Mount-time Reader found; the search was conducted in OpenContainer and across the full .text scan of the entry table.
  • No derivation for image +0x60. Attempted and failed: Byte search in the unpacked payload (Offsets 0..usize-8), Hash over the name in six variants × three hash functions × three shift forms. Therefore: Origin unknown, not "arbitrarily selectable because unverified" — it is freely selectable because no Reader recomputes it (Section 4), not because we failed to find it.
  • Compression Mode 4 (25 entries total in the game) remains undecoded; consequently, one entry from the sample was missed. The code paths at 0xACD8DB show that Mode 4 runs via a separate streaming path for usize > 0x2000000, which never reaches ReadResourceData — thus the checksum test does not apply there anyway.

7. Guide for the Builder

  1. Write payload (raw or Kraken-compressed).
  2. entry+0x40 = csize, entry+0x48 = usize.
  3. entry+0x50 = murmur64b(uncompressed_bytes, 0xDEADBEEF) — from tools.probe_datahash import murmur64b.
  4. entry+0x60:
    • Type ≠ image/extkisclule/cfile/compfile: enter the same value as +0x50. This is exactly what the Retail Generator does (59630 of 59662 across five archives; the 32 exceptions are empty cm payloads with the sentinel 1).
    • Type image and similar: either adopt the Retail value (if the name is unchanged), or choose freely — but then enter the same value at dep+0x18 in every Dependency Record pointing to this entry, otherwise the Stale Check reports "...is stale: embedded resource %s:%s hash mismatch".
  5. Finalize metadata block, then header[0x20], then header[0x08] per re_probes/agent_reports/idclhash.md. Both entry fields flow as bytes into the metadata, so finalize the entry table first, then the header hashes.

8. Artifacts

  • tools/probe_datahash.py — Derivation, Positive/Negative controls, +0x60 collection, TexDB comparison (Py 3.11, stdlib + wolfsdk.oodle)
  • re_probes/datahash/derive_retail.txt — 4499/4499
  • re_probes/datahash/negative_controls.txt — 7 controls, all failed
  • re_probes/datahash/hash60_survey.txt — +0x60 collection and TexDB comparison
  • re_probes/datahash/disasm_datahash.txt — Disassembly evidence (Writer, Reader, Gate, Hash function, all comparison points, strings)

9. For Other Owners

wolfsdk/idcl.py (foreign Ownership) still contains in its Docstring "data_hash is content-derived ... but it matches none of crc32 ... MurmurHash64A ..." and "A written entry therefore keeps the hash it had." The Sweep tested MurmurHash64A, not 64B, and did not test a Seed ≠ 0 — hence the failure. This is now resolved: +0x50 is computable, and Entry.hash_0x60 may be set to data_hash for all non-image types.