osyu/gt_docs.md

Last active October 16, 2024 19:24

Star (3) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/osyu/895c38cf77f08853353988780e52dfc0.js"></script>
Save osyu/895c38cf77f08853353988780e52dfc0 to your computer and use it in GitHub Desktop.

Download ZIP

Ghost Trick remaster docs

Raw

gt_docs.md

(guides assume you're on Windows)

How to extract game assets

short answer

download REtool
download Ghost Trick filepath list
retool -h GTPD_PC_Release.list -x re_chunk_000.pak

long answer

primer:

Ghost Trick runs on Capcom's RE Engine. REtool is an unofficial tool for dealing with the engine's file formats
all resources are stored in a big file named re_chunk_000.pak located in the game's installation directory (which is accessible in steam through [right click] > Manage > Browse local files)
unlike a typical archive format like .zip/.rar/.7z, .pak files don't have filenames plainly stored in a way that REtool can reliably retrieve, so you also need to give it a list of names to read from so it knows what to name the things it's extracting. if you don't, you'll be stuck with a bunch of files without discernable names and your life will become harder

what to do:

download RETool from the link above and extract REtool.exe to the game's installation directory (the same folder as re_chunk_000.pak)
download the filepath list from the link above, into the same folder
open a command prompt in that folder. there are a billion ways to do this but the easiest way is typing cmd into the address bar in file explorer
type retool -h GTPD_PC_Release.list -x re_chunk_000.pak to extract everything. when it's done there'll be a folder named re_chunk_000 with all the stuff
you'll notice there are still hundreds of files without proper names (that look something like 15349567-2389623703.bin). this is normal and most of those files are unimportant/not unique to Ghost Trick, so you can ignore them. it happens because the list we passed to REtool has most but not all filenames, so anything missing will end up with a name like that

How to convert game assets to common formats

(everything here assumes you've just done all the things above)

.tex (textures)

type for /r %f in (*.tex.*) do retool -tex %f. this will recursively convert every .tex file to .dds, which is readable by image editing programs like paint.net

.asrc (audio)

they're just .wav files with an extra header in front. you have two options:
- if you have Python installed, run this script to convert to and from the format
- open up a hex editor (e.g. HxD) and remove the first 78 (0x4e) bytes, then change the file extension to .wav

.mov (video)

they're just .wmv files. change the file extension to .wmv and you're set
except for natives/stm/streaming/movie/igt_op_480x320_500kbps.mov.1, that's an .mp4

.msg (text)

download REMSG_Converter, and type for /r %f in (*.msg.*) do remsg_converter %f to convert recursively to .csv, which you can open in excel or any text editor. add -m json to use json instead

.oft (fonts)

these are encrypted OpenType and TrueType font files. you can encrypt/decrypt them using REE.FontsCryptor from the REE.PAK.Tool repository (as of writing binaries aren't provided, you'll have to build it yourself)

.user/.scn/.pfb (objects, scenes, prefabs)

you can use the RE_RSZ 010 Editor template to read and edit these

Where the hey is everything

(this is a non-exhaustive list)

textures

facepics/portraits: natives/stm/charfiles/3d/08_face
backgrounds: natives/stm/charfiles/2d/02_bg
character animations: natives/stm/charfiles/2d/01_human
object animations: natives/stm/charfiles/2d/03_gimmick
ui: natives/stm/charfiles/3d/06_id

audio

bgm:
- remastered: natives/stm/streaming/sound/bgm_new
- original: natives/stm/streaming/sound/bgm_resource
sound effects: natives/stm/streaming/sound/se_resource

video

cutscenes: natives/stm/streaming/gui/ui010500/tex
unused trailer: natives/stm/streaming/movie

text

it's all in natives/stm/gamedesign/text

fonts

they're in natives/stm/gui/font/outlinefont

Internal names

stages

Junkyard - st01
Pigeon Man's Office - st02
Prison - st03
The Chicken Kitchen - st04
Lady's Red Apartment - st05
Lynne's Apartment - st06
Kamila's Old House - st07
Office of the Troubled Man - st09
Drive to Chicken Kitchen - st10
Special Investigation Unit - st11
Temsik Park - st13
The Yonoa - st14
Prologue & Credits - st15

characters

Sissel (human) - cicel
Sissel (cat) - neko (猫, lit. "cat")
Lynne - linne
Ray - kuneri
Nearsighted Jeego - killera
Commander Sith - sisu
Masked Muscleman - migi (右, lit. "right")
"One Step Ahead" Tengo - killerb
Kamila - girl
Missile - pome ("pome[narian]")
Emma - fujin (婦人, lit. "madam")
Amelie - eimin (永眠, lit. "death")
Justice Minister - daijin (大臣, lit. "minister")
Detective Mccaw - keijig (刑事, lit. "detective g[reen]")
The Blue Detective - keijib (刑事, lit. "detective b[lue]")
Odd Blue Doctor - doc
Inspector Cabanela - kaba
Detective Rindge - caron
Guardian of the Park - birakubari (ビラ配り, lit. "handing out leaflets")
Pigeon Man - kanri (管理, lit. "management")
Chicken Kitchen Cook - cock
Typical Cop - keikan (警官, lit. "policeman")
Officer Bailey - keibia (警備, lit. "guard a")
Rock Jailbird - rock
Curry-loving Jailbird - shujinb
Detective Jowd - jodo
SIU Chief - bucho (部長, lit. "department head")
Memry - waitress
Dandy - yukaib
Beauty - yukaia
Chicken Kitchen Bartender - barten ("barten[der]")
Alma - mama
Minor Crew Hand - zako (雑魚, lit. "small fry")

Modding tools

Fluffy Mod Manager - mod manager for RE Engine games (for modifying assets)
REFramework - scripting and mod framework (for modifying game logic)

niltwill commented Jan 28, 2024

I tried to run asrc31.py through all the audio files of AJ:AA Trilogy, and there are certain ones that it cannot handle. There are two errors, depending on the files.

These are the offenders with the "smpl chunk size" error. AssertionError: assert read_u32(f) == 44 # smpl chunk size

natives\stm\streaming\sound\common\se\get_medal_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_bass_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_click_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_drums_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_keyboards_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_leadguitar_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_rhythmguitar_1648.asrc.31
natives\stm\streaming\sound\gs5\se\script_com\bikkuri.asrc.31
natives\stm\streaming\sound\gs5\se\se_cr003\kokone_pc_tap_flick.asrc.31
natives\stm\streaming\sound\gs5\se\se_cr203\biyoin_chakushin_both.asrc.31
natives\stm\streaming\sound\gs5\se\se_cr203\biyoin_chakushin_hidari.asrc.31
natives\stm\streaming\sound\gs5\se\se_cr203\biyoin_chakushin_migi.asrc.31
natives\stm\streaming\sound\gs5\se\se_cr207\shoko_performance_ken_hikaru.asrc.31
natives\stm\streaming\sound\gs6\se\yamashinop\yamashinop_coin_break_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\animationstudio\se\close_anim_studio_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\animationstudio\se\enter_anim_studio_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\animationstudio\se\se_court_000.asrc.31
natives\stm\streaming\sound\startupscreen\museum\artlibrary\se\close_art_library_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\artlibrary\se\enter_art_library_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\artlibrary\se\paper_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\artlibrary\se\screendown_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\artlibrary\se\screenup_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\orchestrahall\se\curtain_close_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\orchestrahall\se\curtain_open_1648.asrc.31
natives\stm\streaming\sound\startupscreen\se\entering_titlesel_1648.asrc.31
natives\stm\streaming\sound\startupscreen\se\enter_startup_screen_1648.asrc.31

Also here are some others below that output this AssertionError: assert read_u32(f) == 0xffffffff # always -1

natives\stm\streaming\sound\gs5\stream\all\se301_ikuya_yorokobi_rap_strm_lp.asrc.31
natives\stm\streaming\sound\gs5\stream\all\se302_ikuya_ikari_rap_strm_lp.asrc.31
natives\stm\streaming\sound\gs6\stream\se\reifa_v_ima_1644.asrc.31
natives\stm\streaming\sound\gs6\stream\se\reifa_v_minanomono_1644.asrc.31
natives\stm\streaming\sound\gs6\stream\se\reifa_v_sisosamano_1644.asrc.31
natives\stm\streaming\sound\gs6\stream\se\reifa_v_tokutomiyo_1644.asrc.31

For reference (correct this if I'm wrong), the offsets for the asrc.31:

Block(h) 0-3: Has to be "srcd"
Block(h) 8-A: WAV file's size (without the header)
Offset(h) F: Always 32? (default 32-Bit float depth?)
Offset(h) 10: bgm(?) [as it doesn't always appear for BGMs only, my guesses... 1. audio data behaviour: preload audio data (to make sure it plays as soon as the scene appears/starts?)=1/load in background (it's not prioritized, it can wait to load)=0, 2. normalize the audio (to make sure it's not overly loud/clipping)=1, do not normalize it (use source audio as is)=0]
Block(h) 14-17: file id
Block(h) 18-1B: unknown0 (secondary id used for something else?)
Offset(h) 1C: channels? (2 = stereo, 1 = mono?)
Block(h) 20-22: samples
Block(h) 24-25: 44100? The default sample rate for fallback/default? Tends to be 44100 or 48000, or could this be left channel and the other (28-29) for right channel?
Block(h) 28-29: rate
Offset(h) 2C: exported bit depth format? usually 16
Offset(h) 30: Always 1? (perhaps this is indicating that it is not muted?)
Offset(h) 34: Loop enabled if UInt8 = 1?
Block(h) 35-37: LS (loop start)
Block(h) 39-3B: LE (loop length)
Offset(h) 4E: usually 86(?)
Offset(h) 52: usually 44(?)

I checked se301_ikuya_yorokobi_rap_strm_lp.asrc.31 and se302_ikuya_ikari_rap_strm_lp.asrc.31, and figured out those has more length in their header, inserted (+10) at block(h) 40-4F. The reifa ones in gs6 all have different headers. No idea about the other error with the smpl chunk size. I know this doesn't matter much, as it's unlikely that a modder would wish to replace these audio files, but still.

...It might be a good idea to create a gist for this game too, sharing modding stuff there instead, to stay on focus with Ghost Trick here?

Author

osyu commented Jan 29, 2024

I've updated the script to fix those issues, thanks for reporting. My bad for not testing against all the files before posting it...

This is what I have currently for the srcd struct:

u32 magic        : "srcd"
4b  <pad>        : always 0
u32 file_size    : size of audio file
u32 type         : "wav " or "ogg "
u32 strm         : bool; streaming audio flag
u32 id           : audio source id, same as corresponding srch file
u32 unk0         : unknown int, NOT a hash of the audio data
u32 channels     : channel count
u32 samples      : sample count (total, not per channel)
u32 urate        : unknown int, looks like a sample rate
u32 rate         : sample rate
u32 depth        : bit depth
4b  <pad>        : always 1
u8  loop         : bool; loop audio flag
u32 lps          : loop start sample position
u32 lpe          : loop end sample position
u32 mark_count   : marker count
for _ in range(mark_count)...
  u32 type       : marker type (usually 0xffffffff/-1)
  u32 pos        : marker sample position
9b  <pad>        : always 0
u32 unk1         : unknown int, usually 0 but sometimes 5
u32 header_size  : size of header, including this field and the next
u32 data_offset  : offset to start of audio data

I monitored the game's file i/o and confirmed that the "bgm" flag is for streaming audio (which is why I've renamed it to strm).

unk0 is definitely not a hash of the audio, since there are some duplicate .asrc files where the only difference between them is unk0 (even the source id is the same). natives/stm/streaming/sound/common/se/cursor_1644.asrc.31 and natives/stm/streaming/sound/gs6/se/sys/cursor_1644.asrc.31 are examples of this. It might be a murmur hash of some string... maybe the source id is too, who knows.

I don't know what urate is for, changing it doesn't seem to affect anything. No idea what unk1 could be either.

Markers are sample positions that are used primarily to sync the game script to streamed sound effects (via stse_mw and stse_mwn) or BGM (via bgm_mw). You'll see them in things like Andistan'dhin's singing and Rayfa's Divination Séance intro.

niltwill commented Jan 29, 2024 •

edited

Loading

Thanks for the clarification, struct, and the updated script. Only did some shoddy guesses that didn't pay off with unk0 and others.

What's best practice now? Due to these unknown values that cannot be calculated, does one still have to copy these bytes from the original file when re-encoding a modified file? That's a bit cumbersome with batch operations. But that would be the way to go, right? So that nothing breaks, since there is no telling what they do.

It would be great if the script could also take an additional (optional) input (parameter) for the original .asrc audio file (path) with the encoding function (to fetch all unknown data it cannot generate: id, unk0, urate, markers [if any], unk1). That would make it easier for batch operations (rather than manually hex editing every file one by one). Is it possible to pull that off somehow? I will look into it myself eventually, though I'm bad with Python.

Also, did you test the new script? There's a new error that broke it when trying to handle any file now (not only with the problematic files).

Now I get the following error (line 199, in info):

    mark = ','.join(':'.join(str(y) for y in x) for x in mark)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not iterable

Edit: Removed my temporary unclean fix for this after the script update (to make sure it's not getting used in case someone else's reading this).

I tested it with all the aforementioned problematic files, and now it's all OK. Hopefully no other errors now! Yay.

Author

osyu commented Jan 29, 2024 •

edited

Loading

Oops, again... that line formats the marker list before it gets printed out in the info command, but I forgot to skip that step if it's None (i.e. there are no markers); should be fixed now. I should have caught that but that's what I get for working on things at midnight

You should copy all the unknowns using the info command for now; other things like markers also need to stay, if they exist (you have to have the same amount in your new .asrc or the game script will be liable to break).

I already had plans for an r replace command to do what you're asking for batch operations, I'll work on that next

Author

osyu commented Jan 29, 2024 •

edited

Loading

Replace command is done; it takes in the input .wav, a base .asrc and an optional output path (if not specified it overwrites the base), plus the loop points and markers. It'll complain if the base has markers and you didn't specify any (or vice versa), so make sure to adjust those for the new audio.

Author

osyu commented Jan 29, 2024 •

edited

Loading

Updated once more to fix a slew of other bugs, make the out argument required for all commands (to avoid overwriting files you didn't mean to + the splitext stuff wasn't working properly with language suffixes), and add a -cpb option to the replace command which copies the loop positions and markers from the base file rather than using the -lps/-lpe/-mark arguments (useful for batch converting)

niltwill commented Feb 22, 2024

AJ will convert to the DS bytecode format, which can be edited with e.g. phoenixtools.

You get no actual text, only integers using that (with scripting\script-dis from that repo). Here is an example for "sc0_0_op.user.2.en.bin" (after decoding it with your script):

SECTION 0
 0000	text "{57473}"
 0001	music 57442, 00001
 0004	newline
 0005	text "{57473}"
 0006	sound 57451, 00001
 0009	section_setup
 0010	hideperson
 0011	text "{57433}"
 0012	cmd62
.label0_15:
 0013	text "{00610}{57473}"
 0015	fullscreen_text
 0016	text "{57435}"
 0017	section_setup
 0018	section_setup
 0019	section_setup
 0020	text "{57433}"
 0021	cmd62
 0022	text "{00609}{57340}\n{57473}"
 0026	finger_choice_2_args_jmp 57323, 4294967172
 0029	newline
 0030	hideperson
 0031	text "{57433}"
 0032	cmd62
[...]

For reference, it's supposed to look something like the attached image in this post.

My guess is that the AJ:AA Trilogy uses UTF-16LE encoding for the text itself, which is why it doesn't work? Other than manual editing with a hex editor (you can sort of read the text parts in these files), there's no easy way right now...

Author

osyu commented Feb 22, 2024

Yeah, it appears that AJ uses a different encoding here than the DS games, which isn't entirely unexpected since the Unity version of the original trilogy also changed the format some. It segfaults on certain scripts as well (e.g. sc0_0_h00) so it might need more work than just fixing the text parsing. I don't have time right now to look into it further through

mentionmenot commented Apr 10, 2024 •

edited

Loading

Thank you for the helpful information. Could you provide any update info about decoding AJ GS4 script? I'm encountering challenges with it in my localization project.

niltwill commented Apr 10, 2024 •

edited

Loading

Update: Now the script should work with every GS4 binary (I also tested every language).

I made a Python script for the GS4 binary script files. It's not too pretty and the text is hard to edit with it, but it's a start. Also remember that the GS4 script binaries still need to be decoded with this converter first.

Some examples:

ajaat-gs4-script.py decode action_studio.user.2.en.bin
ajaat-gs4-script.py encode action_studio.user.2.en.txt
ajaat-gs4-script.py compare action_studio.user.2.en.bin action_studio.user.2.en.encoded.bin

It supports wildcard matching for encoding and decoding, so you can simply use these commands (while having every script binary in the same folder):

ajaat-gs4-script.py decode *.bin
ajaat-gs4-script.py encode *.txt

With that, the decoding will simply use the same filename with the ".txt" extension.
The encoding will add the ".encoded" prefix (so that you end up with a file like "[name].encoded.bin". In this way, you don't accidentally overwrite the existing binaries, especially since a byte comparison is recommended to make sure there are no discrepancies.

However, there are different problematic files with different languages that need some byte correction after encoding. For English, these are the following:

sc0_1_h02.user.2.en.bin
sc1_0_000.user.2.en.bin
sc1_3_h03.user.2.en.bin
sc2_0_013.user.2.en.bin
sc2_0_016.user.2.en.bin
sc2_3_h00.user.2.en.bin
sc3_0_00a.user.2.en.bin
sc3_3_00c.user.2.en.bin
sc3_3_010.user.2.en.bin

To fix those byte differences that only appear in these 9 files, you can use a command like this: ajaat-gs4-script.py compare --fix sc0_1_h02.user.2.en.bin sc0_1_h02.user.2.en.encoded.bin

(Important to put the original binary file first, then the edited, re-encoded binary.)

Or you can use this batch file, as different languages do have other files that need the byte correction, so it's better to run this through each of these files.

evil-trainer commented Jul 22, 2024 •

edited

Loading

This gist is for the PC version only, you can find info on modding console versions of RE Engine games elsewhere

Sorry to hijack the gist, but I'm a fan translator of Switch games and unfortunately there isn't somewhere else to look, for addressing the Switch tex conversion situation (at least I couldn't find anything anywhere).

But, hope is not lost, since there's a workaround =)

Thanks to the awesome folks from the Modding Haven Discord Server, I learned how to make MHR_Tex CHopper work with Switch textures.

Open up the texture in a hexadecimal editor and replace the 5º byte with the hexadecimal equivalent of "23", which happens to be 1C.

Before:

After:

Now, save your changes and drag and drop the texture onto MHR_Tex Chopper and it'll do its thing ;)

Don't forget to disable "Create MipMap" in Paint.Net BEFORE converting back from .dds to .tex. If you are dealing with UI elements textures, use BC7 linear as the compression or use BC7 sRGB if you're dealing with albedo textures instead. To convert from dds back to .tex, just drag and drop yout edited .dds onto MHR_Tex Chopper and it'll spit out a .tex file. Now, just replace the vanilla .tex with your newly edited one and that's it!

All that's left is create a .pak.patch file and test your edit in-game!

osyu/gt_docs.md

How to extract game assets

short answer

long answer

How to convert game assets to common formats

.tex (textures)

.asrc (audio)

.mov (video)

.msg (text)

.oft (fonts)

.user/.scn/.pfb (objects, scenes, prefabs)

Where the hey is everything

textures

audio

video

text

fonts

Internal names

stages

characters

Modding tools

niltwill commented Jan 28, 2024

Uh oh!

osyu commented Jan 29, 2024

Uh oh!

niltwill commented Jan 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

osyu commented Jan 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

osyu commented Jan 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

osyu commented Jan 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niltwill commented Feb 22, 2024

Uh oh!

osyu commented Feb 22, 2024

Uh oh!

mentionmenot commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niltwill commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evil-trainer commented Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niltwill commented Jan 29, 2024 •

edited

Loading

osyu commented Jan 29, 2024 •

edited

Loading

osyu commented Jan 29, 2024 •

edited

Loading

osyu commented Jan 29, 2024 •

edited

Loading

mentionmenot commented Apr 10, 2024 •

edited

Loading

niltwill commented Apr 10, 2024 •

edited

Loading

evil-trainer commented Jul 22, 2024 •

edited

Loading