Skip to content

Instantly share code, notes, and snippets.

@osyu

osyu/gt_docs.md

Last active October 16, 2024 19:24
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save osyu/895c38cf77f08853353988780e52dfc0 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save osyu/895c38cf77f08853353988780e52dfc0 to your computer and use it in GitHub Desktop.
Download ZIP
Ghost Trick remaster docs
Raw
gt_docs.md

(guides assume you're on Windows)

How to extract game assets

short answer

long answer

primer:

  • Ghost Trick runs on Capcom's RE Engine. REtool is an unofficial tool for dealing with the engine's file formats
  • all resources are stored in a big file named re_chunk_000.pak located in the game's installation directory (which is accessible in steam through [right click] > Manage > Browse local files)
  • unlike a typical archive format like .zip/.rar/.7z, .pak files don't have filenames plainly stored in a way that REtool can reliably retrieve, so you also need to give it a list of names to read from so it knows what to name the things it's extracting. if you don't, you'll be stuck with a bunch of files without discernable names and your life will become harder

what to do:

  • download RETool from the link above and extract REtool.exe to the game's installation directory (the same folder as re_chunk_000.pak)
  • download the filepath list from the link above, into the same folder
  • open a command prompt in that folder. there are a billion ways to do this but the easiest way is typing cmd into the address bar in file explorer
  • type retool -h GTPD_PC_Release.list -x re_chunk_000.pak to extract everything. when it's done there'll be a folder named re_chunk_000 with all the stuff
  • you'll notice there are still hundreds of files without proper names (that look something like 15349567-2389623703.bin). this is normal and most of those files are unimportant/not unique to Ghost Trick, so you can ignore them. it happens because the list we passed to REtool has most but not all filenames, so anything missing will end up with a name like that

How to convert game assets to common formats

(everything here assumes you've just done all the things above)

.tex (textures)

  • type for /r %f in (*.tex.*) do retool -tex %f. this will recursively convert every .tex file to .dds, which is readable by image editing programs like paint.net

.asrc (audio)

  • they're just .wav files with an extra header in front. you have two options:
    • if you have Python installed, run this script to convert to and from the format
    • open up a hex editor (e.g. HxD) and remove the first 78 (0x4e) bytes, then change the file extension to .wav

.mov (video)

  • they're just .wmv files. change the file extension to .wmv and you're set
  • except for natives/stm/streaming/movie/igt_op_480x320_500kbps.mov.1, that's an .mp4

.msg (text)

  • download REMSG_Converter, and type for /r %f in (*.msg.*) do remsg_converter %f to convert recursively to .csv, which you can open in excel or any text editor. add -m json to use json instead

.oft (fonts)

  • these are encrypted OpenType and TrueType font files. you can encrypt/decrypt them using REE.FontsCryptor from the REE.PAK.Tool repository (as of writing binaries aren't provided, you'll have to build it yourself)

.user/.scn/.pfb (objects, scenes, prefabs)

  • you can use the RE_RSZ 010 Editor template to read and edit these

Where the hey is everything

(this is a non-exhaustive list)

textures

  • facepics/portraits: natives/stm/charfiles/3d/08_face
  • backgrounds: natives/stm/charfiles/2d/02_bg
  • character animations: natives/stm/charfiles/2d/01_human
  • object animations: natives/stm/charfiles/2d/03_gimmick
  • ui: natives/stm/charfiles/3d/06_id

audio

  • bgm:
    • remastered: natives/stm/streaming/sound/bgm_new
    • original: natives/stm/streaming/sound/bgm_resource
  • sound effects: natives/stm/streaming/sound/se_resource

video

  • cutscenes: natives/stm/streaming/gui/ui010500/tex
  • unused trailer: natives/stm/streaming/movie

text

  • it's all in natives/stm/gamedesign/text

fonts

  • they're in natives/stm/gui/font/outlinefont

Internal names

stages

  • Junkyard - st01
  • Pigeon Man's Office - st02
  • Prison - st03
  • The Chicken Kitchen - st04
  • Lady's Red Apartment - st05
  • Lynne's Apartment - st06
  • Kamila's Old House - st07
  • Office of the Troubled Man - st09
  • Drive to Chicken Kitchen - st10
  • Special Investigation Unit - st11
  • Temsik Park - st13
  • The Yonoa - st14
  • Prologue & Credits - st15

characters

  • Sissel (human) - cicel
  • Sissel (cat) - neko (猫, lit. "cat")
  • Lynne - linne
  • Ray - kuneri
  • Nearsighted Jeego - killera
  • Commander Sith - sisu
  • Masked Muscleman - migi (右, lit. "right")
  • "One Step Ahead" Tengo - killerb
  • Kamila - girl
  • Missile - pome ("pome[narian]")
  • Emma - fujin (婦人, lit. "madam")
  • Amelie - eimin (永眠, lit. "death")
  • Justice Minister - daijin (大臣, lit. "minister")
  • Detective Mccaw - keijig (刑事, lit. "detective g[reen]")
  • The Blue Detective - keijib (刑事, lit. "detective b[lue]")
  • Odd Blue Doctor - doc
  • Inspector Cabanela - kaba
  • Detective Rindge - caron
  • Guardian of the Park - birakubari (ビラ配り, lit. "handing out leaflets")
  • Pigeon Man - kanri (管理, lit. "management")
  • Chicken Kitchen Cook - cock
  • Typical Cop - keikan (警官, lit. "policeman")
  • Officer Bailey - keibia (警備, lit. "guard a")
  • Rock Jailbird - rock
  • Curry-loving Jailbird - shujinb
  • Detective Jowd - jodo
  • SIU Chief - bucho (部長, lit. "department head")
  • Memry - waitress
  • Dandy - yukaib
  • Beauty - yukaia
  • Chicken Kitchen Bartender - barten ("barten[der]")
  • Alma - mama
  • Minor Crew Hand - zako (雑魚, lit. "small fry")

Modding tools

  • Fluffy Mod Manager - mod manager for RE Engine games (for modifying assets)
  • REFramework - scripting and mod framework (for modifying game logic)
@student197
Copy link

student197 commented Jan 27, 2024

Hi, can you make a program to make it easier to edit dialog files or a script to convert them to .txt or something similar. Just editing them in the 010 editor is inconvenient. P.s And how to edit subtitles in aa5 and aa6? they are stored in the same file format as the dialogues, but there is nothing in them.

@osyu
Copy link
Author

osyu commented Jan 28, 2024

The scripts for each game are in the same formats as they were on their native platforms (i.e. AJ uses the GBA/DS bytecode, DD and SOJ use the 3DS format, etc.); the only difference here is that they're wrapped in RSZ user data.

I imagine tools like Kuriimu will add support for this sometime soon (the same way it supports MT Framework's GMD files for TGAA script editing), but in the meantime I wrote a (sufficiently messy) script to convert to and from common formats: https://gist.github.com/osyu/5bb86d49153edef5415a7aba09a48ca1

DD and SOJ will convert to JSON. AJ will convert to the DS bytecode format, which can be edited with e.g. phoenixtools. The script also supports globbing/wildcards to let you convert the entire directory at once.

Anime subtitles are stored in regular .msg files at natives/stm/gamedesign/text/gs5 (and gs6), which can be converted with REMSG_Converter (linked above in the gist). For example, SOJ's opening cutscene is .../gs6/gs6_gs_01_01.msg.22.

@student197
Copy link

student197 commented Jan 28, 2024

can you please tell us in more detail how to use your script

@osyu
Copy link
Author

osyu commented Jan 28, 2024 •
edited
Loading

gs456scr d <user file> to decode, gs456scr e <json or bin file> to encode. e.g. gs456scr d sce00_c000_0000.user.2.en will create sce00_c000_0000.user.2.en.json which you can edit, and gs456scr e sce00_c000_0000.user.2.en.json will convert the .json back into .user

@student197
Copy link

student197 commented Jan 28, 2024

do I need to convert dgs456scr.ry to exe or how do I use it?

@osyu
Copy link
Author

osyu commented Jan 28, 2024

Install Python and then run the script via cmd

@student197
Copy link

student197 commented Jan 28, 2024

I tried, but it didn't work out, maybe I'm doing something wrong

@osyu
Copy link
Author

osyu commented Jan 28, 2024

I don't want to clog up this gist's comments with support questions, so you should contact me by other means if you need more help

@niltwill
Copy link

niltwill commented Jan 28, 2024

I tried to run asrc31.py through all the audio files of AJ:AA Trilogy, and there are certain ones that it cannot handle. There are two errors, depending on the files.

These are the offenders with the "smpl chunk size" error. AssertionError: assert read_u32(f) == 44 # smpl chunk size

natives\stm\streaming\sound\common\se\get_medal_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_bass_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_click_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_drums_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_keyboards_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_leadguitar_1648.asrc.31
natives\stm\streaming\sound\gs4\stream\bgm\interactivemusic\bgm070_rhythmguitar_1648.asrc.31
natives\stm\streaming\sound\gs5\se\script_com\bikkuri.asrc.31
natives\stm\streaming\sound\gs5\se\se_cr003\kokone_pc_tap_flick.asrc.31
natives\stm\streaming\sound\gs5\se\se_cr203\biyoin_chakushin_both.asrc.31
natives\stm\streaming\sound\gs5\se\se_cr203\biyoin_chakushin_hidari.asrc.31
natives\stm\streaming\sound\gs5\se\se_cr203\biyoin_chakushin_migi.asrc.31
natives\stm\streaming\sound\gs5\se\se_cr207\shoko_performance_ken_hikaru.asrc.31
natives\stm\streaming\sound\gs6\se\yamashinop\yamashinop_coin_break_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\animationstudio\se\close_anim_studio_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\animationstudio\se\enter_anim_studio_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\animationstudio\se\se_court_000.asrc.31
natives\stm\streaming\sound\startupscreen\museum\artlibrary\se\close_art_library_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\artlibrary\se\enter_art_library_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\artlibrary\se\paper_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\artlibrary\se\screendown_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\artlibrary\se\screenup_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\orchestrahall\se\curtain_close_1648.asrc.31
natives\stm\streaming\sound\startupscreen\museum\orchestrahall\se\curtain_open_1648.asrc.31
natives\stm\streaming\sound\startupscreen\se\entering_titlesel_1648.asrc.31
natives\stm\streaming\sound\startupscreen\se\enter_startup_screen_1648.asrc.31

Also here are some others below that output this AssertionError: assert read_u32(f) == 0xffffffff # always -1

natives\stm\streaming\sound\gs5\stream\all\se301_ikuya_yorokobi_rap_strm_lp.asrc.31
natives\stm\streaming\sound\gs5\stream\all\se302_ikuya_ikari_rap_strm_lp.asrc.31
natives\stm\streaming\sound\gs6\stream\se\reifa_v_ima_1644.asrc.31
natives\stm\streaming\sound\gs6\stream\se\reifa_v_minanomono_1644.asrc.31
natives\stm\streaming\sound\gs6\stream\se\reifa_v_sisosamano_1644.asrc.31
natives\stm\streaming\sound\gs6\stream\se\reifa_v_tokutomiyo_1644.asrc.31

For reference (correct this if I'm wrong), the offsets for the asrc.31:

Block(h) 0-3: Has to be "srcd"
Block(h) 8-A: WAV file's size (without the header)
Offset(h) F: Always 32? (default 32-Bit float depth?)
Offset(h) 10: bgm(?) [as it doesn't always appear for BGMs only, my guesses... 1. audio data behaviour: preload audio data (to make sure it plays as soon as the scene appears/starts?)=1/load in background (it's not prioritized, it can wait to load)=0, 2. normalize the audio (to make sure it's not overly loud/clipping)=1, do not normalize it (use source audio as is)=0]
Block(h) 14-17: file id
Block(h) 18-1B: unknown0 (secondary id used for something else?)
Offset(h) 1C: channels? (2 = stereo, 1 = mono?)
Block(h) 20-22: samples
Block(h) 24-25: 44100? The default sample rate for fallback/default? Tends to be 44100 or 48000, or could this be left channel and the other (28-29) for right channel?
Block(h) 28-29: rate
Offset(h) 2C: exported bit depth format? usually 16
Offset(h) 30: Always 1? (perhaps this is indicating that it is not muted?)
Offset(h) 34: Loop enabled if UInt8 = 1?
Block(h) 35-37: LS (loop start)
Block(h) 39-3B: LE (loop length)
Offset(h) 4E: usually 86(?)
Offset(h) 52: usually 44(?)

I checked se301_ikuya_yorokobi_rap_strm_lp.asrc.31 and se302_ikuya_ikari_rap_strm_lp.asrc.31, and figured out those has more length in their header, inserted (+10) at block(h) 40-4F. The reifa ones in gs6 all have different headers. No idea about the other error with the smpl chunk size. I know this doesn't matter much, as it's unlikely that a modder would wish to replace these audio files, but still.

...It might be a good idea to create a gist for this game too, sharing modding stuff there instead, to stay on focus with Ghost Trick here?

@osyu
Copy link
Author

osyu commented Jan 29, 2024

I've updated the script to fix those issues, thanks for reporting. My bad for not testing against all the files before posting it...

This is what I have currently for the srcd struct:

u32 magic        : "srcd"
4b  <pad>        : always 0
u32 file_size    : size of audio file
u32 type         : "wav " or "ogg "
u32 strm         : bool; streaming audio flag
u32 id           : audio source id, same as corresponding srch file
u32 unk0         : unknown int, NOT a hash of the audio data
u32 channels     : channel count
u32 samples      : sample count (total, not per channel)
u32 urate        : unknown int, looks like a sample rate
u32 rate         : sample rate
u32 depth        : bit depth
4b  <pad>        : always 1
u8  loop         : bool; loop audio flag
u32 lps          : loop start sample position
u32 lpe          : loop end sample position
u32 mark_count   : marker count
for _ in range(mark_count)...
  u32 type       : marker type (usually 0xffffffff/-1)
  u32 pos        : marker sample position
9b  <pad>        : always 0
u32 unk1         : unknown int, usually 0 but sometimes 5
u32 header_size  : size of header, including this field and the next
u32 data_offset  : offset to start of audio data

I monitored the game's file i/o and confirmed that the "bgm" flag is for streaming audio (which is why I've renamed it to strm).

unk0 is definitely not a hash of the audio, since there are some duplicate .asrc files where the only difference between them is unk0 (even the source id is the same). natives/stm/streaming/sound/common/se/cursor_1644.asrc.31 and natives/stm/streaming/sound/gs6/se/sys/cursor_1644.asrc.31 are examples of this. It might be a murmur hash of some string... maybe the source id is too, who knows.

I don't know what urate is for, changing it doesn't seem to affect anything. No idea what unk1 could be either.

Markers are sample positions that are used primarily to sync the game script to streamed sound effects (via stse_mw and stse_mwn) or BGM (via bgm_mw). You'll see them in things like Andistan'dhin's singing and Rayfa's Divination Séance intro.

@niltwill
Copy link

niltwill commented Jan 29, 2024 •
edited
Loading

Thanks for the clarification, struct, and the updated script. Only did some shoddy guesses that didn't pay off with unk0 and others.

What's best practice now? Due to these unknown values that cannot be calculated, does one still have to copy these bytes from the original file when re-encoding a modified file? That's a bit cumbersome with batch operations. But that would be the way to go, right? So that nothing breaks, since there is no telling what they do.

It would be great if the script could also take an additional (optional) input (parameter) for the original .asrc audio file (path) with the encoding function (to fetch all unknown data it cannot generate: id, unk0, urate, markers [if any], unk1). That would make it easier for batch operations (rather than manually hex editing every file one by one). Is it possible to pull that off somehow? I will look into it myself eventually, though I'm bad with Python.

Also, did you test the new script? There's a new error that broke it when trying to handle any file now (not only with the problematic files).

Now I get the following error (line 199, in info):

    mark = ','.join(':'.join(str(y) for y in x) for x in mark)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not iterable

Edit: Removed my temporary unclean fix for this after the script update (to make sure it's not getting used in case someone else's reading this).

I tested it with all the aforementioned problematic files, and now it's all OK. Hopefully no other errors now! Yay.

@osyu
Copy link
Author

osyu commented Jan 29, 2024 •
edited
Loading

Oops, again... that line formats the marker list before it gets printed out in the info command, but I forgot to skip that step if it's None (i.e. there are no markers); should be fixed now. I should have caught that but that's what I get for working on things at midnight

You should copy all the unknowns using the info command for now; other things like markers also need to stay, if they exist (you have to have the same amount in your new .asrc or the game script will be liable to break).

I already had plans for an r replace command to do what you're asking for batch operations, I'll work on that next

@osyu
Copy link
Author

osyu commented Jan 29, 2024 •
edited
Loading

Replace command is done; it takes in the input .wav, a base .asrc and an optional output path (if not specified it overwrites the base), plus the loop points and markers. It'll complain if the base has markers and you didn't specify any (or vice versa), so make sure to adjust those for the new audio.

@osyu
Copy link
Author

osyu commented Jan 29, 2024 •
edited
Loading

Updated once more to fix a slew of other bugs, make the out argument required for all commands (to avoid overwriting files you didn't mean to + the splitext stuff wasn't working properly with language suffixes), and add a -cpb option to the replace command which copies the loop positions and markers from the base file rather than using the -lps/-lpe/-mark arguments (useful for batch converting)

@niltwill
Copy link

niltwill commented Feb 22, 2024

AJ will convert to the DS bytecode format, which can be edited with e.g. phoenixtools.

You get no actual text, only integers using that (with scripting\script-dis from that repo). Here is an example for "sc0_0_op.user.2.en.bin" (after decoding it with your script):

SECTION 0
 0000	text "{57473}"
 0001	music 57442, 00001
 0004	newline
 0005	text "{57473}"
 0006	sound 57451, 00001
 0009	section_setup
 0010	hideperson
 0011	text "{57433}"
 0012	cmd62
.label0_15:
 0013	text "{00610}{57473}"
 0015	fullscreen_text
 0016	text "{57435}"
 0017	section_setup
 0018	section_setup
 0019	section_setup
 0020	text "{57433}"
 0021	cmd62
 0022	text "{00609}{57340}\n{57473}"
 0026	finger_choice_2_args_jmp 57323, 4294967172
 0029	newline
 0030	hideperson
 0031	text "{57433}"
 0032	cmd62
[...]

For reference, it's supposed to look something like the attached image in this post.

My guess is that the AJ:AA Trilogy uses UTF-16LE encoding for the text itself, which is why it doesn't work? Other than manual editing with a hex editor (you can sort of read the text parts in these files), there's no easy way right now...

@osyu
Copy link
Author

osyu commented Feb 22, 2024

Yeah, it appears that AJ uses a different encoding here than the DS games, which isn't entirely unexpected since the Unity version of the original trilogy also changed the format some. It segfaults on certain scripts as well (e.g. sc0_0_h00) so it might need more work than just fixing the text parsing. I don't have time right now to look into it further through

@mentionmenot
Copy link

mentionmenot commented Apr 10, 2024 •
edited
Loading

Thank you for the helpful information. Could you provide any update info about decoding AJ GS4 script? I'm encountering challenges with it in my localization project.

@niltwill
Copy link

niltwill commented Apr 10, 2024 •
edited
Loading

Update: Now the script should work with every GS4 binary (I also tested every language).

I made a Python script for the GS4 binary script files. It's not too pretty and the text is hard to edit with it, but it's a start. Also remember that the GS4 script binaries still need to be decoded with this converter first.

Some examples:

ajaat-gs4-script.py decode action_studio.user.2.en.bin
ajaat-gs4-script.py encode action_studio.user.2.en.txt
ajaat-gs4-script.py compare action_studio.user.2.en.bin action_studio.user.2.en.encoded.bin

It supports wildcard matching for encoding and decoding, so you can simply use these commands (while having every script binary in the same folder):

ajaat-gs4-script.py decode *.bin
ajaat-gs4-script.py encode *.txt

With that, the decoding will simply use the same filename with the ".txt" extension.
The encoding will add the ".encoded" prefix (so that you end up with a file like "[name].encoded.bin". In this way, you don't accidentally overwrite the existing binaries, especially since a byte comparison is recommended to make sure there are no discrepancies.

However, there are different problematic files with different languages that need some byte correction after encoding. For English, these are the following:

  • sc0_1_h02.user.2.en.bin
  • sc1_0_000.user.2.en.bin
  • sc1_3_h03.user.2.en.bin
  • sc2_0_013.user.2.en.bin
  • sc2_0_016.user.2.en.bin
  • sc2_3_h00.user.2.en.bin
  • sc3_0_00a.user.2.en.bin
  • sc3_3_00c.user.2.en.bin
  • sc3_3_010.user.2.en.bin

To fix those byte differences that only appear in these 9 files, you can use a command like this: ajaat-gs4-script.py compare --fix sc0_1_h02.user.2.en.bin sc0_1_h02.user.2.en.encoded.bin

(Important to put the original binary file first, then the edited, re-encoded binary.)

Or you can use this batch file, as different languages do have other files that need the byte correction, so it's better to run this through each of these files.

@evil-trainer
Copy link

evil-trainer commented Jul 22, 2024 •
edited
Loading

This gist is for the PC version only, you can find info on modding console versions of RE Engine games elsewhere

Sorry to hijack the gist, but I'm a fan translator of Switch games and unfortunately there isn't somewhere else to look, for addressing the Switch tex conversion situation (at least I couldn't find anything anywhere).

But, hope is not lost, since there's a workaround =)

Thanks to the awesome folks from the Modding Haven Discord Server, I learned how to make MHR_Tex CHopper work with Switch textures.

Open up the texture in a hexadecimal editor and replace the 5º byte with the hexadecimal equivalent of "23", which happens to be 1C.

Before:
image

After:
image

Now, save your changes and drag and drop the texture onto MHR_Tex Chopper and it'll do its thing ;)

image

image

Don't forget to disable "Create MipMap" in Paint.Net BEFORE converting back from .dds to .tex. If you are dealing with UI elements textures, use BC7 linear as the compression or use BC7 sRGB if you're dealing with albedo textures instead. To convert from dds back to .tex, just drag and drop yout edited .dds onto MHR_Tex Chopper and it'll spit out a .tex file. Now, just replace the vanilla .tex with your newly edited one and that's it!

All that's left is create a .pak.patch file and test your edit in-game!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment