P0148
TH04/TH05 decompilation (Text popups, gather circle rendering, player position clamping)
💰 Funded by:
[Anonymous]
🏷️ Tags:
Back after taking way too long to get Touhou Patch Center's MediaWiki
update feature complete… I'm still waiting for more translators to test and
review the new translation interface before delivering and deploying it
all, which will most likely lead to another break from ReC98 within the
next few months. For now though, I'm happy to have mostly addressed the
nagging responsibility I still had after willing that site into existence,
and to be back working on ReC98. 🙂
As announced, the next few pushes will focus on TH04's and TH05's bullet
spawning code, before I get to put all that accumulated TH01 money towards
finishing all of konngara's code in TH01. For a full
picture of what's happening with bullets, we'd really also like to
have the bullet update function as readable C code though.
Clearing all bullets on the playfield will trigger a Bonus!! popup,
displayed as 📝 gaiji in that proportional
font. Unfortunately, TLINK refused to link the code as soon as I referenced
the function for animating the popups at the top of the playfield? Which
can only mean that we have to decompile that function first…
So, let's turn that piece of technical debt into a full push, and first
decompile another random set of previously reverse-engineered TH04 and TH05
functions. Most of these are stored in a different place within the two
MAIN.EXE binaries, and the tried-and-true method of matching
segment names would therefore have introduced several unnecessary
translation units. So I resorted to a segment splitting technique I should
have started using way earlier: Simply creating new segments with names
derived from their functions, at the exact positions they're needed. All
the new segment start and end directives do bloat the ASM code somewhat,
and certainly contributed to this push barely removing any actual lines of
code. However, what we get in return is total freedom as far as
decompilation order is concerned,
📝 which should be the case for any ReC project, really.
And in the end, all these tiny code segments will cancel out anyway.
If only we could do the same with the data segment…
The popup function happened to be the final one I RE'd before my long break
in the spring of 2019. Back then, I didn't even bother looking into that
64-frame delay between changing popups, and what that meant for the game.
Each of these popups stays on screen for 128 frames, during which, of
course, another popup-worthy event might happen. Handling this cleanly
without removing previous popups too early would involve some sort of event
queue, whose size might even be meaningfully limited to the number of
distinct events that can happen. But still, that'd be a data structure, and
we're not gonna have that! Instead, ZUN
simply keeps two variables for the new and current popup ID. During an
active popup, any change to that ID will only be committed once the current
popup has been shown for at least 64 frames. And during that time,
that new ID can be freely overwritten with a different one, which drops any
previous, undisplayed event. But surely, there won't be more than two
events happening within 63 frames, right?
The rest was fairly uneventful – no newly RE'd functions in this push,
after all – until I reached the widely used helper function for applying
the current vertical scrolling offset to a Y coordinate. Its combination of
a function parameter, the pascal calling convention, and no
stack frame was previously thought to be undecompilable… except that it
isn't, and the decompilation didn't even require any new workarounds to be
developed? Good thing that I already forgot how impossible it was to
decompile the first function I looked at that fell into this category!
Oh well, this discovery wasn't too groundbreaking. Looking back at
all the other functions with that combination only revealed a grand total
of 1 additional one where a decompilation made sense: TH05's version of
snd_kaja_interrupt(), which is now compiled from the same C++
file for all 4 games that use it. And well, looks like some quirks really
remain unnoticed and undocumented until you look at a function for the 11th
time: Its return value is undefined if BGM is inactive – that is, if the
user disabled it, or if no FM board is installed. Not that it matters for
the original code, which never uses this function to retrieve anything from
KAJA's drivers. But people apparently do copy ReC98 code into their own
projects, so it is something to keep in mind.
All in all, nothing quite at jank level in this one, but we were surely grazing that tag. Next up, with that out of the way: The bullet update/step function! Very soon in fact, since I've mostly got it done already.
P0117
Rebuilding ZUN.COM + overall position independence
💰 Funded by:
[Anonymous]
🏷️ Tags:
Wouldn't it be a bit disappointing to have TH05 completely
position-independent, but have it still require hex-editing of the
original ZUN.COM to mod its gaiji characters? As in, these
custom "text" glyphs, available to the PC-98 text RAM:
Especially since we now even have a sprite converter… the lack of which
was exactly 📝 what made rebuilding ZUN.COM not that worthwhile before.
So, before the big release, let's get all the remaining
ZUN.COM sub-binaries of TH04 and TH05 dumped into .ASM files,
and re-assembled and linked during the build process.
This is also the moment in which Egor's 2018
reimplementation of O. Morikawa's comcstm finally gets
to shine. Back then, I considered it too early to even bother with
ZUN.COM and reimplementing the .COM wrapper that ZUN
originally used to bundle multiple smaller executables into that single
binary. But now that the time is right, it is nice to have that
code, as it allowed me to get these rebuilds done in half a push.
Otherwise, it would have surely required one or two dedicated ones.
Since we like correctness here, newly dumped ZUN code means that it also
has to be included in the RE%
baseline calculation. This is why TH04's and TH05's overall RE% bars
have gone back a tiny bit… in case you remember how they previously looked
like After all, I would like to figure
out where all that memory allocated during TH04's and TH05's memory check
is freed, if at all.
Alright, one half of a push left… Y'know, getting rid of those last few PI
false positives is actually one of the most annoying chores in this
project, and quite stressful as well: I have to convince myself that the
remaining false positives are, in fact, not memory references, but with
way too little time for in-depth RE and to denote what they are
instead. In that situation, everyone (including myself!)
is anticipating that PI goal, and no one is really interested in RE.
(Well… that is, until they actually get to developing their mod. But more
on that tomorrow. ) Which means that it boils
down to quite some hasty, dumb, and superficial RE around those remaining
numbers.
So, in the hope of making it less annoying for the other 4 games in the
future, let's systematically cover the sources of those remaining false
positives in TH05, over all games. I/O port accesses with either the port
or the value in registers (and thus, no longer as an immediate argument to
the IN or OUT instructions, which the PI counter
can clearly ingore), palette color arithmetic, or heck, 0xFF constants that
obviously just mean "-1" and are not a reference to offset 0xFF in
the data segment. All of this, of course, once again had a way bigger
effect on everything but an almost position-independent TH05… but
hey, that's the sort of thing you reserve the "anything" pushes for. And
that's also how we get some of the single biggest PI% gains we have seen
so far, and will be seeing before the 100% PI mark. And yes, those will
continue in the next push.
Well, that took twice as long as I thought, with the two pushes containing
a lot more maintenance than actual new research. Spending some time
improving both field names and types in
32th System's
TH03 resident structure finally gives us all of those
structures. Which means that we can now cover all the remaining
decompilable ZUN.COM parts at once…
Oh wait, their main() functions have stayed largely identical
since TH02? Time to clean up and separate that first, then… and combine
two recent code generation observations into the solution to a
decompilation puzzle from 4½ years ago. Alright, time to decomp-
Oh wait, we'd kinda like to properly RE all the code in TH03-TH05
that deals with loading and saving .CFG files. Almost every outside
contributor wanted to grab this supposedly low-hanging fruit a lot
earlier, but (of course) always just for a single game, while missing how
the format evolved.
So, ZUN.COM. For some reason, people seem to consider it
particularly important, even though it contains neither any game logic nor
any code specific to PC-98 hardware… All that this decompilable part does
is to initialize a game's .CFG file, allocate an empty resident structure
using master.lib functions, release it after you quit the game,
error-check all that, and print some playful messages~ (OK, TH05's also
directly fills the resident structure with all data from
MIKO.CFG, which all the other games do in OP.EXE.)
At least modders can now freely change and extend all the resident
structures, as well as the .CFG files? And translators can translate those
messages that you won't see on a decently fast emulator anyway? Have fun,
I guess 🤷
And you can in fact do this right now – even for TH04 and TH05,
whose ZUN.COM currently isn't rebuilt by ReC98. There is
actually a rather involved reason for this:
One of the missing files is TH05's GJINIT.COM.
Which contains all of TH05's gaiji characters in hardcoded 1bpp form,
together with a bit of ASM for writing them to the PC-98's hardware gaiji
RAM
Which means we'd ideally first like to have a sprite compiler, for
all the hardcoded 1bpp sprites
Which must compile to an ASM slice in the meantime, but should also
output directly to an OMF .OBJ file (for performance now), as well as to C
code (for portability later)
Which I won't put in as long as the backlog contains actual
progress to drive up the percentages on the front page.
So yeah, no meaningful RE and PI progress at any of these levels. Heck,
even as a modder, you can just replace the zun zun_res
(TH02), zun -5 (TH03), or zun -s (TH04/TH05)
calls in GAME.BAT with a direct call to your modified
*RES*.COM. And with the alternative being "manually typing 0 and 1
bits into a text file", editing the sprites in TH05's
GJINIT.COM is way more comfortable in a binary sprite editor
anyway.
For me though, the best part in all of this was that it finally made sense
to throw out the old Borland C++ run-time assembly slices 🗑 This giant
waste of time
became obvious 5 years ago, but any ASM dump of a .COM
file would have needed rather ugly workarounds without those slices. Now
that all .COM binaries that were originally written in C are
compiled from C, we can all enjoy slightly faster grepping over the entire
repository, which now has 229 fewer files. Productivity will skyrocket!
Next up: Three weeks of almost full-time ReC98 work! Two more PI-focused
pushes to finish this TH05 stretch first, before switching priorities to
TH01 again.
Instead, ZUN
simply keeps two variables for the new and current popup ID. During an
active popup, any change to that ID will only be committed once the current
popup has been shown for at least 64 frames. And during that time,
that new ID can be freely overwritten with a different one, which drops any
previous, undisplayed event. But surely, there won't be more than two
events happening within 63 frames, right? 
