P0317
TH02 RE (Main menu overhaul) / TH03 decompilation (High score menu, part 2/2) / TH02/TH03 debloating (Initial screen tearing fixes)
P0318
TH04/TH05 decompilation (TH04 title screen animation / TH05 All Cast sequence / GENSOU.SCR, part 3/3)
💰 Funded by:
Blue Bolt, [Anonymous], Yanga, Ember2528
🏷️ Tags:
Part 3 of 📝 the 4-post series about the big 2025 PC-98 Touhou portability subproject, and we actually get to move some percentages on the front page with this one! For once, there truly isn't a lot to mention about most of these five disconnected small-feature decompilations, so let's go for more of a touhou-memories style and string together a few shorter bullet points and paragraphs. For even greater brevity, I'll also use the ZUN code issue emoji you might already know from Twitter or Bluesky: 🐞 denotes a bug, 💣 denotes a landmine, and 🎺 denotes a quirk.
This was one of those old decompilations from 2015 that I really wanted to bring up to current standards before the debloated branch would roll out the new more portable and performant blitting code. Replacing the magic-number coordinates with constants and calculations revealed 📝 the usual off-by-one text positioning bugs in the Option menu, despite ZUN still using monospaced text in this game…
As for more unique and exciting details in this screen: ZUN's defined gaiji strings contain an unused adaptation of TH01's blinking ＨＩＴ ＫＥＹ text. On screen, it might have looked something like this:
This string is so unused that we don't even know its intended position, though.
Finishing TH03's High Score menu
At the end of 2021, 📝 I already decompiled most of this menu, but left two functions in ASM due to push scope constraints. Originally, I thought that this menu would need a few changes to address a certain scorefile inconsistency I'll mention in Part 4, but I ended up finding a better solution. Still, we got one interesting discovery per function out of it:
If you've ever entered a score and were too lazy to type a proper name, you know that TH03 just uses the name of the player character in Romaji if you enter either nothing or AAAAAAAA. But did you know that this happens if you enter any letter 8 times?
🐞 When sorting a new score into the list, ZUN does not look at the 9th digit, i.e., the number of continues used. If you ever manage to enter a score whose most significant 8 digits match an existing entry in the current difficulty's score list, those two scores are considered equal and the new score always gets inserted below the old one. If you enter more than one such score, the list will therefore maintain the order in which the scores were entered:
In this example, I first entered 800-million scores with 0, 3, and 1 continues in exactly this order, before entering this new 2-continue score.
TH04's title screen animation
This decompilation was necessary because its palette manipulation code did the very dubious thing of accessing the palette in a freed .PI slot. I don't think that the stylish effect of separately whiting in the image's black outlines is appreciated enough. And yes, that formally was the last non-RE'd tiny bit of any OP.EXE binary!
Also note that single black pixel in Reimu's gohei.
TH05's All Cast sequence
This sequence contained the last not yet decompiled instance of 📝 masked crossfading, which the debloated branch wants to replace with our single optimized implementation.
Most picture and text cues in this sequence are synced to the BGM, using PMD's AH=05h function to retrieve the current measure. And yes, that's measures, which is indeed the only time unit you get from PMD. The cues appear to be timed based on beats rather than measures, but the secret there is that ZUN simply wrote Peaceful Romancer in the internal time signature of 1/4. Just in case anyone tries to mod this BGM and starts wondering why the sequence suddenly progresses more slowly. I'll just use beats below since it's shorter.
Any cues that don't appear synced only do so because of – you guessed it – weird ZUN code issues.
🐞 But first, what happens if you run the game on a system without an FM chip? PMD does remain resident in that case, but enters a reduced-functionality mode that refuses to even process song data, leaving you with no BGM beats to sync to. Due to the various ways of setting the tempo in a .M file, it's impossible to just parse out the tempo without reimplementing the entire format, so it makes sense why ZUN just hardcoded a fixed replacement delay of 44 frames per beat. However, 44 frames translate to (44/56.423) ≈ 780 ms ≈ 76.94 BPM, which is ~1.9× slower than Peaceful Romancer's actual ~145-147 BPM.
Discoveries like these always start out as quirks until I find evidence that would promote them to bugs. And sure enough: ZUN renders this entire sequence at the halved frame rate of 28.212 FPS, that slowdown factor is suspiciously close to 2, and the code actually specifies 22 frames. This looks as if ZUN simply didn't realize that 22 frames would only translate to the slightly more correct 153.88 BPM at the native frame rate of 56.423 FPS.
This bug also applies if you deactivated BGM in the Option menu, since ZUN treats both cases identically.
🎺 The very first crossfading animation doesn't appear to be synced to any beat, though? It starts close to but not exactly on beat 5:
This one is quickly explained: ZUN does enter the first screen within 2 frames of Peaceful Romancer's first downbeat on "beat" 3, but each screen actually starts with a 34-frame fade-out of the previous screen before crossfading in the new picture. Hence, most of this apparent delay is taken up by a fade-out from black to black. The remaining 4 frames between the beat and the first visible on-screen pixels can be attributed to double-buffering at the sequence's halved frame rate.
🎺 Also, why does the crossfading animation only use two of the four mask patterns across its 16 frames? This seems like a typo in the code, but was almost certainly done on purpose to make this sequence feel more languid and relaxed. The dequirked version with all four mask patterns looks almost too hectic, especially compared to the single mask pattern that ZUN used for text.
But even after that initial screen, the first two or three text cues on later screens don't appear in sync with the BGM beats either?
As pointed out by the uneven placement of the Reimu and Rika cues.
These are Yuuka's second and third screens; the fact that each character gets its own sequence of pictures is common knowledge by now, right?
To understand this, we have to look at how ZUN defines the target BGM beat for each cue in the first place. There's only a single variable that defines the target beat for the BGM-syncing delay, and ZUN simply adds a certain number of beats to this variable before every cue. In the case of these text cues, he adds 2 beats, which matches what we can observe for the correctly synced cues in the video above. The very first text cue, however, is placed two beats after… the beat the fade-out was started on, even though we've just spent at least 56 frames on the two fading effects. This means that BGM playback will not only have already reached this beat, but will even have progressed about half a beat beyond. Thus, the game just fades in the text immediately…
💣 …except that it doesn't! All of the above was pretty quirky, but then ZUN adds a definite landmine by loading the .PI file with the picture for the next screen right after the fade-in animation. If you just look at the few lines after that load call, this seems like a productive use of an intended 2-beat delay, but we don't actually get that 2-beat delay, as I explained above. Instead, BGM playback gets to progress even further beyond the target beat, by the CPU-specific amount of frames it takes to load that next .PI image on the system the game happens to run on. I've recorded the video above by running the original game on our target Neko Project 66 MHz configuration, and got an additional 17 frames of cue drift, between frames 101 and 118 inclusive. In the end, it takes the first three text cues for the beat target to catch up with the BGM on this system, and we only return to proper syncing with Meira, where the beat target has finally moved ahead of BGM playback.
That .PI load call would have been much more appropriate before the 30-beat delay in front of the fade-out…
💣 Even worse, ZUN also loads a new image on the last screen, which defines no next image. This causes the game to unconditionally load from a null pointer, resulting in a landmine in 📝 the classic sense of the word: You can completely ignore it on PC-98 because
Real Mode just lets you read from address 0000:0000 without a segmentation fault
The far pointer to the handler for INT 0 is highly unlikely to actually point to the name of an existing file
That file is even less likely to be a valid .PI file
The game won't display that image anyway, and free its buffer once the sequence ends shortly after
But you wouldn't want to rely on null pointers being filtered by the platform layer.
Finishing TH03/TH04/TH05 scorefiles
Well, at least as far as decompilation is concerned. Cleaning up all these binary-specific inconsistencies on the debloated branch will be just as annoying as reconstructing them in the first place, and I won't even get it all the way done within these 11 pushes. TH05 made this even worse by continuing its general trend of taking TH04's slightly bloated but overall fine C++ code and needlessly rewriting it in micro-optimized and only semi-decompilable ASM. If you still believe that the master branch is a good foundation for any kind of serious work, this file should convince you otherwise.
Two more discoveries here:
If you game over and continue in-game while having a score that would qualify for the current character/difficulty list, the game automatically enters it with a CONTINUE name while staying within MAIN.EXE. Of course, this means that both games get yet another dedicated piece of code to mutate the High Score list…
🐞 And so, the TH04 variant of this code also gets its own distinct version of the 📝 C integer promotion issue that limits the technically supported score to 959 million points. In an unexpected twist though, TH05's ASM rewrite actually manages to fix this issue in a surprisingly natural way by explicitly performing the necessary calculations on 8-bit registers. On the other hand, fixing it within C++ would have still been totally possible and natural and code-simplifying…
The single biggest source of inconsistencies can be found in the code that recreates corrupted scorefiles. During my tests of the cleaned-up and improved rewrite on the debloated branch, I regularly had to corrupt these files on purpose. File contents getting fully or partially overwritten with 00 bytes is the most common kind of corruption you'd encounter with modern operating systems and SSDs, but hilariously enough, that happens to be the exact kind of corruption these games might even fail to detect. If these 00 bytes cover an entire character-/difficulty-specific section, all three games consider such a zeroed section as valid, since it passes checksum validation?
The deobfuscation algorithm explains why:
// [key1] and [key2] are `uint8_t` as well.
decoded_byte[i] = (key1 + (std::rotr<uint8_t>(encoded_byte[i + 1], 3) ^ key2) + encoded_byte[i]);
When saving a section within these files, the games generate new random values for key1 and key2 and store them directly in the file. Without any kind of hardcoded nonce to perturb the input, this obfuscation scheme thus fully relies on the combination of keys and data to generate random-looking output. Set both of them to 0, and deobfuscation turns into a no-op. Then, a buffer of 00 also sums to 0, which also matches the 0 checksum in the file. In contrast, TH02's obfuscation scheme lacked any source of randomness, but it did cover this exact case…
Here's how such a fully zeroed-out GENSOU.SCR looks like in TH04's and TH05's High Score viewer:
If you remember how GENSOU.SCR saves scores in 📝 this silly gaiji-offsetted way, these screens almost explain themselves. 0 minus 160 will always be an invalid sprite ID, and since master.lib's super_put() doesn't bounds-check sprite IDs, it blindly accesses invalid sprite data and probably ends up filling every VRAM bitplane with 1 bits. After the game spent way too much time rendering this garbage data, we then only end up seeing the sprites that get rendered after the very last score digit.
The VV characters might look especially weird in place of the usual stage number, but they quickly make sense once you remember that these numbers are gaiji rendered to VRAM. The PC-98's character generator simply can't support a gaiji with an ID of 0, since it would have to be encoded as 0x0056, which is indistinguishable from the halfwidth V in ASCII. And since master.lib assumes that all gaiji are fullwidth, we get two of them next to each other.
The visual result for a zeroed-out YUME.NEM in TH03's High Score screen, however, is much more… well-defined:
Since YUME.NEM stores names, scores, and stage numbers as raw sprite IDs, we get sprite #0 from REGI2.BFT for all of them.
AAAAAAAA AAAAAAAAAA A
Finally, I stumbled over a script bug in TH04's Good Ending for Reimu A:
The 2014 static English patch fixes this issue. That's probably why this isn't talked about anywhere.
This looks unintentional, and the same line in Reimu B's Good Ending confirms that this is indeed a typo:
\p,ed07.pi
\=0,4
魔理沙：なんだよ、そりゃ\ga9\s160\c
\p,ed07.pi
\==0,4
魔理沙：なんだよ、そりゃ\ga9\s160\c
The 📝 cutscene command reference tells us that the line in the Reimu B variant is preceded by \==, the picture crossfading command, followed by both possible parameters, 0 and 4. Reimu A's script, however, lacks that second = and instead spells out \=, the immediate picture display command, which doesn't take a second parameter. Thus, the command stops reading after the 0 and leaves the trailing ,4 as text to be displayed in the newly started box. The line break is then ignored as usual, causing 魔理沙 to be displayed right next to these two characters.
Whew! Once again, this did turn into more of the typical ReC98 research by the end after all. And that was just 75% of the pushes assigned to this post, because the rest already went towards the debloating work. Next up: Concluding this series and actually applying all this research to the games.
Surprise! The last missing main menu in PC-98 Touhou was, in fact, not that hard. Finishing the rest of TH03's OP.EXE took slightly shorter than the expected 2 pushes, which left enough room to uncover an unexpected mystery and take important leaps in position independence…
For TH03, ZUN stepped up the visual quality of the main menu items by exchanging TH02's monospaced font with fixed, pre-composited strings of proportional text. While TH04 would later place its menu text in VRAM, TH03 still wanted to stay with TH02's approach of using gaiji to display the menu items on the PC-98 text layer. Since gaiji have a fixed size of 16×16 pixels, this requires the pre-composited bitmaps to be cut into blocks of that size and padded with blank pixels as necessary:
If your combined amount of text is short enough to fit into the PC-98's 256 gaiji slots, this is a nice way of using hardware features to replace the need for a proportional text renderer. It especially simplifies transitions between menus – simply wiping the entire TRAM is both cheap and certainly less error-prone than (un)blitting pixels in VRAM, which 📝 ZUN was always kind of sloppy at.
However, all this text still needs to be composited and cut into gaiji somewhere. If you do that manually, it's easy to lose sight of how the text is supposed to appear on screen, especially if you decide to horizontally center it. Then, you're in for some awkward coordinate fiddling as you try to place these 16-pixel bricks into the 8-pixel text grid to somehow make it all appear centered:
The VS Start menu actually is correctly centered.
Then again, did ZUN actually want to center the Option menu like this? Even the main menu looks kind of uncanny with perfect centering, probably because I'm so used to the original. Imperfect centering usually counts as a bug, but this case is quirky enough to leave it as is. We might want to perfectly center any future translations, but that would definitely cost a bit as I'd then actually need to write that proportional text renderer.
Apart from that, we're left with only a very short list of actual bugs and landmines:
The Cancel key is not handled inside the VS menu, arrgghh…! 🤬
ZUN almost managed to write a title screen and menu without a 📝 screen📝 tearing landmine, but a single one still managed to sneak into the first frame of the title screen's short fade-in animation. This one will blow up when returning from the Music Room, and can be entirely blamed on that screen's choice to leave 📝 a purple color in hardware palette slot 0. Replacing that color with black before returning would have completely hidden the potential tearing.
There might be another one in the long sliding animation, but I can only tell for sure once I've fully decompiled MAINL.EXE.
While the rest of the code is not free of the usual nitpicks, those don't matter in the grand scheme of things. The code for the sliding 東方夢時空
animation is even better: it makes decent use of the EGC and page flipping, and places the 📝 loading calls for the character selection portraits at sensible points where the animation naturally wants to have a delay anyway. We're definitely ending the main menus of PC-98 Touhou on a high note here.
You might have already spotted some unfamiliar text in the gaiji above, and indeed, we've got three pieces of unused text in these two menus! Starting from the top, the label is entirely unused as none of its gaiji IDs are referenced anywhere in the final code. The label's placement within the gaiji IDs would imply that this option was once part of the main menu, but nothing in the game suggests that the main menu ever had a bigger box that could fit a 7th element. On the contrary, every piece of menu code assumes that the box sprites loaded from OPWIN.BFT are exactly 128 pixels high:
Fun fact: The code doesn't even use the 16 pixels in the middle, and instead just assumes that the pixels between the X coordinates of [8; 16[ and [32; 40[ are identical.
The unused MIDI music option has already been widely documented elsewhere. Changing the first byte in YUME.CFG to 02 has no functional effect because ZUN removed most MIDI-related code before release. He did forget a few instances though, and the surviving dedicated switch case in the Option menu is now the entire reason why you can reveal this text without modifying the binary. Changing the option will always flip its value back to either off or FM(86).
Last but not least, we have the label and its associated numbers. These are the most interesting ones in my eyes; nobody talks about them, even though we have definite proof that they were used for the KeyConfig options at some earlier point in development:
That's all I've got about the menus, so let's talk characters and gameplay! When playing Story Mode, OP.EXE picks the opponents for all stages immediately after the 📝 Select screen has faded out. Each character fights a fixed and hardcoded opponent in Stage 7's Decisive Match:
Player
Stage 7 opponent
Reimu
Mima
Mima
Reimu
Marisa
Reimu
Ellen
Marisa
Kotohime
Reimu
Kana
Ellen
Rikako
Kana
Chiyuri
Kotohime
Yumemi
Rikako
The opponents for the first 6 stages, however, are indeed completely random, and picked by master.lib's reimplementation of the Borland RNG. The game only needs to ensure that no character is picked twice, which it does like this:
const int stage_7_opponent = HARDCODED_STAGE_7_OPPONENT_FOR[playchar];
bool opponent_seen[7] = { false };
for(int stage = 0; stage < 6; stage++) {
int candidate;
do {
// Pick a random character between Reimu and Rikako
candidate = (irand() % 7);
} while(opponent_seen[candidate] || (stage_7_opponent == candidate));
opponent_seen[candidate] = true;
story_opponent[stage] = candidate;
}
Characters are numbered from 0 ( Reimu) to 8 ( Yumemi), following the order in the Stage 7 table above.
Yup. For every stage, ZUN re-rolls until the RNG returns a character who hasn't yet been seen in a previous stage – even in Stage 6 where there's only one possible character left. Since each successive stage makes it harder for the inner loop to find a valid answer, you start to wonder if there is some unlucky combination of seed and player character that causes the game to just hang forever.
So I tested all possible 232 seed values for all 9 player characters and… nope, Borland's RNG is good enough to eventually always return the only possible answer. The inner loop for Stage 6 does occasionally run for a disproportionate number of iterations, with the worst case being 134 re-rolls when playing Rikako's Story Mode with a seed value of 0x099BDA86. But even that is many orders of magnitude away from manifesting as any kind of noticeable delay. And on average, it just takes 17.15 iterations to determine all 6 random opponents.
The attract demos are another intriguing aspect that I initially didn't even have on my radar for the main menu. touhou-memories raises an interesting question: The demos start at Gauge and Boss Attack level 9, which would imply Lunatic difficulty, but the enemy formations don't match what you'd normally get on Lunatic. So, which difficulty were they recorded on?
Our already RE'd code clears up the first part of that question. TH03's demos are not recordings, but simply regular VS rounds in CPU vs. CPU mode that automatically quit back to the title screen after 7,000 frames. They can only possibly appear pre-recorded because the game cycles through a mere four hardcoded character pairings with fixed RNG seeds:
Demo #
P1
P2
Seed
1
Mima
Reimu
600
2
Marisa
Rikako
1000
3
Ellen
Kana
3200
4
Kotohime
Marisa
500
Certainly an odd choice if your game already had the feature to let arbitrary CPU-controlled characters fight each other. That would have even naturally worked for the trial version, which doesn't contain demos at all.
Then again, even a "random" character selection would have appeared deterministic to an outside observer. As usual for PC-98 Touhou, the RNG seed is initialized to 0 at startup and then simply increments after every frame you spend on the title screen and inside the top-level main, Option, and character selection menus – and yes, it does stay constant inside the VS Start menu. But since these demos always start after waiting exactly 520 frames on the title screen without pressing any key to enter the main menu, there's no actual source of randomness anywhere. ZUN could have classically initialized the RNG with the current system time, which is what we used to do back in the day before operating systems had easily accessible APIs for true randomness, but he chose not to, for whatever reason.
The difficulty question, however, is not so easy to answer. The demo startup code in the main menu doesn't override the configured difficulty, and neither does any other of the binaries depending on the demo ID. This seems to suggest that the demos simply run at the difficulty you last configured in the Option menu, just like regular VS matches. But then, you'd expect them to run differently depending on that difficulty, which they demonstrably don't. They always start on Gauge and Boss Attack level 9, and their last frame before the exit animation is always identical, right down to the score, reinforcing the pre-recorded impression:
Note that it takes much longer than the expected 2:04 minutes for the game to reach this end state. Each WARNING!! You are forced to evade / Your life is in peril popup freezes gameplay for 26 frames which don't count toward the demo frame counter. That's why these popups will provide such a great 📝 resynchronization opportunity for netplay. It's almost as if Versus Touhou was designed from the start with rollback netcode in mind!
With quite a bit of time left over in the second push, it made sense to look at a bit of code around the Gauge and Boss Attack levels to hopefully get a better idea of what's going on there. The Gauge Attack levels are very straightforward – they can range from 1 to 16 inclusive, which matches the range that the game can depict with its gaiji, and all parts of the game agree about how they're interpreted:
Stored in GAMEFT.BFT.
The same can't be said about the Boss Attack level though, as the gauge and the WARNING!! popup interpret the same internal variable as two different levels?
This apparent inconsistency raises quite a few questions. After all, these gaiji have to be addressed by adding an offset from 0 to 15 to the ID of the 1 gaiji, but the levels are supposed to range from 1 to 16. Does this mean that one of these two displays has an off-by-one error? You can't fire a Level 0 Boss Attack because the level always increments before every attack, but would 0 still be a technically valid Boss Attack level?
Decompiling the static HUD code debunks at least the first question as ZUN resolves the apparent off-by-one error by explicitly capping the displayed level to 16. And indeed, if a round lasts until the maximum Boss Attack level, the two numbers end up matching:
This suggests that the popup indicates the level of the incoming attack while the gauge indicates the level of the next one to be fired by any player. That said, this theory not only needs tons of comments to explain it within the code, but also contradicts 夢時空.TXT, which explicitly describes the level next to the gauge as the 現在のＢＯＳＳアタックのレベル. Still, it remains our best bet until we've decompiled a few of the Boss Attacks and saw how they actually use this single variable.
So, what does this tell us about the demo difficulty? Now that we can search the code for these variables, we quickly come across the dedicated demo-specific branch that initializes these levels to the observable fixed values, along with two other variables I haven't researched so far. This confirms that demos run at a custom difficulty, as the two other variables receive slightly different values in regular gameplay.
However, it's still a good idea to check the code for any other potential effects of the difficulty setting. Maybe they're just hard to spot in demos? Doesn't difficulty typically affect a whole lot of other things in Touhou game code? Well, not in TH03 – MAIN.EXE only ever looks at the configured difficulty in three places, and all of them are part of the code that initializes a new round.
This reveals the true nature of difficulty in TH03: It's exclusively specified in terms of these five variables, and the Easy/Normal/Hard/Lunatic/"Demo" settings can be thought of as simply being presets for them. Story Mode adds 📝 the AI's number of safety frames to the list of variables and factors the current stage number into their values, but the concept stays the same. In this regard, TH03's design is unusually clean, making it perhaps the only Touhou game with not even a single "if difficulty is this, then do that" branch in script code. It's certainly the only PC-98 Touhou game with this property.
But it gets even better if we consider what this means for netplay. We now know that the configured difficulty is part of the match-defining parameters that must be synced between both players, just like the selected characters and the RNG seed. But why stop there? How about letting players not just choose between the presets, but allowing them to customize each of the five variables independently? Boom, we've just skyrocketed the replay value of netplay. 🚀 It's discoveries like these that justify my decision to start the road toward netplay by decompiling all of OP.EXE: In-engine menus are the cleanest and most friendly way of allowing players to configure all these variables, and now they're also the easiest and most natural choice from a technical point of view.
But wait, there's still some time left in that second push! The remaining fraction of the OP.EXE reverse-engineering contribution had repeating decimals, so let's do some quick TH02 PI work to remove the matching instance of repeating decimals from the backlog. This was very much a continuation of 📝 last year's light PI work; while the regular TH02 decompilation progress has focused and will continue to focus on the big features, it still left plenty of low-hanging PI fruit in boss code. Back then, we left with the positions of the Five Magic Stones, where ZUN's choice of storing them in arrays was almost revolutionary compared to what we saw in TH01. The same now applies to the state flags and total damage amount of not just the boss of Stage 3, but also the two independently damageable entities of the stage's midboss. In total, all of the newly identified arrays made up 3.36% of all memory references in TH02, and we're not even done with Stage 3.
Actually, you know what, let's round out that second push with even more low-hanging PI fruit and ensure 📝 technical position independence for TH03's MAINL.EXE. This was very helpful considering that I'm going to build netplay into the anniversary branch, whose debloated foundation 📝 aims to merge every game into as few executables as possible. Due to TH03's overall lower level of bloat and the dedicated SPRITE16-based rendering code in MAIN.EXE, it might not make as much sense to merge all three of TH03's .EXE binaries as it did for TH01, and MAIN.EXE's lack of position independence currently prevents this anyway. However, merging just OP.EXE and MAINL.EXE makes tremendous sense not just for TH03, but for the other three games as well. These binaries have a much smaller ratio of ZUN code to library code, and use the same file formats and subsystems.
But that's not even the best part! Once we've factored out all the invisible inconsistencies between the games, we get to share all of this code across all of the four games. Hence, technical position independence for TH03's MAINL.EXE also was the final obstacle in the way of a single consistent and ultimately portable version of all of this code. 🙌
So, how do we go from here to 📝 the short-term half-PC-98/half-modern netplay option that Ember2528 is now funding? Most of the netcode will be unrelated to TH03 in particular, but we'd obviously still want to reverse-engineer more of MAIN.EXE to ensure a high-quality integration. So how about alternating the upcoming deliveries between pure RE work and any new or modded code? Next up, therefore, I'll go for the latter and debloat OP.EXE so that I can later add the netplay features without pulling my hair out. At that point, it also makes sense to take the first steps into portability; I've got some initial ideas I'm excited to implement, and Congrio's tiny bit of funding just begs to be removed from the backlog.
(And I'm definitely going to defuse all the tearing landmines because my goodness are they infuriating when slowing down the game or working with screen recordings.)
P0148
TH04/TH05 decompilation (Text popups, gather circle rendering, player position clamping)
💰 Funded by:
[Anonymous]
🏷️ Tags:
Back after taking way too long to get Touhou Patch Center's MediaWiki
update feature complete… I'm still waiting for more translators to test and
review the new translation interface before delivering and deploying it
all, which will most likely lead to another break from ReC98 within the
next few months. For now though, I'm happy to have mostly addressed the
nagging responsibility I still had after willing that site into existence,
and to be back working on ReC98. 🙂
As announced, the next few pushes will focus on TH04's and TH05's bullet
spawning code, before I get to put all that accumulated TH01 money towards
finishing all of konngara's code in TH01. For a full
picture of what's happening with bullets, we'd really also like to
have the bullet update function as readable C code though.
Clearing all bullets on the playfield will trigger a Bonus!! popup,
displayed as 📝 gaiji in that proportional
font. Unfortunately, TLINK refused to link the code as soon as I referenced
the function for animating the popups at the top of the playfield? Which
can only mean that we have to decompile that function first…
So, let's turn that piece of technical debt into a full push, and first
decompile another random set of previously reverse-engineered TH04 and TH05
functions. Most of these are stored in a different place within the two
MAIN.EXE binaries, and the tried-and-true method of matching
segment names would therefore have introduced several unnecessary
translation units. So I resorted to a segment splitting technique I should
have started using way earlier: Simply creating new segments with names
derived from their functions, at the exact positions they're needed. All
the new segment start and end directives do bloat the ASM code somewhat,
and certainly contributed to this push barely removing any actual lines of
code. However, what we get in return is total freedom as far as
decompilation order is concerned,
📝 which should be the case for any ReC project, really.
And in the end, all these tiny code segments will cancel out anyway.
If only we could do the same with the data segment…
The popup function happened to be the final one I RE'd before my long break
in the spring of 2019. Back then, I didn't even bother looking into that
64-frame delay between changing popups, and what that meant for the game.
Each of these popups stays on screen for 128 frames, during which, of
course, another popup-worthy event might happen. Handling this cleanly
without removing previous popups too early would involve some sort of event
queue, whose size might even be meaningfully limited to the number of
distinct events that can happen. But still, that'd be a data structure, and
we're not gonna have that! Instead, ZUN
simply keeps two variables for the new and current popup ID. During an
active popup, any change to that ID will only be committed once the current
popup has been shown for at least 64 frames. And during that time,
that new ID can be freely overwritten with a different one, which drops any
previous, undisplayed event. But surely, there won't be more than two
events happening within 63 frames, right?
The rest was fairly uneventful – no newly RE'd functions in this push,
after all – until I reached the widely used helper function for applying
the current vertical scrolling offset to a Y coordinate. Its combination of
a function parameter, the pascal calling convention, and no
stack frame was previously thought to be undecompilable… except that it
isn't, and the decompilation didn't even require any new workarounds to be
developed? Good thing that I already forgot how impossible it was to
decompile the first function I looked at that fell into this category!
Oh well, this discovery wasn't too groundbreaking. Looking back at
all the other functions with that combination only revealed a grand total
of 1 additional one where a decompilation made sense: TH05's version of
snd_kaja_interrupt(), which is now compiled from the same C++
file for all 4 games that use it. And well, looks like some quirks really
remain unnoticed and undocumented until you look at a function for the 11th
time: Its return value is undefined if BGM is inactive – that is, if the
user disabled it, or if no FM board is installed. Not that it matters for
the original code, which never uses this function to retrieve anything from
KAJA's drivers. But people apparently do copy ReC98 code into their own
projects, so it is something to keep in mind.
All in all, nothing quite at jank level in this one, but we were surely grazing that tag. Next up, with that out of the way: The bullet update/step function! Very soon in fact, since I've mostly got it done already.
P0117
Rebuilding ZUN.COM + overall position independence
💰 Funded by:
[Anonymous]
🏷️ Tags:
Wouldn't it be a bit disappointing to have TH05 completely
position-independent, but have it still require hex-editing of the
original ZUN.COM to mod its gaiji characters? As in, these
custom "text" glyphs, available to the PC-98 text RAM:
Especially since we now even have a sprite converter… the lack of which
was exactly 📝 what made rebuilding ZUN.COM not that worthwhile before.
So, before the big release, let's get all the remaining
ZUN.COM sub-binaries of TH04 and TH05 dumped into .ASM files,
and re-assembled and linked during the build process.
This is also the moment in which Egor's 2018
reimplementation of O. Morikawa's comcstm finally gets
to shine. Back then, I considered it too early to even bother with
ZUN.COM and reimplementing the .COM wrapper that ZUN
originally used to bundle multiple smaller executables into that single
binary. But now that the time is right, it is nice to have that
code, as it allowed me to get these rebuilds done in half a push.
Otherwise, it would have surely required one or two dedicated ones.
Since we like correctness here, newly dumped ZUN code means that it also
has to be included in the RE%
baseline calculation. This is why TH04's and TH05's overall RE% bars
have gone back a tiny bit… in case you remember how they previously looked
like After all, I would like to figure
out where all that memory allocated during TH04's and TH05's memory check
is freed, if at all.
Alright, one half of a push left… Y'know, getting rid of those last few PI
false positives is actually one of the most annoying chores in this
project, and quite stressful as well: I have to convince myself that the
remaining false positives are, in fact, not memory references, but with
way too little time for in-depth RE and to denote what they are
instead. In that situation, everyone (including myself!)
is anticipating that PI goal, and no one is really interested in RE.
(Well… that is, until they actually get to developing their mod. But more
on that tomorrow. ) Which means that it boils
down to quite some hasty, dumb, and superficial RE around those remaining
numbers.
So, in the hope of making it less annoying for the other 4 games in the
future, let's systematically cover the sources of those remaining false
positives in TH05, over all games. I/O port accesses with either the port
or the value in registers (and thus, no longer as an immediate argument to
the IN or OUT instructions, which the PI counter
can clearly ingore), palette color arithmetic, or heck, 0xFF constants that
obviously just mean "-1" and are not a reference to offset 0xFF in
the data segment. All of this, of course, once again had a way bigger
effect on everything but an almost position-independent TH05… but
hey, that's the sort of thing you reserve the "anything" pushes for. And
that's also how we get some of the single biggest PI% gains we have seen
so far, and will be seeing before the 100% PI mark. And yes, those will
continue in the next push.
Well, that took twice as long as I thought, with the two pushes containing
a lot more maintenance than actual new research. Spending some time
improving both field names and types in
32th System's
TH03 resident structure finally gives us all of those
structures. Which means that we can now cover all the remaining
decompilable ZUN.COM parts at once…
Oh wait, their main() functions have stayed largely identical
since TH02? Time to clean up and separate that first, then… and combine
two recent code generation observations into the solution to a
decompilation puzzle from 4½ years ago. Alright, time to decomp-
Oh wait, we'd kinda like to properly RE all the code in TH03-TH05
that deals with loading and saving .CFG files. Almost every outside
contributor wanted to grab this supposedly low-hanging fruit a lot
earlier, but (of course) always just for a single game, while missing how
the format evolved.
So, ZUN.COM. For some reason, people seem to consider it
particularly important, even though it contains neither any game logic nor
any code specific to PC-98 hardware… All that this decompilable part does
is to initialize a game's .CFG file, allocate an empty resident structure
using master.lib functions, release it after you quit the game,
error-check all that, and print some playful messages~ (OK, TH05's also
directly fills the resident structure with all data from
MIKO.CFG, which all the other games do in OP.EXE.)
At least modders can now freely change and extend all the resident
structures, as well as the .CFG files? And translators can translate those
messages that you won't see on a decently fast emulator anyway? Have fun,
I guess 🤷
And you can in fact do this right now – even for TH04 and TH05,
whose ZUN.COM currently isn't rebuilt by ReC98. There is
actually a rather involved reason for this:
One of the missing files is TH05's GJINIT.COM.
Which contains all of TH05's gaiji characters in hardcoded 1bpp form,
together with a bit of ASM for writing them to the PC-98's hardware gaiji
RAM
Which means we'd ideally first like to have a sprite compiler, for
all the hardcoded 1bpp sprites
Which must compile to an ASM slice in the meantime, but should also
output directly to an OMF .OBJ file (for performance now), as well as to C
code (for portability later)
Which I won't put in as long as the backlog contains actual
progress to drive up the percentages on the front page.
So yeah, no meaningful RE and PI progress at any of these levels. Heck,
even as a modder, you can just replace the zun zun_res
(TH02), zun -5 (TH03), or zun -s (TH04/TH05)
calls in GAME.BAT with a direct call to your modified
*RES*.COM. And with the alternative being "manually typing 0 and 1
bits into a text file", editing the sprites in TH05's
GJINIT.COM is way more comfortable in a binary sprite editor
anyway.
For me though, the best part in all of this was that it finally made sense
to throw out the old Borland C++ run-time assembly slices 🗑 This giant
waste of time
became obvious 5 years ago, but any ASM dump of a .COM
file would have needed rather ugly workarounds without those slices. Now
that all .COM binaries that were originally written in C are
compiled from C, we can all enjoy slightly faster grepping over the entire
repository, which now has 229 fewer files. Productivity will skyrocket!
Next up: Three weeks of almost full-time ReC98 work! Two more PI-focused
pushes to finish this TH05 stretch first, before switching priorities to
TH01 again.
The remaining 4 frames between the beat and the first visible on-screen pixels can be attributed to double-buffering at the sequence's halved frame rate.
And that was just 75% of the pushes assigned to this post, because the rest already went towards the debloating work. Next up: Concluding this series and actually applying all this research to the games.
label is entirely unused as none of its gaiji IDs are referenced anywhere in the final code. The label's placement within the gaiji IDs would imply that this option was once part of the main menu, but nothing in the game suggests that the main menu ever had a bigger box that could fit a 7th element. On the contrary, every piece of menu code assumes that the box sprites loaded from 
label and its associated numbers. These are the most interesting ones in my eyes; nobody talks about them, even though we have definite proof that they were used for the KeyConfig options at some earlier point in development:
Reimu
Mima
Marisa
Ellen
Kotohime
Kana
Rikako
Chiyuri
Yumemi
