⮜ Blog

⮜ List of tags

Showing all posts tagged hidden-content- and waste-

📝 Posted:
🚚 Summary of:
P0128, P0129
Commits:
dc65b59...dde36f7, dde36f7...f4c2e45
💰 Funded by:
Yanga
🏷 Tags:
rec98+ th01+ file-format+ gameplay+ card-flipping+ waste- hidden-content- bug+

So, only one card-flipping function missing, and then we can start decompiling TH01's two final bosses? Unfortunately, that had to be the one big function that initializes and renders all gameplay objects. #17 on the list of longest functions in all of PC-98 Touhou, requiring two pushes to fully understand what's going on there… and then it immediately returns for all "boss" stages whose number is divisible by 5, yet is still called during Sariel's and Konngara's initialization 🤦

Oh well. This also involved the final file format we hadn't looked at yet – the STAGE?.DAT files that describe the layout for all stages within a single 5-stage scene. Which, for a change is a very well-designed form– no, of course it's completely weird, what did you expect? Development must have looked somewhat like this:

  • Weirdness #1: :zunpet: "Hm, the stage format should include the file names for the background graphics and music… or should it?" And so, the 22-byte header still references some music and background files that aren't part of the final game. The game doesn't use anything from there, and instead derives those file names from the scene ID.
    That's probably nothing new to anyone who has ever looked at TH01's data files. In a slightly more interesting discovery though, seeing the+ 📝 .GRF extension, in some of the file names that are short enough to not cut it off, confirms that .GRF was initially used for background images. Probably before ZUN learned about .PI, and how it achieves better compression than his own per-bitplane RLE approach?
  • Weirdness #2: :zunpet: "Hm, I might want to put obstacles on top of cards?" You'd probably expect this format to contain one single array for every stage, describing which object to place on every 32×32 tile, if any. Well, the real format uses two arrays: One for the cards, and a combined one for all "obstacles" – bumpers, bumper bars, turrets, and portals. However, none of the card-flipping stages in the final game come with any such overlaps. That's quite unfortunate, as it would have made for some quite interesting level designs:

    As you can see, the final version of the blitting code was not written with such overlaps in mind either, blitting the cards on top of all the obstacles, and not the other way round.

  • Weirdness #3: :zunpet: "In contrast to obstacles, of which there are multiple types, cards only really need 1 bit. Time for some bit twiddling!" Not the worst idea, given that the 640×336 playfield can fit 20×10 cards, which would fit exactly into 25 bytes if you use a single bit to indicate card or no card. But for whatever reason, ZUN only stored 4 card bits per byte, leaving the other 4 bits unused, and needlessly blowing up that array to 50 bytes. 🤷

    Oh, and did I mention that the contents of the STAGE?.DAT files are loaded into the main data segment, even though the game immediately parses them into something more conveniently accessible? That's another 1250 bytes of memory wasted for no reason…

  • Weirdness #4: :zunpet: "Hm, how about requiring the player to flip some of the cards multiple times? But I've already written all this bit twiddling code to store 4 cards in 1 byte. And if cards should need anywhere from 1 to 4 flips, that would need at least 2 more bits, which won't fit into the unused 4 bits either…" This feature must have come later, because the final game uses 3 "obstacle" type IDs to act as a flip count modifier for a card at the same relative array position. Complete with lookup code to find the actual card index these modifiers belong to, and ridiculous switch statements to not include those non-obstacles in the game's internal obstacle array. :tannedcirno:

With all that, it's almost not worth mentioning how there are 12 turret types, which only differ in which hardcoded pellet group they fire at a hardcoded interval of either 100 or 200 frames, and that they're all explicitly spelled out in every single switch statement. Or how the layout of the internal card and obstacle SoA classes is quite disjointed. So here's the new ZUN bugs you've probably already been expecting!


Cards and obstacles are blitted to both VRAM pages. This way, any other entities moving on top of them can simply be unblitted by restoring pixels from VRAM page 1, without requiring the stationary objects to be redrawn from main memory. Obviously, the backgrounds behind the cards have to be stored somewhere, since the player can remove them. For faster transitions between stages of a scene, ZUN chose to store the backgrounds behind obstacles as well. This way, the background image really only needs to be blitted for the first stage in a scene.

All that memory for the object backgrounds adds up quite a bit though. ZUN actually made the correct choice here and picked a memory allocation function that can return more than the 64 KiB of a single x86 Real Mode segment. He then accesses the individual backgrounds via regular array subscripts… and that's where the bug lies, because he stores the returned address in a regular far pointer rather than a huge one. This way, the game still can only display a total of 102 objects (i. e., cards and obstacles combined) per stage, without any unblitting glitches.
What a shame, that limit could have been 127 if ZUN didn't needlessly allocate memory for alpha planes when backing up VRAM content. :onricdennat:

And since array subscripts on far pointers wrap around after 64 KiB, trying to save the background of the 103rd object is guaranteed to corrupt the memory block header at the beginning of the returned segment. :zunpet:. When TH01 runs in test mode, it correctly reports a corrupted heap in this case.


Finally, some unused content! Upon discovering TH01's debug mode, probably everyone tried to access Stage 21, just to see what happens, and indeed landed in an actual stage, with a black background and a weird color palette. Turns out that ZUN did ship an unused scene in SCENE7.DAT, which is exactly was loaded there.
Unfortunately, it's easy to believe that this is just garbage data (as I initially did): At the beginning of "Stage 22", the game seems to enter an infinite loop somewhere during the flip-in animation.

Well, we've had a heap overflow above, and the cause here is nothing but a stack buffer overflow – a perhaps more modern kind of classic C bug, given its prevalence in the Windows Touhou games. Explained in a few lines of code:

void stageobjs_init_and_render()
{
	int card_animation_frames[50]; // even though there can be up to 200?!
	int total_frames = 0;

	(code that would end up resetting total_frames if it ever tried to reset
	card_animation_frames[50]…)
}
The number of cards in "Stage 22"? 76. There you have it.

But of course, it's trivial to disable this animation and fix these stage transitions. So here they are, Stages 21 to 24, as shipped with the game in STAGE7.DAT:


Wow, what a mess. All that was just a bit too much to be covered in two pushes… Next up, assuming the current subscriptions: Taking a vacation with one smaller TH01 push, covering some smaller functions here and there to ensure some uninterrupted Konngara progress later on.

📝 Posted:
🚚 Summary of:
P0124, P0125
Commits:
72dfa09...056b1c7, 056b1c7...f6a3246
💰 Funded by:
Blue Bolt, [Anonymous]
🏷 Tags:
rec98+ th02+ th04+ pc98+ menu+ waste- master.lib+ waste-

Turns out that TH04's player selection menu is exactly three times as complicated as TH05's. Two screens for character and shot type rather than one, and a way more intricate implementation for saving and restoring the background behind the raised top and left edges of a character picture when moving the cursor between Reimu and Marisa. TH04 decides to backup precisely only the two 256×8 (top) and 8×244 (left) strips behind the edges, indicated in red in the picture below.

These take up just 4 KB of heap memory… but require custom blitting functions, and expanding this explicitly hardcoded approach to TH05's 4 characters would have been pretty annoying. So, rather than, uh, not explicitly hardcoding it all, ZUN decided to just be lazy with the backup area in TH05, saving the entire 640×400 screen, and thus spending 128 KB of heap memory on this rather simple selection shadow effect. :zunpet:


So, this really wasn't something to quickly get done during the first half of a push, even after already having done TH05's equivalent of this menu. But since life is very busy right now, I also used the occasion to start addressing another code organization annoyance: master.lib's single master.h header file.

  • Now that ReC98 is trying to develop (or at least mimic) a more type-safe C++ foundation to model the PC-98 hardware, a pure C header (with counter-productive C++ extensions) is becoming increasingly unidiomatic. By moving some of the original assumptions about function parameters into the type system, we can also reduce the reliance on its Japanese-only documentation without having to translate it :tannedcirno:
  • It's far from complete with regards to providing compile-time PC-98 hardware constants and helpful types. In fact, we started to add these in our own header files a long time ago.
  • It's quite bloated, with at least 2800 lines of code that currently are #included into the vast majority of files, not counting master.h's recursively included C standard library headers. PC-98 Touhou only makes direct use of a rather small fraction of its contents.
  • And finally, all the DOS/V compatibility definitions are especially useless in the context of ReC98. As I've noted 📝 time and 📝 time again, porting PC-98 Touhou to IBM-compatible DOS won't be easy, and MASTER_DOSV won't be helping much. Therefore, my upstream version of ReC98 will never include all of master.lib. There's no point in lengthening compile times for everyone by default, and those will be getting quite noticeable after moving to a full 16-bit build process.
    (Actually, what retro system ports should rather be doing: Get rid of master.lib's original ASM code, replace it with readable, modern C++, and then simply convert the optimized assembly output of modern compilers to your ISA of choice. Improving the landscape of such assembly or object file converters would benefit everyone!)

So, time to start a new master.hpp header that would contain just the declarations from master.h that PC-98 Touhou actually needs, plus some semantic (yes, semantic) sugar. Comparing just the old master.h to just the new master.hpp after roughly 60% of the transition has been completed, we get median build times of 319 ms for master.h, and 144 ms for master.hpp on my (admittedly rather slow) DOSBox setup. Nice!
As of this push, ReC98 consists of 107 translation units that have to be compiled with Turbo C++ 4.0J. Fully rebuilding all of these currently takes roughly 37.5 seconds in DOSBox. After the transition to master.hpp is done, we could therefore shave some 10 to 15 seconds off this time, simply by switching header files. And that's just the beginning, as this will also pave the way for further #include optimizations. Life in this codebase will be great!


Unfortunately, there wasn't enough time to repay some of the actual technical debt I was looking forward to, after all of this. Oh well, at least we now also have nice identifiers for the three different boldface options that are used when rendering text to VRAM, after procrastinating that issue for almost 11 months. Next up, assuming the existing subscriptions: More ridiculous decompilations of things that definitely weren't originally written in C, and a big blocker in TH03's MAIN.EXE.