Secured a 30-hour RL workweek to leave plenty of time for this project, Touhou Patch Center's commissioned MediaWiki update work is also nearing completion, time to reopen the store! Since it's been a long time, here's an overview of where we currently are in each game and binary, and what the next logical step would be:

• TH01:
  • OP.EXE: Main menu and game initialization.
  • REIIDEN.EXE: Konngara's boss script code. We finished 2020 by reverse-engineering everything it references, so Konngara is expected to get done quite quickly – especially in terms of RE% numbers.
  • FUUIN.EXE: The verdict screen, followed by the boss slideshow.
• TH02:
  • MAIN.EXE: Got a few variables and functions here and there, but not a single gameplay-related structure yet. The game seems to have a weird sprite system; looks like that'll be first.
  • MAINE.EXE: The ending sequences.
• TH03:
  • OP.EXE: Story Mode initialization.
  • MAIN.EXE: The enemy structure? Or the bullet structure, maybe? Either way, we're still missing a lot of essential gameplay structures. At least we did make 📝 first 📝 steps in this game, in contrast to TH02.
  • MAINL.EXE: The stage end / VS dialogue followed by the stage start screen, or the High Score menu, after some groundwork.
• TH04:
  • OP.EXE: The game title slide-in animation, followed by the High Score menu and the Music Room. The latter will be done in parallel with the TH05 version.
  • MAIN.EXE:
    • Turns out that this game has a 📝 shared structure for custom entity types after all. Looking at that one would also get us pretty close to 100% PI.
    • Otherwise, we could decompile the main bullet update functions, …
    • … or the big "Master Spark" laser, with Gengetsu's boss script afterward, …
    • … or HUD rendering (in parallel with TH05).
  • MAINE.EXE: High score name entry, or ending script parsing in parallel with TH05.
• TH05:
  • OP.EXE: High Score menu, followed by the Music Room. Will be done in parallel with TH04's version.
  • MAIN.EXE: Finishing PI created quite a lot of segment splits where we could immediately continue:
    • bullet spawning (self-modifying code that can't be decompiled to C++ in this game, unfortunately)
    • items and extends
    • the Mai & Yuki fight script
    • the EX-Alice fight script
    • bomb rendering
    • Shinki's background animations
    • HUD rendering (in parallel with TH04, and getting us right in front of some boss sprite animations)
    • player shot collision detection and rendering
  • MAINE.EXE: The staff roll animation, or ending script parsing in parallel with TH04.

But as always, you can request pretty much any other part of any game. We're now at a pretty good place as far as arbitrary requests are concerned, as I simply can't decide myself where to put all the current pending contributions in the funding backlog. 😅 By spending the missing or , you can capture any of those "fractional" contributions towards a specific goal.

The next specific requests are going to set the priorities of this project for quite some time! The best strategy: Spend a low amount of money on something very specific, and watch as existing generic contributions will necessarily have to be put towards making that specific goal happen 😛

(Wait, with -Tom-'s 66% discount on TH05, does that mean you could secure that a pending 0.66 contribution will go towards TH05 by spending as little as , with -Tom- providing the remaining to make a full push? Same with a pending 0.33 contribution and spending …)

Whoops, the build was broken again? Since P0127 from mid-November 2020, on TASM32 version 5.3, which also happens to be the one in the DevKit… That version changed the alignment for the default segments of certain memory models when requesting .386 support. And since redefining segment alignment apparently is highly illegal and absolutely has to be a build error, some of the stand-alone .ASM translation units didn't assemble anymore on this version. I've only spotted this on my own because I casually compiled ReC98 somewhere else – on my development system, I happened to have TASM32 version 5.0 in the PATH during all this time.
At least this was a good occasion to get rid of some weird segment alignment workarounds from 2015, and replace them with the superior convention of using the USE16 modifier for the .MODEL directive.

ReC98 would highly benefit from a build server – both in order to immediately spot issues like this one, and as a service for modders. Even more so than the usual open-source project of its size, I would say. But that might be exactly because it doesn't seem like something you can trivially outsource to one of the big CI providers for open-source projects, and quickly set it up with a few lines of YAML.
That might still work in the beginning, and we might get by with a regular 64-bit Windows 10 and DOSBox running the exact build tools from the DevKit. Ideally, though, such a server should really run the optimal configuration of a 32-bit Windows 10, allowing both the 32-bit and the 16-bit build step to run natively, which already is something that no popular CI service out there offers. Then, we'd optimally expand to Linux, every other Windows version down to 95, emulated PC-98 systems, other TASM versions… yeah, it'd be a lot. An experimental project all on its own, with additional hosting costs and probably diminishing returns, the more it expands…
I've added it as a category to the order form, let's see how much interest there is once the store reopens (which will be at the beginning of May, at the latest). That aside, it would 📝 also be a great project for outside contributors!

So, technical debt, part 8… and right away, we're faced with TH03's low-level input function, which 📝 once 📝 again 📝 insists on being word-aligned in a way we can't fake without duplicating translation units. Being undecompilable isn't exactly the best property for a function that has been interesting to modders in the past: In 2018, spaztron64 created an ASM-level mod that hardcoded more ergonomic key bindings for human-vs-human multiplayer mode: 2021-04-04-TH03-WASD-2player.zip However, this remapping attempt remained quite limited, since we hadn't (and still haven't) reached full position independence for TH03 yet. There's quite some potential for size optimizations in this function, which would allow more BIOS key groups to already be used right now, but it's not all that obvious to modders who aren't intimately familiar with x86 ASM. Therefore, I really wouldn't want to keep such a long and important function in ASM if we don't absolutely have to…

… and apparently, that's all the motivation I needed? So I took the risk, and spent the first half of this push on reverse-engineering TCC.EXE, to hopefully find a way to get word-aligned code segments out of Turbo C++ after all.

And there is! The -WX option, used for creating DPMI applications, messes up all sorts of code generation aspects in weird ways, but does in fact mark the code segment as word-aligned. We can consider ourselves quite lucky that we get to use Turbo C++ 4.0, because this feature isn't available in any previous version of Borland's C++ compilers.
That allowed us to restore all the decompilations I previously threw away… well, two of the three, that lookup table generator was too much of a mess in C. But what an abuse this is. The subtly different code generation has basically required one creative workaround per usage of -WX. For example, enabling that option causes the regular PUSH BP and POP BP prolog and epilog instructions to be wrapped with INC BP and DEC BP, for some reason:

a_function_compiled_with_wx proc
	inc 	bp    	; ???
	push	bp
	mov 	bp, sp
	    	      	; [… function code …]
	pop 	bp
	dec 	bp    	; ???
	ret
a_function_compiled_with_wx endp

Luckily again, all the functions that currently require -WX don't set up a stack frame and don't take any parameters.
While this hasn't directly been an issue so far, it's been pretty close: snd_se_reset(void) is one of the functions that require word alignment. Previously, it shared a translation unit with the immediately following snd_se_play(int new_se), which does take a parameter, and therefore would have had its prolog and epilog code messed up by -WX. Since the latter function has a consistent (and thus, fakeable) alignment, I simply split that code segment into two, with a new -WX translation unit for just snd_se_reset(void). Problem solved – after all, two C++ translation units are still better than one ASM translation unit. Especially with all the previous #include improvements.

The rest was more of the usual, getting us 74% done with repaying the technical debt in the SHARED segment. A lot of the remaining 26% is TH04 needing to catch up with TH03 and TH05, which takes comparatively little time. With some good luck, we might get this done within the next push… that is, if we aren't confronted with all too many more disgusting decompilations, like the two functions that ended this push. If we are, we might be needing 10 pushes to complete this after all, but that piece of research was definitely worth the delay. Next up: One more of these.

Alright, back to continuing the master.hpp transition started in P0124, and repaying technical debt. The last blog post already announced some ridiculous decompilations… and in fact, not a single one of the functions in these two pushes was decompilable into idiomatic C/C++ code.

As usual, that didn't keep me from trying though. The TH04 and TH05 version of the infamous 16-pixel-aligned, EGC-accelerated rectangle blitting function from page 1 to page 0 was fairly average as far as unreasonable decompilations are concerned.
The big blocker in TH03's MAIN.EXE, however, turned out to be the .MRS functions, used to render the gauge attack portraits and bomb backgrounds. The blitting code there uses the additional FS and GS segment registers provided by the Intel 386… which

1. are not supported by Turbo C++'s inline assembler, and
2. can't be turned into pointers, due to a compiler bug in Turbo C++ that generates wrong segment prefix opcodes for the _FS and _GS pseudo-registers.

Apparently I'm the first one to even try doing that with this compiler? I haven't found any other mention of this bug…
Compiling via assembly (#pragma inline) would work around this bug and generate the correct instructions. But that would incur yet another dependency on a 16-bit TASM, for something honestly quite insignificant.

What we can always do, however, is using __emit__() to simply output x86 opcodes anywhere in a function. Unlike spelled-out inline assembly, that can even be used in helper functions that are supposed to inline… which does in fact allow us to fully abstract away this compiler bug. Regular if() comparisons with pseudo-registers wouldn't inline, but "converting" them into C++ template function specializations does. All that's left is some C preprocessor abuse to turn the pseudo-registers into types, and then we do retain a normal-looking poke() call in the blitting functions in the end. 🤯

Yeah… the result is batshit insane. I may have gone too far in a few places…

One might certainly argue that all these ridiculous decompilations actually hurt the preservation angle of this project. "Clearly, ZUN couldn't have possibly written such unreasonable C++ code. So why pretend he did, and not just keep it all in its more natural ASM form?" Well, there are several reasons:

• Future port authors will merely have to translate all the pseudo-registers and inline assembly to C++. For the former, this is typically as easy as replacing them with newly declared local variables. No need to bother with function prolog and epilog code, calling conventions, or the build system.
• No duplication of constants and structures in ASM land.
• As a more expressive language, C++ can document the code much better. Meticulous documentation seems to have become the main attraction of ReC98 these days – I've seen it appreciated quite a number of times, and the continued financial support of all the backers speaks volumes. Mods, on the other hand, are still a rather rare sight.
• Having as few .ASM files in the source tree as possible looks better to casual visitors who just look at GitHub's repo language breakdown. This way, ReC98 will also turn from an "Assembly project" to its rightful state of "C++ project" much sooner.
• And finally, it's not like the ASM versions are gone – they're still part of the Git history.

Unfortunately, these pushes also demonstrated a second disadvantage in trying to decompile everything possible: Since Turbo C++ lacks TASM's fine-grained ability to enforce code alignment on certain multiples of bytes, it might actually be unfeasible to link in a C-compiled object file at its intended original position in some of the .EXE files it's used in. Which… you're only going to notice once you encounter such a case. Due to the slightly jumbled order of functions in the 📝 second, shared code segment, that might be long after you decompiled and successfully linked in the function everywhere else.

And then you'll have to throw away that decompilation after all 😕 Oh well. In this specific case (the lookup table generator for horizontally flipping images), that decompilation was a mess anyway, and probably helped nobody. I could have added a dummy .OBJ that does nothing but enforce the needed 2-byte alignment before the function if I really insisted on keeping the C version, but it really wasn't worth it.

Now that I've also described yet another meta-issue, maybe there'll really be nothing to say about the next technical debt pushes? Next up though: Back to actual progress again, with TH01. Which maybe even ends up pushing that game over the 50% RE mark?

And indeed, I got to end my vacation with a lot of image format and blitting code, covering the final two formats, .GRC and .BOS. .GRC was nothing noteworthy – one function for loading, one function for byte-aligned blitting, and one function for freeing memory. That's it – not even a unblitting function for this one. .BOS, on the other hand…

…has no generic (read: single/sane) implementation, and is only implemented as methods of some boss entity class. And then again for Sariel's dress and wand animations, and then again for Reimu's animations, both of which weren't even part of these 4 pushes. Looking forward to decompiling essentially the same algorithms all over again… And that's how TH01 became the largest and most bloated PC-98 Touhou game. So yeah, still not done with image formats, even at 44% RE.

This means I also had to reverse-engineer that "boss entity" class… yeah, what else to call something a boss can have multiple of, that may or may not be part of a larger boss sprite, may or may not be animated, and that may or may not have an orb hitbox?
All bosses except for Kikuri share the same 5 global instances of this class. Since renaming all these variables in ASM land is tedious anyway, I went the extra mile and directly defined separate, meaningful names for the entities of all bosses. These also now document the natural order in which the bosses will ultimately be decompiled. So, unless a backer requests anything else, this order will be:

1. Konngara
2. Sariel
3. Elis
4. Kikuri
5. SinGyoku
6. (code for regular card-flipping stages)
7. Mima
8. YuugenMagan

As everyone kind of expects from TH01 by now, this class reveals yet another… um, unique and quirky piece of code architecture. In addition to the position and hitbox members you'd expect from a class like this, the game also stores the .BOS metadata – width, height, animation frame count, and 📝 bitplane pointer slot number – inside the same class. But if each of those still corresponds to one individual on-screen sprite, how can YuugenMagan have 5 eye sprites, or Kikuri have more than one soul and tear sprite? By duplicating that metadata, of course! And copying it from one entity to another
At this point, I feel like I even have to congratulate the game for not actually loading YuugenMagan's eye sprites 5 times. But then again, 53,760 bytes of waste would have definitely been noticeable in the DOS days. Makes much more sense to waste that amount of space on an unused C++ exception handler, and a bunch of redundant, unoptimized blitting functions

(Thinking about it, YuugenMagan fits this entire system perfectly. And together with its position in the game's code – last to be decompiled means first on the linker command line – we might speculate that YuugenMagan was the first boss to be programmed for TH01?)

So if a boss wants to use sprites with different sizes, there's no way around using another entity. And that's why Girl-Elis and Bat-Elis are two distinct entities internally, and have to manually sync their position. Except that there's also a third one for Attacking-Girl-Elis, because Girl-Elis has 9 frames of animation in total, and the global .BOS bitplane pointers are divided into 4 slots of only 8 images each.
Same for SinGyoku, who is split into a sphere entity, a person entity, and a… white flash entity for all three forms, all at the same resolution. Or Konngara's facial expressions, which also require two entities just for themselves.

And once you decompile all this code, you notice just how much of it the game didn't even use. 13 of the 50 bytes of the boss entity class are outright unused, and 10 bytes are used for a movement clamping and lock system that would have been nice if ZUN also used it outside of Kikuri's soul sprites. Instead, all other bosses ignore this system completely, and just party on the X/Y coordinates of the boss entities directly.

As for the rendering functions, 5 out of 10 are unused. And while those definitely make up less than half of the code, I still must have spent at least 1 of those 4 pushes on effectively unused functionality.
Only one of these functions lends itself to some speculation. For Elis' entrance animation, the class provides functions for wavy blitting and unblitting, which use a separate X coordinate for every line of the sprite. But there's also an unused and sort of broken one for unblitting two overlapping wavy sprites, located at the same Y coordinate. This might indicate that Elis could originally split herself into two sprites, similar to TH04 Stage 6 Yuuka? Or it might just have been some other kind of animation effect, who knows.

After over 3 months of TH01 progress though, it's finally time to look at other games, to cover the rest of the crowdfunding backlog. Next up: Going back to TH05, and getting rid of those last PI false positives. And since I can potentially spend the next 7 weeks on almost full-time ReC98 work, I've also re-opened the store until October!

TH01 pellets are coming up next, and for the first time, we'll have the chance to move hardcoded sprite data from ASM land to C land. As it would turn out, bad luck with the 2-byte alignment at the end of REIIDEN.EXE's data segment pretty much forces us to declare TH01's pellet sprites in C if we want to decompile the final few pellet functions without ugly workarounds for the float literals there. And while I could have just converted them into a C array and called it a day, it did raise the question of when we are going to do this The Right And Moddable Way, by auto-converting actual image files into ASM or C arrays during the build process. These arrays are even more annoying to edit in C, after all – unlike TASM, the old C++ we have to work with doesn't support binary number literals, only hexadecimal or, gasp, octal.
Without the explicit funding for such a converter, I reached out to GitHub, asking backers and outside contributors whether they'd be in favor of it. As something that requires no RE skills and collides with nothing else, it would be a perfect task for C/C++ coders who want to support ReC98 with something other than money.

And surprisingly, those still exist! Jonathan Campbell, of DOSBox-X fame, went ahead and implemented all the required functionality, within just a few days. Thanks again! The result is probably a lot more portable than it would have been if I had written it. Which is pretty relevant for future port authors – any additional tooling we write ourselves should not add to the list of problems they'll have to worry about.

Right now, all of the sprites are #included from the big ASM dump files, which means that they have to be converted before those files are assembled during the 32-bit build part. We could have introduced a third distinct build step there, perhaps even a 16-bit one so that we can use Turbo C++ 4.0J to also compile the converter… However, the more reasonable option was to do this at the beginning of the 32-bit build step, and add a 32-bit Windows C++ compiler to the list of tools required for ReC98's build process.
And the best choice for ReC98 is, in fact… 🥁… the 20-year-old Borland C++ 5.5 freeware release. See the README for a lengthy justification, as well as download links.

So yes, all sprites mentioned in the GitHub issue can now be modded by simply editing .BMP files, using an image editor of your choice. 🖌
And now that that's dealt with, it's finally time for more actual progress! TH01 pellets coming tomorrow.

Did WindowsTiger just cover 2% over all games on his own? While not all of that passed my review, +1.59% RE and +1.66% PI over all 5 games is still pretty noteworthy, and comfortably pushes TH05 over the 25% mark in RE, and the 60% mark in PI.

However.

While I definitely do appreciate such contributions, reviewing and adapting these to my current code organization standards also takes more time than I'd like it to take. And taken to this level, it does kind of undermine this crowdfunding project, causing both a literal denial of service and exactly the stress that this crowdfunding was designed to avoid. Most of the time, I can't merge all of that as-is without knowingly creating annoyances down the line. But I don't want to just ignore it either, or reject every non-perfect commit…
That's also why I let it slide this time, due to some of the RE work in there being genuinely amazing. In the future though, be aware that your chance of having your work merged diminishes the further you move ahead of my current master branch. In extreme cases like this one, I'll then just be waiting until enough generic reverse-engineering pushes have accrued, and treat the merge as regular work.

But now, time to continue with the regular programming… I am kind of exhausted from all of this, so no bullets for the next two Touhou Patch Center pushes, still… Good thing there's still plenty of simpler things with big percentage gains to be done:

• WindowsTiger mostly focused on OP.EXE which I tended to neglect, as the big MAIN executables seemed to be more interesting to my backers. (It's not like anyone ever requested OP to be done either – like, who even cares about boring menu source code, right?) Good that I therefore sort of left it as low-hanging fruit to be grabbed by outside contributors – because now, TH04's and TH05's OP.EXE are close to 100% position-independence. The GENSOU.SCR format is pretty much the only thing missing there right now, so let's finally go all the way there, I'd say.
• And in TH05, there's still Reimu's shot type functions left to be decompiled.

With no feedback to 📝 last week's blog post, I assume you all are fine with how things are going? Alright then, another one towards position independence, with the same approach as before…

Since -Tom- wanted to learn something about how the PC-98 EGC is used in TH04 and TH05, I took a look at master.lib's egc_shift_*() functions. These simply do a hardware-accelerated memmove() of any VRAM region, and are used for screen shaking effects. Hover over the image below for the raw effect:

Then, I finally wanted to take a look at the bullet structures, but it required way too much reverse-engineering to even start within ¾ of a position independence push. Even with the help of uth05win – bullet handling was changed quite a bit from TH04 to TH05.

What I ultimately settled on was more raw, "boring" PI work based around an already known set of functions. For this one, I looked at vector construction… and this time, that actually made the games a little bit more position-independent, and wasn't just all about removing false positives from the calculation. This was one of the few sets of functions that would also apply to TH01, and it revealed just how chaotically that game was coded. This one commit shows three ways how ZUN stored regular 2D points in TH01:

• "regularly", like in master.lib's Point structure (X first, Y second)
• reversed, (Y first and X second), then obviously with two distinct variables declared next to each other
• and with multiple points stored in a structure of arrays.

… yeah. But in more productive news, this did actually lay the groundwork for TH04 and TH05 bullet structures. Which might even be coming up within the next big, 5-push order from Touhou Patch Center? These are the priorities I got from them, let's see how close I can get!

So, here we have the first two pushes with an explicit focus on position independence… and they start out looking barely different from regular reverse-engineering? They even already deduplicate a bunch of item-related code, which was simple enough that it required little additional work? Because the actual work, once again, was in comparing uth05win's interpretations and naming choices with the original PC-98 code? So that we only ended up removing a handful of memory references there?

(Oh well, you can mod item drops now!)

So, continuing to interpret PI as a mere by-product of reverse-engineering might ultimately drive up the total PI cost quite a bit. But alright then, let's systematically clear out some false positives by looking at master.lib function calls instead… and suddenly we get the PI progress we were looking for, nicely spread out over all games since TH02. That kinda makes it sound like useless work, only done because it's dictated by some counting algorithm on a website. But decompilation will want to convert all of these values to decimal anyway. We're merely doing that right now, across all games.

Then again, it doesn't actually make any game more position-independent, and only proves how position-independent it already was. So I'm really wondering right now whether I should just rush actual position independence by simply identifying structures and their sizes, and not bother with members or false positives until that's done. That would certainly get the job done for TH04 and TH05 in just a few more pushes, but then leave all the proving work (and the road to 100% PI on the front page) to reverse-engineering.

I don't know. Would it be worth it to have a game that's „maybe fully position-independent“, only for there to maybe be rare edge cases where it isn't?

Or maybe, continuing to strike a balance between identifying false positives (fast) and reverse-engineering structures (slow) will continue to work out like it did now, and make us end up close to the current estimate, which was attractive enough to sell out the crowdfunding for the first time… 🤔

Please give feedback! If possible, by Friday evening UTC+1, before I start working on the next PI push, this time with a focus on TH04.