📝 Posted:

2022-08-15 23:57 UTC

🚚 Summary of:

P0216

⌨ Commits:

3123c9d...a0ff3f1

💰 Funded by:

JonathKane

🏷 Tags:

th01

rec98

meta

On August 15, 1997, at Comiket 52, an unknown doujin developer going by the name of ZUN released his first game, 東方靈異伝　～ The Highly Responsive to Prayers, marking the start of the Touhou Project game series that keeps running to this day. Today, exactly 25 years later, the C++ source code to version 1.10 of that game has been completely and perfectly reconstructed, reviewed, and documented.

And with that, a warm welcome to all game journalists who have (re-)discovered this project through these news! Here's a summary for everyone who doesn't want to go through 3 years worth of blog posts:

What does this mean?

• All code that ZUN wrote as part of a TH01 installation has now been decompiled to C++ code. The only parts left in assembly are two third-party libraries (master.lib and PiLoad), which were originally written in assembly, and are built from their respective official source code.
• You can clone the ReC98 repository, set up the build environment, and get a binary with an identical program image. The hashes of the resulting executables won't match those of ZUN's original release, but all differences there stem from details in the .EXE header that don't influence program execution, such as the on-disk order of the conceptually unordered set of x86 memory segment relocations. If you're interested in that level of correctness, you can order Easier verification against original binaries from the store. For now though, use mzdiff for verifying the builds against ZUN's binaries.
• Ever since this crowdfunding has started 3 years ago, the goal of this project has shifted more and more towards a full-on code review rather than being just a mechanical decompilation:
  • Hardcoded constants were derived from as few truly hardcoded values as possible, which uncovered their intended meaning and highlighted any inconsistencies
  • Code was deduplicated to a perhaps obsessive level (I'm still trying to find a balance)
  • Tons of comments everywhere to put everything into context
  • And, of course, 2½ years worth of blog posts summarizing any highlights, glitches, and secrets. (There might still be some left to be discovered!)
• As a result, modding the games and porting them away from the PC-98 platform is now a lot easier.

What does this not mean?

• This is not a piracy release. ReC98 only provides the code that the game's .EXE and .COM files are built out of. Without the rest of the original data files, supplied from a pre-existing game copy, the code won't do very much.
• Even apart from ZUN's own code quality, the ReC98 repository is not as polished and consistent as it could be, having seen multiple code structure evolutions over the 8 years of its existence.
• TH01 hasn't magically reached Doom levels of easy portability now. As a decompilation of the exact code that ZUN wrote for the PC-98 platform, it is very PC-98-native, and wildly mixes game logic with hardware accesses. As ZUN's first foray into game development, he understandably didn't see the need for writing an engine or hardware abstraction layer yet.

So while this milestone opened the floodgates to PC-98-native mods, I wouldn't advise trying to attempt a port away from PC-98 right now. But then again, I have a financial interest in being a part of the porting process, and who knows, maybe you can just merge in a PC-98 emulator core and get started with something halfway decent in a short amount of time. After all, TH01 is by far the easiest PC-98 Touhou game to port to other systems, as it makes the least use of hardware features. (Edit (2023-03-30): 📝 Turns out that this crown actually goes to TH02. It features the least amount of ZUN-written PC-98-specific rendering code out of all the 5 games, with most of it being decently abstracted via master.lib.)

However, this game in particular raises the question of what exactly one would even want to port. TH01 is a broken flicker-fest that overwhelmingly suffers the drawbacks of PC-98 hardware rather than using it to its advantage. Out of the 78 bugs that I ended up labeling as such, the majority are sprite blitting issues, while you can count the instances of good hardware use on one hand.
And even at the level of game logic, this game features a lot of weird, inconsistent behavior. Less rigorous projects such as uth05win would probably promptly identify these issues as bugs and fix them. On the one hand, this shows that there is a part of the community that wants sane versions of these games which behave as expected. In other parts of the community though, such projects quickly gain the reputation of being too inaccurate to bother about them.

Some terminology might help here. If you look over the ReC98 codebase, you'll find that I classified any weird code into three categories. Edit (2023-03-05): These have been overhauled with a new landmine category for invisible issues. Check CONTRIBUTING.md for the complete and current current definition of all weird code categories.

• 🔗 ZUN bugs: Broken code that results from logic errors or incorrect programming language/API/hardware use, with enough evidence in the code to indicate that ZUN did not intend the bug. Fixing these issues must not affect hypothetical replay compatibility, and any resulting visual changes must match ZUN's provable intentions.
• 🔗 ZUN quirks: Weird code that looks incorrect in context. Fixing these issues would change gameplay enough to desync a hypothetical replay recorded on the original version, or affect the visuals so much that the result is no longer faithful to ZUN's original release. It might very well be called a fangame at that point.
• 🔗 ZUN bloat: Code that wastes memory, CPU cycles, or even just the mental capacity of anyone trying to read and understand the code. If you want to write a particularly resource-intensive mod, these are the places you could claim some of those resources from.

Some examples:

• All crashes are bugs
• All blitting issues related to inappropriate VRAM byte alignment are bugs
• All the 📝 incorrect coordinate calculations in the YuugenMagan fight are quirks
• The overly high damage of TH10's MarisaB 3.00-3.95 power shot is a quirk, despite having been proven to be a typo.
• The idea of splitting TH01 across three executables is its biggest source of bloat. It wastes disk space, the game doesn't even make use of the memory gained from unloading unneeded code and data, it complicates the build process and code structure with inconsistencies between the individual binaries, and the required inter-process communication via shared memory adds another piece of global state mutation headache.

Since I'm not in the business of writing fanfiction, I won't offer any option that fixes quirks. That's where all of you can come in, and use ReC98 as a base for remasters and remakes. As for bloat and bugs though, there are many ways we could go from here:

• If you want to ultimately try porting the game yourself, but still support ReC98 somehow, I can recommend the ZUN code cleanup goal. This is the most conservative option that leaves all bugs and quirks in place and only removes bloat, rearchitecting the codebase so that it's easier to work with.
• For an improved gameplay experience on PC-98, choose the TH01 Anniversary Edition goal. In addition to the above code cleanup, this goal fixes every bug with the game, most notably all the sprite flickering by implementing a completely new renderer, while maintaining hypothetical replay compatibility to ZUN's original release.
• If you're mainly interested in seeing any variety of TH01 ported away from PC-98 to any system, choose the Portability to non-PC-98 systems goal. In this one, I'm going to develop the abstraction layers that would ultimately bring this game to the aforementioned Doom level of portability, while still keeping it running with better than original performance on PC-98.
• Replay support is also something you could order…
• … as is Multilingual translation support (on PC-98), for those sweet non-ASCII characters if that's your thing.

Then again, with all these choices in mind, maybe we should just let TH01 be what it is: ZUN's first game, evidence for the truth that no programmer writes good code the first time around, and more of a historical curiosity than anything you'd want to maintain and modernize. The idea of moving on to the next game and decompiling all 5 PC-98 Touhou games in order has certainly shown to be popular among the backers who funded this 100% goal.

Since the beginning of the year, I've been dramatically raising the level of quality and care I've been putting into this project, leading to 9 of the 10 longest blog posts having been written in the past 8 months. The community reception has been even more supportive as well, with all of you still regularly selling out the store in return. To match the level of quality with the community demand, I'm raising push prices from to per push, as of this blog post. 📝 As usual, I'm going to deliver any existing orders in the backlog at the value they were originally purchased at. Due to the way the cap has to be calculated, these contributions now appear to have increased in value by 25%.

However, I do realize that this might make regular pushes prohibitively expensive for some. This could especially prevent all these exciting modding goals from ever getting off the ground. Thinking about it though, the push system is only really necessary for the core reverse-engineering business, where longer, concentrated stretches of work allow me to study a new piece of code in a larger context and improve the quality of the final result. In contrast, modding-related goals could theoretically be segmented into arbitrarily small portions of work, as I have a clear idea of where I want to go and how to get there.
Thus, I'm introducing microtransactions, now available for all modding-related goals. These allow you to order fractional pieces of work for as low as 1 €, which I will immediately deliver without requiring others to fund a full push first. Edit (2022-08-16): And then the store still sold out with a single regular contribution by nrook towards more reverse-engineering. Guess that this experiment will have to wait a little while longer, then… 😅

Next up: Taking a break and recovering from crunch time by improving video playback on this blog and working on Shuusou Gyoku, before returning to Touhou in September.