Blog

📝 Posted:
🚚 Summary of:
P0072, P0073, P0074, P0075
Commits:
4bb04ab...cea3ea6, cea3ea6...5286417, 5286417...1807906, 1807906...222fc99
💰 Funded by:
[Anonymous], -Tom-, Myles

Long time no see! And this is exactly why I've been procrastinating bullets while there was still meaningful progress to be had in other parts of TH04 and TH05: There was bound to be quite some complexity in this most central piece of game logic, and so I couldn't possibly get to a satisfying understanding in just one push.

Or in two, because their rendering involves another bunch of micro-optimized functions adapted from master.lib.

Or in three, because we'd like to actually name all the bullet sprites, since there are a number of sprite ID-related conditional branches. And so, I was refining things I supposedly RE'd in the the commits from the first push until the very end of the fourth.

When we talk about "bullets" in TH04 and TH05, we mean just two things: the white 8×8 pellets, with a cap of 240 in TH04 and 180 in TH05, and any 16×16 sprites from MIKO16.BFT, with a cap of 200 in TH04 and 220 in TH05. These are by far the most common types of… err, "things the player can collide with", and so ZUN provides a whole bunch of pre-made motion, animation, and n-way spread / ring / stack pattern options for those, which can be selected by simply setting a few fields in the bullet template. All the other "non-bullets" have to be fired and controlled individually.

Which is nothing new, since uth05win covered this part pretty accurately – I don't think anyone could just make up these structure member overloads. The interesting insights here all come from applying this research to TH04, and figuring out its differences compared to TH05. The most notable one there is in the default patterns: TH05 allows you to add a stack to any single bullet, n-way spread or ring, but TH04 only lets you create stacks separately from n-way spreads and rings, and thus gets by with fewer fields in its bullet template structure. On the other hand, TH04 has a separate "n-way spread with random angles, yet still aimed at the player" pattern? Which seems to be unused, at least as far as midbosses and bosses are concerned; can't say anything about stage enemies yet.

In fact, TH05's larger bullet template structure illustrates that these distinct pattern types actually are a rather redundant piece of over-engineering. You can perfectly indicate any permutation of the basic patterns through just the stack bullet count (1 = no stack), spread bullet count (1 = no spread), and spread delta angle (0 = ring instead of spread). Add a 4-flag bitfield to cover the rest (aim to player, randomize angle, randomize speed, force single bullet regardless of difficulty or rank), and the result would be less redundant and even slightly more capable.

Even those 4 pushes didn't quite finish all of the bullet-related types, stopping just shy of the most trivial and consistent enum that defines special movement. This also left us in a 📝 TH03-like situation, in which we're still a bit away from actually converting all this research into actual RE%. Oh well, at least this got us way past 50% in overall position independence. On to the second half! 🎉

For the next push though, we'll first have a quick detour to the remaining C code of all the ZUN.COM binaries. Now that the 📝 TH04 and TH05 resident structures no longer block those, -Tom- has requested TH05's RES_KSO.COM to be covered in one of his outstanding pushes. And since 32th System recently RE'd TH03's resident structure, it makes sense to also review and merge that, before decompiling all three remaining RES_*.COM binaries in hopefully a single push. It might even get done faster than that, in which case I'll then review and merge some more of WindowsTiger's research.

📝 Posted:
🚚 Summary of:
P0071
Commits:
327021b...4bb04ab
💰 Funded by:
KirbyComment, -Tom-

Turns out that covering TH03's 128-byte player structure was way more insightful than expected! And while it doesn't include every bit of per-player data, we still got to know quite a bit about the game from just trying to name its members:

  • 50 frames of invincibility when starting a new round
  • 110 frames of invincibility when getting hit
  • 64 frames of knockback when getting hit
  • 128 frames before a charged up gauge/boss attack is fired automatically
  • The damage a player will take from the next hit starts out at ½ heart at the beginning of each round, and increases by another ½ heart every 1024 frames, capped at a maximum of 3 hearts. This guarantees that a player will always survive at least two hits.
  • In Story Mode, hit damage is biased in favor of the player for the first 6 stages. The CPU will always take an additional 1½ hearts of damage in stages 1 and 2, 1 heart in stages 3 and 4, and ½ heart in stages 5 and 6, plus the above frame-based and capped damage amount. So while it's therefore possible to cause 4½ hearts of damage in Stages 1 and 2 if the first hit is somehow delayed for at least 5120 frames, you'd still win faster if the CPU gets hit as soon as possible.
  • CPU players will charge up a gauge/boss attack as soon as their gauge has reached a certain level. These levels are now proved to be random; at the start of every round, the game generates a sequence of 64 gauge level positions (from 1 to 4), separately for each player. If a round were to last long enough for a CPU player to fire all 64 of those predetermined attacks, you'd observe that sequence repeating.
    • Yes, that means that in theory, these levels can be RNG-manipulated. More details on that once we got this game's resident structure, where the seed is stored.
  • CPU players follow two main strategies: trying to not get hit, and… not quite doing that once they've survived for a certain safety threshold of frames. For the first 2000 frames of a round, this safety frame counter is reset to 0 every 64 frames, leading the CPU to switch quickly between the two strategies in the first few Story Mode stages on lower difficulties, where this safety threshold is less than 64. The calculation of the actual value is a bit more complex; more on that also once we got this game's resident structure.
  • Section 13 of 夢時空.TXT states that Boss Attacks are only counted towards the Clear Bonus if they were caused by reaching a certain number of spell points. This is incorrect; manually charged Level 4 Boss Attacks are counted as well.

The next TH03 pushes can now cover all the functions that reference this structure in one way or another, and actually commit all this research and translate it into some RE%. Since the non-TH05 priorities have become a bit unclear after the last 50 € RE contribution though (as of this writing, it's still 10 € to decide on what game to cover in two RE pushes!), I'll be returning to TH05 until that's decided.

📝 Posted:
🚚 Summary of:
P0070
Commits:
a931758...327021b
💰 Funded by:
KirbyComment

As noted in 📝 P0061, TH03 gameplay RE is indeed going to progress very slowly in the beginning. A lot of the initial progress won't even be reflected in the RE% – there are just so many features in this game that are intertwined into each other, and I only consider functions to be "reverse-engineered" once we understand every involved piece of code and data, and labeled every absolute memory reference in it. (Yes, that means that the percentages on the front page are actually underselling ReC98's progress quite a bit, and reflect a pretty low bound of our actual understanding of the games.)

So, when I get asked to look directly at gameplay code right now, it's quite the struggle to find a place that can be covered within a push or two and that would immediately benefit scoreplayers. The basics of score and combo handling themselves managed to fit in pretty well, though:

  • Just like TH04 and TH05, TH03 stores the current score as 8 binary-coded decimal digits. Since the last constant 0 is not included, the maximum score displayable without glitches therefore is 999,999,990 points, but the game will happily store up to 24,699,999,990 points before the score wraps back to 0.
  • There are (surprisingly?) only 6 places where the game actually adds points to the score. Not quite sure about all of them yet, but they (of course) include ending a combo, killing enemies, and the bonus at the end of a round.
  • Combos can be continued for 80 frames after a 2-hit. The hit counter can only be increased in the first 48, and effectively resets to 0 for the last 32, when the Spell Point value starts blinking.
  • TH03 can track a total of 16 independent "hit combo sources" per player, simultaneously. These are not related to the number of actual explosions; rather, each explosion is assigned to one of the 16 slots when it spawns, and all consecutive explosions spawned from that one will then add to the hit combo in that slot. The hit number displayed in the top left is simply the largest one among all these.

Oh well, at least we still got a bit of PI% out of this one. From this point though, the next push (or two) should be enough to cover the big 128-byte player structure – which by itself might not be immediately interesting to scoreplayers, but surely is quite a blocker for everything else.

📝 Posted:
🚚 Summary of:
P0067, P0068, P0069
Commits:
e55a48b...ebb30ce, ebb30ce...2ac00d4, e0d0dcd...0f18dbc
💰 Funded by:
Splashman, Yanga, [Anonymous]

Now that's more like the speed I was expecting! After a few more unused functions for palette fading and rectangle blitting, we've reached the big line drawing functions. And the biggest one among them, drawing a straight line at any angle between two points using Bresenham's algorithm, actually happens to be the single longest function present in more than one binary in all of PC-98 Touhou, and #23 on the list of individual longest functions.

And it technically has a ZUN bug! If you pass a point outside the (0, 0) - (639, 399) screen range, the function will calculate a new point at the edge of the screen, so that the resulting line will retain the angle intended by the points given. Except that it does so by calculating the line slope using an integer division rather than a floating-point one :zunpet: Doesn't seem like it actually causes any weirdly skewed lines to be drawn in-game, though; that case is only hit in the Mima boss fight, which draws a few lines with a bottom coordinate of 400 rather than the maximum of 399. It might also cause the wrong background pixels to be restored during parts of the YuugenMagan fight, leading to flickering sprites, but seriously, that's an issue everywhere you look in this game.

Together with the rendering-text-to-VRAM function we've mostly already known from TH02, this pushed the total RE percentage well over 20%, and almost doubled the TH01 RE percentage, all within three pushes. And comparatively, it went really smoothly, to the point (ha) where I even had enough time left to also include the single-point functions that come next in that code segment. Since about half of the remaining functions in OP.EXE are present in more than just itself, I'll be able to at least keep up this speed until OP.EXE hits the 70% RE mark. That is, as long as the backers' priorities continue to be generic RE or "giving some love to TH01"… we don't have a precedent for TH01's actual game code yet.

And that's all the TH01 progress funded for January! Next up, we actually do have a focus on TH03's game and scoring mechanics… or at least the foundation for that.

📝 Posted:
🚚 Summary of:
P0066
Commits:
042b780...e55a48b
💰 Funded by:
Yanga, Splashman

So, the thing that made me so excited about TH01 were all those bulky C reimplementations of master.lib functions. Identical copies in all three executables, trivial to figure out and decompile, removing tons of instructions, and providing a foundation for large parts of the game later. The first set of functions near the end of that shared code segment deals with color palette handling, and master.lib's resident palette structure in particular. (No relation to the game's resident structure.) Which directly starts us out with pretty much all the decompilation difficulties imaginable:

  • iteration over internal DOS structures via segment pointers – Turbo C++ doesn't support a lot of arithmetic on those, requiring tons of casts to make it work
  • calls to a far function near the beginning of a segment from a function near the end of a segment – these are undecompilable until we've decompiled both functions (and thus, the majority of the segment), and need to be spelled out in ASM for the time being. And if the caller then stores some of the involved variables in registers, there's no way around the ugliest of workarounds, spelling out opcode bytes
  • surprising color format inconsistencies – apparently, GRB (rather than RGB) is some sort of wider standard in PC-98 inter-process communication, because it matches the order of the hardware's palette register ports? (0AAh = green, 0ACh = red, 0AEh = blue)? Yet the game's actual palette still uses RGB…

And as it turns out, the game doesn't even use the resident palette feature. Which adds yet another set of functions to the, uh, learning experience that ZUN must have chosen this game to be. I wouldn't be surprised if we manage to uncover actual scrapped beta game content later on, among all the unused code that's bound to still be in there.

At least decompilation should get easier for the next few TH01 pushes now… right?

📝 Posted:
🚚 Summary of:
P0065
Commits:
9faa29a...042b780
💰 Funded by:
Touhou Patch Center

A~nd resident structures ended up being exactly the right thing to start off the new year with. WindowsTiger and spaztron64 have already been pushing for them with their own reverse-engineering, and together with my own recent GENSOU.SCR RE work, we've clarified just enough context around the harder-to-explain values to make both TH04's and TH05's structures fit nicely into the typical time frame of a single push.

With all the apparently obvious and seemingly just duplicated values, it has always been easy to do a superficial job for most of the structure, then lose motivation for the last few unknown fields. Pretty glad to got this finally covered; I've heard that people are going to write trainer tools now?

Also, where better to slot in a push that, in terms of figures, seems to deliver 0% RE and only miniscule PI progress, than at the end of Touhou Patch Center's 5-push order that already had multiple pushes yielding above-average progress? :onricdennat: As usual, we'll be reaping the rewards of this work in the next few TH04/TH05 pushes…

…whenever they get funded, that is, as for January, the backers have shifted the priorities towards TH01 and TH03. TH01 especially is something I'm quite excited about, as we're finally going to see just how fast this bloated game is really going to progress. Are you excited?

📝 Posted:
🚚 Summary of:
P0064
Commits:
80cec5b...9faa29a
💰 Funded by:
Touhou Patch Center

🎉 TH04's and TH05's OP.EXE are now fully position-independent! 🎉

What does this mean?

You can now add any data or code to the main menus of the two games, by simply editing the ReC98 source, writing your mod in ASM or C/C++, and recompiling the code. Since all absolute memory addresses have now been converted to labels, this will work without causing any instability. See the position independence section in the FAQ for a more thorough explanation about why this is a problem.

What does this not mean?

The original ZUN code hasn't been completely reverse-engineered yet, let alone decompiled. Pretty much all of that is still ASM, which might make modding a bit inconvenient right now.

Since this push was otherwise pretty unremarkable, I made a video demonstrating a few basic things you can do with this: (click to enable external content from YouTube)

Now, what to do for the last outstanding Touhou Patch Center push? Bullets, or resident structures? :thonk:

📝 Posted:
🚚 Summary of:
P0063
Commits:
034ae4b...8dbb450
💰 Funded by:
-Tom-

Almost!

Just like most of the time, it was more sensible to cover GENSOU.SCR, the last structure missing in TH05's OP.EXE, everywhere it's used, rather than just rushing out OP.EXE position independence. I did have to look into all of the functions to fully RE it after all, and to find out whether the unused fields actually are unused. The only thing that kept this push from yielding even more above-average progress was the sheer inconsistency in how the games implemented the operations on this PC-98 equivalent of score*.dat:

  • OP.EXE declares two structure instances, for simultaneous access to both Reimu and Marisa scores. TH05 with its 4 playable characters instead uses a single one, and overwrites it successively for each character when drawing the high score menu – meaning, you'd only see Yuuka's scores when looking at the structure inside the rendered high score menu. However, it still declares the TH04 "Marisa" structure as a leftover… and also decodes it and verifies its checksum, despite nothing being ever loaded into it
  • MAIN.EXE uses a separate ASM implementation of the decoding and encoding functions :godzun:
  • TH05's MAIN.EXE also reimplements the basic loading functions in ASM – without the code to regenerate GENSOU.SCR with default data if the file is missing or corrupted. That actually makes sense, since any regeneration is already done in OP.EXE, which always has to load that file anyway to check how much has been cleared
  • However, there is a regeneration function in TH05's MAINE.EXE… which actually generates different default data: OP.EXE consistently sets Extra Stage records to Stage 1, while MAINE.EXE uses the same place-based stage numbering that both versions use for the regular ranks

Technically though, TH05's OP.EXE is position-independent now, and the rest are (should be? :tannedcirno:) merely false positives. However, TH04's is still missing another structure, in addition to its false positives. So, let's wait with the big announcement until the next push… which will also come with a demo video of what will be possible then.

📝 Posted:
🚚 Summary of:
P0062
Commits:
1d6fbb8...f275e04
💰 Funded by:
Touhou Patch Center

Big gains, as expected, but not much to say about this one. With TH05 Reimu being way too easy to decompile after 📝 the shot control groundwork done in October, there was enough time to give the comprehensive PI false-positive treatment to two other sets of functions present in TH04's and TH05's OP.EXE. One of them, master.lib's super_*() functions, was used a lot in TH02, more than in any other game… I wonder how much more that game will progress without even focusing on it in particular.

Alright then! 100% PI for TH04's and TH05's OP.EXE upcoming… (Edit: Already got funding to cover this!)

📝 Posted:

Did WindowsTiger just cover 2% over all games on his own? While not all of that passed my review, +1.59% RE and +1.66% PI over all 5 games is still pretty noteworthy, and comfortably pushes TH05 over the 25% mark in RE, and the 60% mark in PI.

However.

While I definitely do appreciate such contributions, reviewing and adapting these to my current code organization standards also takes more time than I'd like it to take. And taken to this level, it does kind of undermine this crowdfunding project, causing both a literal denial of service and exactly the stress that this crowdfunding was designed to avoid. Most of the time, I can't merge all of that as-is without knowingly creating annoyances down the line. But I don't want to just ignore it either, or reject every non-perfect commit…
That's also why I let it slide this time, due to some of the RE work in there being genuinely amazing. In the future though, be aware that your chance of having your work merged diminishes the further you move ahead of my current master branch. In extreme cases like this one, I'll then just be waiting until enough generic reverse-engineering pushes have accrued, and treat the merge as regular work.

But now, time to continue with the regular programming… I am kind of exhausted from all of this, so no bullets for the next two Touhou Patch Center pushes, still… Good thing there's still plenty of simpler things with big percentage gains to be done:

  • WindowsTiger mostly focused on OP.EXE which I tended to neglect, as the big MAIN executables seemed to be more interesting to my backers. (It's not like anyone ever requested OP to be done either – like, who even cares about boring menu source code, right?) Good that I therefore sort of left it as low-hanging fruit to be grabbed by outside contributors – because now, TH04's and TH05's OP.EXE are close to 100% position-independence. The GENSOU.SCR format is pretty much the only thing missing there right now, so let's finally go all the way there, I'd say.
  • And in TH05, there's still Reimu's shot type functions left to be decompiled.
📝 Posted:
🚚 Summary of:
P0061
Commits:
96684f4...5f4f5d8
💰 Funded by:
Touhou Patch Center

… nope, with a game whose MAIN.EXE is still just 5% reverse-engineered and which naturally makes heavy use of structures, there's still a lot more PI groundwork to be done before RE progress can speed up to the levels that we've now reached with TH05. The good news is that this game is (now) way easier to understand: In contrast to TH04 and TH05, where we needed to work towards player shots over a two-digit number of pushes, TH03 only needed two for SPRITE16, and a half one for the playfield shaking mechanism. After that, I could even already decompile the per-frame shot update and render functions, thanks to TH03's high number of code segments. Now, even the big 128-byte player structure doesn't seem all too far off.

Then again, as TH03 shares no code with any other game, this actually was a completely average PI push. For the remaining three, we'll return to TH04 and TH05 though, which should more than make up for the slight drop in RE speed after this one.

In other news, we've now also reached peak C++, with the introduction of templates! TH03 stores movement speeds in a 4.4 fixed-point format, which is an 8-bit spin on the usual 16-bit, 12.4 fixed-point format.

📝 Posted:
🚚 Summary of:
P0060
Commits:
29385dd...73f5ae7
💰 Funded by:
Touhou Patch Center

So, where to start? Well, TH04 bullets are hard, so let's procrastinate start with TH03 instead :tannedcirno: The 📝 sprite display functions are the obvious blocker for any structure describing a sprite, and therefore most meaningful PI gains in that game… and I actually did manage to fit a decompilation of those three functions into exactly the amount of time that the Touhou Patch Center community votes alloted to TH03 reverse-engineering!

And a pretty amazing one at that. The original code was so obviously written in ASM and was just barely decompilable by exclusively using register pseudovariables and a bit of goto, but I was able to abstract most of that away, not least thanks to a few helpful optimization properties of Turbo C++… seriously, I can't stop marveling at this ancient compiler. The end result is both readable, clear, and dare I say portable?! To anyone interested in porting TH03, take a look. How painful would it be to port that away from 16-bit x86?

However, this push is also a typical example that the RE/PI priorities can only control what I look at, and the outcome can actually differ greatly. Even though the priorities were 65% RE and 35% PI, the progress outcome was +0.13% RE and +1.35% PI. But hey, we've got one more push with a focus on TH03 PI, so maybe that one will include more RE than PI, and then everything will end up just as ordered? :onricdennat:

📝 Posted:
🚚 Summary of:
P0059
Commits:
01de290...8b62780
💰 Funded by:
[Anonymous], -Tom-

With no feedback to 📝 last week's blog post, I assume you all are fine with how things are going? Alright then, another one towards position independence, with the same approach as before…

Since -Tom- wanted to learn something about how the PC-98 EGC is used in TH04 and TH05, I took a look at master.lib's egc_shift_*() functions. These simply do a hardware-accelerated memmove() of any VRAM region, and are used for screen shaking effects. Hover over the image below for the raw effect:

Then, I finally wanted to take a look at the bullet structures, but it required way too much reverse-engineering to even start within ¾ of a position independence push. Even with the help of uth05win – bullet handling was changed quite a bit from TH04 to TH05.

What I ultimately settled on was more raw, "boring" PI work based around an already known set of functions. For this one, I looked at vector construction… and this time, that actually made the games a little bit more position-independent, and wasn't just all about removing false positives from the calculation. This was one of the few sets of functions that would also apply to TH01, and it revealed just how chaotically that game was coded. This one commit shows three ways how ZUN stored regular 2D points in TH01:

  • "regularly", like in master.lib's Point structure (X first, Y second)
  • reversed, (Y first and X second), then obviously with two distinct variables declared next to each other
  • and with multiple points stored in a structure of arrays.

… yeah. But in more productive news, this did actually lay the groundwork for TH04 and TH05 bullet structures. Which might even be coming up within the next big, 5-push order from Touhou Patch Center? These are the priorities I got from them, let's see how close I can get!

📝 Posted:
🚚 Summary of:
P0057, P0058
Commits:
1cb9731...ac7540d, ac7540d...fef0299
💰 Funded by:
[Anonymous], -Tom-

So, here we have the first two pushes with an explicit focus on position independence… and they start out looking barely different from regular reverse-engineering? They even already deduplicate a bunch of item-related code, which was simple enough that it required little additional work? Because the actual work, once again, was in comparing uth05win's interpretations and naming choices with the original PC-98 code? So that we only ended up removing a handful of memory references there?

(Oh well, you can mod item drops now!)

So, continuing to interpret PI as a mere by-product of reverse-engineering might ultimately drive up the total PI cost quite a bit. But alright then, let's systematically clear out some false positives by looking at master.lib function calls instead… and suddenly we get the PI progress we were looking for, nicely spread out over all games since TH02. That kinda makes it sound like useless work, only done because it's dictated by some counting algorithm on a website. But decompilation will want to convert all of these values to decimal anyway. We're merely doing that right now, across all games.

Then again, it doesn't actually make any game more position-independent, and only proves how position-independent it already was. So I'm really wondering right now whether I should just rush actual position independence by simply identifying structures and their sizes, and not bother with members or false positives until that's done. That would certainly get the job done for TH04 and TH05 in just a few more pushes, but then leave all the proving work (and the road to 100% PI on the front page) to reverse-engineering.

I don't know. Would it be worth it to have a game that's „maybe fully position-independent“, only for there to maybe be rare edge cases where it isn't?

Or maybe, continuing to strike a balance between identifying false positives (fast) and reverse-engineering structures (slow) will continue to work out like it did now, and make us end up close to the current estimate, which was attractive enough to sell out the crowdfunding for the first time… 🤔

Please give feedback! If possible, by Friday evening UTC+1, before I start working on the next PI push, this time with a focus on TH04.

📝 Posted:
🚚 Summary of:
P0056
Commits:
b28cefc...c09446a
💰 Funded by:
rosenrose, [Anonymous]

No priorities, again…?! Please don't do this to me… 😕

Well, let's not continue with TH05 then 😛 And instead use the occasion to commit this interesting discovery, made by @m1yur1 last year. Yup, TH03's "ZUNSP" sprite driver is actually a "rebranded" version of Promisence Soft's SPRITE16.COM. Sure, you were allowed to use this driver in your own game, but replacing the copyright with your own isn't exactly the nicest thing to do… That now makes three library programmers that ZUN didn't credit. Makes me wonder what makes M. Kajihara so special. Probably the fact that Touhou has always been about the music for ZUN, first and foremost.

But what makes this more than a piece of trivia is the fact that Promiscence Soft's SPRITE16 sample game StormySpace was bundled with documentation on the driver. Shoutout to the Neo Kobe PC-98 collection for preserving he original release!

That means more documented third-party code that we don't necessarily have to reverse-engineer, just like master.lib or KAJA's PMD driver. However, the PC-98 EGC is rather complex and definitely not designed for alpha-tested 16-color sprite blitting. So it (once again) took quite a while to make sense of SPRITE16's code and the available documentation on the EGC, to come up with satisfying function names. As a result, I'm going to distribute the entire RE work related to TH03's SPRITE16 interface across a total of three pushes, this one being the first of them.

  • The second one will reverse-engineer the SPRITE16 code reachable from its interrupt handler, and also come with somewhat detailed English documentation on the PC-98 EGC raster ops in particular,
  • and the third one will cover TH03's sprite drawing functions, which look pretty undecompilable once again… (Update: 📝 Delivered and actually decompiled in P0060)

Next up, we do have more TH05 progress though, now for the first time specifically with position independence in mind. Pretty excited for this one!

📝 Posted:
🚚 Summary of:
P0036, P0037
Commits:
a533b5d...82b0e1d, 82b0e1d...e7e1cbc
💰 Funded by:
zorg

And just in time for zorg's last outstanding pushes, the TH05 shot type control functions made the speedup happen!

  • TH05 as a whole is now 20% reverse-engineered, and 50% position independent,
  • TH05's MAIN.EXE is now even below TH02's in terms of not yet RE'd instructions,
  • and all price estimates have now fallen significantly.

It would have been really nice to also include Reimu's shot control functions in this last push, but figuring out this entire system, with its weird bitflags and switch statement micro-optimizations, was once again taking way longer than it should have. Especially with my new-found insistence on turning this obvious copy-pasta into something somewhat readable and terse…

But with such a rather tabular visual structure, things should now be moddable in hopefully easily consistent way. Of course, since we're only at 54% position independence for MAIN.EXE, this isn't possible yet without crashing the game, but modifiying damage would already work.

Despite my earlier claims of ZUN only having used C++ in TH01, as it's the only game using new and delete, it's now pretty much confirmed that ZUN used it for all games, as inlined functions (and by extension, C++ class methods) are the only way to get certain instructions out of the Turbo C++ code generator. Also, I've kept my promise and started really filling that decompilation pattern file.

And now, with the reverse-engineering backlog finally being cleared out, we wait for the next orders, and the direction they might focus on…

📝 Posted:
🚚 Summary of:
P0034, P0035
Commits:
6cdd229...6f1f367, 6f1f367...a533b5d
💰 Funded by:
zorg

Deathbombs confirmed, in both TH04 and TH05! On the surface, it's the same 8-frame window as in most Windows games, but due to the slightly lower PC-98 frame rate of 56.4 Hz, it's actually slightly more lenient in TH04 and TH05.

The last function in front of the TH05 shot type control functions marks the player's previous position in VRAM to be redrawn. But as it turns out, "player" not only means "the player's option satellites on shot levels ≥ 2", but also "the explosion animation if you lose a life", which required reverse-engineering both things, ultimately leading to the confirmation of deathbombs.

It actually was kind of surprising that we then had reverse-engineered everything related to rendering all three things mentioned above, and could also cover the player rendering function right now. Luckily, TH05 didn't decide to also micro-optimize that function into un-decompilability; in fact, it wasn't changed at all from TH04. Unlike the one invalidation function whose decompilation would have actually been the goal here…

But now, we've finally gotten to where we wanted to… and only got 2 outstanding decompilation pushes left. Time to get the website ready for hosting an actual crowdfunding campaign, I'd say – It'll make a better impression if people can still see things being delivered after the big announcement.

📝 Posted:
🚚 Summary of:
P0031, P0032, P0033
Commits:
dea40ad...9f764fa, 9f764fa...e6294c2, e6294c2...6cdd229
💰 Funded by:
zorg

The glacial pace continues, with TH05's unnecessarily, inappropriately micro-optimized, and hence, un-decompilable code for rendering the current and high score, as well as the enemy health / dream / power bars. While the latter might still pass as well-written ASM, the former goes to such ridiculous levels that it ends up being technically buggy. If you enjoy quality ZUN code, it's definitely worth a read.

In TH05, this all still is at the end of code segment #1, but in TH04, the same code lies all over the same segment. And since I really wanted to move that code into its final form now, I finally did the research into decompiling from anywhere else in a segment.

Turns out we actually can! It's kinda annoying, though: After splitting the segment after the function we want to decompile, we then need to group the two new segments back together into one "virtual segment" matching the original one. But since all ASM in ReC98 heavily relies on being assembled in MASM mode, we then start to suffer from MASM's group addressing quirk. Which then forces us to manually prefix every single function call

  • from inside the group
  • to anywhere else within the newly created segment
with the group name. It's stupidly boring busywork, because of all the function calls you mustn't prefix. Special tooling might make this easier, but I don't have it, and I'm not getting crowdfunded for it.

So while you now definitely can request any specific thing in any of the 5 games to be decompiled right now, it will take slightly longer, and cost slightly more.
(Except for that one big segment in TH04, of course.)

Only one function away from the TH05 shot type control functions now!

📝 Posted:
🚚 Summary of:
P0029, P0030
Commits:
6ff427a...c7fc4ca, c7fc4ca...dea40ad
💰 Funded by:
zorg

Here we go, new C code! …eh, it will still take a bit to really get decompilation going at the speeds I was hoping for. Especially with the sheer amount of stuff that is set in the first few significant functions we actually can decompile, which now all has to be correctly declared in the C world. Turns out I spent the last 2 years screwing up the case of exported functions, and even some of their names, so that it didn't actually reflect their calling convention… yup. That's just the stuff you tend to forget while it doesn't matter.

To make up for that, I decided to research whether we can make use of some C++ features to improve code readability after all. Previously, it seemed that TH01 was the only game that included any C++ code, whereas TH02 and later seemed to be 100% C and ASM. However, during the development of the soon to be released new build system, I noticed that even this old compiler from the mid-90's, infamous for prioritizing compile speeds over all but the most trivial optimizations, was capable of quite surprising levels of automatic inlining with class methods…

…leading the research to culminate in the mindblow that is 9d121c7 – yes, we can use C++ class methods and operator overloading to make the code more readable, while still generating the same code than if we had just used C and preprocessor macros.

Looks like there's now the potential for a few pull requests from outside devs that apply C++ features to improve the legibility of previously decompiled and terribly macro-ridden code. So, if anyone wants to help without spending money…

📝 Posted:
🚚 Summary of:
P0028
Commits:
6023f5c...6ff427a
💰 Funded by:
zorg

Back to actual development! Starting off this stretch with something fairly mechanical, the few remaining generic boss and midboss state variables. And once we start converting the constant numbers used for and around those variables into decimal, the estimated position independence probability immediately jumped by 5.31% for TH04's MAIN.EXE, and 4.49% for TH05's – despite not having made the game any more position- independent than it was before. Yup… lots of false positives in there, but who can really know for sure without having put in the work.

But now, we've RE'd enough to finally decompile something again next, 4 years after the last decompilation of anything!

📝 Posted:
🚚 Summary of:
P0016, P0017
Commits:
(Website) 98b5090...bca833b, (Website) 3f81d1f...d5b9ea2
💰 Funded by:
qp

Website development time: 12/12

Calculating the average speed of the previous crowdfunded pushes, we arrive at estimated "goals" of…

So, time's up, and I didn't even get to the entire PayPal integration and FAQ parts… 😕 Still got to clarify a couple of legal questions before formally starting this, though. So for now, let's continue with zorg's next 5 TH05 reverse-engineering and decompilation pushes, and watch those prices go down a bit… hopefully quite significantly!

📝 Posted:
🚚 Summary of:
P0013, P0014, P0015
Commits:
(Website) b9805d2...efeddd8, (Website) 31474a0...9dc9632, (Website) 9dc9632...8d3652f
💰 Funded by:
qp

Website development time: 10/12

In order to be able to calculate how many instructions and absolute memory references are actually being removed with each push, we first need the database with the previous pushes from the Discord crowdfunding days. And while I was at it, I also imported the summary posts from back then.

Also, we now got something resembling a web design!

📝 Posted:
🚚 Summary of:
P0012
Commits:
(Website) b9918cc...b9805d2
💰 Funded by:
qp

Website development time: 7/12

So yeah, "upper bound" and "probability". In reality it's certainly better than the numbers suggest, but as I keep saying, we can't say much about position independence without having reverse-engineered everything.

Next up: Money goals.

📝 Posted:
🚚 Summary of:
P0011
Commits:
(Website) 40c1e98...b9918cc
💰 Funded by:
qp

Website development time: 6/12

Here we go, overall ReC98 reverse-engineering progress. Now viewable for every commit on the page.

📝 Posted:
🚚 Summary of:
P0010, P0054, P0055
Commits:
(Website) cbda977...94127fb, (Website) 94127fb...3161d7e, (Website) 3161d7e...40c1e98
💰 Funded by:
DTM, Egor

Website development time: 5/12

Now with the number of not yet RE'd x86 instructions the you might have seen in the thpatch Discord. They're a bit smaller now, didn't filter out a couple of directives back then.

Yes, requesting these currently is super slow. That's why I didn't want to have everyone here yet!

Next step: Figuring out the actual total number of game code instructions, for that nice "% done". Also, trying to do the same for position independence.

📝 Posted:
🚚 Summary of:
P0051, P0052, P0053
Commits:
6ed8e6033493...3ba536a2d894
💰 Funded by:
-Tom-

Boss explosions! And… urgh, I really also had to wade through that overly complicated HUD rendering code. Even though I had to pick -Tom-'s 7th push here as well, the worst of that is still to come. TH04 and TH05 exclusively store the current and high score internally as unpacked little-endian BCD, with some pretty dense ASM code involving the venerable x86 BCD instructions to update it.

So, what's actually the goal here. Since I was given no priorities :onricdennat:, I still haven't had to (potentially) waste time researching whether we really can decompile from anywhere else inside a segment other than backwards from the end. So, the most efficient place for decompilation right now still is the end of TH05's main_01_TEXT segment. With maybe 1 or 2 more reverse-engineering commits, we'd have everything for an efficient decompilation up to sub_123AD. And that mass of code just happens to include all the shot type control functions, and makes up 3,007 instructions in total, or 12% of the entire remaining unknown code in MAIN.EXE.

So, the most reasonable thing would be to actually put some of the upcoming decompilation pushes towards reverse-engineering that missing part. I don't think that's a bad deal since it will allow us to mod TH05 shot types in C sooner, but zorg and qp might disagree :thonk:

Next up: thcrap TL notes, followed by finally finishing GhostPhanom's old ReC98 future-proofing pushes. I really don't want to decompile without a proper build system.

📝 Posted:
🚚 Summary of:
P0049, P0050
Commits:
893bd46969bf...6ed8e6033493
💰 Funded by:
-Tom-

Sometimes, "strategically picking things to reverse-engineer" unfortunately also means "having to move seemingly random and utterly uninteresting stuff, which will only make sense later, out of the way". Really, this was so boring. Gonna get a lot more exciting in the next ones though.

📝 Posted:
🚚 Summary of:
P0047, P0048
Commits:
9a2c6f72ba23...893bd46969bf
💰 Funded by:
-Tom-

So, let's continue with player shots! …eh, or maybe not directly, since they involve two other structure types in TH05, which we'd have to cover first. One of them is a different sort of sprite, and since I like me some context in my reverse-engineering, let's disable every other sprite type first to figure out what it is.

One of those other sprite types were the little sparks flying away from killed stage enemies, midbosses, and grazed bullets; easy enough to also RE right now. Turns out they use the same 8 hardcoded 8×8 sprites in TH02, TH04, and TH05. Except that it's actually 64 16×8 sprites, because ZUN wanted to pre-shift them for all 8 possible start pixels within a planar VRAM byte (rather than, like, just writing a few instructions to shift them programmatically), leading to them taking up 1,024 bytes rather than just 64.
Oh, and the thing I wanted to RE *actually* was the decay animation whenever a shot hits something. Not too complex either, especially since it's exclusive to TH05.

And since there was some time left and I actually have to pick some of the next RE places strategically to best prepare for the upcoming 17 decompilation pushes, here's two more function pointers for good measure.

📝 Posted:
🚚 Summary of:
P0046
Commits:
612beb815b9a...deb45ea14e01
💰 Funded by:
-Tom-

Stumbled across one more drawing function in the way… which was only a duplicated and seemingly pointlessly micro-optimized copy of master.lib's super_roll_put_tiny() function, used for fast display of 4-color 16×16 sprites.

With this out of the way, we can tackle player shot sprite animation next. This will get rid of a lot of code, since every power level of every character's shot type is implemented in its own function. Which makes up thousands of instructions in both TH04 and TH05 that we can nicely decompile in the future without going through a dedicated reverse-engineering step.

📝 Posted:
🚚 Summary of:
P0043, P0044, P0045
Commits:
261d5037ed39...612beb815b9a
💰 Funded by:
-Tom-

Turns out I had only been about half done with the drawing routines. The rest was all related to redrawing the scrolling stage backgrounds after other sprites were drawn on top. Since the PC-98 does have hardware-accelerated scrolling, but no hardware-accelerated sprites, everything that draws animated sprites into a scrolling VRAM must then also make sure that the background tiles covered by the sprite are redrawn in the next frame, which required a bit of ZUN code. And that are the functions that have been in the way of the expected rapid reverse-engineering progress that uth05win was supposed to bring. So, looks like everything's going to go really fast now?

📝 Posted:
🚚 Summary of:
P0025, P0026, P0027
Commits:
0cde4b7cd370...261d5037ed39
💰 Funded by:
zorg

… yeah, no, we won't get very far without figuring out these drawing routines.
Which process data that comes from the .STD files. Which has various arrays related to the background… including one to specify the scrolling speed. And wait, setting that to 0 actually is what starts a boss battle?

So, have a TH05 Boss Rush patch. Theoretically, this should have also worked for TH04, but for some reason, the Stage 3 boss gets stuck on the first phase if we do this?

Here's the diff for the Boss Rush. Turning it into a thcrap-style Skipgame patch is left as an exercise for the reader.

📝 Posted:
🚚 Summary of:
P0023, P0024
Commits:
807df3d5a749...0cde4b7cd370
💰 Funded by:
zorg

Actually, I lied, and lasers ended up coming with everything that makes reverse-engineering ZUN code so difficult: weirdly reused variables, unexpected structures within structures, and those TH05-specific nasty, premature ASM micro-optimizations that will waste a lot of time during decompilation, since the majority of the code actually was C, except for where it wasn't.

📝 Posted:
🚚 Summary of:
P0042
Commits:
f3b6137fe24d...807df3d5a749
💰 Funded by:
-Tom-

Laser… is not difficult. In fact, out of the remaining entity types I checked, it's the easiest one to fully grasp from uth05win alone, as it's only drawn using master.lib's line, circle, and polygon functions. Everything else ends up calling… something sprite-related that needs to be RE'd separately, and which uth05win doesn't help with, at all.

Oh, and since the speed of shoot-out lasers (as used by TH05's Stage 2 boss, for example) always depends on rank, we also got this variable now.

This only covers the structure itself – uth05win's member names for the LASER structure were not only a bit too unclear, but also plain wrong and misleading in one instance. The actual implementation will follow in the next one.

📝 Posted:
🚚 Summary of:
P0041
Commits:
b03bc9135749...f3b6137fe24d
💰 Funded by:
-Tom-

So, after introducing instruction number statistics… let's go for over 2,000 lines that won't show up there immediately :tannedcirno: That being (mid-)boss HP, position, and sprite ID variables for TH04/TH05. Doesn't sound like much, but it kind of is if you insist on decimal numbers for easier comparison with uth05win's source code.

📝 Posted:
🚚 Summary of:
P0040
Commits:
d7483c09cce1...b03bc9135749
💰 Funded by:
-Tom-

Let's start this stretch with a pretty simple entity type, the growing and shrinking circles shown during bomb animations and around bosses in TH04 and TH05. Which can be drawn in varying colors… wait, what's all this inlined and duplicated GRCG mode and color setting code? Let's move that out into macros too, it takes up too much space when grepping for constants, and will raise a "wait, what was that I/O port doing again" question for most people reading the code again after a few months.

🎉 With this push, we've also hit a milestone! Less than 200,000 unknown x86 instructions remain until we've completely reverse-engineered all of PC-98 Touhou.

📝 Posted:
🚚 Summary of:
P0009
Commits:
79cc3ed71ccf...141baa4ca92b
💰 Funded by:
DTM

While we're waiting for Bruno to release the next thcrap build with ANM header patching, here are the resulting commits of the ReC98 CDG/CD2 special offer purchased by DTM, reverse-engineering all code that covers these formats.

📝 Posted:
🚚 Summary of:
P0019, P0020, P0021, P0022
Commits:
c592464, cbe8a37, 8dfc2cd, 79cc3ed
💰 Funded by:
zorg
> OK, let's do a quick ReC98 update before going back to thcrap, shouldn't take long > Hm, all that input code is kind of in the way, would be nice to cover that first to ease comparisons with uth05win's source code > What the hell, why does ZUN do this? Need to do more research > … > OK, research done, wait, what are those other functions doing? > Wha, everything about this is just ever so slightly awkward

Which ended up turning this one update into 2/10, 3/10, 4/10 and 5/10 of zorg's reverse-engineering commits. But at least we now got all shared input functions of TH02-TH05 covered and well understood.

📝 Posted:
🚚 Summary of:
P0018
Commits:
746681d...178d58940853
💰 Funded by:
zorg

What do you do if the TH06 text image feature for thcrap should have been done 3 days™ ago, but keeps getting more and more complex, and you have a ton of other pushes to deliver anyway? Get some distraction with some light ReC98 reverse-engineering work. This is where it becomes very obvious how much uth05win helps us with all the games, not just TH05.

5a5c347 is the most important one in there, this was the missing substructure that now makes every other sprite-like structure trivial to figure out.

📝 Posted:
🚚 Summary of:
P0008
Commits:
47e5601613ff...d62dd06ddffb
💰 Funded by:
-Tom-

You could use this to get a homing Mima, for example.