📝 Posted:

2026-03-16 22:58 UTC

🚚 Summary of:

P0330 TH03 RE (Enemies / fireballs / explosions, part 1: Preparations + enemy state)
P0331 TH03 decompilation (Enemies / fireballs / explosions, part 2: Fireball state / enemy functions)
P0332 TH03 decompilation (Enemies / fireballs / explosions, part 3: Enemy functions) + Tooling (TH03 ENEDAT.DAT decompiler)
P0333 TH03 decompilation (Enemies / fireballs / explosions, part 4: Fireball functions + Chaining + Boss Attacks/Panics)
P0334 TH03 decompilation (Enemies / fireballs / explosions, part 5: Collision detection + Combos)
P0335 TH03 research (Enemies / fireballs / explosions, part 6: Score reduction and extend glitches) / Twitter→Fediverse migration, part 2 + Website (Multi-row tabs)

💰 Funded by:

[Anonymous], Ember2528, LeyDud

🏷️ Tags:

th01

+

th03

+

rec98

+

website

-

gameplay

+

score

+

enemy

+

file-format

+

contribution-ideas

+

kotohime

+

glitch

+

Most of my blog posts are way too long because they tend to cover multiple and often unrelated topics. I'm well aware that this blog would reach way more people if I just split these posts into multiple smaller and more thematically focused ones. This time, however, there would have been absolutely no way I could have split off anything about TH03's enemy, fireball, explosion, chaining, and combo systems while still leaving you with a coherent understanding. If this were a multi-post series, you'd be clicking back and forth to even just fully understand one of those systems in its entirety.

Really, what is this mess? I've procrastinated RE work on any of these systems ever since 📝 2022, when it became clear just how much enemies, fireballs, and explosions are connected with each other. Because both enemies and fireballs are things that can explode, ZUN decided to use the same 34-byte structure for them and the explosions they turn into, forming a single 64-element array of explodable entities.
This might remind you of 📝 TH04's and 📝 TH05's custom entity structures. Aren't these even used for 6 different things in each of the two games? Well, the difference between those games and TH03 is like night and day:

• The custom entity structure is only 26 bytes large, and uses much more consistent semantics for most of the fields. Only 3 (TH04) or 4 (TH05) fields are truly generic and are used in completely different ways within the custom subclasses…
• …if these fields are even used at all. These subclasses are always used in isolation, by a small and self-contained gameplay system. At first, TH03 doesn't seem too bad, because explosions aren't actually much of a standalone system. "Exploding" is simply another state that both enemies and fireballs can be in, and both of them update and render their explosions independently from each other, despite using the same sprites. Collision detection, however, is shared, and suddenly introduces a shared "base class" that reaches way further than it has any right to. (Also, enemies are by no means a small system.)
• But then, all of this also interacts with TH03's chaining system. Which is even less well-defined and isolated, and only exists as ad-hoc code within the collision detection of all three systems. Hence, I couldn't RE the chaining system in isolation because there was no way of deducing the intent behind these very specific calculations without knowing the conditions in the enemy/fireball/explosion mess that triggered them.

This was very much not code written to be read and understood. This was code that naturally evolved from an explorative process of playing around with the data, with the goal of creating an exciting game whose exact mechanics are hard to figure out from the outside.
But the result is a ≥2,200-line mess of tangled spaghetti that not only accesses the ostensibly same structure in subtly different ways, but also splashes an unhealthy dose of mutable global state on top, adding an extra layer of intractability. It's so convoluted that I would have loved to refactor it immediately before continuing with anything else in this project. It certainly makes sense to do so earlier rather than later because TH03 is the single most ideal PC-98 Touhou game for these sorts of sweeping architecture changes. Its highly dynamic gameplay makes it highly likely for small accidental deviations from ZUN's gameplay logic to snowball into wildly different game states, and we can easily observe these by comparing the demo replays against their 📝 original state. Alas, we're still 719 potential memory references away from position independence making that possible.

This has been the single hardest to understand piece of game logic in all of PC-98 Touhou. The RE content of this post alone beats 📝 the previous record holder for the post with the largest amount of PC-98 Touhou RE content by 2.38×, and would be the second-heaviest post by HTML size that this project has seen so far, just behind 📝 the giant Shuusou Gyoku waveform BGM project. What else could possibly be left in these games that would be harder than this? A lot of the not yet RE'd code is either an isolated gameplay system or isolated to specific characters or bosses, and 📝 a lot of script code will be highly repetitive… yeah, I'm pretty certain that PC-98 Touhou won't have a single remaining RE task that will take as long to complete as this one.
How do you structure a blog post that covers so many intertwined systems? Let's try building this up in the logical order from most to least intertwined:

1. An important note about durations in TH03
2. TH03's chaining system
3. Difficulty-specific tuning
   • round_speed
   • enemy_speed
     • Angle tuning
4. TH03's combo system
   • Boss Attacks and Panics
5. TH03's enemy formation scripts
   • Gameplay details
   • Function reference
6. TH03's enemies themselves
7. TH03's explosions
8. TH03's fireballs
   • Destroying fireballs with regular player shots or Charge Shots
9. TH03's score reduction and extend glitches
   • The fix (← new builds here!)
10. Completing the Fediverse migration
    • Polls
    • Bluesky rant, part 2

Before I can talk about anything else, I need to clarify the most important fact about in-game durations in TH03:

Any gameplay-related frame time or real-time duration in TH03 does not include the first 27 frames of every WARNING!! popup shown throughout a round. These frames freeze gameplay precisely because they run in a separate blocking function that runs its own VSync delay loop.

Obviously, this applies both to this blog and to anything you can read in the code. The gameplay community has been rather confused as far as precise durations are concerned, so I'm going to occasionally link back to this explanation in this post.

With that out of the way, let's start with…

The chaining system

…which orchestrates most of TH03's wildly escalating gameplay. Internally, chaining works like this:

• Each chain tracks four unsigned 8-bit values:
  • Number of hits. The Hit! counter in the HUD shows the single highest one of these among all active chains. Contrary to what its two digits might suggest, this internal field is actually clamped at 255, not 99; that's how the defeat screen at the end of each round gets to show the correct overall highest hit count using three digits.
  • Charge values for fireballs (charge_fireball) and Extra Attacks (charge_exatt). These cause the game to fire one of each once the charge hits a certain value (see below), after which they get reset to 0. Note how this means that you can't carry over any excess charge across individual fireballs and Extra Attacks, and that you don't get multiple fireballs or Extra Attacks if charge_* has reached a multiple of the required value.
  • A pellet_and_fireball_value that increases as the chain destroys pellets (+1), blue fireballs (+6), and red fireballs (+9). It controls how much charge explosions add to charge_fireball and charge_exatt upon destroying pellets and fireballs (not enemies), as shown in the table below.
  Notably, this list doesn't contain the amount of points that each chain contributes to the total bonus shown in the top-left corner of the playfield during an active combo. More on that in the combo section below.
• Both these charges and the bonuses described below rely on entities being destroyed by explosions. Therefore, destroying individual enemies with no other entities next to them doesn't impact the chain charges or the combo system in any way.
• The game can track up to 16 chains per player. On the surface, it looks as if these are organized as a ring buffer and assigned to explosions as such, but there are three cases where the game just adds charge to a different and unrelated chain – and yes, this foreshadows the much nastier details later on in this post. Since new chains also only reset their hit counter and the pellet/fireball value and retain their previous charges, they're best thought of as a regular, mutably written piece of state.
• Internally, these values are stored as a structure of four 16×2-element arrays, which will also become very relevant later on.
• New chains are started by killing enemies and creating new explosions with anything other than an existing explosion. The explosion is then linked to that specific chain slot.
• These explosions will then hit any pellets, enemies, and fireballs within their hitbox. Destroying the latter two with an explosion will then create new explosions that are linked to the same chain slot.

The charge values added for destroying enemies with explosions require further explanation, but we can already cover the pellet_and_fireball_value-dependent values added when destroying those two:

A particularly interesting detail about those: If more than one explosion hits a pellet or fireball within the same frame, these charges are added separately for each colliding explosion, to their respective chains. For red fireballs, TH03 then also adds a final +1 to the charge_exatt value of the chain whose colliding explosion has the highest index within the overall array of explodable entities.

Explaining the combo system would be the logical next step, but that system has a dependency on…

Difficulty-specific tuning

Where we get right to the final variable that would 📝 define TH03's difficulty in a round of netplay:

round_speed

This is a Q4.4 fixed-point value that is limited to a range from 0.0 to 7.9375 inclusive, increments by 0.0625 every 64 frames, and starts out at the following values:

This variable is then used to derive a whole variety of speeds and limits:

• Speed of pellets and fireballs while they transfer to the other playfield: (4.375 + ⌊^round_speed/₃⌋)
• Speed of the single pellet fired when killing an enemy with an explosion whose chain has a hit count of ≥3 and ≤(14 - (⌊^round_speed/_2.0⌋ × 2)), after it transferred to the other player's field: (1.5 + (^{⌊3 × round_speed⌋}/₁₆))
• Additional fireball speed added on top of the randomized base speed between 1.25 and 3.25: ⌊^round_speed/₂⌋
• Required charge_exatt within a chain to fire an Extra Attack after destroying a fireball with an explosion: (7 - ⌊^round_speed/_2.0⌋)
• Required charge_exatt within a chain to fire an Extra Attack after destroying a fireball with a regular player shot or Charge Shot, or an enemy with an explosion: (6 - ⌊^round_speed/_2.0⌋)
• Required charge_fireball within a chain to fire a fireball after destroying an enemy, fireball, or pellet with an explosion (12 - ⌊^round_speed/_1.0⌋)
• Two other things in character-specific Extra Attack code I haven't RE'd so far

Yup – the difficulty setting in the Option menu merely controls where all of these values start out at. After 6,144 frames of gameplay, or 1:49 minutes, even an Easy round will have accelerated to Lunatic levels.

enemy_speed

Whenever the game spawns a new formation, it updates this variable by integer-dividing round_speed by 2.0, resulting in a variable that ranges from 0 to 3 inclusive.
This value is then used to scale several speed-related variables using rather weird formulas. For example, ZUN scales movement durations using a formula of

Which looks like you could simplify it down to

But as the ⌊floor signs⌋ indicate, all of these divisions operate, once again, on integers, where additional divisions lead to additionally truncated remainders. In this case, they introduce an error of ±1 compared to the simplified formula at regular intervals. Such an error might look like ZUN tried to correctly round the non-truncated floating-point result, but that result would sometimes be even further away from what you get from the janky double-division formula. Let's take a duration of 25, for example:

Hence, this makes no sense on any level. The simplified expressions would take up less space in both the C++ code and the binary and execute faster.
The same happens for the speeds that enemies move at. In the code, ZUN uses a formula of

which looks like

but actually isn't, and suffers from the same off-by-one errors at regular intervals.

The final and most complex enemy_speed-dependent formula is used for tuning the angles for wavy or circular motions. This time around, ZUN actually used 16-bit angles, treating each enemy's angle field as effectively a Q8.8 fixed-point value. The movement vectors themselves are still calculated by indexing master.lib's 8-bit sine and cosine lookup tables with the top 8 bits of such an angle, but the extra 8 bits below add extra precision when adding the tuned angle_speed on every frame of such a motion.
This extended value range allowed ZUN to reduce the impact of enemy_speed to, literally, a very small degree:

This formula would have benefited most from a simplification to

which would have only introduced a maximum discrete error of 1.7 per frame, or an error of 0.0094° when converted to 360° angles. But the more critical issue with this formula lies in the first term, ⌊^{(512 × angle_speed)}/₃⌋. Multiplying an 8-bit angle_speed by 512 effectively left-shifts the value by 9 bits, requiring 17 bits to store all possible resulting values. However, ZUN then assigns this temporary result to a signed 16-bit variable, causing an overflow into negative numbers – and thus, an incorrectly reversed enemy rotation – as soon as angle_speed reaches ≥0x40, rather than the ≥0x80 where the sign is actually supposed to flip.
Then again, who would seriously add ±90° to an enemy's movement angle per frame. -0x08 and +0x08 are the highest speeds used in the original scripts, so this counts as neither a quirk nor a landmine in my book.

But now we can look at…

The combo system

As you might have already guessed from the list of per-chain fields, the game indeed only tracks the total combo bonus for each player as a single divided-by-10 16-bit value that every chain with ≥2 hits adds onto. For clarity, I'll only use the displayed values in this post. Once the displayed value reaches 655,350, it therefore becomes impossible to cancel the 📝 80-frame cooldown period by adding more bonus points. This also means that you lose all bonus points you would have earned within these last 80 frames.
The bonus value itself grows as you destroy certain entity types with an explosion:

With f being:

• 0.5 if round_speed ≤ 3.0
• 1.0 if round_speed > 3.0 and < 6.0
• 2.0 if round_speed ≥ 6.0

The thick border in the table indicates once again that the pellet and fireball bonuses are awarded multiple times if the entity is destroyed by more than one explosion on the same frame, just like the charges of its chain. Here are some messy screenshots of that case happening:

The bonus difference between the two frames is 31,600, which is exactly

With a round_speed of 7.125, we can further deduce that this screenshot was made about 21 seconds into a round at Lunatic difficulty, which checks out when looking at the score. Conclusion: The best way to make things happen in TH03 is to destroy red fireballs with explosions, and preferably multiple explosions in a single frame.

Boss Attacks and Panics

People seem to have the impression that this system and its bonus point (or Spell Point) requirements are complicated, so let's describe it all in just a few bullet points:

• The bonus point requirement is shared among both players.
• This requirement is only used for firing bonus-based Boss Attacks.
• Every time a player has increased their bonus by destroying something with an explosion and is not already displaying a WARNING!! popup on the other player's field, the game checks whether to fire a Boss Attack or Boss Panic:
  • Boss Attacks are fired once the player's bonus value has reached the point requirement and if they don't already have an active boss on the other player's field. Then, the game also increases the requirement by 51,200, and resets all non-hit fields of the triggering explosion's chain – that is, pellet_and_fireball_value, charge_fireball, and charge_exatt.
  • Boss Panics are additional 📝 Level 3 Gauge Attacks that are fired for players that do already have an active boss on the other player's field, once per combo if its value has reached 300,000 points. For this condition, bosses are already counted as active on the first frame of their entrance animation.
    Just like the Boss Attacks they depend on, the game delays firing a Boss Panic until all required conditions are met. Panics also do not increase the point requirement for regular Boss Attacks.
• The initial point requirement is 51,200.
• Manually firing a Boss Attack by charging the gauge up to Level 4 increases the requirement by 102,400.
• Destroying a boss, timing it out, or starting a new round will reset the requirement back to 51,200.
• Internally, the requirement is stored in the same divided-by-10 point format inside an unsigned 16-bit variable. Contrary to the bonus itself, however, none of the additions to the requirement are clamped, so it can and will overflow.
• The term spell points seems to have been made up by Rukaroa in 2005. Not too bad of a term to use outside of the codebase, considering that it's a nice two syllables and doesn't collide with any other gameplay feature of TH03.

Now that we know most of what there is to know about scoring, let's get the whole system going by placing some enemies in more or less chainable positions on the playfield!

Enemy formations

These are loaded from ENEDAT.DAT, which is the only bytecode format in the entirety of TH03. The original game does come with hardcoded limits of up to 16 enemies per formation and up to 24 formations in total, but the actual number of formations is neatly taken from the file itself.
The original ENEDAT.DAT defines 18 different formations, which I all recorded and posted to the Fediverse along with screenshots of their reconstructed script code, in their original order within the file. Isn't it nice that my primary PR channel can now hold lossless AV1 videos and lossless images that are just as self-hosted as this blog is, and that I can just link to without them taking up space in a blog post? Since there's nothing beyond the internal IDs in the game's memory that you could possibly want to cross-reference these formations with, I chose to number them using the same 0-based scheme. For the same reason, the names aren't important because nothing in the game needs to name these formations, so I just used the names that the gameplay community came up with, taken from these videos by Christian Azinn and KirbyComment.

Obviously, all these source code images mean that I've also written a dumper for the bytecode format, supporting the 13 functions that ZUN actually used in the original ENEDAT.DAT. Since we 📝 no longer build 32-bit Windows binaries as part of our build process, the ReC98 build process currently exclusively compiles this dumper as a DOS binary to bin\Pipeline\enedat.com. This allowed me to use the game's own header files at the expense of not having a trivially buildable native Windows or Linux version at this point. Such a build wouldn't be all that interesting without a compile option either, I'd say – and that part would definitely fall into the realm of contribution-ideas or require dedicated funding.

Gameplay details

These are quickly summarized:

• At the beginning of each Story Mode stage and VS round, TH03 reloads these formations into a dynamically allocated buffer. Then, it randomizes them into a single global cycle of 256 formations (while making sure not to show the same formation twice in a row) and 256 random flags that indicate whether the respective formation's positions and velocities should be horizontally mirrored. This means that you'll see this cycle of formations repeating if a round lasts long enough – although, given the average formation length of 3.93 seconds, this would take about 16:45 minutes of gameplay.
• Once the enemy system reports no living enemies for a player, they are advanced to the next formation on the next frame, where the game then spawns all of the formation's enemies at once. The check treats enemies as no longer alive once they have started playing the 36-frame explosion animation, which means that TH03 will always at least try to render either an enemy or an enemy-originating explosion on every single frame of gameplay. Enemies might get clipped and not actually be blitted, but the game at least tries to. This fact will become very important later on.
  

As usual for these sorts of script file-formats, I once again wrote a full…

Command reference

…which only requires a few additional bits of context to understand:

• All parameters are 1 byte large. Data types of common values are as follows:
  • angle*: 📝 8-bit angles
  • clip*: bool
  • duration: Unsigned
  • speed*: Q4.4 fixed-point, unsigned
  • velocity*: Q4.4 fixed-point, signed
• Most of these functions mutate a few state fields of each enemy, whose last values can therefore be shared across functions:
  • angle: Used for linear, wavy, and circular motions. This field uses the 16-bit angle type described above.
  • speed: The Q4.4 subpixel value that scales every fully recalculated 2D velocity vector calculated from angle.
  • velocity: The Q12.4 delta added to the enemy's X and Y coordinates during all movement functions.
• Functions are executed on the same frame until the script hits a blocking function with a duration parameter.
• Unused commands are in gray.

With no way of deduplicating the multiple enemies that fly along the same path in most formations, the scripts in ENEDAT.DAT end up highly copy-pasted, with individual enemies often only differing in the initial delay. It's hard to criticize this though, as this simple non-abstracted approach also provided the flexibility for ZUN to have enemies on multiple paths within the same formation in the first place.

If there is one design flaw in this format, it's the mere existence of the stop instruction. It does fulfill the role of terminating the script interpreter, whose instruction pointer would otherwise continue past the end of an enemy's script, but even that detail wouldn't justify its existence if we consider the design of these enemy formations. Every enemy of every formation is designed to fly past some edge of the playfield, get clipped, despawn, and end script execution that way, yet they still remove themselves using this stop instruction instead of relying on playfield edge clipping to do so.
As usual, I wouldn't mention this if it didn't cause at least one quirk in the game. If we look closely at formation #13 (Folding "7"), we see that the center enemies on the straight vertical path call stop() just a bit too early and despawn themselves just a few pixels above the bottom edge of the playfield:

You could equally argue that this quirk is simply caused by the human error of passing a duration to move_linear() that is shorter than it should have been. Since the playfield is 368 pixels tall, a 64-pixel enemy spawned at a top Y position of (-32 - (⁶⁴/₂)) = -64 and moving at a velocity of 4 pixels per second would need to keep moving for (^{(64 + 368)}/₄) = 108 frames rather than just 100. But removing the stop instruction would have made that quirk obvious by causing clearly visible glitches as a result of the script instruction pointer no longer being kept from running past the end of the respective enemy scripts.

Enemies

Despite all the explanations above, there's still a bit left to be said about enemies themselves:

• The enemy cap is 40 across both playfields. This is only 8 enemies more than the 16×2 enemies that formation #8 (Four Lines of Four), the formation with the highest enemy count, would need when running simultaneously for both players. Due to how TH03 instantly spawns all enemies in a formation and the fact that exploding enemies still take up an enemy slot, it is absolutely possible to hit this cap during regular gameplay and cause interesting formation-related quirks. The simplest test case would be this one:

  • Randomize enemy formations as usual, but start the game with one of the various 12-enemy formations and then continue with formation #8. Let's pick formation #0 (Loop-de-Loop), which has a generous 1-second window where all 12 enemies are visible on the playfield.
  • During this window, both players then need to fire a bomb at roughly the same time, turning all 24 living enemies into explosions.
  • Then, the game will advance both players to formation #8, with P1 taking precedence over P2 if both players bombed on the exact same frame. However, the game will then only have enough free enemy slots to exactly fit the 16 enemies for the one player that spawns them first.

  If both players do bomb on the exact same frame, this happens:

  The spawn call for every single one of P2's enemies ends up doing nothing because all enemy slots are occupied. Consequently, the enemy system continues to report no living enemies for P2, which causes the formation system to advance P2 to the next formation on the following frame. This process repeats for 29 more frames until the first three of both P1's and P2's enemies have finished their explosion animation. Then, P2 finally finds itself with enough free enemy slots to spawn at least 6 of the 8 enemies of formation #9 (Inverse "Y"), its then current 30^th formation of the randomized sequence – and a formation sequence cursor that has moved far ahead of P1's.
  This way of seeing into the future of the formation sequence might give P1 an advantage in competitive matches, as they can then prepare for the correct chaining route in advance. Then again, this would still require P2's cooperation – if just one of the players bombs a 12-enemy formation that is followed by the 16-enemy formation #8, the 40-enemy cap will be exactly enough to fit P1's then 28 enemies without further quirks.

• The difference between ghost, star, heart, and crescent enemies is purely visual, in case anyone was still wondering.
• The formation scripts might have already hinted at the fact that enemy HP are 0-based. Thus, enemies are only destroyed once they would reach -1 HP, not 0. Each enemy can also only lose up to 1 HP per frame.
• Killing enemies with an explosion adds the following charge values, depending on the hit count of the explosion's chain, to… not the explosion's chain, but the chain in slot ⌊^round_speed/_2.0⌋, which may or may not be active? Wait, what?! This is most likely why these charges aren't reset when starting a new chain.

  Hit count of explosion's chain	Effect
  ≤2	charge_fireball +1
  ≤(9 - ⌊round_speed/2.0⌋)	charge_fireball +2
  ≤(14 - (⌊round_speed/2.0⌋ × 2))	charge_fireball +3, charge_exatt +1
  (greater)	charge_fireball +4, charge_exatt +1

  Contrary to pellets and fireballs, these are not multiplied if more than one explosion hits an enemy on the same frame.
• On the 📝 collision bitmap against players, each enemy is represented by a striped rectangle that's 0.375× the size of the enemy and placed in its center.
• The hitboxes against player shots are very generous. Perhaps they might look overly so in the figure below, but that's mainly due to TH03 collision-detecting each pair of player shots as a single 32×16 rectangle. The basic hitbox of each enemy is a square in its center that's 0.875× its size, except for 16-pixel enemies, which receive a 20×20 square rather than the 14×14 one they'd otherwise get.

  

• Final fun fact: Kotohime's Charge Shot one-hits enemies regardless of HP. This is a deliberate feature that ZUN deliberately activates by setting a dedicated flag.

For reasons that will soon become apparent, I'll skip over fireballs for now. Instead, let's continue with…

Explosions

Collision detection is the only piece of explosion-related code shared between enemies and fireballs, so we can visualize most of the gritty internal details by simply looking at the hitbox frame data… and of course you're getting all 30 permutations of explosion size/source and destroyable entity:

We can summarize these 30 videos in just 5 bullet points:

• Explosions can only ever deal damage once every 4 frames. Against pellets and fireballs, this applies across all 36 frames of an explosion, resulting in 9 possible frames they can be destroyed on.
• Hits against enemies are further restricted to a window of frames that roughly corresponds to the frames of animation with lots of sparkles in a circle. Since the game flips between these two animation frames on the same 4-frame cycle that the collision detection runs on, the damaging frames are always indicated by a sprite change.
• For fireball-originating explosions, this window starts 4 frames later than it does for enemy-originating explosions. Whether this was a deliberate design choice or just an unintended inconsistency resulting from outdated copy-pasted code is anybody's guess.
• This leaves us with only 5 or even just 4 frames where an explosion can possibly damage an enemy: frame 13 (for enemy-originating explosions), 17, 21, 25, and 29.
• Fireball-originating explosions use the same sprites as explosions that originate from 48×48-pixel enemies, but are rendered 8 pixels lower than their internal position used for collision detection. This is the only fixable bug in all of the systems covered in this post, and it's already fixed in the new Anniversary Edition release.

But there is one more obscure detail that we can't see in these frame data videos. Even within the window where explosions can hit enemies, TH03 further caps the total amount of damage that a single explosion can deal to them. Each enemy-originating explosion can deal up to (size_words × 2) HP of damage in total, while fireball-originating explosions can deal up to 4 HP if the explosion originated from a blue fireball and 6 HP if it originated from a red one. ZUN's code suggests that this detail is among the most accidentally emerged features in the entire game – because of course he did not separately store the damage cap in one of the 14 padding bytes within the explodable structure, but simply reinterpreted size_words (for enemies) or the variant field (for fireballs) as half of the cap's value.
This limitation even seems largely pointless from a game design standpoint at first:

• The formation scripts always send more than one enemy along the same path.
• Hence, explosions from earlier enemies will cause subsequent enemies to explode right next to existing enemies.
• These new explosions will come with a fresh hit counter, which in turn will ensure the destruction of the next enemy on the same path, and so on.

Sure, once we consider the small number of frames where explosions can actually hit enemies, this might make it less likely for older explosions to kill enemies further along the formation's path, but does it really matter in practice?
Turns out that we have to look no further than formation #4 (Racetrack), which sends 12 enemies with increasing HP values along the same path. If we colorize explosions based on the total amount of damage they've dealt, we can clearly see the desync just 48 frames after the first explosion:

And now imagine a more complex game state where these earlier explosions might in turn cause hit-count-dependent pellets, charged fireballs, or charged Extra Attacks to fire earlier…

Only one left to go then!

Fireballs

As usual, let's start with a few facts in bullet-point form:

• Fireballs can, indeed, only initially enter the playfield as a result of a chain's fireball charge being sufficiently high.
• The cap for fireballs is 24, again across both playfields. Since this cap doesn't interact with other gameplay systems, this seems plenty enough.
• The target center X coordinates for newly spawned fireballs are random, but the possible value range alternates every 256 frames between
  • the initial setting of covering the entire playfield (i.e., [0.0; 288.0[), and
  • being restricted to the 32 pixels to the right of the target player's center X coordinate at the time the fireball was spawned.
  Just like bullets, fireballs also need to 📝 physically travel through the 32-pixel border between the two playfields during their transfer phase.
• The HP of fireballs behaves similarly to enemies: Fireballs can also only lose up to 1 HP per frame, and are destroyed once they would reach -1 HP, not 0. This is why the code says that blue fireballs have 2 HP and red fireballs have 3 HP, despite them taking 3 and 4 player shots to destroy, respectively.
  The main difference lies in how these HP only matter against regular player shots and most Charge Shots. Both explosions and, again, Kotohime's Charge Shot destroy fireballs in a single hit.
• There is code to move falling fireballs on the X-axis as well, but it won't have any effect in the original game because the X velocity of falling fireballs is always set to 0.
• Bombing immediately removes all fireballs from the field of the player who fired a bomb.
• Fireballs only turn into explosions when they are caught in other explosions, whose chains must have ultimately started by destroying an enemy.
• On the 📝 collision bitmap against players, each falling fireball is represented by a striped 16×16 rectangle in its center.
• Once again, the hitbox against player shots is roughly where you would expect, and not worth worrying about:

  

The more interesting parts about fireballs all relate to their destruction. Let's start with destruction by explosions, which seems to always send one new red fireball to the other player's field. In reality, these spawns are limited by a "generation number" system: Each blue fireball starts with a generation number of 0, while each red fireball starts, on paper, with 1 plus the generation number of whatever fireball was involved in its creation. In the case of explosions, the game obviously uses the generation number of the exploding fireball itself, but will only spawn such a new fireball if that number is <4.
This might look like even more of a rarely-triggered and expendable gameplay detail than the enemy damage cap we've seen above. How often does the "same" fireball really get transferred between both playfields 4 times in a row, without any player dropping a single generation? Well, exactly this case happens a mere 41.8 real-time seconds into 📝 the very first demo of Reimu vs. Mima. Removing the generation limit would fork gameplay as an explosion would then spawn another 5^th-generation fireball that otherwise wouldn't have been there.

Sometimes, however, the game also appears to spawn additional red fireballs in response to destroying only a single one with an explosion. The cleanest example I could find in the four demos happens 1:05 minutes into the Kotohime vs. Marisa demo:

With all the features we've previously looked at, it's easy to explain why we get that second fireball – because the chain's fireball charge value was high enough to spawn one. But why is this new fireball red? Doesn't 夢時空.TXT say that red [fireballs] have been sent back at least once?

赤いのは、１回以上送りかえされたことのある奴です。

Yup, 📝 another case where the manual is flat out wrong. The color variant of newly spawned fireballs is read from a global variable that is set to red at the beginning of the fireball collision detection function, set back to blue before returning, and not modified anywhere else. Hence, this color variable does not just apply to the fireballs spawned directly in the collision handler itself, but also to any fireballs spawned indirectly via the chaining system as a result of fireballs colliding with explosions. Given the fact that there's a significant gameplay difference between red fireballs and blue ones, this is quite a significant error in the manual, I'd say…

But it gets even quirkier. Which generation number do these chain-charged red fireballs spawn with? Logically, you'd expect 0, but red fireballs are always spawned with a generation number of at least 1. Apart from that, though, there is no other generation number that the code could meaningfully propagate. These fireballs are spawned by the same generic explosion collision handler that would also spawn blue fireballs upon destroying pellets, which only has access to a hitbox and not to the structure instance of the object it tests all explosions against.
But once again, the fireball spawn function reads the previous generation number from a global variable, so it will assign… 1 plus whatever generation number the previous explosion-destroyed fireball had.
And yes, the previously destroyed fireball, not the current one, because the collision handler that spawns these chain-charged fireballs is called before the global variable is updated with the affected fireball's generation number.

You can also destroy a fireball without an explosion though. Internally, this is done by removing the old fireball and spawning a new transferring red one at the same position. This new fireball will start with the old fireball's generation number incremented by 1, matching the explanation of the generation system from above. Unlike explosion-destroyed fireballs, there is no limit to the number of times a particular fireball can be transferred in this way. So you can go higher than 5 generations as long as you keep using shots to transfer fireballs back and forth – and if you manage 256 of those transfers, the generation counter will overflow back to 0.
But then, the game logic goes completely off the rails in the rest of the collision handler. For starters, this kind of fireball destruction still increases the Extra Attack charge of a chain, despite the fact that player shots and Charge Shots exist completely separate from the chaining system and the fact that the respective player may not even have any active explosions on their playfield. ZUN simply takes the chain slot that the next explosion would be assigned to, and adds +1 (for blue fireballs) or +2 (for red fireballs) to its charge_exatt value. Second reason why these charges aren't reset when starting a new chain, I guess…

And then, the code performs another charge_exatt firing check without adding any new charge, in what could have only possibly been a leftover function call that was supposed to go into the branch that handles fireball destruction by explosions. On this non-explosion code path, the chain_slot variable used in this check remains uninitialized, which leads to a potential read and write access outside of the bounds of this array…

…wait a moment, this immediately reminds me of certain bug reports from players that smelled like they were caused by out-of-bounds array accesses in unrelated parts of the game. I've been waiting to find one of these accesses ever since I heard about these bugs, and it's great that I immediately stumbled over the issue on my first RE pass over the fireball collision handler. Could it be that I've just found…

TH03's score reduction and extend glitches

Yup. In their thousands of hours of mastering this game's intricate systems, the gameplay community has encountered two very rare and seemingly random glitches without any obvious cause:

1. Random score reduction
2. Two extra Story Mode lives out of nowhere

And indeed, both of them could be caused if charge_exatt were indexed with a chain_slot ≥16, which would cause certain bytes to accidentally get reinterpreted as an Extra Attack charge value, compared against round_speed, and reset to 0 if their value is high enough. Want to take a guess at which particular pieces of data lie particularly close to charge_exatt in TH03's original memory layout? Well…

This gives us the following chain_slot ranges for the two glitches, depending on which player destroys the fireball:

1. If chain_slot is between 0x26 and 0x35 inclusive (P1) 0x16 and 0x25 inclusive (P2) , the charge check will hit one of the score digits, causing the score reduction variant of the glitch if the digit's value is high enough.

2. If chain_slot is exactly 0x3E (P1) 0x2E (P2) , the charge check will hit the byte that controls the number of Story Mode extends gained, causing the extend variant of the glitch if the byte's value is high enough.
   To understand how this can work, we need to look into how ZUN implemented TH03's extend system:

   • At the start of each round, extends_gained is initialized to the value of P1's second-highest score digit, or to 255 if the overall score is ≥20 million.
   • On every frame, the game then compares P1's second-highest score digit against extends_gained. If the digit is higher, it then grants an extend and increments the byte accordingly.
   • At a score of ≥20 million, extends_gained is set to 255, which is higher than any possible single digit and thus blocks the game from granting any further extends. Otherwise, you'd get additional ones at 110 million, 120 million, 210 million, …

   If you edit memory and set extends_gained from any nonzero value to 0, the game will therefore think that you haven't received any extends yet, and newly grant you as many of them as you're supposed to have with P1's current score: one additional extend if the score is ≥10 million, and two if your score is ≥20 million. There are no other checks that prevent the game from granting extends under this condition, so you could repeat this process until you've reached 255 lives, the maximum possible value.
   The "blocking" value of 255 also explains how this glitch can exist in the first place. It can only work if the game sets extends_gained to 0 on its own as a result of the Extra Attack charge check, and the regular values of 0, 1, and 2 are all smaller than the smallest possible Extra Attack charge value of 3. 255, on the other hand, is larger than any possible charge value. Thus, this glitch can only happen if you have ≥20 million points, but then, it will consistently happen whenever its other conditions are met.

But how likely could it possibly be for the chain_slot to fall within these exact ranges? After all, this bug is reported to only happen very rarely.
Turns out that explaining how and why we actually get these effects is not trivial in the slightest. Let's look at the stack layout across the relevant functions during a random frame of gameplay, and try to reason about the value we will actually find in this uninitialized local variable at runtime:

Where:

• The purple byte in fireballs_hittest() corresponds to chain_slot.
• The blue and red lines indicate the values of the BP and SP registers.
• INST and INED are the offsets of the start and end of the "init function" array that holds pointers to functions that should run before main(), annotated using #pragma startup. crt0 happens to load these offsets into the SI and DI registers prior to calling main(). Borland's calling conventions then require the previous value of these registers to be preserved by every function that needs these registers, which is done by pushing them on the stack. And since neither main() nor round_main() use SI or DI, we end up seeing additional stack copies of its original values – i.e., INST and INED – in a few other functions further up on the call stack.
• Labeled addresses prefixed with → correspond to the offset of the following instruction at the current function's call site, where execution will return to.
• en_* are local variables in the single-enemy rendering function. (Top-left X/Y coordinate, half of its size in screen pixels, and a pointer to the enemy instance.)
• All values are in hexadecimal, obviously.

That only leaves us with a handful of explicit numbers in this table, all of which are copies of the previous function's base pointer (BP). These get saved onto the stack as part of the typical x86 function prolog, which works like this:

1. The CALL instruction pushes the caller's code segment (for far calls) and the offset of the next instruction on the call site, before jumping to the address indicated by CALL's operand.
2. The new function then calls either ENTER <stack size>, 0 or the more RISC-y and consistently faster equivalent sequence of PUSH BP / MOV BP, SP / SUB SP, <stack size>, saving the previous function's base pointer and making room for all required local variables. The key insight here is that SP is simply subtracted. This is exactly where the "garbage values" of uninitialized variables come from, since they will start out with the value of whatever was previously written to their corresponding location on the stack.
3. If the function needs to modify SI or DI, it also pushes those registers, which further decreases SP.
4. Upon returning, the function executes the epilog counterpart of these instructions in reverse order:
   1. POP DI and POP SI if needed
   2. LEAVE or the equivalent sequence of MOV SP, BP / POP BP
   3. RETN (near) or RETF (far) to pop the instruction pointer and (for RETF) the caller's code segment

And it's these explicit numbers that reveal that ZUN got incredibly lucky:

• The initial 0x1000 value for BP we see in main() exactly corresponds to Turbo C++ 4.0J's default stack size of 4,096 bytes.
• Since ZUN makes very little use of the stack between main() and the two blitting functions, the top 8 bits of BP are guaranteed to be 0F by the time these two functions push that register onto the stack.
• The address of the local chain_slot variable within fireballs_hittest() will then fall on exactly that stack location where the blitting functions previously wrote the 0F to…
• … and since chain_slot is an 8-bit integer, it will therefore start out with an "uninitialized" value of 15, which just happens to be the very last valid index into the 16-element Extra Attack charge arrays.

This is also why it matters that the game renders at least one enemy or enemy-originating explosion on every frame of gameplay. The only time it renders neither is during the round start animation, but at that time, there also aren't any fireballs to be destroyed. Therefore, this exact tree of function calls is indeed the only one we need to look at, and proves that the formally undefined initial value of chain_slot is indeed deterministic on the one and only code path where it would be read from.

But from this alone, there should be no chance of the game ever writing outside the bounds of the Extra Attack charge array. So what could we possibly be missing here?
Well, just looking at MAIN.EXE's code flow doesn't actually give us the full picture. This is still a PC-98 game, and needs a full PC-98 system to run in the first place. And once we consider all the subsystems involved in running the game, we notice that three of them need to take control of the x86 instruction pointer at regular intervals:

• Keyboard input (calling interrupt 0x09)
• VSync (calling interrupt 0x0A), and
• PMD's background task (calling interrupt 0x14)

If any of the corresponding IRQs fires, the CPU has to immediately call the corresponding interrupt handler. This requires saving the current CPU flags, code segment, and offset onto the stack, very much like a call to a regular function. While this would only affect the "inactive" area of the stack below SP at the time of the interrupt call, this area exactly controls the "uninitialized" value of future local variables. Thus, interrupts can easily modify these values beyond the state they would have as a result of the normal flow of code.
This theory is supported by the fact that players have reported the extend glitch in particular to occur much more frequently on underclocked PC-98 systems. If each clock cycle takes up a bigger fraction of a second and the three interrupts have to run at regular intervals, we'd get more of them on every frame of gameplay. Thus, slower clock speeds increase the chance of such an interrupt to fall within the exact window of instructions to influence the initial value of the chain_slot variable.
Usually, running PC-98 Touhou over and over in underclocked emulators would make for an awful research and debugging experience. Thankfully, DOSBox-X's Turbo (fast-forward) option counteracts the slowdown in exactly the right way: While the reduced clock speed will stretch each 📝 logical frame to multiple real frames, fast-forwarding the emulation will then display all these frames as fast as possible and without blocking at the emulated VSync signal. The result runs very close to the intended 56.423 FPS, but with way more of those IRQs firing per second.

With this setup, let's see what happens if PMD's timer interrupt happens to fire somewhere near the end of enemies_render:

And there we have it. If any of those three interrupts is serviced within the very specific window of x86 instructions between the last rendered enemy and the call to fireballs_hittest(), the call to the interrupt handler will modify the stack in such a way that the future chain_slot variable will start out with the top 8 bits of the address of MAIN.EXE's fourth code segment, instead of its usually guaranteed value of 15. This will be more likely if the last living or exploding enemy was spawned into a slot close to the beginning of the 48-element enemy array, as TH03 will then spend more time looping over the rest of the structure without calling either of the two *_put() functions.

Alright, now we know what the exact variant of the glitch depends on. But how likely is it for MAIN.EXE's fourth code segment to actually fall within the affected ranges of memory, and how can we influence this placement?
First off, since DOS predates ASLR, any specific DOS system will always load its kernel, the shell, and any other drivers at deterministic addresses every time, as long as you don't change any of the parameters that influence this placement. Consequently, the system will also end up with the exact same free regions of memory every time, causing DOS to load a game at the exact same address as well. Of course, since DOS is an open platform where you can get arbitrary code execution by just, uh, writing code, building, and running it, you can easily push the game higher in memory by writing a TSR that reserves the desired amount of memory and executing it before TH03. Pushing TH03 lower, however, would require either

1. the infamous oldskool wizardry of freeing up as much conventional RAM as possible,
2. changing the DOS kernel, or
3. changing the 📝 version or type of the PMD driver loaded from GAME.BAT. (Note that this won't apply to the debloated or Anniversary Editions if you directly launch them via the debloat or anniv binaries because 📝 the integrated TSR spawning code pushes these drivers above the TH03 process).

Option 2) is particularly impactful, especially if we compare earlier DOS kernels with later ones:

• The first few major versions of MS-DOS load their kernel as a single large contiguous block near the start of conventional RAM, which pushes PMD and the game to the highest segment values I've observed in my tests.
• With MS-DOS 5, the kernel started pushing more and more of its code into the Upper Memory Area. This causes the MS-DOS 6.20 in a certain widely circulating .HDI of TH04 to load PMD and the game at addresses that will be roughly ~50 KB lower than the ones we see in MS-DOS 3.3.
• DOSBox-X doesn't need to reserve any conventional memory for a kernel because its entire kernel is implemented in native code. However, it still needs to spend about ~16 KB of low-memory space on run-time data structures for compatibility with programs that directly want to read them, and then chooses to add just as much padding to work around issues in certain games. With COMMAND.COM consistently at segment 0x0800 and the games following immediately after, the resulting segment values strongly resemble those of MS-DOS 6.20.
• The MS-DOS 7 bundled with Windows 95 features a slightly larger kernel and COMMAND.COM, which again pushes the games slightly higher compared to MS-DOS 6.20.

And this trend matches exactly with the addresses for segment #4 that we can observe when we go out into the wild and compare various distributions of TH03 against each other:

Cross-referencing these with the memory map above then gives us the following affected addresses:

The fix

So, we've got two glitches that may or may not appear on any given PC-98 setup, and whose effects depend on the combination of operating system, sound card model, and machine speed. That should make it abundantly clear that we can classify the underlying code issue as a landmine. There is nothing worth preserving about system-specific behavior on any ReC98 branch other than master, especially since we want to get away from the architecture in the long term.
It's also obvious that we must fix this to retain gameplay integrity in any build that wants to support netplay:

• Gameplay would desync if the two players run different PC-98 setups with different glitch behavior. If the glitch on a local player's system then spawns an additional Extra Attack onto their simulation of the other player's field, it wouldn't appear in the remote player's own simulation. Thus, they wouldn't be able to see and dodge it. Obviously, we'd like to strive for crossplay between emulated (and even real) PC-98 systems and ports to modern systems, and only cut that feature if there are good arguments against it. But dumb system-specific glitches don't qualify in the slightest.
• Also, we'd like to skip rendering during rollback. But that would also skip the call to enemies_render(), which is the whole reason why this glitch occurs with this exact frequency.

But how do we fix it? The offending call to chain_fire_charged_exatt() might look redundant at first, but it has definite effects on gameplay on both branches:

• The explosion branch contains its own copy of the Extra Attack charge check, with a well-defined chain slot but a higher threshold value. A chain might only meet the lower requirement of the usual function.
• The non-explosion branch uses the probably inactive chain slot at chain_ring_p[pid], which will be different from the uninitialized chain_slot value of 15 in the vast majority of cases. Two calls with two different chain slots might therefore cause a destroyed fireball to spawn two Extra Attacks if both charges are high enough.

The ideal fix is equally simple, though. Just initializing chain_slot to 15 removes all undefined behavior while simultaneously locking down the deterministic and observable effect of normal code flow. We can also easily implement this without breaking the original position-dependent binary by taking the 4 necessary additional bytes from the function's own bloat. And boy is there a lot of it; removing just even the most obvious single piece of bloat in this one function freed up 23 bytes, leaving 19 bytes unused. Any concerns about memory budgets for minor mods are vastly overblown.
This means that we're already up to the second release of TH03's debloated and Anniversary Editions!

ReC98 (version P0335) 2026-03-16-ReC98.zip

So that was 3,374 words to explain the ramifications of a single uninitialized local variable, leaving us at 5.375 pushes in total. What better way to round out this one than to finish the unfinished subproject from last time:

Completing the Fediverse migration

📝 As of last time, my Fediverse presence was only missing three pieces of data to function as a full-on replacement for Twitter:

1. All posts since 2023-07-02, which had twice as much media attached to them as the previous 8½ years combined
2. Alt text for images
3. Polls and their results, if this is even possible without assigning each vote to an existing Fediverse user

With the latter two not being part of Twitter's data archive, I already expected that I'd have to cobble together some very awkward code to obtain this data. But I didn't expect that this would cast an even worse light on Twitter:

• The frontend has been written in client-side-rendered React for many years, making scripted HTML scraping impossible without executing JavaScript.
• The Googlebot User-Agent trick that people documented in 2020 no longer works and immediately returns 403 Forbidden.
• https://publish.twitter.com/oembed?query=? No longer works either.
• But surely there is some kind of free API I could use to properly retrieve this dat– nope, API access is fully paywalled nowadays.

And I am certainly not paying just to efficiently retrieve data that should have been part of my data archive all along if I can at all avoid it.

Aren't there a few evil third-party projects that could help here? Nitter, for example, is well-known for retrieving Twitter timelines and displaying them on efficient, server-rendered, and easily scrapeable pages, bypassing the need for both activated JavaScript and a Twitter account just for reading posts. Sadly, it didn't render alt text on its frontend at the time in early February when I wanted to complete the Fediverse import. But it's open-source – and although I couldn't comprehend the exact mechanisms of the unofficial Twitter API they use, it might be the best starting point.

And sure enough, someone else previously needed alt text as well, found the string in the API response that Nitter was already processing, and wrote the code to pass it on to the frontend. All I had to do then was to cherry-pick these commits, adjust them for the current API's response schema, and run my own Nitter instance. This was the first time I hacked around in anything written in Nim, but I encountered no issues when building the project on Linux, although the build system is definitely on the slower end as far as systems languages are concerned.
Since this was ridiculous enough for what I wanted to do, I then pushed the updated versions of these commits, in the hope that someone else could save those 10-20 minutes of fixing merge conflicts. Since that issue was languishing for over three years, I certainly didn't expect that Nitter's maintainer would actually merge these commits 1½ weeks later. Even better, though! Thanks to ReC98, alt text is now shown on the main nitter.net instance. As of this blog post, the widget in Nitter's frontend still suffers from a visual overlapping bug in its unexpanded form when displaying multi-line alt text, but having that text in the server-rendered DOM at all is all I was asking for…

…except that my GoToSocial importer can't just automatically scrape this text from nitter.net because they prevent scraping, and have put a lot of effort into keeping it prevented. Specifically, any request made with the Python Requests library will return a 200 OK response code but an empty response body.
The Nitter wiki links a few other public instances, but none of those would work for us either. Some of these are still running a Nitter version older than 1c06a67, as of this blog post. And the ones that do run a current version employ similar scraping prevention techniques: They either respond with a 403 Forbidden, a redirect to a JavaScript challenge page, or just outright close the HTTP connection. Working around that would be way beyond reasonable, considering the budget that this task was supposed to take up… and if they don't want people to scrape them, I should respect that. Yet another example of AI crawlers ruining everything, I guess…
zedeus did recommend Twikit for programmatic access to Twitter data in the issue I linked above, but that library is currently unusable as well. A simple get_tweet_by_id() call returns 403 Forbidden, suggesting that you have to log in first, but any attempt to login() gets blocked by CloudFlare.

So yeah. You might only need to run such an import script once, but if you want a complete migration with alt text and polls, there's no way around temporarily self-hosting a Nitter instance, as far as I can tell. Lovely.
Speaking of polls, though…

Polls

With a scraping setup in place, retrieving poll data and calculating the exact votes from the given percentages on Nitter's frontend was no big deal. Instead, all of the annoyance here lies in then getting that data onto a Fediverse server:

1. Very understandably, GoToSocial lacks an API to pre-fill poll results,
2. but it also refuses to create backdated statuses with polls altogether.

These two points defeat any lasting automated migration code I could add to my GoToSocial importer. Printing out the imported data to stdout is the best I can do; any efficient method of importing backdated polls relies on removing the condition from 2) and compiling your own build of GoToSocial.
Thankfully, this is no big deal even when cross-compiling from Windows to Linux. Then, your POST /api/v1/statuses requests can at least include the poll's question labels and return the database ID of the newly created poll for further manipulation at the SQL level. Then, you can reformat the stdout output into a single query and run it manually:

UPDATE polls SET
	voters = 87,
	votes = '[25, 49, 11, 2]',
	expires_at = '2022-04-16 23:53:20.000000+00:00'
WHERE id = {poll_id};

But wait, really? You can manipulate a poll's results, both inside GoToSocial and on all other clients, with just a single SQL query? If these results don't need to be associated with specific users or at least their originating servers, doesn't this mean that polls on the Fediverse are inherently untrustworthy?
Of course, it makes little sense to verify poll results before displaying them on a server's own frontend. You can always just place a reverse proxy in front of the server and rig a poll in that way. But the inbox mechanism of ActivityPub already technically breaks anonymity across servers from the point of view of the server admin, so I would have expected both clients and other federated servers to at least run some sort of validation upon receiving a poll. Although that would definitely cause heavy load on servers and clients as these polls receive more and more answers, within a protocol that is already quite chatty…

In any case, this kind of validation would have to exist on an entirely different layer that has nothing to do with ActivityPub. That protocol – or rather, the underlying ActivityStreams vocabulary – only specifies how Questions and their associated answers are sent between servers. But once these votes are on a server, they only need to be stored in the minimal way expected by the usual client API. Which, as these words suggest, actually isn't even part of ActivityPub: Although that standard does attempt to specify a client API in addition to the server-to-server protocol, its development and adoption have stalled. Instead, most clients in common use simply implement the API that Mastodon once came up with, as a de facto standard. Mastodon's poll feature was quickly implemented in the most basic way, and no one has ever even proposed adding any validation features on top, as far as I can tell. And so, the GET /api/v1/statuses/{id} endpoint of any Mastodon-compatible Fediverse server is simply expected to return

{
	…
	"poll": {
		"options": [
			{ "title": "Option 1", "votes_count": 0 },
			{ "title": "Option 2", "votes_count": 0 },
		]
	}
	…
}

without any reference to where these votes are coming from, expecting the client to fully trust the server as far as the legitimacy of these votes is concerned. Of course, I'm a pretty trustworthy person if I may say so myself, but this detail turns the Fediverse into a pretty bad place to conduct even just semi-serious polls…
This is also why I haven't pointed any of the old links to Twitter polls on this website to my ActivityPub server yet. As long as the @ReC98Project account still exists and you can still view individual tweets without a Twitter account, it remains the authoritative and ultimately more trustworthy home of these poll results.

So in the end, centralized systems, and Twitter in particular, would still be indispensable for at least the occasional poll I tend to run. Bluesky, unfortunately, still doesn't natively support polls and instead forces people to build custom poll systems in the meantime. While slightly more trustworthy than Fediverse clients on paper, the ones I've seen are just Strawpoll clones that don't come with any kind of authentication, and often even allow people to place multiple votes by simply opening a new private browser window. Unsurprisingly, I therefore received several random and apparenly botted answers within seconds of creating a poll on both the systems I've used so far, making their results even more unusable. One of them also went offline within roughly 6 months of me using it, taking its one associated post with it.
Or maybe I should just conduct polls by asking for thought-out replies and rewarding people with, like, of ReC98 budget going to a goal of their choice…

On the topic of Bluesky, though…

The hell is this video quality? You upload lossless AV1 through the bsky.app frontend, and they blow up the video to 15.5× its original size by applying every single one of the worst possible processing steps:

1. For starters, they re-encode your video as H.264 using the High profile with YUV420P chroma subsampling. This is the same state of the art from 2005 that Twitter still expects you to produce yourself 21 years later. I'd expect any competitor to be better than that, but with Bluesky, that's just the start.
2. Because then, they enforce 30 FPS, duplicating or dropping frames as needed. This is inexcusable and disgusting.
3. They also bilinearly scale your video to 720 pixels in its shortest dimension. At least they maintain its original aspect ratio…
4. But this scaling process also targets a constant bitrate of 3,000 kbit/s, regardless of whether the video actually needs that much. I'm getting flashbacks of a certain dumb and uninformed if MP3, then 320kbps CBR, everything below sucks mentality.
5. Those bits would have been much better invested in keyframes. Of which we get exactly zero, causing seeking lags at consistent spots throughout these videos.

But maybe that's just bsky.app's backend doing its own video conversion. Maybe its developers did collect statistics and then made the conscious and informed decision that those exact processing steps would deliver the optimal trade-off between quality, traffic, and compatibility for the kind of HD content that people typically want to upload to such a platform nowadays. Maybe, my use case is just a rare exception… wait, gamers are unhappy too? 🤨
Surely then, the AT Protocol network still stores the original lossless video I uploaded, and an alternate client could play it directly, right? That's certainly what I would expect from a platform that advertises self-hosting, control over one's own data, and account portability.

So let's check Bluesky's data archive, and… the hell is a CAR file? A proprietary binary format in place of the standard .zip file you'd get from even the most evil corporations on the planet? Why the hell would they confront people trying to access their own data with a blog post containing Go code, directly linked from the data export modal? Usually, platforms are hard to leave due to network effects and people's inherent laziness to open another browser tab, but I've never seen a service also put up such a deliberate technical hurdle. How expensive could it have possibly been to set up a server that zips up all of a user's data on request? Why did they have to externalize these costs to their users?
The biggest joke, though: All code in that blog post is taken from this repository with example code, but the README in that directory says:

Check out the goat command line tool, which does the same thing and is actively maintained.

Except that the goat subcommand that actually downloads post-attached blobs, goat blob export, downloads directly from the AT Protocol network and doesn't even need the CAR file! Which, by the way, you could also download via goat repo export. And why wouldn't you, because you sure need this tool anyway to do anything because no existing tooling works with this format!
Anyway. Let's look at the blob of one of those videos, and…

$ ffprobe -hide_banner 03_source.webm
Input #0, matroska,webm, from '03_source.webm':
  Metadata:
    ENCODER         : Lavf62.3.100



  Duration: 00:00:05.12, start: 0.000000, bitrate: 454 kb/s
  Stream #0:0:
    Video:
      av1 (libdav1d) (High), gbrp(pc, gbr/bt709/iec61966-2-1, progressive),
      288x368, SAR 1:1 DAR 18:23, 56.42 fps, 56.42 tbr, 1k tbn
    Metadata:
      ENCODER         : Lavc62.11.100 libaom-av1
      DURATION        : 00:00:05.122000000

$ ffprobe -hide_banner bafkreid7mp5u7mb32o2a5n7zl2hbfxkiiaiebh7x77vik27alu2hfvuq5q
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bafkreid7mp5u7mb32o2a5n7zl2hbfxkiiaiebh7x77vik27alu2hfvuq5q':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.1.100
  Duration: 00:00:05.17, start: 0.000000, bitrate: 193 kb/s
  Stream #0:0[0x1](und):
    Video:
      h264 (High) (avc1 / 0x31637661), yuv420p(tv, unknown/bt709/iec61966-2-1, progressive),
      288x368 [SAR 1:1 DAR 18:23], 189 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc61.3.100 libx264

That's basically the same video before the scaling. Still YUV420P, still 30 FPS, still no keyframes.
I hate everything about this – and most importantly, the fact that I even have to bother with this complete waste of human engineering effort just because these kinds of platform choices are decided by politics and marketing for the vast majority of people. If I didn't land the occasional viral hit on there, I would have discontinued Bluesky immediately after this discovery. Maybe I should just trade one corporate platform for another and start looking into Tumblr, as touhou-memories recommended…

Oh well. Next up: Headless Shuusou Gyoku, preparing the automated replay validation I've wanted for years. Since Shuusou Gyoku's new replay system is long overdue now, I will put all future budget from Ember2528 into that system until it's done. If you'd like more TH03 done sooner, please fund it separately – but with of free budget left, there should be plenty of room for everyone.
That said, my life priorities do seem to have shifted for good by now. So I might need to reduce the cap if all of you really end up commissioning that much… But let's see how fast things will actually progress in the future, now that this worst possible piece of reverse-engineering is finally done.

📝 Posted:

2025-12-31 22:43 UTC

🚚 Summary of:

P0327 TH03 RE (Character-specific attack function pointers / Bullet dependencies / Bullet structure)
P0328 TH03 decompilation (Bullets, part 1)
P0329 TH03 decompilation (Bullets, part 2) / Twitter→Fediverse migration, part 1

💰 Funded by:

[Anonymous], Ember2528

🏷️ Tags:

th03

+

rec98

+

website

-

gameplay

+

bullet

+

micro-optimization

+

mima-th03

+

kana

+

chiyuri

+

glitch

+

pc98

+

blitting

+

meta

+

Alright! I've been announcing a big look at TH03's in-game systems all throughout 2025, and I technically still made it before the end of the year. TH03's enemy, fireball, and explosion systems are a great fit for this occasion: They fulfill both of the netplay-relevant criteria I mentioned 📝 at the end of the previous blog post, but also unfortunately share the same structure and overload some of their fields with vastly different meanings, much like 📝 TH04's and TH05's custom entities. Hence, they will take rather long to untangle, which ensures that the resulting look will be appropriately big.

Until I noticed that explosions spawn bullets in a not-so-straightforward way that basically requires complete knowledge of the bullet system. What a great discovery to make 2½ pushes into development… Oh well, we have the budget, and bullets also happen to match our netplay-relevant criteria, so let's get those done first.

1. A cursory look at character-specific attacks
2. TH03's bullet logic
   • High-level overview
   • Trail sprites
   • Rings and other groups
   • Transferred pellets
3. TH03's bullet renderer
   • Pellets, and unresolved VRAM alignment questions
   • Delay clouds, and the unfortunate lack of a SPRITE16 feature
4. Migrating away from Twitter
   • Twitter's technical downfall (not clickbait)
   • Trying a Bluesky PDS
   • Trying Mastodon
   • Deciding on GoToSocial as a Fediverse server

As usual, we'd also like to identify and name all character-specific functions in the ASM code so that we can immediately correlate certain interesting features of the bullet system with the characters and attacks that use them. In TH03, this is particularly worthwhile because it's all we need for a 100% complete overview of how bullets are used. Apart from the transferred pellets fired from exploding enemies outside of Gauge or Boss Attacks, every bullet pattern in the game is part of such a hardcoded and character-specific Extra, Gauge, or Boss Attack, since enemy scripts cannot fire bullets in this game.
This quick look also showed how ZUN implemented the 9 characters in a highly consistent manner. Gauge Attacks in particular follow a predictable convention:

• The "Level 2" attack (available at 50% gauge and consuming 25% gauge) always fires 8×8 pellets.
• The "Level 3" attack (available at 75% gauge and consuming 50% gauge) always fires 16×16 bullets.
• The game only provides a single function pointer for each of the two levels, which gets called as part of game logic and before the game starts rendering the current frame. With no room for custom rendering calls, characters can only define these attacks as patterns that are made up of common entities.

The funny part in all of this: All characters follow these conventions, yet ZUN still architected TH03 as if they don't. Each of the two Gauge Attack levels gets a separate per-player function pointer, but every character just uses these two functions to call a single common function with a flag that indicates Level 2 or Level 3. This common function then follows a similar structure for all 9 characters as well. The same trend continues with the Boss Attacks, where we find 9 copies of the more or less unchanged update and rendering boilerplate… so yeah, ZUN basically copy-pasted the same code 9 times with minor variations.
And now we can be very hyped for the future of TH03 decompilation. Lots of duplicates of the same functionality means that I'll basically only have to decompile them once, which means that TH03 decompilation is very likely to progress very quickly in terms of absolute numbers once I get the basic gameplay systems done. And there's not a lot of that code left either: After this delivery, we're left with a mere 132 undecompiled foundational functions that are not related to any specific character. After the next delivery, that number will drop to 99. I'm expecting a return to the glorious days of 2020, where the 3 copies of TH01's foundational graphics code allowed me to decompile 10% of its entire code within 2½ weeks. With 9 copies tripling that speed, we may even get to finish this game next year?

Onto bullets then! As you'd expect, TH03's bullet system is based on 📝 TH02's system we looked at earlier this year, which in turn was based on TH01's system. In some respects, it's a minor iteration of TH02's system adapted to the new features in TH03, but some of these new features also form the missing link between TH02 and 📝 TH04. The high-level overview:

• Like most entities in TH03, bullets are stored in a single array that is shared between both players. Each bullet has a structure field that denotes which playfield it is moving on and constrained to.

• The total bullet cap shared among both players is 320, slightly more than twice the 150-bullet cap we saw in TH02.

• Just like in TH02, this system covers both 8×8 pellets and 16×16 sprite bullets. The former are once again hardcoded and rendered using the GRCG, while the latter are rendered by SPRITE16.

• The system defines a default set of four adjacent 16×16 bullet sprites, starting at (64, 0) within 📝 SPRITE16's sprite area. These can be used in two ways:

  1. Four animation frames for a single bullet type, animated at the maximum speed of 1 cel per frame. This is how they are used by most characters.
  2. Alternatively, they can represent one non-animated bullet type and three trail sprites, as seen in Mima's and Chiyuri's Boss Attacks.

  Patterns can override the default 16×16 sprites with an arbitrary other set of four adjacent sprites, but this feature is only used in Kana's Extra Attack.

  Character	Sprites
  Reimu
  Mima
  Marisa
  Ellen
  Kotohime
  Kana
  Rikako
  Chiyuri
  Yumemi

• As we already found out 📝 in 2022, both 8×8 pellets and 16×16 bullets have the same "hitbox" – a single 2×2-pixel tile in the game's collision bitmap that gets compared against the 8×8 square surrounding the player's center.

• Delay clouds are back after their absence in TH02. They are still limited to pellets, though.

• The 📝 bullet template makes its debut in this game, replacing TH01's and TH02's spawn function parameters with a single piece of global data. This introduces the usual trade-offs with this sort of thing: Code size savings in patterns that spawn multiple groups with minor variations to their parameters, in exchange for the usual confusion that comes with widely mutated global state. As a result, code quality suffers greatly, especially when it comes to the derived transfer pellets fired within the bullet system itself. Sure, there are lots of patterns where retained state comes in handy, but local per-pattern template instances would have solved that as well.

• This decline in code quality also extends to the rest of the logic code. Continuing his general trend of micro-optimizing based purely on vibes we've seen 📝 time and 📝 time again, ZUN wrote a significant part of TH03's bullet logic in ASM, especially in the update function. And once again, it's the same tragic conclusion: While TH04's and TH05's full-on ASM approach might later bring some measurable runtime benefits to bullet logic via self-modifying code and whatnot, TH03 is left with pretty much only the downsides of its partial ASM approach. If ZUN just wrote idiomatic C++ code without any optimization tricks and inlined just one function, the whole bullet update code would have been 87 lines of C++ shorter and exactly as large when compiled. And that's with all the redundant code still in place! TH02's not-great-but-passable implementation of bullets indeed marked the high point for the PC-98 series, and it only went downhill from there.

Three features of the bullet system deserve a deeper look:

Trail sprites

These work by remembering the last 6 positions of a bullet and rendering the sprites at the 2^nd, 4^th, and 6^th position, respectively:

Obviously, this requires (6 × 2 × 2) = 24 additional bytes per bullet. Adding these to the regular bullet structure would waste (24 × 320) = 7,680 bytes of conventional RAM, which would in no way be justified for a feature that ZUN used in a grand total of two patterns. For once, ZUN agreed, and instead provided a single ring buffer that can hold these 6 additional positions for up to 48 bullets. This allowed ZUN to reduce the per-bullet cost to 3 bytes: 1 byte for the has trail flag, and 2 bytes for a near pointer into the ring buffer. That's still two more bytes than absolutely needed, and the debloated branch will definitely free up these wasted 640 bytes for portability reasons alone.

This trail sprite cap of 48 seems a bit random at first. Unlike the regular bullet cap that the game enforces by just not spawning any new bullets if all 320 slots are occupied, the trail sprite cap is not enforced or even just checked in any way. Due to the circular nature of the buffer, the 49^th simultaneously active bullet with a trail sprite will then share its position memory with the 1^st trail sprite bullet, leading to one additional position memory update per frame and trail sprites appearing in wrong positions.
Thus, it's the game design's responsibility to make minimal use of trail sprites to avoid these glitches. On the surface, it certainly looks as if ZUN was careful here:

1. The cap happens to exactly match the 48 bullets fired as part of the ring group in Chiyuri's pattern, which is definitely the more bullet-intensive pattern of the two.
2. Both trail-using patterns are part of Boss Attacks, and only a single player's Boss Attack can be active at any given time.
3. The ring groups in Chiyuri's pattern move fast and are separated by a 96-frame delay, as captured in the video above. By the time Chiyuri spawns the next group, every bullet of the previous one should have long been removed due to flying past the edges of the playfield.
4. Mima fires her alternating 5- and 4-spreads at a much shorter interval, but that interval is still long enough to never leave more than 5 of these 9-bullet subpatterns on screen at once.

Until you test Chiyuri's pattern on the (📝 announced) Boss Attack level 1 and notice that the bullets move slow enough for 2) to no longer apply. The result:

Missing bullets on every group beyond the first, and even sporadic trail sprites at mixed-up X and Y coordinates. This nicely demonstrates how these trail sprites are not just cosmetic, but also take control of clipping and affect gameplay as a result. For obvious optical reasons, trail sprite bullets will only get removed after the 6^th remembered position lies outside of the clipping area – i.e., 6 frames later than bullets without trail sprites. If the remembered positions are then shared with a second bullet, the game would also clip that bullet if the clipping condition of the first one is met – regardless of the fact that the second bullet's main sprite might be nowhere close to the boundaries of the playfield. This clipping then either happens on the same frame if the second bullet's slot number within the 320-element bullet array is higher than the slot number of the first one, or on the next frame if the second bullet's slot number is lower.
The mixed-up X and Y positions on frames 139 and 235 can also be explained by clipping. The update function processes the X and Y coordinates independently from each other: It starts with the horizontal clipping checks, updates the position memory for the X coordinate, and then repeats both steps for the Y coordinate, immediately removing the bullet and moving on to the next one if it failed the respective clipping check. If the vertical clipping checks fail in a situation where two bullets share the same position memory, you'll end up with a mismatched X/Y pair where X comes from a clipped bullet and Y comes from an active one… for a single frame, until the same clipping check is applied to the other bullet and removes it as well. Hence, this is the only "fixable" bug in the bullet system that won't affect gameplay, as the mixed-up positions are unrelated to the result of the clipping condition that ultimately removes both bullets.

It's rare for Chiyuri's Boss Attack to launch the same pattern multiple times in a row, and once the (announced) Boss Attack level is ≥3, bullets already move fast enough to prevent this quirk from happening. But it's definitely possible to run into it during regular gameplay.

For a clearer and more extreme demonstration of the resulting glitches, let's turn Mima's trail sprite pattern into a 64-ring:

Rings and other groups

If there's one aspect where TH03's bullet system shows its TH02 lineage most clearly, it's the set of predefined bullet groups. The 2-, 3-, 4-, and 5-spreads with fixed narrow, medium, and wide arc angles, as well as the multi-bullet groups with randomized angles and speeds, are not only available in TH03 once again, but reuse the exact same code from TH02.
Instead, TH03's main innovation can be found in its ring system. Rings can now have any number of bullets between 0 and 255, and are no longer limited to the first six powers of 2. This allowed ZUN to fine-tune most ring groups based on the Gauge or Boss Attack level, and to also just have a few static ring patterns with non-power-of-two bullet counts. Chiyuri's aforementioned 48-ring trail sprite pattern falls in this category, and the rotating 5-ring pattern seen in Kana's Boss Attack is another example.

And yes, storing the number of ring bullets in a regular uint8_t field now also allows patterns to spawn 0-rings. And sure enough, the bug that would later cause 📝 Kurumi's Divide Error crash in TH04 was actually introduced in TH03! The underlying code wasn't modified between the two games, which further proves that TH04's bullet system also traces back to TH03 and wasn't rewritten from scratch, at least concerning this aspect. TH03 just doesn't have any (known) way of triggering the bug in the unmodded original game.
Interestingly, TH02's ring system with predefined power-of-2 bullet counts is still part of TH03, and ZUN does use it for some ring groups in a few Boss Attack patterns. Did he do this because it's shorter than adding a second line of code that sets bullet_template.count? Did he deliberately need to preserve the previous value of bullet_template.count across groups? Or did he code these patterns at an earlier time in development when the arbitrary ring system didn't exist yet? Until I've decompiled every single bullet pattern in this game, we can only guess.

However, ZUN also removed two of TH02's group-related features from TH03:

1. The eight special motion types have been reduced to a single gravity type. While gravity is now a separate flag in the bullet structure and template that can now be applied to any group, this vast removal of options still severely limits the expressivity of bullet patterns in TH03. This means that every non-gravity bullet in the game moves at a constant velocity.
   Gravity is also exclusively used by Kana, in both her Extra attack with the alternative 16×16 bullet sprites as well as in one certain pattern of her Boss Attack, which demonstrates gravity in combination with a ring group.

2. The auto-stacking system was removed without any direct replacement. With TH03's more 📝 numeric method of defining difficulty, ZUN no longer needed this quick mechanism to 📝 distinguish Easy and Normal from Hard and Lunatic. This was one of the better changes between the two games though; the auto-stacking system added a quite annoying asterisk to the documentation of the random groups that is no longer needed in TH03.
   Manually creating stacks is obviously still possible by spawning separate versions of the same group with gradually reduced speed. This might be considered another practical advantage of the global bullet template, since you only need to mutate a single field before calling bullets_add() again. But really, nothing justifies global data.

Transferred pellets

This is the final gameplay feature that deserves its own section. Let's follow the pellet's X coordinate on its way from the spawn point to its destination on the other playfield:

1. ZUN calculates the destination coordinate on the target playfield as a random Q12.4 X coordinate between 0 and 288.
2. This subpixel coordinate is translated to screen space, adding either 16 for the left playfield or 336 for the right one. ZUN does this using the regular pixel-space conversion function that is typically used to calculate blitting coordinates, losing subpixel precision in the process and forming a very minor quirk.
3. The pellet's movement angle is calculated in screen space, aiming a screen-space version of the pellet's origin point at the coordinate from 2).
4. The screen-space pixel coordinate from 2) is translated back to a Q12.4 subpixel coordinate on the pellet's originating playfield. The result will deliberately lie outside the boundaries of this playfield: For a pellet flying from left to right, it will be between 320 and 608, while it will be between -320 and -32 for a pellet flying from right to left.
5. The pellet then flies to this out-of-bounds coordinate while internally staying on the playfield it originated on. This means that neither the update nor the rendering code can clip the pellet at the borders of its originating playfield. Once it flew past the border, it only visually appears on the other playfield because that's what the out-of-bounds X coordinate translates to when the renderer converts it to screen space.
6. Once the pellet's X coordinate has approached or flown past this relative target coordinate from its respective movement direction, the pellet is removed and respawned as a delay cloud.

This shows that the 32-pixel border between the two playfields is not just visual, but an actual part of the simulated game world. We can visualize this by removing the black cells on the text layer:

Speaking of, there's also a lot to cover in…

TH03's bullet renderer

And at first, it looks pretty good! TH03 retains the best idea from TH02 and batches rendering into three passes:

1. 16×16 bullets, rendered normally via SPRITE16
2. 32×32 delay clouds, rendered via SPRITE16's monochrome mode
3. 8×8 pellets, rendered from hardcoded sprites via the GRCG

The second pass is skipped if the first pass didn't detect at least a single active delay cloud. However, this skip can only remove 320 out of the 960 iterations over the entire bullet array, every frame. Combine that with the most unlucky allocation of registers, and the resulting instructions end up wasting a low 5-digit number of CPU cycles per frame on a 486 in the worst case of no bullets being active. Same game that wrote large parts of its bullet update function in ASM, by the way.
While that number is still an order of magnitude away from causing significant performance problems, this issue became serious enough in TH04 for ZUN to introduce a display list for at least pellets that would cut down the number of iterations.

And then, we look at…

Pellet rendering

… and are greeted by the single strangest set of hardcoded sprites across all of PC-98 Touhou so far. TH03's pellets are not only the first time we see a doubly-preshifted sprite sheet, but 2 of the 16 variants for the transfer pellet sprites are also shifted incorrectly:

📝 Looks familiar? Now we know that ZUN still didn't have a tool for automatically preshifting sprites by the time he developed TH03 – something that 📝 I considered a necessity 5½ years ago.

The doubly-preshifted nature of this sprite sheet, on the other hand, raises a whole lot of PC-98 blitting performance questions. This only possibly makes sense as an attempt at optimizing away the unaligned 16-bit VRAM writes you'd naturally run into when shifting an 8-wide sprite to cover two bytes.
Let's look at a regular 8-wide pellet sprite that was singly-preshifted to 16 pixels/bits. If we want to blit such a sprite to a left X position of 12 with the minimum amount of instructions, we would perform a single 2-byte write to VRAM address 0x0001, which itself is not divisible by 2:

On most 16-bit architectures, unaligned memory writes like these are either slower than aligned writes or entirely unsupported. The x86 MOV and MOVS instructions fall into the first category, so it makes sense to think that the GRCG might add a performance penalty of its own on top of the already higher latency of these instructions.
The natural workaround, then, is to add a second set of preshifted sprites to cover the remaining 8 possible start bit positions within a 16-pixel VRAM word. This would expand pellet sprites to a total width of 23 pixels. Understandably, ZUN also wanted to optimize for the low instruction counts, so he had to round up the physical width of the sprite to 32 pixels. Then, every preshifted variant could be blitted with a single MOVSD instruction:

But does this actually matter for the PC-98 and the GRCG? Are unaligned writes actually slow enough to justify writing 2× as much sprite data per frame and hardcoding 4× as many bytes? Unfortunately, I don't know of any hardware-level documentation about the GRCG that would conclusively answer this question. All the usual books and text files are disappointingly surface-level and only document the same programmer interfaces over and over, and hardware researchers are still waiting for EGC and GRCG die shots to even get started.
There are a few signs that this might be a good idea:

• The GDC reads out VRAM in 16-bit words, as repeatedly stated on almost every page of its hardware manual. This probably means the least though, because all Graphics Chargers sit at the other side of the dual-ported VRAM.
• Any VRAM-reading EGC operation must use aligned 16-bit accesses, which probably has a deeper reason that goes beyond the size of its internal shift register. And since you activate the EGC by first activating the GRCG in TDW mode…
• Neko Project spends the same number of clock cycles on both 8- and 16-bit GRCG writes. The absence of a dedicated 32-bit write handler suggests that real hardware breaks down 32-bit writes into two 16-bit writes, implying that we don't also have to worry about 32-bit alignment of our single MOVSD instruction.
• The MAME developers removed the 8-bit GRCG write function during a massive refactor 4 years ago, without explaining why. The MAME-typical high-level hardware explanation is still missing as well.
• The PC-9800Series Technical Data Book Hardware contains this little note in its description of the GRCG's RMW mode on page 192:

  バイトアクセス可能

  Shouldn't byte access be a given? Clearly, this would only deserve special mention if it wasn't because the previous contents of this book heavily implied some sort of 16-bit nature and I just missed it.

But without documentation or benchmarks, none of this means anything.
This is also why I haven't yet explored this whole field of optimizing VRAM writes for alignment. It would always involve branching to alignment-respecting code similar to how master.lib does it, but code like this is at odds with the more tangible goal of minimizing instruction counts in the generic case. Not to mention that we'll once again have to test this across every PC-98 hardware generation and possibly even GRCG revision if we ever go down to that level of optimization…

But even if alignment matters, ZUN's unconditional MOVSD instructions approach still appears to be slower on average. Consider the optimal 56.25% of cases where the sprite does lie within a single 16-bit word:

Keep in mind that we still use the GRCG here, and that it will also have to perform its fast-but-not-entirely-free four-plane Read-Modify-Write operation for the empty sprite bytes 3 and 4. Unconditional 32-bit writes would only be worth it if the GRCG somehow optimizes away empty writes at the microarchitecture level. That assumption is even more of a stretch, because 📝 why would master.lib even check for emptiness if that were true?

In the end, doubly-preshifted sprites slow down 56.25% of all blitting operations in a dubious attempt to speed up the other 43.75%. Unaligned 16-bit writes would have to be really slow to justify this approach – and judging from the fact that TH04 went back to single-byte preshifting, this is not the case. Maybe I'll write a benchmark for this someday, but honestly, this is the least interesting PC-98 benchmark question I've encountered so far. There are slowdown issues at 📝 our performance target of 66 MHz in Neko Project, but pellet sprite alignment is unlikely to significantly contribute to those.

16×16 bullets are simply rendered using standard SPRITE16 calls, nothing special there. That only leaves…

Pellet delay clouds

Just like the 48×48 📝 hit circle, these 32×32 sprites are rendered using SPRITE16's single-color render-path, which uses the EGC's GRCG-equivalent mode. Last year, I took a very brief look at this mode and wondered whether this was actually faster than just using the GRCG. 1½ years and 📝 one benchmark won by the EGC later, it certainly seems so, especially since we want to blit these to unaligned X positions. The EGC's hardware-accelerated pixel shifting seems highly preferable once sprite widths exceed 24 pixels and you can't fit a row of pixels in a 32-bit register anymore.
Stepping through SPRITE16 reveals that this GRCG-equivalent mode matches the GRCG even in how it doesn't read monochrome sprite data from VRAM, but from SPRITE16's 1bpp alpha mask buffer in conventional RAM.

But that only raises the question of why you'd want to use SPRITE16 over the raw EGC. It makes sense why SPRITE16 would have this feature; flashing existing sprites in a single color every once in a while is a useful thing to have in a game-focused rendering API. But using this feature for sprites that are only rendered in this monochrome mode just wastes the VRAM that these sprites occupy in SPRITE16's sprite area. You still blit such a sprite by passing a byte offset into the sprite area, which then gets interpreted as an offset into SPRITE16's alpha mask buffer.
If SPRITE16 had a function for directly blitting from a pointer to 1bpp data, ZUN could have freed up quite a bit of VRAM and maybe even added more sprites for character-specific attacks. Conceptually, it makes sense why SPRITE16 would restrict itself to a single sprite source, but it is quite an unfortunate omission, I'd say.

And that was the last PC-98 Touhou bullet system we were still missing! But at a little over 2 pushes, I have to find something else to do to round out the third one… wait, what about that one incident?

Migrating away from Twitter

On November 6, Twitter was hit by an automated ban wave that suspended all accounts that were using the OldTweetDeck extension. After Twitter discontinued the official free TweetDeck frontend on 2023-08-17, I quickly switched to OldTweetDeck – not just because it was free, but because it supported multiple accounts and was simply more performant than X's own premium offering at the time. In return, I gladly paid my €10 a month to dimden instead, who deserved it much more for continuously updating OldTweetDeck to all of Twitter's API changes over the years. It's very impressive how he kept it running for 2⅓ years without any such critical issues and still keeps maintaining it to this day.
Aside from Touhou Patch Center and all of my accounts, the ban wave affected enough people that Twitter decided to gradually revert it a day later. But without any public postmortem or excuse, this feels more like an act of gratitude that we shouldn't take for granted.

Ever since Elon took over, the Internet has been full of sensationalist doomposts about Twitter's imminent downfall any moment now OMG. For the longest time, I could ignore all these pundits because nothing of what they were complaining about was affecting my little corner. But sudden account suspensions are an existential threat to my business, and finally provided the first actual technical and non-political argument to get my data off Twitter in the medium term. I've put too much effort into all of the content there to let it be exclusively controlled by any one company.

Hilariously, things only got worse from there. Until two days ago, Twitter's data download option was inaccessible due to an infinite redirection bug. Call it malice or incompetence, but leaving such an issue unfixed for weeks is a definite sign of a platform in decline. Thus, I had to run the import on an older archive I happened to request on 2023-07-02.
And then I looked inside that archive and noticed that it was missing at least three key pieces of data that Twitter demonstrably stores for my account:

1. Poll options
2. Alt text for images. (Also known as the actually most annoying and time-consuming part of every tweet if you actually want to properly explain an image with all its context and implications. AI won't help with that as long as its context window doesn't span every piece of knowledge related to this project. 🤷)
3. The original version of each uploaded image, which they do have for a fact because it's shown in the /status view. The archive only contains the processed versions shown in the timeline, which were resized to at most 1200 pixels along their larger dimension and which may or may not have been converted to JPEG based on rules I didn't bother to reverse-engineer.

That's a rather selective interpretation of Art. 20 GDPR. If the argument is that you can just scrape that data out of the HTML yourself, why are they even bothering with sending me anything more than a nested list of tweet IDs, then? 🤨 Someone with more time and care could probably turn this into a lawsuit…
Presenting all media in its original quality is one of the more important reasons for moving to a self-hosted service as far as I'm concerned, especially since mainstream media conversion pipelines are infamous for destroying pixel art. So I went through my hard drives and replaced Twitter's images with the original versions of all 167 non-retweeted images I had uploaded to Twitter until July 2023. The videos also desperately needed to be replaced with their original AV1 versions; Twitter's enforced x264 YUV420P format has been the single worst implementation detail of that entire platform…

Trying a Bluesky PDS

So, time to complete the 📝 Bluesky self-hosting plan from last year? Setting up a PDS isn't all too annoying, but the import process leaves a lot to be desired:

• You can backdate posts by modifying their creation time, but Bluesky's crawlers will also record the indexed time when they first saw each post on the network. Unfortunately, the bsky.app frontend that everyone uses will then present this indexed time as a post's main timestamp, demoting your intended creation time to an archival time that Bluesky can't confirm the authenticity of:

  

  The PDS database schema does track an indexedAt timestamp in addition to the createdAt timestamp you specify during the import, but indexedAt might as well not exist because it doesn't seem to be used anywhere.
  There is a PR that would slightly improve the UI in this case, but it's been languishing unmerged throughout most of 2025. Probably because it has to be merged by the same people who came up with the current UI in the first place, and who prioritized resilience against pranks and disinformation campaigns.
  But even if the UI is fixed, these imports would spam the timeline of everyone who follows the existing Bluesky account that we obviously want to import into.

• Why does the best-known importing script convert any non-JPEG image to JPEG? 🤨 Removing the conversion and just uploading the PNG files from Twitter seems to cause any tweet with a post to simply not get imported at all. Of course, 📝 as we know since April, lossless WebP is preferable over PNG anyway, but, uh, what the hell, are they serious?!

Figuring out and confirming that first issue required remote debugging of the PDS server written in Node.js. Visual Studio Code's LSP quickly ran up against my server's low amount of RAM, which forced me to upgrade my server just to efficiently navigate through the source code…
Typical Node.js criticisms aside, the architecture of the PDS server is quite bizarre. A whole lot of the apparent API surface is never directly called, but generically proxied to some other node in the AT Protocol network at the byte level. If you log into your PDS via bsky.app, it seems as if the AppView calls API endpoints like /xrpc/app.bsky.unspecced.getPostThreadV2 on your PDS, but good luck meaningfully intercepting any of these requests, or even just getting your debugger to break on them.
Together with lots of bulky API schema descriptions in the form of lexicons, all this XRPC code makes up a big proportion of the code in the @atproto/pds package. But for… what exactly? Why would the PDS server need a thick layer of type safety and validation for payloads it doesn't look at, and that the relays will have to verify anyway? Why do they install all this dead code that will confuse most people who are trying to understand this system?
In the end, we just can't thoroughly backdate our imported posts because the crawl timestamps are set by the relays, whose code we have no control over. Now, I could ignore all these issues and still upload some sort of full archive to the platform that now houses ¹/₆ of my following, but this just doesn't match the quality I expect from the canonical, definitive source of my short-form news posts.
Edit (2026-03-16): And things look 📝 even worse once you take a closer look at their video processing pipeline and the hoops you have to jump through to get your data out of the AT Protocol network…

Trying Mastodon

That leaves the Fediverse as the only remaining alternative for a service where people can still follow, like, and repost my content using relatively commonly used clients. Among the various ActivityPub implementations, Misskey is particularly popular among the Japanese Touhou community, but I've only heard bad things about its resource usage. Mastodon isn't the most lightweight option either – as aptly implied by its name – but you can make the argument that it's become the default option across the Fediverse over the years. Thus, there'll be at least a slight chance that people will be familiar with the web UI of what I'm about to self-host.

Too bad that I didn't even get through the first page of the setup guide before being stumped by obscure asset precompilation errors that apparently no one else has ever faced. In a way, it's commendable that a project would exclusively explain a bare-metal from-source setup in the Docker-dominated DevOps seascape of 2025. But why would you want to do this for a project that requires servers to be infested with npm and Postgres and a bleeding-edge self-compiled version of Ruby and several -dev packages for C dependencies of certain Ruby gems? Unsurprisingly, Japanese Python behaves just like Dutch Ruby in how the community effectively treats every minor version as a major version because there are no adults left in the room to put all the children and Ph.Ds in their place…

Fortunately, ActivityPub is relatively simple to implement and there are plenty of existing servers that are better suited to the kind of PR channel I'm actually looking for. After a very quick search, I settled on…

GoToSocial

…which immediately impresses in pretty much every single area:

• An explicit backdating feature for imports? In an API that's compatible with Mastodon's and requires minimal adjustments to existing Mastodon import scripts, instead of requiring that feature to be temporarily modded into the server?
• After the two previous bloatfests, it's very refreshing to see a single binary next to a bunch of static assets. Sure, 87.4 MiB is certainly way more bulky than necessary, but still much smaller than either of our two competitors.
• The documentation is extremely well-organized and polished, especially for a project that's on version 0.20.2.
• I can write Markdown in posts!
• WebP and AV1? Just work too, without any attempt to convert the main image or video that gets attached to a post. Sure, the thumbnailer does convert images, but that's way less critical…
• …and you can effectively bypass it by passing some five-digit size to media-thumb-max-pixels.
• The whole thing works exactly like a lightweight server should work: A single binary serving posts from a SQLite database and media attachments from static files lying next to it. With easy access to every piece of data, fixing typos and import errors after the fact is trivial. Applying these might need a server restart for caching reasons, but they're immediately reflected in whatever app is accessing the data.
• Drawbacks? The database schema is highly redundant, poster image conversion for videos results in weirdly green images for every one of my AV1 source files, and the paginated timeline view could use just a few more navigation options and customizability. Other seemingly missing features like posting and search are handled by third-party clients like the very admirable Pinafore. And except for the first issue, these are all relatively minor, and I might even fix them myself one day. That's how you get new contributors to your free software project.

And just in case you ever want to import a Twitter archive onto a GoToSocial instance, here is the no-nonsense importer I used:

• twitter-archive-to-gotosocial

So if you've got an account on Misskey, Mastodon, or another ActivityPub server, please follow @rec98@nmlgc.net. I'll keep posting everything to both Twitter and Bluesky for the time being, but will no longer advertise either of them. If they ever go down, I'll make no attempt at restoring them.

And that was 2025! It surely brought lots of words, breaking even last year's record by an additional 37% of blog post content. 😮 Here's to 2026 bringing more of the actual reverse-engineering we've been sorely lacking with all the modding and porting projects over the past few years. And with at least four TH03 gameplay pushes queued up, things are already looking quite promising…
Next up: Enemies! Formation scripts! Fireballs! Explosions! Combos, or at least the first part of them! And a slightly more common glitch that players have been wondering about for many years…

📝 Posted:

2024-11-22 04:00 UTC

🚚 Summary of:

P0296 Website / Blog (Post list page + Navigation and post header redesign)
P0297 TH03 decompilation (Character selection screen)

💰 Funded by:

[Anonymous], 32th System

🏷️ Tags:

th03

+

website

-

rec98

+

pc98

+

emulation

+

animation

+

menu

+

performance

+

palette

+

Remember when ReC98 was about researching the PC-98 Touhou games? After over half a year, we're finally back with some actual RE and decompilation work. The 📝 build system improvement break was definitely worth it though, the new system is a pure joy to use and injected some newfound excitement into day-to-day development.
And what game would be better suited for this occasion than TH03, which currently has the highest number of individual backers interested in it. Funding the full decompilation of TH03's OP.EXE is the clearest signal you can send me that 📝 you want your future TH03 netplay to be as seamlessly integrated and user-friendly as possible. We're just two menu screens away from reaching that goal anyway, and the character selection screen fits nicely into a single push.

1. TH03's character selection screen
   • Loading a whole lot of images
   • The background curve animation
     • Performance issues
     • Bugs and quirks
   • VRAM and palette tearing
2. Improved blog navigability

The code of a menu typically starts with loading all its graphics, and TH03's character selection already stands out in that regard due to the sheer amount of image data it involves. Each of the game's 9 selectable characters comes with

1. a 192×192-pixel portrait (??SL.CD2),
2. a 32×44-pixel pictogram describing her Extra Attack (in SLEX.CD2), and
3. a 128×16-pixel image of her name (in CHNAME.BFT). While this image just consists of regular boldfaced versions of font ROM glyphs that the game could just render procedurally, pre-rendering these names and keeping them around in memory does make sense for performance reasons, as we're soon going to see. What doesn't make sense, though, is the fact that this is a 16-color BFNT image instead of a monochrome one, wasting both memory and rendering time.

Luckily, ZUN was sane enough to draw each character's stats programmatically. If you've ever looked through this game's data, you might have wondered where the game stores the sprite for an individual stat star. There's SLWIN.CDG, but that file just contains a full stat window with five stars in all three rows. And sure enough, ZUN renders each character's stats not by blitting sprites, but by painting (5 - value) yellow rectangles over the existing stars in that image.

Together with the EXTRA🎔 window and the question mark portrait for Story Mode, all of this sums up to 255,216 bytes of image data across 14 files. You could remove the unnecessary alpha plane from SLEX.CD2 (-1,584 bytes) or store CHNAME.BFT in a 1-bit format (-6,912 bytes), but using 3.3% less memory barely makes a difference in the grand scheme of things.
From the code, we can assume that loading such an amount of data all at once would have led to a noticeable pause on the game's target PC-98 models. The obvious alternative would be to just start out with the initially visible images and lazy-load the data for other characters as the cursors move through the menu, but the resulting mini-latencies would have been bound to cause minor frame drops as well. Instead, ZUN opted for a rather creative solution: By segmenting the loading process into four parts and moving three of these parts ahead into the main menu, we instead get four smaller latencies in places where they don't stick out as much, if at all:

1. The loading process starts at the logo animation, with Ellen's, Kotohime's, and Kana's portraits getting loaded after the 東方夢時空 letters finished sliding in. Why ZUN chose to start with characters #3, #4, and #5 is anyone's guess.
2. Reimu's, Mima's, and Marisa's portraits as well as all 9 EXTRA🎔 attack pictograms are loaded at the end of the flash animation once the full title image is shown on screen and before the game is waiting for the player to press a key.
3. The stat and EXTRA🎔 windows are loaded at the end of the main menu's slide-in animation… together with the question mark portrait for Story Mode, even though the player might not actually want to play Story Mode.
4. Finally, the game loads Rikako's, Chiyuri's, and Yumemi's portraits after it cleared VRAM upon entering the Select screen, regardless of whether the latter two are even unlocked.

I don't like how ZUN implemented this split by using three separately named standalone functions with their own copy-pasted character loop, and the load calls for specific files could have also been arranged in a more optimal order. But otherwise, this has all the ingredients of good-code. As usual, though, ZUN then definitively ruins it all by counteracting the intended latency hiding with… deliberately added latency frames:

• The entire initialization process of the character selection screen, including Step #4 of image loading, is enforced to take at least 30 frames, with the count starting before the switch to the Selection theme. Presumably, this is meant to give the player enough time to release the Z key that entered this menu, because holding it would immediately select Reimu (in Story mode) or the previously selected 1P character (in VS modes) on the very first frame. But this is a workaround at best – and a completely unnecessary one at that, given that regular navigation in this menu already needs to lock keys until they're released. In the end, you can still auto-select the default choice by just not releasing the Z key.
• And if that wasn't enough, the 1P vs. 2P variant of the menu adds 16 more frames of startup delay on top.

Sure, maybe loading the fourth part's 69,120 bytes from a highly fragmented hard drive might have even taken longer than 30 frames on a period-correct PC-98, but the point still stands that these delays don't solve the problem they are supposed to solve.

But the unquestionable main attraction of this menu is its fancy background animation. Mathematically, it consists of Lissajous curves with a twist: Instead of calculating each point as x = sin((fx·t)+ẟx) y = sin((fy·t)+ẟy) , TH03 effectively calculates its points as x = cos(fx·((t+ẟx) % 0xFF)) y = sin(fy·((t+ẟy) % 0xFF)) , due to t and ẟ being 📝 8-bit angles. Since the result of the addition remains 8-bit as well, it can and will regularly overflow before the frequency scaling factors fx and fy are applied, thus leading to sudden jumps between both ends of the 8-bit value range. The combination of this overflow and the gradual changes to fx and fy create all these interesting splits along the 360° of the curve:

In a rather unusual display of mathematical purity, ZUN fully re-calculates all variables and every point on every frame from just the single byte of state that indicates the current time within the animation's 128-frame cycle. However, that beauty is quickly tarnished by the sheer cost of fully recalculating these curves every frame:

• In total, the effect calculates, clips, and plots 16 curves: 2 main ones, with up to 7×2 = 14 darker trailing curves.
• Each of these curves is made up of the 256 maximum possible points you can get with 8-bit angles, giving us 4,096 points in total.
• Each of these points takes at least 333 cycles on a 486 if it passes all clipping checks, not including VRAM latencies or the performance impact of the 📝 GRCG's RMW mode.
• Due to the larger curve's diameter of 440 pixels, a few of the points at its edges are needlessly calculated only to then be discarded by the clipping checks as they don't fit within the 400 VRAM rows. Still, >1.3 million cycles for a single frame remains a reasonable ballpark assumption.

This is decidedly more than the 1.17 million cycles we have between each VSync on the game's target 66 MHz CPUs. So it's not surprising that this effect is not rendered at 56.4 FPS, but instead drops the frame rate of the entire menu by targeting a hardcoded 1 frame per 3 VSync interrupts, or 18.8 FPS. Accordingly, I reduced the frame rate of the video above to represent the actual animation cycle as cleanly as possible.
Apparently, ZUN also tested the game on the 33 MHz PC-98 model that he targeted with TH01, and realized that 4,096 points were way too much even at 18.8 FPS. So he also added a mechanism that decrements the number of trailing curves if the last frame took ≥5 VSync interrupts, down to a minimum of only a single extra curve. You can see this in action by underclocking the CPU in your Neko Project fork of choice.

But were any of these measures really necessary? Couldn't ZUN just have allocated a 12 KiB ring buffer to keep the coordinates of previous curves, thus reducing per-frame calculations to just 512 points? Well, he could have, but we now can't use such a buffer to optimize the original animation. The 8-bit main angle offset/animation cycle variable advances by 0x02 every frame, but some of the trailing curves subtract odd numbers from this variable and thus fall between two frames of the main curves.
So let's shelve the idea of high-level algorithmic optimizations. In this particular case though, even micro-optimizations can have massive benefits. The sheer number of points magnifies the performance impact of every suboptimal code generation decision within the inner point loop:

• Inlining grcg_pset() would save 42 cycles by cutting out one far function call and the required stack and register shuffling for passing two parameters. (Remember how 📝 TH01's EGC-powered unblitter suffered from the same function call performance issue?)
• Frequency scaling works by multiplying the 8-bit angles with a fixed-point Q8.8 factor. The result is then scaled back to regular integers via… two divisions by 256 rather than two bitshifts? That's another ≥46 cycles where ≥10 would have sufficed.
  Edit (2025-08-29): The initial version of this post miscounted the number of required cycles as ≥4, or 2× the cycle count of a single SAR instruction. That number didn't consider that the frequency scaling multiplication occasionally produces negative numbers, which 📝 must be conditionally rounded up when replacing signed divisions with arithmetic bitshifts to still produce the exact original animation. This conditional rounding adds ≥8 cycles in the more common positive case, and ≥6 in the rarer negative case.
• The biggest gains, however, would come from inlining the two far calls to the 5-instruction function that calculates one dimension of a polar coordinate, saving another ≥100 cycles.

Multiplied by the number of points, even these low-hanging fruit already save a whopping ≥729,088 cycles per frame on an i486, without writing a single line of ASM! On Pentium CPUs such as the one in the PC-9821Xa7 that ZUN supposedly developed this game on, the savings are slightly smaller because far calls are much faster, but still come in at a hefty ≥466,944 cycles. Thus, this animation easily beats 📝 TH01's sprite blitting and unblitting code, which just barely hit the 6-digit mark of wasted cycles, and snatches the crown of being the single most unoptimized code in all of PC-98 Touhou.
The incredible irony here is that TH03 is the point where ZUN 📝 really 📝 started 📝 going 📝 overboard with useless ASM micro-optimizations, yet he didn't even begin to optimize the one thing that would have actually benefitted from it. Maybe he 📝 once again went for the 📽️ cinematic look 📽️ on purpose?

Unlike TH01's sprites though, all this wasted performance doesn't really matter much in the end. Sure, optimizing the animation would give us more trailing curves on slower PC-98 models, but any attempt to increase the frame rate by interpolating angles would send us straight into fanfiction territory. Due to the 0x02/2.8125° increment per cycle, tripling the frame rate of this animation would require a change to a very awkward (log₂384) = 8.58-bit angle format, complete with a new 384-entry sine/cosine lookup table. And honestly, the effect does look quite impressive even at 18.8 FPS.

There are three more bugs and quirks in this animation that are unrelated to performance:

• If you've tried counting the number of trailing dots in the video above, you might have noticed that the very first frame actually renders 8×2 trailing curves instead of 7×2, thus rendering an even higher 4,608 points. What's going on there is that ZUN actually requested 8 trailing curves, but then forgot to reset the VSync counter after the initial 30-frame delay. As a result, the game always thinks that the first frame of the menu took ≥30 VSync interrupts to render, thus causing the decrement mechanism to kick in and deterministically reduce the trailing curve count to 7.
  This is a textbook example of my definition of a ZUN bug: The code unmistakably says 8, and we only don't get 8 because ZUN forgot to mutate a piece of global state.

• The small trailing curves have a noticeable discontinuity where they suddenly get rotated by ±90° between the last and first frame of the animation cycle.
  This quirk comes down to the small curve's ẟy angle offset being calculated as ((c/2)-i), with i being the number of the trailing curve. Halving the main cycle variable effectively restricts this smaller curve to only the first half of the sine oscillation, between [0x00, 0x80[. For the main curve, this is fine as i is always zero. But once the trailing curves leave us with a negative value after the subtraction, the resulting angle suddenly flips over into the second half of the sine oscillation that the regular curve never touches. And if you recall how a sine wave looks, the resulting visual rotation immediately makes sense:
  Removing the division would be the most obvious fix, but that would double the speed of the sine oscillation and change the shape of the curve way beyond ZUN's intentions. The second-most obvious fix involves matching the trailing curves to the movement of the main one by restricting the subtraction to the first half of the oscillation, i.e., calculating ẟy as (((c/2)-i) % 0x80) instead. With c increasing by 0x02 on each frame of the animation, this fix would only affect the first 8 frames.
• ZUN decided to plot the darker trailing curves on top of the lighter main ones. Maybe it should have been the other way round?

If you want to play with the math in a more user-friendly and high-res way, here's a Desmos graph of the full animation, converted to 360° angles and with toggles for the discontinuity and trail count fixes.

Now that we fully understand how the curve animation works, there's one more issue left to investigate. Let's actually try holding the Z key to auto-select Reimu on the very first frame of the Story Mode Select screen:

Stepping through the individual frames of the video above reveals quite a bit of tearing, particularly when VRAM is cleared in frame 1 and during the menu's first page flip in frame 49. This might remind you of 📝 the tearing issues in the Music Rooms – and indeed, this tearing is once again the expected result of ZUN landmines in the code, not an emulation bug. In fact, quite the contrary: Scanline-based rendering is a mark of quality in an emulator, as it always requires more coding effort and processing power than not doing it. Everyone's favorite two PC-98 emulators from 20 years ago might look nicer on a per-frame basis, but only because they effectively hide ZUN's frequent confusion around VRAM page flips.
To understand these tearing issues, we need to consider two more code details:

1. If a frame took longer than 3 VSync interrupts to render, ZUN flips the VRAM pages immediately without waiting for the next VSync interrupt.
2. The hardware palette fade-out is the last thing done at the end of the per-frame rendering loop, but before busy-waiting for the VSync interrupt.

The combination of 1) and the aforementioned 30-frame delay quirk explains Frame 49. There, the page flip happens within the second frame of the three-frame chunk while the electron beam is drawing row #156. DOSBox-X doesn't try to be cycle-accurate to specific CPUs, but 1 menu frame taking 1.39 real-time frames at 56.4 FPS is roughly in line with the cycle counting we did earlier.
Frame 97 is the much more intriguing one, though. While it's mildly amusing to see the palette actually go brighter for a single frame before it fades out, the interesting aspect here is that 2) practically guarantees its palette changes to happen mid-frame. And since the CRT's electron beam might be anywhere at that point… yup, that's how you'd get more than 16 colors out of the PC-98's 16-color graphics mode. 🎨
Let's exaggerate the brightness difference a bit in case the original difference doesn't come across too clearly on your display:

This reproduces on both DOSBox-X and Neko Project 21/W, although the latter needs the Screen → Real palettes option enabled to actually emulate a CRT electron beam. Unfortunately, I couldn't confirm it on real hardware because my PC-9821Nw133's screen vinegar'd at the beginning of the year. But just as with the image loading times, TH03's remaining code sorts of indicate that mid-frame palette changes were noticeable on real hardware, by means of this little flag I RE'd way back in March 2019. Sure, palette_show() takes >2,850 cycles on a 486 to downconvert master.lib's 8-bit palette to the GDC's 4-bit format and send it over, and that might add up with more than one palette-changing effect per frame. But tearing is a way more likely explanation for deferring all palette updates until after VSync and to the next frame.

And that completes another menu, placing us a very likely 2 pushes away from completing TH03's OP.EXE! Not many of those left now…

To balance out this heavy research into a comparatively small amount of code, I slotted in 2024's Part 2 of my usual bi-annual website improvements. This time, they went toward future-proofing the blog and making it a lot more navigable. You've probably already noticed the changes, but here's the full changelog:

• The Progress blog link in the main navigation bar now points to a new list page with just the post headers and each post's table of contents, instead of directly overwhelming your browser with a view of every blog post ever on a single page.
  If you've been reading this blog regularly, you've probably been starting to dread clicking this link just as much as I've been. 14 MB of initially loaded content isn't too bad for 136 posts with an increasing amount of media content, but laying out the now 2 MB of HTML sure takes a while, leaving you with a sluggish and unresponsive browser in the meantime. The old one-page view is still available at a dedicated URL in case you want to Ctrl-F over the entire history from time to time, but it's no longer the default.

• The new 🔼 and 🔽 buttons now allow quick jumps between blog posts without going through the table of contents or the old one-page view. These work as expected on all views of the blog: On single-post pages, the buttons link to the adjacent single-post pages, whereas they jump up and down within the same page on the list of posts or the tag-filtered and one-page views.

• The header section of each post now shows the individual goals of each push that the post documents, providing a sort of title. This is much more useful than wasting space with meaningless commit hashes; just like in the log, links to the commit diffs don't need to be longer than a GitHub icon.

• The web feeds that 📝 handlerug implemented two years ago are now prominently displayed in the new blog navigation sub-header. Listing them using <link rel="alternate"> tags in the HTML <head> is usually enough for integrated feed reader extensions to automatically discover their presence, but it can't hurt to draw more attention to them. Especially now that Twitter has been locking out unregistered users for quite some time…

Speaking of microblogging platforms, I've now also followed a good chunk of the Touhou community to Bluesky! The algorithms there seem to treat my posts much more favorably than Twitter has been doing lately, despite me having less than ¹/₁₀ of mostly automatically migrated followers there. For now, I'm going to cross-post new stuff to both platforms, but I might eventually spend a push to migrate my entire tweet history over to a self-hosted PDS to own the primary source of this data.

Next up: Staying with main menus, but jumping forward to TH04 and TH05 and finalizing some code there. Should be a quick one.

📝 Posted:

2023-11-30 23:56 UTC

🚚 Summary of:

P0262 Decompilation (TH04/TH05 main/option menu)
P0263 Decompilation (TH04/TH05 first-launch sound setup menu + TH05 title screen animation)

💰 Funded by:

Blue Bolt, [Anonymous]

🏷️ Tags:

th04

+

th05

+

rec98

+

website

-

menu

+

waste

+

file-format

+

animation

+

And once again, the Shuusou Gyoku task was too complex to be satisfyingly solved within a single month. Even just finding provably correct loop sections in both the original and arranged MIDI files required some rather involved detection algorithms. I could have just defined what sounded like correct loops, but the results of these algorithms were quite surprising indeed. Turns out that not even Seihou is safe from ZUN quirks, and some tracks technically loop much later than you'd think they do, or don't loop at all. And since I then wanted to put these MIDI loops back into the game to ensure perfect synchronization between the recordings and MIDI versions, I ended up rewriting basically all the MIDI code in a cross-platform way. This rewrite also uncovered a pbg bug that has traveled from Shuusou Gyoku into Windows Touhou, where it survived until ZUN ultimately removed all MIDI code in TH11 (!)…

Fortunately, the backlog still had enough general PC-98 Touhou funds that I could spend on picking some soon-important low-hanging fruit, giving me something to deliver for the end of the month after all. TH04 and TH05 use almost identical code for their main/option menus, so decompiling it would make number go up quite significantly and the associated blog post won't be that long…

Wait, what's this, a bug report from touhou-memories concerning the website?

1. Tab switchers tended to break on certain Firefox versions, and
2. video playback didn't work on Microsoft Edge at all?

Those are definitely some high-priority bugs that demand immediate attention.

1. Microsoft Edge's anti-support of AV1
2. TH04/TH05's main/option menu
3. TH04/TH05's first-launch sound setup menu
4. TH05's title animation ☯️

The tab switcher issue was easily fixed by replacing the previous z-index trickery with a more robust solution involving the hidden attribute. The second one, however, is much more aggravating, because video playback on Edge has been broken ever since I 📝 switched the preferred video codec to AV1.
This goes so far beyond not supporting a specific codec. Usually, unsupported codecs aren't supposed to be an issue: As soon as you start using the HTML <video> tag, you'll learn that not every browser supports all codecs. And so you set up an encoding pipeline to serve each video in a mix of new and ancient formats, put the <source> tag of the most preferred codec first, and rest assured that browsers will fall back on the best-supported option as necessary. Except that Edge doesn't even try, and insists on staying on a non-playing AV1 video. 🙄

The codecs parameter for the <source> type attribute was the first potential solution I came across. Specifying the video codec down to the finest encoding details right in the HTML markup sounds like a good idea, similar to specifying sizes of images and videos to prevent layout reflows on long pages during the initial page load. So why was this the first time I heard of this feature? The fact that there isn't a simple ffprobe -show_html_codecs_string command to retrieve this string might already give a clue about how useful it is in practice. Instead, you have to manually piece the string together by grepping your way through all of a video's metadata…
…and then it still doesn't change anything about Edge's behavior, even when also specifying the string for the VP9 and VP8 sources. Calling the infamously ridiculous HTMLMediaElement.canPlayType() method with a representative parameter of "video/webm; codecs=av01.1.04M.08.0.000.01.13.00.0" explains why: Both the AV1-supporting Chrome and Edge return "probably", but only the former can actually play this format. 🤦

But wait, there is an AV1 video extension in the Microsoft Store that would add support to any unspecified favorite video app. Except that it stopped working inside Edge as of version 116. And even if it did: If you can't query the presence of this extension via JavaScript, it might as well not exist at all.
Not to mention that the favorite video app part is obviously a lie as a lot of widely preferred Windows video apps are bundled with their own codecs, and have probably long supported AV1.

In the end, there's no way around the utter desperation move of removing the AV1 <source> for Edge users. Serving each video in two other formats means that we can at least do something here – try visiting the GitHub release page of the P0234-1 TH01 Anniversary Edition build in Edge and you also don't get to see anything, because that video uses AV1 and GitHub understandably doesn't re-encode every uploaded video into a variety of old formats.
Just for comparison, I tried both that page and the ReC98 blog on an old Android 6 phone from 2014, and even that phone picked and played the AV1 videos with the latest available Chrome and Firefox versions. This was the phone whose available Firefox version didn't support VP9 in 2019, which was my initial reason for adding the VP8 versions. Looks like it's finally time to drop those… 🤔 Maybe in the far future once I start running out of space on this server.

Removing the <source> tags can be done in one of two places:

1. server-side, detecting Edge via the User-Agent header, or
2. client-side, using navigator.userAgentData.brands.

I went with 2) because more dynamic server-side code would only move us further away from static site generation, which would make a lot of sense as the next evolutionary step in the architecture of this website. The client-side solution is much simpler too, and we can defer the deletion until a user actually hovers over a specific video.
And while we're at it, let's also add a popup complaining about this whole state of affairs. Edge is heavily marketed inside Windows as "the modern browser recommended by Microsoft", and you sure wouldn't expect low-quality chroma-subsampled VP9 from such a tagline. With such a level of anti-support for AV1, Edge users deserve to know exactly what's going on, especially since this post also explains what they will encounter on other websites.

Alright, where was I? For TH01, the main menu was the last thing I decompiled before the 100% finalization mark, so it's rather anticlimactic to already cover the TH04/TH05 one now, with both of the games still being very far away from 100%, just because people will soon want to translate the description text in the bottom-right corner of the screen. But then again, the ZUN Soft logo animation would make for an even nicer final piece of decompiled code, especially since the bouncing-ball logo from TH01, TH02, and TH03 was the very first decompilation I did, all the way back in 2015.

The code quality of ZUN's VRAM-based menus has barely increased between TH01 and TH05. Both the top-level and option menu still need to know the bounding rectangle of the other one to unblit the right pixels when switching between the two. And since ZUN sure loved hardcoded and copy-pasted numbers in the PC-98 days, the coordinates both tend to be excessively large, and excessively wrong. Luckily, each menu item comes with its own correct unblitting rectangle, which avoids any graphical glitches that would otherwise occur.
As for actual observable quirks and bugs, these menus only contain one of each, and both are exclusive to TH04:

• Quitting out of the Music Room moves the cursor to the Start option. In TH05, it stays on Music Room.
• Changing the S.E. mode seems to do nothing within TH04's menus, and would only take effect if you also change the Music mode afterward, or launch into the game.

Now that 100% finalization of their OP.EXE binaries is within reach, all this bloat made me think about the viability of a 📝 single-executable build for TH04's and TH05's debloated and anniversary versions. It would be really nice to have such a build ready before I start working on the non-ASCII translations – not just because they will be based on the anniversary branch by default, but also because it would significantly help their development if there are 4 fewer executables to worry about.
However, it's not as simple for these games as it was for TH01. The unique code in their OP.EXE and MAINE.EXE binaries is much larger than Borland's easily removed C++ exception handler, so I'd have to remove a lot more bloat to keep the resulting single binary at or below the size of the original MAIN.EXE. But I'm sure going to try.

Speaking of code that can be debloated for great effect: The second push of this delivery focused on the first-launch sound setup menu, whose BGM and sound effect submenus are almost complete code duplicates of each other. The debloated branch could easily remove more than half of the code in there, yielding another ≈800 bytes in case we need them.
If hex-editing MIKO.CFG is more convenient for you than deleting that file, you can set its first byte to FF to re-trigger this menu. Decompiling this screen was not only relevant now because it contains text rendered with font ROM glyphs and it would help dig our way towards more important strings in the data segment, but also because of its visual style. I can imagine many potential mods that might want to use the same backgrounds and box graphics for their menus.

With the two submenus being shown in a fixed sequence, there's not a lot of room for the code to do anything wrong, and it's even more identical between the two games than the main menu already was. Thankfully, ZUN just reblits the respective options in the new color when moving the cursor, with no 📝 palette tricks. TH04's background image only uses 7 colors, so he could have easily reserved 3 colors for that. In exchange, the TH05 image gets to use the full 16 colors with no change to the code.

Rounding out this delivery, we also got TH05's rolling Yin-Yang Orb animation before the title screen… and it's just more bloat and landmines on a smaller scale that might be noticeable on slower PC-98 models. In total, there are three unnecessary inter-page copies of the entire VRAM that can easily insert lag frames, and two minor page-switching landmines that can potentially lead to tearing on the first frame of the roll or fade animation. Clearly, ZUN did not have smoothness or code quality in mind there, as evidenced by the fact that this animation simply displays 8 .PI files in sequence. But hey, a short animation like this is 📝 another perfectly appropriate place for a quick-and-dirty solution if you develop with a deadline.
And that's 1.30% of all PC-98 Touhou code finalized in two pushes! We're slowly running out of these big shared pieces of ASM code…

I've been neglecting TH03's OP.EXE quite a bit since it simply doesn't contain any translatable plaintext outside the Music Room. All menu labels are gaiji, and even the character selection menu displays its monochrome character names using the 4-plane sprites from CHNAME.BFT. Splitting off half of its data into a separate .ASM file was more akin to getting out a jackhammer to free up the room in front of the third remaining Music Room, but now we're there, and I can decompile all three of them in a natural way, with all referenced data.
Next up, therefore: Doing just that, securing another important piece of text for the upcoming non-ASCII translations and delivering another big piece of easily finalized code. I'm going to work full-time on ReC98 for almost all of December, and delivering that and the Shuusou Gyoku SC-88Pro recording BGM back-to-back should free up about half of the slightly higher cap for this month.

📝 Posted:

2023-09-30 21:01 UTC

🚚 Summary of:

P0252 Seihou / Shuusou Gyoku (Building SDL from source within Tup)
P0253 Seihou / Shuusou Gyoku (Game logic portability, part 1/? + Cross-platform APIs, part 1/?: Window creation via SDL)
P0254 Seihou / Shuusou Gyoku (Cross-platform APIs, part 2/?: Sound via miniaudio)
P0255 Seihou / Shuusou Gyoku (Cross-platform APIs, part 3/?: Input via SDL)
P0256 Seihou / Shuusou Gyoku (Restoring the screenshot feature + Translating inline ASM code to C++)
P0257 Website (Video player code cleanup + audio support)

💰 Funded by:

Arandui, Ember2528, [Anonymous]

🏷️ Tags:

seihou

+

sh01

+

website

-

build-process

+

input

+

micro-optimization

+

And now we're taking this small indie game from the year 2000 and porting its game window, input, and sound to the industry-standard cross-platform API with "simple" in its name.

Why did this have to be so complicated?! I expected this to take maybe 1-2 weeks and result in an equally short blog post. Instead, it raised so many questions that I ended up with the longest blog post so far, by quite a wide margin. These pushes ended up covering so many aspects that could be interesting to a general and non-Seihou-adjacent audience, so I think we need a table of contents for this one:

1. Evaluating Zig
2. Visual Studio doesn't implement concepts correctly?
3. Reusable building blocks for Tup
4. Compiling SDL 2
5. The new frame rate limiter
6. Audio via SDL or SDL_mixer? (Nope, neither)
7. miniaudio
8. Resampling defective sound effects (including FLAC not always being lossless)
9. Joypad input with SDL
10. Restoring the original screenshot feature
11. Integer math in hand-written ASM

Before we can start migrating to SDL, we of course have to integrate it into the build somehow. On Linux, we'd ideally like to just dynamically link to a distribution's SDL development package, but since there's no such thing on Windows, we'd like to compile SDL from source there. This allows us to reuse our debug and release flags and ensures that we get debug information, without needing to clone build scripts for every C++ library ever in the process or something.
So let's get my Tup build scripts ready for compiling vendored libraries… or maybe not? Recently, I've kept hearing about a hot new technology that not only provides the rare kind of jank-free cross-compiling build system for C/C++ code, but innovates by even bundling a C++ compiler into a single 279 MiB package with no further dependencies. Realistically replacing both Visual Studio and Tup with a single tool that could target every OS is quite a selling point. The upcoming Linux port makes for the perfect occasion to evaluate Zig, and to find out whether Tup is still my favorite build system in 2023.

Even apart from its main selling point, there's a lot to like about Zig:

• First and foremost: It's a modern systems programming language with seamless C interop that we could gradually migrate parts of the codebase to. The feature set of the core language seems to hit the sweet spot between C and C++, although I'd have to use it more to be completely sure.
• A native, optimized Hello World binary with no string formatting is 4 KiB when compiled for Windows, and 6.4 KiB when cross-compiled from Windows to Linux. It's so refreshing to see a systems language in 2023 that doesn't bundle a bulky runtime for trivial programs and then defends it with the old excuse of "but all this runtime code will come in handy the larger your program gets". With a first impression like this, Zig managed to realize the "don't pay for what you don't use" mantra that C++ typically claims for itself, but only pulls off maybe half of the time.
• You can directly target specific CPU models, down to even the oldest 386 CPUs?! How amazing is that?! In contrast, Visual Studio only describes its /arch:IA32 compatibility option in very vague terms, leaving it up to you to figure out that "legacy 32-bit x86 instruction set without any vector operations" actually means "i586/P5 Pentium, because the startup code still includes an unconditional CPUID instruction". In any case, it means that Zig could also cover the i586 build.
  • Even better, changing Zig's CPU model setting recompiles both its bundled C/C++ standard library and Zig's own compiler-rt polyfill library for that architecture. This ensures that no unsupported instructions ever show up in the binary, and also removes the need for any CPUID checks. This is so much better than the Visual Studio model of linking against a fixed pre-compiled standard library because you don't have to trust that all these newer instructions wouldn't actually be executed on older CPUs that don't have them.
• I love the auto-formatter. Want to lay out your struct literal into multiple lines? Just add a trailing comma to the end of the last element. It's very snappy, and a joy to use.
• Like every modern programming language, Zig comes with a test framework built into the language. While it's not all too important for my grand plan of having one big test that runs a bunch of replays and compares their game states against the original binary, small tests could still be useful for protecting gameplay code against accidental changes. It would be great if I didn't have to evaluate and choose among the many testing frameworks for C++ and could just use a language standard.
• Package management is still in its infancy, but it's looking pretty good so far, resembling Go's decentralized approach of just pointing to a URL but with specific version selection from the get-go.

However, as a version number of 0.11.0 might already suggest, the whole experience was then bogged down by quite a lot of issues:

• While Zig's C/C++ compilation feature is very well architected to reuse the C/C++ standard libraries of GCC and MinGW and thus automatically keeps up with changes to the C++ standard library, it's ultimately still just a Clang frontend. If you've been working with a Visual Studio-exclusive codebase – which, as we're going to see below, can easily happen even if you compile in C++23 mode – you'd now have to migrate to Clang and Zig in a single step. Obviously, this can't ever be fixed without Microsoft open-sourcing their C++ compiler. And even then, supporting a separate set of command-line flags might not be worth it.
• The standard library is very poorly documented, especially in the build-related parts that are meant to attract the C++ audience.
• Often, the only documentation is found in blog posts from a few years ago, with example code written against old Zig versions that doesn't compile on the newest version anymore. It's all very far from stable.
• However, Zig's project generation sub-commands (zig init-exe and friends) do emit well-documented boilerplate code? It does make sense for that code to double as a comprehensive example, but Zig advertises itself as so simple that I didn't even think about bootstrapping my project with a CLI tool at first – unlike, say, Rust, where a project always starts with filling out a small form in Cargo.toml.
• There's no progress output for C/C++ compilation? Like, at all?
• This hurts especially because compilation times are significantly longer than they were with Visual Studio. By default, the current Tupfile builds Shuusou Gyoku in both debug and release configurations simultaneously. If I fully rebuild everything from a clean cache, Visual Studio finishes such a build in roughly the same amount of time that Zig takes to compile just a debug build.
• The --global-cache-dir option is only supported by specific subcommands of the zig CLI rather than being a top-level setting, and throws an error if used for any other subcommand. Not having a system-wide way to change it and being forced into writing a wrapper script for that is fine, but it would be nice if said wrapper script didn't have to also parse and switch over the subcommand just to figure out whether it is allowed to append the setting.
• compiler-rt still needs a bit of dead code elimination work. As soon as your program needs a single polyfilled function, you get all of them, because they get referenced in some exception-related table even if nothing uses them? Changing the link_eh_frame_hdr option had no effect.
• And that was not the only std.Build.Step.Compile option that did nothing. Worse, if I just tweaked the options and changed nothing about the code itself, Zig simply copied a previously built executable out of its build cache into the output directory, as revealed by the timestamp on the .EXE. While I am willing to believe that Zig correctly detects that all these settings would just produce the same binary, I do not like how this behavior inspires distrust and uncertainty in Zig's build process as a whole. After all, we still live in a world where clearing the build cache is way too often the solution for weird problems in software, especially when using CMake. And it makes sense why it would be: If you develop a complex system and then try solving the infamously hard problem of cache invalidation on top, the risk of getting cache invalidation wrong is, by definition, higher than if that was the only thing your system did. That's the reason why I like Tup so much: It solely focuses on getting cache invalidation right, and rather errs on the side of caution by maybe unnecessarily rebuilding certain files every once in a while because the compiler may have read from an environment variable that has changed in the meantime. But this is the one job I expect a build system to do, and Tup has been delivering for years and has become fundamentally more trustworthy as a result.
• Zig activates Clang's UBSan in debug builds by default, which executes a program-crashing UD2 instruction whenever the program is about to rely on undefined C++ behavior. In theory, that's a great help for spotting hidden portability issues, but it's not helpful at all if these crashes are seemingly caused by C++ standard library code?! Without any clear info about the actual cause, this just turned into yet another annoyance on top of all the others. Especially because I apparently kept searching for the wrong terms when I first encountered this issue, and only found out how to deactivate it after I already decided against Zig.
• Also, can we get /PDBALTPATH? Baking absolute paths from the filesystem of the developer's machine into released binaries is not only cringe in itself, but can also cause potential privacy or security accidents.

So for the time being, I still prefer Tup. But give it maybe two or three years, and I'm sure that Zig will eventually become the best tool for resurrecting legacy C++ codebases. That is, if the proposed divorce of the core Zig compiler from LLVM isn't an indication that the productive parts of the Zig community consider the C/C++ building features to be "good enough", and are about to de-emphasize them to focus more strongly on the actual Zig language. Gaining adoption for your new systems language by bundling it with a C/C++ build system is such a great and unique strategy, and it almost worked in my case. And who knows, maybe Zig will already be good enough by the time I get to port PC-98 Touhou to modern systems.

(If you came from the Zig wiki, you can stop reading here.)

A few remnants of the Zig experiment still remain in the final delivery. If that experiment worked out, I would have had to immediately change the execution encoding to UTF-8, and decompile a few ASM functions exclusive to the 8-bit rendering mode which we could have otherwise ignored. While Clang does support inline assembly with Intel syntax via -fms-extensions, it has trouble with ; comments and instructions like REP STOSD, and if I have to touch that code anyway… (The REP STOSD function translated into a single call to memcpy(), by the way.)

Another smaller issue was Visual Studio's lack of standard library header hygiene, where #including some of the high-level STL features also includes more foundational headers that Clang requires to be included separately, but I've already known about that. Instead, the biggest shocker was that Visual Studio accepts invalid syntax for a language feature as recent as C++20 concepts:

// Defines the interface of a text rendering session class. To simplify this
// example, it only has a single `Print(const char* str)` method.
template <class T> concept Session = requires(T t, const char* str) {
	t.Print(str);
};

// Once the rendering backend has started a new session, it passes the session
// object as a parameter to a user-defined function, which can then freely call
// any of the functions defined in the `Session` concept to render some text.
template <class F, class S> concept UserFunctionForSession = (
	Session<S> && requires(F f, S& s) {
		{ f(s) };
	}
);

// The rendering backend defines a `Prerender()` method that takes the
// aforementioned user-defined function object. Unfortunately, C++ concepts
// don't work like this: The standard doesn't allow `auto` in the parameter
// list of a `requires` expression because it defines another implicit
// template parameter. Nevertheless, Visual Studio compiles this code without
// errors.
template <class T, class S> concept BackendAttempt = requires(
	T t, UserFunctionForSession<S> auto func
) {
	t.Prerender(func);
};

// A syntactically correct definition would use a different constraint term for
// the type of the user-defined function. But this effectively makes the
// resulting concept unusable for actual validation because you are forced to
// specify a type for `F`.
template <class T, class S, class F> concept SyntacticallyFixedBackend = (
	UserFunctionForSession<F, S> && requires(T t, F func) {
		t.Prerender(func);
	}
);

// The solution: Defining a dummy structure that behaves like a lambda as an
// "archetype" for the user-defined function.
struct UserFunctionArchetype {
	void operator ()(Session auto& s) {
	}
};

// Now, the session type disappears from the template parameter list, which
// even allows the concrete session type to be private.
template <class T> concept CorrectBackend = requires(
	T t, UserFunctionArchetype func
) {
	t.Prerender(func);
};

What's this, Visual Studio's infamous delayed template parsing applied to concepts, because they're templates as well? Didn't they get rid of that 6 years ago? You would think that we've moved beyond the age where compilers differed in their interpretation of the core language, and that opting into a current C++ standard turns off any remaining antiquated behaviors…

So let's actually get my Tup build scripts ready for compiling vendored libraries, because the 📝 previous 70 lines of Lua definitely weren't. For this use case, we'd like to have some notion of distinct build targets that can have a unique set of compilation and linking flags. We'd also like to always build them in debug and release versions even if you only intend to build your actual program in one of those versions – with the previous system of specifying a single version for all code, Tup would delete the other one, which forces a time-consuming and ultimately needless rebuild once you switch to the other version.

The solution I came up with treats the set of compiler command-line options like a tree whose branches can concatenate new options and/or filter the versions that are built on this branch. In total, this is my 4^th attempt at writing a compiler abstraction layer for Tup. Since we're effectively forced to write such layers in Lua, it will always be a bit janky, but I think I've finally arrived at a solid underlying design that might also be interesting for others. Hence, I've split off the result into its own separate repository and added high-level documentation and a documented example. And yes, that's a Code Nutrition label! I've wanted to add one of these ever since I first heard about the idea, since it communicates nicely how seriously such an open-source project should be taken. Which, in this case, is actually not all too seriously, especially since development of the core Tup project has all but stagnated. If Zig does indeed get better and better at being a Clang frontend/build system, the only niches left for Tup will be Visual Studio-exclusive projects, or retrocoding with nonstandard toolchains (i.e., ReC98). Quite ironic, given Tup's Unix heritage…
Oh, and maybe general Makefile-like tasks where you just want to run specific programs. Maybe once the general hype swings back around and people start demanding proper graph-based dependency tracking instead of just a command runner…

Alright, alternatives evaluated, build system ready, time to include SDL! Once again, I went for Git submodules, but this time they're held together by a batch file that ensures that the intended versions are checked out before starting Tup. Git submodules have a bad rap mainly because of their usability issues, and such a script should hopefully work around them? Let's see how this plays out. If it ends up causing issues after all, I'll just switch to a Zig-like model of downloading and unzipping a source archive. Since Windows comes with curl and tar these days, this can even work without any further dependencies.

Compiling SDL from a non-standard build system requires a bit of globbing to include all the code that is being referenced, as well as a few linker settings, but it's ultimately not much of a big deal. I'm quite happy that it was possible at all without pre-configuring a build, but hey, that's what maintaining a Visual Studio project file does to a project.
By building SDL with the stock Windows configuration, we then end up with exactly what the SDL developers want us to use… which is a DLL. You can statically link SDL, but they really don't want you to do that. So strongly, in fact, that they not merely argue how well the textbook advantages of dynamic linking have worked for them and gamers as a whole, but implemented a whole dynamic API system that enforces overridable dynamic function loading even in static builds. Nudging developers to their preferred solution by removing most advantages from static linking by default… that's certainly a strategy. It definitely fits with SDL's grassroots marketing, which is very good at painting SDL as the industry standard and the only reliable way to keep your game running on all originally supported operating systems. Well, at least until SDL 3 is so stable that SDL 2 gets deprecated and won't receive any code for new backends…

However, dynamic linking does make sense if you consider what SDL is. Offering all those multiple rendering, input, and sound backends is what sets it apart from its more hip competition, and you want to have all of them available at any time so that SDL can dynamically select them based on what works best on a system. As a result, everything in SDL is being referenced somewhere, so there's no dead code for the linker to eliminate. Linking SDL statically with link-time code generation just prolongs your link time for no benefit, even without the dynamic API thwarting any chance of SDL calls getting inlined.
There's one thing I still don't like about all this, though. The dynamic API's table references force you to include all of SDL's subsystems in the DLL even if your game doesn't need some of them. But it does fit with their intention of having SDL2.dll be swappable: If an older game stopped working because of an outdated SDL2.dll, it should be possible for anyone to get that game working again by replacing that DLL with any newer version that was bundled with any random newer game. And since that would fail if the newer SDL2.dll was size-optimized to not include some of the subsystems that the older game required, they simply removed (or de-prioritized) the possibility altogether. Maybe that was their train of thought? You can always just use the official Windows DLL, whose whole point is to include everything, after all. 🤷

So, what do we get in these 1.5 MiB? There are:

• renderer backends for Direct3D 9/11/12, regular OpenGL, OpenGL ES 2.0, Vulkan, and a software renderer,
• input backends for DirectInput, XInput, Raw Input, and all the official game console controllers that can be connected via USB,
• and audio backends for WinMM, DirectSound, WASAPI, and direct-to-disk recording.

Unfortunately, SDL 2 also statically references some newer Windows API functions and therefore doesn't run on Windows 98. Since this build of Shuusou Gyoku doesn't introduce any new features to the input or sound interfaces, we can still use pbg's original DirectSound and DirectInput code for the i586 build to keep it working with the rest of the platform-independent game logic code, but it will start to lag behind in features as soon as we add support for SC-88Pro BGM or more sophisticated input remapping. If we do want to keep this build at the same feature level as the SDL one, we now have a choice: Do we write new DirectInput and DirectSound code and get it done quickly but only for Shuusou Gyoku, or do we port SDL 2 to Windows 98 and benefit all other SDL 2 games as well? I leave that for my backers to decide.

Immediately after writing the first bits of actual SDL code to initialize the library and create the game window, you notice that SDL makes it very simple to gradually migrate a game. After creating the game window, you can call SDL_GetWindowWMInfo() to retrieve HWND and HINSTANCE handles that allow you to continue using your original DirectDraw, DirectSound, and DirectInput code and focus on porting one subsystem at a time.
Sadly, D3DWindower can no longer turn SDL's fullscreen mode into a windowed one, but DxWnd still works, albeit behaving a bit janky and insisting on minimizing the game whenever its window loses focus. But in exchange, the game window can surprisingly be moved now! Turns out that the originally fixed window position had nothing to do with the way the game created its DirectDraw context, and everything to do with pbg blocking the Win32 "syscommand" that allows a window to be moved. By deleting a system menu… seriously?! Now I'm dying to hear the Raymond Chen explanation for how this behavior dates back to an unfortunate decision during the Win16 days or something.
As implied by that commit, I immediately backported window movability to the i586 build.

However, the most important part of Shuusou Gyoku's main loop is its frame rate limiter, whose Win32 version leaves a bit of room for improvement. Outside of the uncapped [おまけ] DrawMode, the original main loop continuously checks whether at least 16 milliseconds have elapsed since the last simulated (but not necessarily rendered) frame. And by that I mean continuously, and deliberately without using any of the Windows system facilities to sleep the process in the meantime, as evidenced by a commented-out Sleep(1) call. This has two important effects on the game:

• The 60Fps DrawMode actually corresponds to a frame rate of (1000 / 16) = 62.5 FPS, not 60. Since the game didn't account for the missing ²/₃ ms to bring the limit down to exactly 60 FPS, 62.5 FPS is Shuusou Gyoku's actual official frame rate in a non-VSynced setting, which we should also maintain in the SDL port.
• Not sleeping the process turns Shuusou Gyoku's frame rate limitation into a busy-waiting loop, which always uses 100% of a single CPU core just to wait for the next frame.

Unsurprisingly, SDL features a delay function that properly sleeps the process for a given number of milliseconds. But just specifying 16 here is not exactly what we want:

1. Sure, modern computers are fast, but a frame won't ever take an infinitely fast 0 milliseconds to render. So we still need to take the current frame time into account.
2. SDL_Delay()'s documentation says that the wake-up could be further delayed due to OS scheduling.

To address both of these issues, I went with a base delay time of 15 ms minus the time spent on the current frame, followed by busy-waiting for the last millisecond to make sure that the next frame starts on the exact frame boundary. And lo and behold: Even though this still technically wastes up to 1 ms of CPU time, it still dropped CPU usage into the 0%-2% range during gameplay on my Intel Core i5-8400T CPU, which is over 5 years old at this point. Your laptop battery will appreciate this new build quite a bit.

Time to look at audio then, because it sure looks less complicated than input, doesn't it? Loading sounds from .WAV file buffers, playing a fixed number of instances of every sound at a given position within the stereo field and with optional looping… and that's everything already. The DirectSound implementation is so straightforward that the most complex part of its code is the .WAV file parser.
Well, the big problem with audio is actually finding a cross-platform backend that implements these features in a way that seamlessly works with Shuusou Gyoku's original files. DirectSound really is the perfect sound API for this game:

• It doesn't require the game code to specify any output sample format. Just load the individual sound effects in their original format, and playback just works and sounds correctly.
• Its final sound stream seems to have a latency of 10 ms, which is perfectly fine for a game running at 62.5 FPS. Even 15 ms would be OK.
• Sound effect looping? Specified by passing the DSBPLAY_LOOPING flag to IDirectSoundBuffer::Play().
• Stereo panning balancing? One method call.
• Playing the same sound multiple times simultaneously from a single memory buffer? One method call. (It can fail though, requiring you to copy the data after all.)
• Pausing all sounds while the game window is not focused? That's the default behavior, but it can be equally easily disabled with just a single per-buffer flag.
• Future streaming of waveform BGM? No problem either. Windows Touhou has always done that, and here's some code I wrote 12½ years ago that would even work without DirectSound 8's notification feature.
• No further binary bloat, because it's part of the operating system.

The last point can't really be an argument against anything, but we'd still be left with 7 other boxes that a cross-platform alternative would have to tick. We already picked SDL for our portability needs, so how does its audio subsystem stack up? Unfortunately, not great:

• It's fully DIY. All you get is a single output buffer, and you have to do all the mixing and effect processing yourself. In other words, it's the masochistic approach to cross-platform audio.
• There are helper functions for resampling and mixing, but the documentation of the latter is full of FUD. With a disclaimer that so vehemently discourages the use of this function, what are you supposed to do if you're newly integrating SDL audio into a game? Hunt for a separate sound mixing library, even though your only quality goal is parity with stone-age DirectSound? 🙄
• It forces the game to explicitly define the PCM sampling rate, bit depth, and channel count of the output buffer. You can't just pass a nullptr to SDL_OpenAudioDevice(), and if you pass a zeroed SDL_AudioSpec structure, SDL just defaults to an unacceptable 22,050 Hz sampling rate, regardless of what the audio device would actually prefer. It took until last year for them to notice that people would at least like to query the native format. But of course, this approach requires the backend to actually provide this information – and since we've seen above that DirectSound doesn't care, the DirectSound version of this function has to actually use the more modern WASAPI, and remains unimplemented if that API is not available.
  Standardizing the game on a single sampling rate, bit depth, and channel count might be a decent choice for games that consistently use a single format for all its sounds anyway. In that case, you get to do all mixing and processing in that format, and the audio backend will at most do one final conversion into the playback device's native format. But in Shuusou Gyoku, most sound effects use 22,050 Hz, the boss explosion sound effect uses 11,025 Hz, and the future SC-88Pro BGM will obviously use 44,100 Hz. In such a scenario, you would have to pick the highest sampling rate among all sound sources, and resample any lower-quality sounds to that rate. But if the audio device uses a different sampling rate, those lower-quality sounds would get resampled a second time.
  I know that this will be fixed in SDL 3, but that version is still under heavy development.
• Positives? Uh… the callback-based nature means that BGM streaming is rather trivial, and would even be comparatively less complicated than with DirectSound. Having a mutex to prevent writes to your sound instance structures while they're being read by the audio thread is nice too.

OK, sure, but you're not supposed to use it for anything more than a single stream of audio. SDL_mixer exists precisely to cover such non-trivial use cases, and it even supports sound effect looping and panning with just a single function call! But as far as the rest of the library is concerned, it manages to be an even bigger disappointment than raw SDL audio:

• As it sits on top of SDL's audio subsystem, it still can't just use your audio device's native sample format.
• Even worse, it insists on initializing the audio device itself, and thus always needs to duplicate whatever you would do for raw SDL.
• It only offers a very opinionated system for streaming – and of course, its opinion is wrong. 😛 The fact that it only supports a single streaming audio track wouldn't matter all too much if you could switch to another track at sample precision. But since you can't, you're forced to implement looping BGM using a single file…
• …which brings us to the unfortunate issue of loop point definitions. And, perhaps most importantly, the complete lack of any way to set them through the API?! It doesn't take long until you come up with a theory for why the API only offers a function to retrieve loop points: The "music" abstraction is so format-agnostic that it even supports MIDI and tracker formats where a typical loop point in PCM samples doesn't make sense. Both of these formats already have in-band ways of specifying loop points in their respective time units. They might not be standardized, but it's still much better than usual single-file solutions for PCM streams where the loop point has to be stored in an out-of-band way – such as in a metadata tag or an entirely separate file.
  • Speaking of MIDI, why is it so common among these APIs to not have any way of specifying the MIDI device? The fact that Windows Vista removed the Control Panel option for specifying the system-wide default MIDI output device is no excuse for your API lacking the option as well. In fact, your MIDI API now needs such a setting more than it was needed in the Windows XP and 9x days.
  • Actually, wait, the API does have a function that is exclusive to tracker formats. Which means that they aren't actually insisting on a clean, consistent, and minimal API here… 🤔
• Funnily enough, they did once receive a patch for a function to set loop points which was never upstreamed… and this patch came from the main developer behind PyTouhou, who needed that feature for obvious reasons. The world sure is a small place.
• As a result, they turned loop points into a property that each individual format may or may not have. Want to loop MP3 files at sample precision? Tough luck, time to reconvert to another lossy format. 🙄 This is the exact jank I decided against when I implemented BGM modding for thcrap back in 2018, where I concluded that separate intro and loop files are the way to go.
  But OK, we only plan to use FLAC and Ogg Vorbis for the SC-88Pro BGM, for which SDL_mixer does support loop points in the form of Vorbis comments, and hey, we can even pass them at sample accuracy. Sure, it's wrong and everything, but nothing I couldn't work with…
• However, the final straw that makes SDL_mixer unsuitable for Shuusou Gyoku is its core sound mixing paradigm of distributing all sound effects onto a fixed number of channels, set to 8 by default. Which raises the quite ridiculous question of how many we would actually need to cover the maximum amount of sounds that can simultaneously be played back in any game situation. The theoretic maximum would be 41, which is the combined sum of individual sound buffer instances of all 20 original sound effects. The practical limit would surely be a lot smaller, but we could only find out that one through experiments, which honestly is quite a silly proposition.
  • It makes you wonder why they went with this paradigm in the first place. And sure enough, they actually use the aforementioned SDL core function for mixing audio. Yes, the same function whose current documentation advises against using it for this exact use case. 🙄 What's the argument here? "Sure, 8 is significantly more than 2, but any mixing artifacts that will occur for the next 6 sounds are not worrying about, but they get really bad after the 8^th sound, so we're just going to protect you from that"?

There is a fork that does add support for an arbitrary number of music streams, but the rest of its features leave me questioning the priorities and focus of this project. Because surely, when I think about missing features in an audio backend, I immediately think about support for a vast array of chiptune file formats… 🤪
And wait, what, they merged this piece of bloat back into the official SDL_mixer library?! Thanks for opening up a vast attack surface for potential security vulnerabilities in code that would never run for the majority of users, just to cover some niche formats that nobody would seriously expect in a general audio library. And that's coming from someone who loves listening to that stuff!
At this rate, I'm expecting SDL_mixer to gain a mail client by the end of the decade. Hmm, what's the closest audio thing to a mail client… oh, right, WebRTC! Yeah, let's just casually drop a giant part of the Chromium codebase into SDL_mixer, what could possibly go wrong?

This dire situation made me wonder if SDL was the wrong choice for Shuusou Gyoku to begin with. Looking at other low-level cross-platform game libraries, you'll quickly notice that all of them come with mostly equally capable 2D renderers these days, and mainly differentiate themselves in minute API details that you'd only notice upon a really close look.
raylib is another one of those libraries and has been getting exceptionally popular in recent years, to the point of even having more than twice as many GitHub stars as SDL. By restricting itself to OpenGL, it can even offer an abstraction for shaders, which we'd really like for the 西方Ｐｒｏｊｅｃｔ lens ball effect.
In the case of raylib's audio system, the lack of sound effect looping is the minute API detail that would make it annoying to use for Shuusou Gyoku. But it might be worth a look at how raylib implements all this if it doesn't use SDL… which turned out to be the best look I've taken in a long time, because raylib builds on top of miniaudio which is exactly the kind of audio library I was hoping to find. Let's check the list from above:

• 🟢 miniaudio's high-level API initialization defaults to the native sample format of the playback device. Its internal processing uses 32-bit floating-point samples and only converts back to the native bit depth as necessary when writing the final stream into the backend's audio buffer. WASAPI, for example, never needs any further conversion because it operates with 32-bit floats as well.
• 🟢 The final audio stream uses the same 10 ms update period (and thus, sound effect latency) that I was getting with DirectSound.
• 🟢 Stereo panning balancing? ma_sound_set_pan(), although it does require a conversion from Shuusou Gyoku's dB units into a linear attenuation factor.
• 🟢 Sound effect looping? ma_sound_set_looping().
• 🟢 Playing the same sound multiple times simultaneously from a single memory buffer? Perfectly possible, but requires a bit of digging in the header to find the best solution. More on that below.
• 🟢 Future streaming of waveform BGM? Just call ma_sound_init_from_file() with the MA_SOUND_FLAG_STREAM flag.
  • 👍 It also comes with a FLAC decoder in the core library and an Ogg Vorbis one as part of the repo, …
  • 🤩 … and even supports gapless switching between the intro and loop files via a single declarative call to ma_data_source_set_next()!
    (Oh, and it also has ma_data_set_loop_point_in_pcm_frames() for anyone who still believes in obviously and objectively inferior out-of-band loop points.)
• 🟢 Pausing all sounds while the game window is not focused? It's not automatic, but adding new functions to the sound interface and calling ma_engine_stop() and ma_engine_start() does the trick, and most importantly doesn't cause any samples to be lost in the process.
• 🟡 Sound control is implemented in a lock-free way, allowing your main game thread to call these at any time without causing glitches on the audio thread. While that looks nice and optimal on the surface, you now have to either believe in the soundness (ha) of the implementation, or verify that atomic structure fields actually are enough to not cause any race conditions (which I did for the calls that Shuusou Gyoku uses, and I didn't find any). "It's all lock-free, don't worry about it" might be easier, but I consider SDL's approach of just providing a mutex to prevent the output callback from running while you mutate the sound state to actually be simpler conceptually.
• 🟡 miniaudio adds 247 KB to the binary in its minimum configuration, a bit more than expected. Some of that is bloat from effect code that we never use, but it does include backends for all three Windows audio subsystems (WASAPI, DirectSound, and WinMM).
• ✅ But perhaps most importantly: It natively supports all modern operating systems that one could seriously want to port this game to, and could be easily ported to any other backend, including SDL.

Oh, and it's written by the same developer who also wrote the best FLAC library back in 2018. And that's despite them being single-file C libraries, which I consider to be massively overrated…

The drawback? Similar to Zig, it's only on version 0.11.18, and also focuses on good high-level documentation at the expense of an API reference. Unlike Zig though, the three issues I ran into turned out to be actual and fixable bugs: Two minor ones related to looping of streamed sounds shorter than 2 seconds which won't ever actually affect us before we get into BGM modding, and a critical one that added high-frequency corruption to any mono sound effect during its expansion to stereo. The latter took days to track down – with symptoms like these, you'd immediately suspect the bug to lie in the resampler or its low-pass filter, both of which are so much more of a fickle and configurable part of the conversion chain here. Compared to that, stereo expansion is so conceptually simple that you wouldn't imagine anyone getting it wrong.
While the latter PR has been merged, the fix is still only part of the dev branch and hasn't been properly released yet. Fortunately, raylib is not affected by this bug: It does currently ship version 0.11.16 of miniaudio, but its usage of the library predates miniaudio's high-level API and it therefore uses a different, non-SSE-optimized code path for its format conversions.

The only slightly tricky part of implementing a miniaudio backend for Shuusou Gyoku lies in setting up multiple simultaneously playing instances for each individual sound. The documentation and answers on the issue tracker heavily push you toward miniaudio's resource manager and its file abstractions to handle this use case. We surely could turn Shuusou Gyoku's numeric sound effect IDs into fake file names, but it doesn't really fit the existing architecture where the sound interface just receives in-memory .WAV file buffers loaded from the SOUND.DAT packfile.
In that case, this seems to be the best way:

• Call ma_decode_memory() to decode from any of the supported audio formats to a buffer of raw PCM samples.
  At this point, you can choose between
  1. decoding into the original format the sound effect is stored in, which would require it to be converted to the playback format every time it's played, or
  2. decoding into 32-bit floats (the native bit depth of the miniaudio engine) and the native sampling rate of the playback device, which avoids any further resampling and floating-point conversion, but takes up more memory.
  Nowadays, it's not clear at all which of the two approaches is faster. Does it actually matter if we save the audio thread from doing all those floating-point operations on every sample? Or is that no longer true these days because the audio thread is probably running on a different CPU core, the rest of the game largely doesn't touch the floating-point parts of your CPU anyway, and you'd rather want to keep sound effects small so that they can better fit into the CPU cache? That would be an interesting question to benchmark, but just like the similar text rendering question from the last blog posts, it doesn't matter for this tiny 2000s retro game. 😌
  I went with 2) mainly because it simplified all the debugging I was doing. At a sampling rate of 48,000 Hz, this increases the memory usage for all sound effects from 379 KiB to 3.67 MiB. At least I'm not channel-expanding all sound effects as well here… We've seen earlier that mono➜stereo expansion is SSE-optimized, so it's very hard to justify a further doubling of the memory usage here.
• Then, for each instance of the sound, call
  • ma_audio_buffer_ref_init() to create a reference buffer with its own playback cursor, and
  • ma_sound_init_from_data_source() to create a new high-level sound node that will play back the reference buffer.

As a side effect of hunting that one critical bug in miniaudio, I've now learned a fair bit about audio resampling in general. You'll probably need some knowledge about basic digital signal behavior to follow this section, and that video is still probably the best introduction to the topic.

So, how could this ever be an issue? The only time I ever consciously thought about resampling used to be in the context of the Opus codec and its enforced sampling rate of 48,000 Hz, and how Opus advocates claim that resampling is a solved problem and nothing to worry about, especially in the context of a lossy codec. Still, I didn't add Opus to thcrap's BGM modding feature entirely because the mere thought of having to downsample to 44,100 Hz in the decoder was off-putting enough. But even if my worries were unfounded in that specific case: Recording the Stereo Mix of Shuusou Gyoku's now two audio backends revealed that apparently not every audio processing chain features an Opus-quality resampler…

If we take a look at the material that resamplers actually have to work with here, it quickly becomes obvious why their results are so varied. As mentioned above, Shuusou Gyoku's sound effects use rather low sampling rates that are pretty far away from the 48,000 Hz your audio device is most definitely outputting. Therefore, any potential imaging noise across the extended high-frequency range – i.e., from the original Nyquist frequencies of 11,025 Hz/5,512.5 Hz up to the new limit of 24,000 Hz – is still within the audible range of most humans and can clearly color the resulting sound.
But it gets worse if the audio data you put into the resampler is objectively defective to begin with, which is exactly the problem we're facing with over half of Shuusou Gyoku's sound effects. Encoding them all as 8-bit PCM is definitely excusable because it was the turn of the millennium and the resulting noise floor is masked by the BGM anyway, but the blatant clipping and DC offsets definitely aren't:

KEBARI	TAME	LASER	LASER2	BOMB	SELECT	HIT	CANCEL	WARNING	SBLASER	BUZZ	MISSILE	JOINT	DEAD	SBBOMB	BOSSBOMB	ENEMYSHOT	HLASER	TAMEFAST	WARP

Wait a moment, true peaks? Where do those come from? And, equally importantly, how can we even observe, measure, and store anything above the maximum amplitude of a digital signal?

The answer to the first question can be directly derived from the Xiph.org video I linked above: Digital signals are lollipop graphs, not stairsteps as commonly depicted in audio editing software. Converting them back to an analog signal involves constructing a continuous curve that passes through each sample point, and whose frequency components stay below the Nyquist frequency. And if the amplitude of that reconstructed wave changes too strongly and too rapidly, the resulting curve can easily overshoot the maximum digital amplitude of 0 dBFS even if none of the defined samples are above that limit.

But I can assure you that I did not create the waveform images above by recording the analog output of some speakers or headphones and then matching the levels to the original files, so how did I end up with that image? It's not an Audacity feature either because the development team argues that there is no "true waveform" to be visualized as every DAC behaves differently. While this is correct in theory, we'd be happy just to get a rough approximation here.
ffmpeg's ebur128 filter has a parameter to measure the true peak of a waveform and fairly understandable source code, and once I looked at it, all the pieces suddenly started to make sense: For our purpose of only looking at digital signals, 💡 resampling to a floating-point signal with an infinite sampling rate is equivalent to a DAC. And that's exactly what this filter does: It picks 192,000 Hz and 64-bit float as a format that's close enough to the ideal of "analog infinity" for all practical purposes that involve digital audio, and then simply converts each incoming 100 ms of audio and keeps the sample with the largest floating-point value.

So let's store the resampled output as a FLAC file and load it into Audacity to visualize the clipped peaks… only to find all of them replaced with the typical kind of clipping distortion? 😕 Turns out that I've stumbled over the one case where the FLAC format isn't lossless and there's actually no alternative to .WAV: FLAC just doesn't support floating-point samples and simply truncates them to discrete integers during encoding. When we measured inter-sample peaks above, we weren't only resampling to a floating-point format to avoid any quantization to discrete integer values, but also to make it possible to store amplitudes beyond the 0 dBFS point of ±1.0 in the first place. Once we lose that ability, these amplitudes are clipped to the maximum value of the integer bit depth, and baked into the waveform with no way to get rid of them again. After all, the resampled file now uses a higher sampling rate, and the clipping distortion is now a defined part of what the sound is.
Finally, storing a digital signal with inter-sample peaks in a floating-point format also makes it possible for you to reduce the volume, which moves these peaks back into the regular, unclipped amplitude range. This is especially relevant for Shuusou Gyoku as you'll probably never listen to sound effects at full volume.

Now that we understand what's going on there, we can finally compare the output of various resamplers and pick a suitable one to use with miniaudio. And immediately, we see how they fall into two categories:

• High-quality resamplers are the ones I described earlier: They cleanly recreate the signal at a higher sampling rate from its raw frequency representation and thus add no high-frequency noise, but can lead to inter-sample peaks above 0 dBFS.
• Linear resamplers use much simpler math to merely interpolate between neighboring samples. Since the newly interpolated samples can only ever stay within 0 dBFS, this approach fully avoids inter-sample clipping, but at the expense of adding high-frequency imaging noise that has to then be removed using a low-pass filter.

miniaudio only comes with a linear resampler – but so does DirectSound as it turns out, so we can get actually pretty close to how the game sounded originally:

And yes, these are indeed the first videos on this blog to have sound! I spent another push on preparing the 📝 video conversion pipeline for audio support, and on adding the highly important volume control to the player. Web video codecs only support lossy audio, so the sound in these videos will not exactly match the spectrum image, but the lossless source files do contain the original audio as uncompressed PCM streams.

Compared to that whole mess of signals and noise, keyboard and joypad input is indeed much simpler. Thanks to SDL, it's almost trivial, and only slightly complicated because SDL offers two subsystems with seemingly identical APIs:

• SDL_Joystick simply numbers all axes and buttons of a joypad and doesn't assign any meaning to these numbers, but works with every joypad ever.
• SDL_GameController provides a consistent interface for the typical kind of modern gamepad with two analog sticks, a D-pad, and at least 4 face and 2 shoulder buttons. This API is implemented by simply combining SDL_Joystick with a long list of mappings for specific controllers, and therefore doesn't work with joypads that don't match this standard.

To match Shuusou Gyoku's original WinMM backend, we'd ideally want to keep the best aspects from both APIs but without being restricted to SDL_GameController's idea of a controller. The Joy Pad menu just identifies each button with a numeric ID, so SDL_Joystick would be a natural fit. But what do we do about directional controls if SDL_Joystick doesn't tell us which joypad axes correspond to the X and Y directions, and we don't have the SDL-recommended configuration UI yet? Doing that right would also mean supporting POV hats and D-pads, after all… Luckily, all joypads we've tested map their main X axis to ID 0 and their main Y axis to ID 1, so this seems like a reasonable default guess.

Fortunately, there is a solution for our exact issue. We can still try to open a joypad via SDL_GameController, and if that succeeds, we can use a function to retrieve the SDL_Joystick ID for the main X and Y axis, close the SDL_GameController instance, and keep using SDL_Joystick for the rest of the game.
And with that, the SDL build no longer needs DirectInput 7, certain antivirus scanners will no longer complain about its low-level keyboard hook, and I turned the original game's single-joypad hot-plugging into multi-joypad hot-plugging with barely any code. 🎮

The necessary consolidation of the game's original input handling uncovered several minor bugs around the High Score and Game Over screen that I sufficiently described in the release notes of the new build. But it also revealed an interesting detail about the Joy Pad screen: Did you know that Shuusou Gyoku lets you unbind all these actions by pressing more than one joypad button at the same time? The original game indicated unbound actions with a [Button 0] label, which is pretty confusing if you have ever programmed anything because you now no longer know whether the game starts numbering buttons at 0 or 1. This is now communicated much more clearly.

With that, we're finally feature-complete as far as this delivery is concerned! Let's send a build over to the backers as a quick sanity check… a~nd they quickly found a bug when running on Linux and Wine. When holding a button, the game randomly stops registering directional inputs for a short while on some joypads? Sounds very much like a Wine bug, especially if the same pad works without issues on Windows.
And indeed, on certain joypads, Wine maps the buttons to completely different and disconnected IDs, as if it simply invents new buttons or axes to fill the resulting gaps. Until we can differentiate joypad bindings per controller, it's therefore unlikely that you can use the same joypad mapping on both Windows and Linux/Wine without entering the Joy Pad menu and remapping the buttons every time you switch operating systems.

Still, by itself, this shouldn't cause any issues with my SDL event handling code… except, of course, if I forget a break; in a switch case. 🫠
This completely preventable implicit fallthrough has now caused a few hours of debugging on my end. I'd better crank up the warning level to keep this from ever happening again. Opting into this specific warning also revealed why we haven't been getting it so far: Visual Studio did gain a whole host of new warnings related to the C++ Core Guidelines a while ago, including the one I was looking for, but actually getting the compiler to throw these requires activating a separate static analysis mode together with a plugin, which significantly slows down build times. Therefore I only activate them for release builds, since these already take long enough.

But that wasn't the only step I took as a result of this blunder. In addition, I now offer free fixes for regressions in my mod releases if anyone else reports an issue before I find it myself. I've already been following this policy 📝 earlier this year when mu021 reported the unblitting bug in the initial release of the TH01 Anniversary Edition, and merely made it official now. If I was the one who broke a thing, I'll fix it for free.

Since all that input debugging already started a 5^th push, I might as well fill that one by restoring the original screenshot feature. After all, it's triggered by a key press (and is thus related to the input backend), reads the contents of the frame buffer (and is thus related to the graphics backend), and it honestly looks bad to have this disclaimer in the release notes just because we're one small feature away from 100% parity with pbg's original binary.
Coincidentally, I had already written code to save a DirectDraw surface to a .BMP file for all the debugging I did in the last delivery, so we were basically only missing filename generation. Except that Shuusou Gyoku's original choice of mapping screenshots to the PrintScreen key did not age all too well:

• As of Windows XP's 64-bit version, you can no longer use standard window messages to detect that this key is being pressed.
• And as of Windows 11, the OS takes full control of the key by binding it to the Snipping Tool by default, complete with a UI that politely steals focus when hitting that key.

As a result, both Arandui and I independently arrived at the idea of remapping screenshots to the P key, which is the same screenshot key used by every Windows Touhou game since TH08.

The rest of the feature remains unchanged from how it was in pbg's original build and will save every distinct frame rendered by the game (i.e., before flipping the two framebuffers) to a .BMP file as long as the P key is being held. At a 32-bit color depth, these screenshots take up 1.2 MB per frame, which will quickly add up – especially since you'll probably hold the P key for more than ¹/₆₀ of a second and therefore end up saving multiple frames in a row. We should probably compress them one day.

Since I already translated some of Shuusou Gyoku's ASM code to C++ during the Zig experiment, it made sense to finish the fifth push by covering the rest of those functions. The integer math functions are used all throughout the game logic, and are the main reason why this goal is important for a Linux port, or any port to a 64-bit architecture for that matter. If you've ever read a micro-optimization-related blog post, you'll know that hand-written ASM is a great recipe that often results in the finest jank, and the game's square root function definitely delivers in that regard, right out of the gate.
What slightly differentiates this algorithm from the typical definition of an integer square root is that it rounds up: In real numbers, √3 is ≈ 1.73, so isqrt(3) returns 2 instead of 1. However, if the result is always rounded down, you can determine whether you have to round up by simply squaring the calculated root and comparing it to the radicand. And even that is only necessary if the difference between the two doesn't naturally fall out of the algorithm – which is what also happens with Shuusou Gyoku's original ASM code, but pbg didn't realize this and squared the result regardless.

That's one suboptimal detail already. Let's call the original ASM function in a loop over the entire supported range of radicands from 0 to 2³¹ and produce a list of results that I can verify my C++ translation against… and watch as the function's linear time complexity with regard to the radicand causes the loop to run for over 15 hours on my system. 🐌 In a way, I've found the literal opposite of Q_rsqrt() here: Not fast, not inverse, no bit hacks, and surely without the awe-inspiring kind of WTF.
I really didn't want to run the same loop over a literal C++ translation of the same algorithm afterward. Calculating integer square roots is a common problem with lots of solutions, so let's see if we can go better than linear.

And indeed, Wikipedia also has a bitwise algorithm that runs in logarithmic time, uses only additions, subtractions, and bit shifts, and even ends up with an error term that we can use to round up the result as necessary, without a multiplication. And this algorithm delivers the exact same results over the exact same range in… 50 seconds. 🏎️ And that's with the I/O to print the first value that returns each of the 46,341 different square root results.

"But wait a moment!", I hear you say. "Why are you bothering with an integer square root algorithm to begin with? Shouldn't good old round(sqrt(x)) from <math.h> do the trick just fine? Our CPUs have had SSE for a long time, and this probably compiles into the single SQRTSD instruction. All that extra floating-point hardware might mean that this instruction could even run in parallel with non-SSE code!"
And yes, all of that is technically true. So I tested it, and my very synthetic and constructed micro-benchmark did indeed deliver the same results in… 48 seconds. That's not enough of a difference to justify breaking the spirit of treating the FPU as lava that permeates Shuusou Gyoku's code base. Besides, it's not used for that much to begin with:

• pre-calculating the 西方Ｐｒｏｊｅｃｔ lens ball effect
• the fade animation when entering and leaving stages
• rendering the circular part of stationary lasers
• pulling items to the player when bombing

After a quick C++ translation of the RNG function that spells out a 32-bit multiplication on a 32-bit CPU using 16-bit instructions, we reach the final pieces of ASM code for the 8-bit atan2() and trapezoid rendering. These could actually pass for well-written ASM code in how they express their 64-bit calculations: atan8() prepares its 64-bit dividend in the combined EDX and EAX registers in a way that isn't obvious at all from a cursory look at the code, and the trapezoid functions effectively use Q32.32 subpixels. C++ allows us to cleanly model all these calculations with 64-bit variables, but unfortunately compiles the divisions into a call to a comparatively much more bloated 64-bit/64-bit-division polyfill function. So yeah, we've actually found a well-optimized piece of inline assembly that even Visual Studio 2022's optimizer can't compete with. But then again, this is all about code generation details that are specific to 32-bit code, and it wouldn't be surprising if that part of the optimizer isn't getting much attention anymore. Whether that optimization was useful, on the other hand… Oh well, the new C++ version will be much more efficient in 64-bit builds.

And with that, there's no more ASM code left in Shuusou Gyoku's codebase, and the original DirectXUTYs directory is slowly getting emptier and emptier.

Phew! Was that everything for this delivery? I think that was everything. Here's the new build, which checks off 7 of the 15 remaining portability boxes:

Shuusou Gyoku P0256

Next up: Taking a well-earned break from Shuusou Gyoku and starting with the preparations for multilingual PC-98 Touhou translatability by looking at TH04's and TH05's in-game dialog system, and definitely writing a shorter blog post about all that…

📝 Posted:

2023-05-01 23:56 UTC

🚚 Summary of:

P0238 Website (TypeScript migration + Stripe integration, part 1/2: Basic support)
P0239 Website (Stripe integration, part 2/2: Subscription support) + TH01 Anniversary Edition (Minor bugfixes)

💰 Funded by:

Ember2528

🏷️ Tags:

th01

+

website

-

rec98

+

meta

+

anniversary-edition

+

debloating

+

master.lib

+

cutscene

+

animation

+

debug

+

Stripe is now properly integrated into this website as an alternative to PayPal! Now, you can also financially support the project if PayPal doesn't work for you, or if you prefer using a provider out of Stripe's greater variety. It's unfortunate that I had to ship this integration while the store is still sold out, but the Shuusou Gyoku OpenGL backend has turned out way too complicated to be finished next to these two pushes within a month. It will take quite a while until the store reopens and you all can start using Stripe, so I'll just link back to this blog post when it happens.

Integrating Stripe wasn't the simplest task in the world either. At first, the Checkout API seems pretty friendly to developers: The entire payment flow is handled on the backend, in the server language of your choice, and requires no frontend JavaScript except for the UI feedback code you choose to write. Your backend API endpoint initiates the Stripe Checkout session, answers with a redirect to Stripe, and Stripe then sends a redirect back to your server if the customer completed the payment. Superficially, this server-based approach seems much more GDPR-friendly than PayPal, because there are no remote scripts to obtain consent for. In reality though, Stripe shares much more potential personal data about your credit card or bank account with a merchant, compared to PayPal's almost bare minimum of necessary data.
It's also rather annoying how the backend has to persist the order form information throughout the entire Checkout session, because it would otherwise be lost if the server restarts while a customer is still busy entering data into Stripe's Checkout form. Compare that to the PayPal JavaScript SDK, which only POSTs back to your server after the customer completed a payment. In Stripe's case, more JavaScript actually only makes the integration harder: If you trigger the initial payment HTTP request from JavaScript, you will have to improvise a bit to avoid the CORS error when redirecting away to a different domain.

But sure, it's all not too bad… for regular orders at least. With subscriptions, however, things get much worse. Unlike PayPal, Stripe kind of wants to stay out of the way of the payment process as much as possible, and just be a wrapper around its supported payment methods. So if customers aren't really meant to register with Stripe, how would they cancel their subscriptions?
Answer: Through the… merchant? Which I quite dislike in principle, because why should you have to trust me to actually cancel your subscription after you requested it? It also means that I probably should add some sort of UI for self-canceling a Stripe subscription, ideally without adding full-blown user accounts. Not that this solves the underlying trust issue, but it's more convenient than contacting me via email or, worse, going through your bank somehow. Here is how my solution works:

• When setting up a Stripe subscription, the server will generate a random ID for authentication. This ID is then used as a salt for a hash of the Stripe subscription ID, linking the two without storing the latter on my server.
• The thank you page, which is parameterized with the Stripe Checkout session ID, will use that ID to retrieve the subscription ID via an API call to Stripe, and display it together with the above salt. This works indefinitely – contrary to what the expiry field in the Checkout session object suggests, Stripe sessions are indeed stored forever. After all, Stripe also displays this session information in a merchant's transaction log with an excessive amount of detail. It might have been better to add my own expiration system to these pages, but this had been taking long enough already. For now, be aware that sharing the link to a Stripe thank you page is equivalent to sharing your subscription cancellation password.
• The salt is then used as the key for a subscription management page. To cancel, you visit this page and enter the Stripe subscription ID to confirm. The server then checks whether the salt and subscription ID pair belong to each other, and sends the actual cancellation request back to Stripe if they do.

I might have gone a bit overboard with the crypto there, but I liked the idea of not storing any of the Stripe session IDs in the server database. It's not like that makes the system more complex anyway, and it's nice to have a separate confirmation step before canceling a subscription.

But even that wasn't everything I had to keep in mind here. Once you switch from test to production mode for the final tests, you'll notice that certain SEPA-based payment providers take their sweet time to process and activate new subscriptions. The Checkout session object even informs you about that, by including a payment status field. Which initially seems just like another field that could indicate hacking attempts, but treating it as such and rejecting any unpaid session can also reject perfectly valid subscriptions. I don't want all this control… 🥲
Instead, all I can do in this case is to tell you about it. In my test, the Stripe dashboard said that it might take days or even weeks for the initial subscription transaction to be confirmed. In such a case, the respective fraction of the cap will unfortunately need to remain red for that entire time.

And that was 1½ pushes just to replicate the basic functionality of a simple PayPal integration with the simplest type of Stripe integration. On the architectural site, all the necessary refactoring work made me finally upgrade my frontend code to TypeScript at least, using the amazing esbuild to handle transpilation inside the server binary. Let's see how long it will now take for me to upgrade to SCSS…

With the new payment options, it makes sense to go for another slight price increase, from up to per push. The amount of taxes I have to pay on this income is slowly becoming significant, and the store has been selling out almost immediately for the last few months anyway. If demand remains at the current level or even increases, I plan to gradually go up to by the end of the year.
📝 As 📝 usual, I'm going to deliver existing orders in the backlog at the value they were originally purchased at. Due to the way the cap has to be calculated, these contributions now appear to have increased in value by a rather awkward 13.33%.

This left ½ of a push for some more work on the TH01 Anniversary Edition. Unfortunately, this was too little time for the grand issue of removing byte-aligned rendering of bigger sprites, which will need some additional blitting performance research. Instead, I went for a bunch of smaller bugfixes:

• ANNIV.EXE now launches ZUNSOFT.COM if MDRV98 wasn't resident before. In hindsight, it's completely obvious why this is the right thing to do: Either you start ANNIV.EXE directly, in which case there's no resident MDRV98 and you haven't seen the ZUN Soft logo, or you have made a single-line edit to GAME.BAT and replaced op with anniv, in which case MDRV98 is resident and you have seen the logo. These are the two reasonable cases to support out of the box. If you are doing anything else, it shouldn't be that hard to adjust though?

  You might be wondering why I didn't just include all code of ZUNSOFT.COM inside ANNIV.EXE together with the rest of the game. The reason: ZUNSOFT.COM has almost nothing in common with regular TH01 code. While the rest of TH01 uses the custom image formats and bad rendering code I documented again and again during its RE process, ZUNSOFT.COM fully relies on master.lib for everything about the bouncing-ball logo animation. Its code is much closer to TH02 in that respect, which suggests that ZUN did in fact write this animation for TH02, and just included the binary in TH01 for consistency when he first sold both games together at Comiket 52. Unlike the 📝 various bad reasons for splitting the PC-98 Touhou games into three main executables, it's still a good idea to split off animations that use a completely different set of rendering and file format functions. Combined with all the BFNT and shape rendering code, ZUNSOFT.COM actually contains even more unique code than OP.EXE, and only slightly less than FUUIN.EXE.

• The optional AUTOEXEC.BAT is now correctly encoded in Shift-JIS instead of accidentally being UTF-8, fixing the previous mojibake in its final ECHO line.

• The command-line option that just adds a stage selection without other debug features (anniv s) now works reliably.

  This one's quite interesting because it only ever worked because of a ZUN bug. From a superficial look at the code, it shouldn't: While the presence of an 's' branch proves that ZUN had such a mode during development, he nevertheless forgot to initialize the debug flag inside the resident structure within this branch. This mode only ever worked because master.lib's resdata_create() function doesn't clear the resident structure after allocation. If anything on the system previously happened to write something other than 0x00, 0x01, or 0x03 to the specific byte that then gets repurposed as the debug mode flag, this lack of initialization does in fact result in a distinct non-test and non-debug stage selection mode.
  This is what happens on a certain widely circulated .HDI copy of TH01 that boots MS-DOS 3.30C. On this system, the memory that master.lib will allocate to the TH01 resident structure was previously used by DOS as stack for its kernel, which left the future resident debug flag byte at address 9FF6:0012 at a value of 0x12. This might be the entire reason why game s is even widely documented to trigger a stage selection to begin with – on the widely circulated TH04 .HDI that boots MS-DOS 6.20, or on DOSBox-X, the s parameter doesn't work because both DOS systems leave the resident debug flag byte at 0x00. And since ANNIV.EXE pushes MDRV98 into that area of conventional DOS RAM, anniv s previously didn't work even on MS-DOS 3.30C.

• Both bugs in the 📝 1×1 particle system during the Mima fight have been fixed. These include the off-by-one error that killed off the very first particle on the 80^th frame and left it in VRAM, and, just like every other entity type, a replacement of ZUN's EGC unblitter with the new pixel-perfect and fast one. Until I've rearchitected unblitting as a whole, the particles will now merely rip barely visible 1×1 holes into the sprites they overlap.

• The 📝 score popups for flipped cards are now displayed without the two frames of flicker.

• The bomb value shown in the lowest line of the in-game debug mode output is now right-aligned together with the rest of the values. This ensures that the game always writes a consistent number of characters to TRAM, regardless of the magnitude of the bomb value, preventing the seemingly wrong timer values that appeared in the original game whenever the value of the bomb variable changed to a lower number of digits:

• Finally, I've streamlined VRAM page access changes, which allowed me to consistently replace ZUN's expensive function call with the optimal two inlined x86 instructions. Interestingly, this change alone removed 2 KiB from the binary size, which is almost all of the difference between 📝 the P0234-1 release and this one. Let's see how much longer we can make each new release of ANNIV.EXE smaller than the previous one.

The final point, however, raised the question of what we're now going to do about 📝 a certain issue in the 地獄/Jigoku Bad Ending. ZUN's original expensive way of switching the accessed VRAM page was the main reason behind the lag frames on slower PC-98 systems, and search-replacing the respective function calls would immediately get us to the optimized version shown in that blog post. But is this something we actually want? If we wanted to retain the lag, we could surely preserve that function just for this one instance…
The discovery of this issue predates the clear distinction between bloat, quirks, and bugs, so it makes sense to first classify what this issue even is. The distinction comes all down to observability, which I defined as changes to rendered frames between explicitly defined frame boundaries. That alone would be enough to categorize any cause behind lag frames as bloat, but it can't hurt to be more explicit here.

Therefore, I now officially judge observability in terms of an infinitely fast PC-98 that can instantly render everything between two explicitly defined frames, and will never add additional lag frames. If we plan to port the games to faster architectures that aren't bottlenecked by disappointing blitter chips, this is the only reasonable assumption to make, in my opinion: The minimum system requirements in the games' README files are minimums, after all, not recommendations. Chasing the exact frame drop behavior that ZUN must have experienced during the time he developed these games can only be a guessing game at best, because how can we know which PC-98 model ZUN actually developed the games on? There might even be more than one model, especially when it comes to TH01 which had been in development for at least two years before ZUN first sold it. It's also not like any current PC-98 emulator even claims to emulate the specific timing of any existing model, and I sure hope that nobody expects me to import a bunch of bulky obsolete hardware just to count dropped frames.

That leaves the tearing, where it's much more obvious how it's a bug. On an infinitely fast PC-98, the ドカーン frame would never be visible, and thus falls into the same category as the 📝 two unused animations in the Sariel fight. With only a single unconditional 2-frame delay inside the animation loop, it becomes clear that ZUN intended both frames of the animation to be displayed for 2 frames each:

TH01 Anniversary Edition, version P0239 2023-05-01-th01-anniv.zip

Next up: Taking the oldest still undelivered push and working towards TH04 position independence in preparation for multilingual translations. The Shuusou Gyoku OpenGL backend shouldn't take that much longer either, so I should have lots of stuff coming up in May afterward.

📝 Posted:

2022-10-31 23:58 UTC

🚚 Summary of:

P0218 Website (Video pipeline, part 1/2: Preparations / Source file gathering)
P0219 Website (Video pipeline, part 2/2: Encoding / Tweaking settings / Caching)
P0220 Website (Video player, part 1/3: Basic controls and frame-accurate seeking)
P0221 Website (Video player, part 2/3: Tabs and markers)
P0222 Website (Video player, part 3/3: Dynamic captions and useful fullscreen mode)

💰 Funded by:

[Anonymous], Yanga, Ember2528

🏷️ Tags:

website

-

Yes, I'm still alive. This delivery was just plagued by all of the worst luck: Data loss, physical hard drive failure, exploding phone batteries, minor illness… and after taking 4 weeks to recover from all of that, I had to face this beast of a task. 😵

Turns out that neither part of improving video performance and usability on this blog was particularly easy. Decently encoding the videos into all web-supported formats required unexpected trade-offs even for the low-res, low-color material we are working with, and writing custom video player controls added the timing precision resistance of HTML <video> on top of the inherent complexity of frontend web development. Why did this need to be 800 lines of commented JavaScript and 200 lines of commented CSS, and consume almost more than 5 pushes?! Apparently, the latest price increase also seemed to have raised the minimum level of acceptable polish in my work, since that's more than the maximum of 3.67 pushes it should have taken. To fund the rest, I stole some of the reserved JIS trail word rendering research pushes, which means that the next towards anything will go back towards that goal.

The codec situation is especially sad because it seems like so much of a solved problem. ZMBV, the lossless capture codec introduced by DOSBox, is both very well suited for retro game footage and remarkably simple too: DOSBox-X's implementation of both an encoder and decoder comes in at under 650 lines of C++, excluding the Deflate implementation. Heck, the AVI container around the codec is more complicated to write than the compressed video data itself, and AVI is already the easiest choice you have for a widely supported video container format.
Currently, this blog contains 9:02 minutes of video across 86 files, with a total frame count of 24,515. In case this post attracts a general video encoding audience that isn't familiar with what I'm encoding here: The maximum resolution is 640×400, and most of the video uses 16 colors, with some parts occasionally using more. With ZMBV, the lossless source files take up 43.8 MiB, and that's even with AVI's infamously bad overhead. While you can always spend more time on any compression task and precisely tune your algorithm to match your source data even better, 43.8 MiB looks like a more than reasonable amount for this type of content.

Especially compared with what I actually have to ship here, because sadly, ZMBV is not supported by browsers. 😔 Writing a WebAssembly player for ZMBV would have certainly been interesting, but it already took 5 pushes to get to what we have now. So, let's instead shell out to ffmpeg and build a pipeline to convert ZMBV to the ill-suited codecs supported by web browsers, replacing the previously committed VP9 and VP8 files. From that point, we can then look into AV1, the latest and greatest web-supported video codec, to save some additional bandwidth.

But first, we've got to gather all the ZMBV source files. While I was working on the 📝 2022-07-10 blog post, I noticed some weirdly washed-out colors in the converted videos, leading to the shocking realization that my previous, historically grown conversion script didn't actually encode in a lossless way. 😢 By extension, this meant that every video before that post could have had minor discolorations as well.
For the majority of videos, I still had the original ZMBV capture files straight out of DOSBox-X, and reproducing the final videos wasn't too big of a deal. For the few cases where I didn't, I went the extra mile, took the VP9 files, and manually fixed up all the minor color errors based on reference videos from the same gameplay stage. There might be a huge ffmpeg command line with a complicated filter graph to do the job, but for such a small 4-digit number of frames, it is much more straightforward to just dump each frame as an image and perform the color replacement with ImageMagick's -opaque and -fill options.

So, time to encode our new definite collection of source files into AV1, and what the hell, how slow is this codec? With ffmpeg's libaom-av1, fully encoding all 86 videos takes almost 9 hours on my mid-range development system, regardless of the quality selected.
But sure, the encoded videos are managed by a cache, and this obviously only needs to be done once. If the results are amazing, they might even justify these glacial encoding speeds. Unfortunately, they don't: In its lossless -crf 0 mode, AV1 performs even worse than VP9, taking up 222 MiB rather than 182 MiB. It might not sound bad now, but as we're later going to find out, we want to have a lot of keyframes in these videos, which will blow up video sizes even further.

So, time to go lossy and maybe take a deep dive into AV1 tuning? Turns out that it only gets worse from there:

• The alternative libsvtav1 encoder is fast and creates small files… but even on the highest-quality settings, -crf 0 and -qp 0, the video quality resembled the terrible x264 YUV420P format that Twitter enforces on uploaded videos.
• I don't remember the librav1e results, but they sure weren't convincing either.
• libaom-av1's -usage realtime option is a complete joke. 771 MiB for all videos, and it doesn't even compress in real time on my system, more like 2.5× real-time. For comparison, a certain stone-age technology by the name of "animated GIF" would take 54.3 MiB, encode in sub-realtime (0.47×), and the only necessary tuning you need is an easily googled palette generation and usage filter. Why can't I just use those in a <video> tag?! These results have clearly proven the top-voted just use modern video codecs Stack Overflow answers wrong.
• What you're actually supposed to do is to drop -cpu-used to maybe 2 or 3, and then selectively add back prediction filters that suit your type of content. In our case, these are
  • -enable-palette
  • -enable-rect-partitions and friends
  • -enable-intrabc (source)
  and maybe others, depending on much time you want to waste.

Because that's what all this tuning ended up being: a complete waste of time. No matter which tuning options I tried, all they did was cut down encoding time in exchange for slightly larger files on average. If there is a magic tuning option that would suddenly cause AV1 to maybe even beat ZMBV, I haven't found it. Heck, at particularly low settings, -enable-intrabc even caused blocky glitches with certain pellet patterns that looked like the internal frame block hashes were colliding all over the place. Unfortunately, I didn't save the video where it happened.

So yeah, if you've already invested the computation time and encoded your content by just specifying a -crf value and keeping the remaining settings at their time-consuming defaults, any further tuning will make no difference. Which is… an interesting choice from a usability perspective. I would have expected the exact opposite: default to a reasonably fast and efficient profile, and leave the vast selection of tuning options for those people to explore who do want to wait 5× as long for their encoder for that additional 5% of compression efficiency. On the other hand, that surely is one way to get people to extensively study your glorious engineering efforts, I guess? You know what would maybe even motivate people to intrinsically do that? Good documentation, with examples of the intent behind every option and its optimal use case. Nobody needs long help strings that just spell out all of the abbreviations that occur in the name of the option…
But hey, that at least means there's no reason to not use anything but ZMBV for storing and archiving the lossless source files. Best compression efficiency, encodes in real-time, and the files are much easier to edit.

OK, end of rant. To understand why anyone could be hyped about AV1 to begin with, we just have to compare it to VP9, not to ZMBV. In that light, AV1 is pretty impressive even at -crf 1, compressing all 86 videos to 68.9 MiB, and even preserving 22.3% of frames completely losslessly. The remaining frames exhibit the exact kind of quality loss you'd want for retro game footage: Minor discoloration in individual pixels, so minuscule that subtracting the encoded image from the source yields an almost completely black image. Even after highlighting the errors by normalizing such a difference image, they are barely visible even if you know where to look. If "compressed PNG size of the normalized difference between ZMBV and AV1 -crf 1" is a useful metric, this would be its median frame among the 77.7% of non-lossless frames:

For comparison, here's the 13th worst one. The codec only resorts to color bleeding with particularly heavy effects, exactly where it doesn't matter:

Whether you can actually spot the difference is pretty much down to the glass between the physical pixels and your eyes. In any case, it's very hard, even if you know where to look. As far as I'm concerned, I can confidently call this "visually lossless", and it's definitely good enough for regular watching and even single-frame stepping on this blog.
Since the appeal of the original lossless files is undeniable though, I also made those more easily available. You can directly download the one for the currently active video with the ⍗ button in the new video player – or directly get all of them from the Git repository if you don't like clicking.

Unfortunately, even that only made up for half of the complexity in this pipeline. As impressive as the AV1 -crf 1 result may be, it does in fact come with the drawback of also being impressively heavy to decode within today's browsers. Seeking is dog slow, with even the latencies for single-frame stepping being way beyond what I'd consider tolerable. To compensate, we have to invest another 78 MiB into turning every 10^th frame into a keyframe until single-stepping through an entire video becomes as fast as it could be on my system.
But fine, 146 MiB, that's still less than the 178 MiB that the old committed VP9 files used to take up. However, we still want to support VP9 for older browsers, older hardware, and people who use Safari. And it's this codec where keyframes are so bad that there is no clear best solution, only compromises. The main issue: The lower you turn VP9's -crf value, the slower the seeking performance with the same number of keyframes. Conversely, this means that raising quality also requires more keyframes for the same seeking performance – and at these file sizes, you really don't want to raise either. We're talking 1.2 GiB for all 86 videos at -crf 10 and -g 5, and even on that configuration, seeking takes 1.3× as long as it would in the optimal case.

Thankfully, a full VP9 encode of all 86 videos only takes some 30 minutes as opposed to 9 hours. At that speed, it made sense to try a larger number of encoding settings during the ongoing development of the player. Here's a table with all the trials I've kept:

Yup, the compromise ended up including a chroma subsampling conversion to YUV422P. That's the one thing you don't want to do for retro pixel graphics, as it's the exact cause behind washed-out colors and red fringing around edges:

But there simply was no satisfying solution around the ~200 MiB mark with RGB colors, and even this compromise is still a disappointment in both size and seeking speed. Let's hope that Safari users do get AV1 support soon… Heck, even VP8, with its exclusive support for YUV420P, performs much better here, with the impact of -crf on seeking speed being much less pronounced. Encoding VP8 also just takes 3 minutes for all 86 videos, so I could have experimented much more. Too bad that it only matters for really ancient systems…
Two final takeaways about VP9:

• -tune-content screen and the tile options make no difference at all.
• All results used two-pass encoding. VP9 is the only codec where two passes made a noticeable difference, cutting down the final encoded size from 224 MiB to 207 MiB. For AV1, compression even seems to be slightly worse with two passes, yielding 154,201,892 bytes rather than the 153,643,316 bytes we get with a single pass. But that's a difference of 0.36%, and hardly significant.

Alright, now we're done with codecs and get to finish the work on the pipeline with perhaps its biggest advantage. With a ffmpeg conversion infrastructure in place, we can also easily output a video's first frame as a poster image to be passed into the <video> tag. If this image is kept at the exact resolution of the video, the browser doesn't need to wait for an indeterminate amount of "video metadata" to be loaded, and can reserve the necessary space in the page layout much faster and without any of these dreaded loading spinners. For the big /blog page, this cuts down the minimum amount of required resources from 69.5 MB to 3.6 MB, finally making it usable again without waiting an eternity for the page to fully load. It's become pretty bad, so I really had to prioritize this task before adding any more blog posts on top.

That leaves the player itself, which is basically a sum of lots of little implementation challenges. Single-frame stepping and seeking to discrete frames is the biggest one of them, as it's technically not possible within the <video> tag, which only returns the current time as a continuous value in seconds. It only sort of works for us because the backend can pass the necessary FPS and frame count values to the frontend. These allow us to place a discrete grid of frame "frets" at regular intervals, and thus establish a consistent mapping from frames to seconds and back. The only drawback here is a noticeably weird jump back by one frame when pausing a video within the second half of a frame, caused by snapping the continuous time in seconds back onto the frame grid in order to maintain a consistent frame counter. But the whole feature of frame-based seeking more than makes up for that.
The new scrubbable timeline might be even nicer to use with a mouse or a finger than just letting a video play regularly. With all the tuning work I put into keyframes, seeking is buttery smooth, and much better than the built-in <video> UI of either Chrome or Firefox. Unfortunately, it still costs a whole lot of CPU, but I'd say it's worth it. 🥲

Finally, the new player also has a few features that might not be immediately obvious:

• Keybindings for almost everything you might want them for, indicated by hovering on top of each button. The tab switchers additionally support the ↑ Up and ↓ Down keys to cycle through all tabs, or the number keys to jump to a specific tab. Couldn't find a way to indicate these mappings in the UI yet.
• Per-video captions now reserve the maximum height of any caption in the layout. This prevents layout reflows when switching through such videos, which previously caused quite annoying lag on the big /blog page.
• Useful fullscreen modes on both desktop and mobile, including all markers and the video caption. Firefox made this harder than it needed to be, and if it weren't for display: contents, the implementation would have been even worse. In the end though, we didn't even need any video pixel sizes from the backend – just as it should be…
• … and supporting Firefox was definitely worth it, as it's the only browser to support nearest-neighbor interpolation on videos.
• As some of the Unicode codepoints on the buttons aren't covered by the default fonts of some operating systems, I've taken them from the Catrinity font, licensed under the SIL Open Font License. With all the edits I did on this font, that license definitely was necessary. I hope I applied it correctly though; it's not straightforward at all how to properly license a Modified Version of an original font with a Reserved Font Name.

And with that, development hell is over, and I finally get to return to the core business! Just more than one month late. Next up: Shipping the oldest still pending order, covering the TH04/TH05 ending script format. Meanwhile, the Seihou community also wants to keep investing in Shuusou Gyoku, so we're also going to see more of that on the side.

📝 Posted:

2021-12-15 23:21 UTC

🚚 Summary of:

P0170 Website (Technical debt visualization)
P0171 Website (Resource caching + new funding goals)

💰 Funded by:

[Anonymous]

🏷️ Tags:

website

-

meta

+

The "bad" news first: Expanding to Stripe in order to support Google Pay requires bureaucratic effort that is not quite justified yet, and would only be worth it after the next price increase.

Visualizing technical debt has definitely been overdue for a while though. With 1 of these 2 pushes being focused on this topic, it makes sense to summarize once again what "technical debt" means in the context of ReC98, as this info was previously kind of scattered over multiple blog posts. Mainly, it encompasses

• any ZUN-written code
• that we did name and reverse-engineer,
• but which we simply moved out into dedicated files that are then #included back into the big .ASM translation units,
• without worrying about decompilation or proving undecompilability for now.

Technically (ha), it would also include all of master.lib, which has always been compiled into the binaries in this way, and which will require quite a bit of dedicated effort to be moved out into a properly linkable library, once it's feasible. But this code has never been part of any progress metric – in fact, 0% RE is defined as the total number of x86 instructions in the binary minus any library code. There is also no relation between instruction numbers and the time it will take to finalize master.lib code, let alone a precedent of how much it would cost.

If we now want to express technical debt as a percentage, it's clear where the 100% point would be: when all RE'd code is also compiled in from a translation unit outside the big .ASM one. But where would 0% be? Logically, it would be the point where no reverse-engineered code has ever been moved out of the big translation units yet, and nothing has ever been decompiled. With these boundary points, this is what we get:

Not too bad! So it's 6.22% of total RE that we will have to revisit at some point, concentrated mostly around TH04 and TH05 where it resulted from a focus on position independence. The prices also give an accurate impression of how much more work would be required there.

But is that really the best visualization? After all, it requires an understanding of our definition of technical debt, so it's maybe not the most useful measurement to have on a front page. But how about subtracting those 6.22% from the number shown on the RE% bars? Then, we get this:

Which is where we get to the good news: Twitter surprisingly helped me out in choosing one visualization over the other, voting 7:2 in favor of the Finalized version. While this one requires you to manually calculate € finalized - € RE'd to obtain the raw financial cost of technical debt, it clearly shows, for the first time, how far away we are from the main goal of fully decompiling all 5 games… at least to the extent it's possible.

Now that the parser is looking at these recursively included .ASM files for the first time, it needed a small number of improvements to correctly handle the more advanced directives used there, which no automatic disassembler would ever emit. Turns out I've been counting some directives as instructions that never should have been, which is where the additional 0.02% total RE came from.

One more overcounting issue remains though. Some of the RE'd assembly slices included by multiple games contain different if branches for each game, like this:

; An example assembly file included by both TH04's and TH05's MAIN.EXE:
if (GAME eq 5)
	; (Code for TH05)
else
	; (Code for TH04)
endif

Currently, the parser simply ignores if, else, and endif, leading to the combined code of all branches being counted for every game that includes such a file. This also affects the calculated speed, and is the reason why finalization seems to be slightly faster than reverse-engineering, at currently 471 instructions per push compared to 463. However, it's not that bad of a signal to send: Most of the not yet finalized code is shared between TH04 and TH05, so finalizing it will roughly be twice as fast as regular reverse-engineering to begin with. (Unless the code then turns out to be twice as complex than average code… ).

For completeness, finalization is now also shown as part of the per-commit metrics. Now it's clearly visible what I was doing in those very slow five months between P0131 and P0140, where the progress bar didn't move at all: Repaying 3.49% of previously accumulated technical debt across all games. 👌

As announced, I've also implemented a new caching system for this website, as the second main feature of these two pushes. By appending a hash string to the URLs of static resources, your browser should now both cache them forever and re-download them once they did change on the server. This avoids the unnecessary (and quite frankly, embarrassing) re-requests for all static resources that typically just return a 304 Not Modified response. As a result, the blog should now load a bit faster on repeated visits, especially on slower connections. That should allow me to deliberately not paginate it for another few years, without it getting all too slow – and should prepare us for the day when our first game reaches 100% and the server will get smashed. However, I am open to changing the progress blog link in the navigation bar at the top to the list of tags, once people start complaining.

Apart frome some more invisible correctness and QoL improvements, I've also prepared some new funding goals, but I'll cover those once the store reopens, next year. Syntax highlighting for code snippets would have also been cool, but unfortunately didn't make it into those two pushes. It's still on the list though!

Next up: Back to RE with the TH03 score file format, and other code that surrounds it.

📝 Posted:

2021-06-06 00:53 UTC

🚚 Summary of:

P0143 Website (Progress number caching)
P0144 Website (Blog tag system, part 1: Manual and automatic tag assignment, blog filtering, design)
P0145 Website (Blog tag system, part 2: Combining tags, per-tag descriptions)

💰 Funded by:

[Anonymous], Yanga, Lmocinemod

🏷️ Tags:

website

-

Who said working on the website was "fun"? That code is a mess. This right here is the first time I seriously wrote a website from (almost) scratch. Its main job is to parse over a Git repository and calculate numbers, so any additional bulky frameworks would only be in the way, and probably need to be run on some sort of wobbly, unmaintainable "stack" anyway, right? 😛 📝 As with the main project though, I'm only beginning to figure out the best structure for this, and these new features prompted quite a lot of upfront refactoring…

Before I start ranting though, let's quickly summarize the most visible change, the new tag system for this blog!

• Yes, I manually went through every one of the 82 posts I've written so far, and assigned labels to them.
• The per-project (rec98 and website) and per-game (th01 th02 th03 th04 th05) tags are automatically generated from the database and the Git commit history, respectively. That might have ended us up with a fair bit of category clutter, as any single change to a tiny aspect is enough for a blog post to be tagged with an otherwise unrelated game. For now, it doesn't seem too much of an issue though.
• Filtering already works for an arbitrary number of tags. Right now, these are always combined with AND – no arbitrary boolean expressions for tag filtering yet.
• Adding filters simply works by adding components to the URL path: https://rec98.nmlgc.net/blog/tag/tag1/tag2/tag3/… and so on.
• Hovering over any tag shows a brief description of what that tag is about. Some of the terms really needed a definition, so I just added one for all of them. Hope you all enjoy them!
• These descriptions are also shown on the new tag overview page, which now kind of doubles as a glossary.

Finally, the order page now shows the exact number of pushes a contribution will fund – no more manual divisions required. Shoutout to the one email I received, which pointed out this potential improvement!

As for the "invisible" changes: The one main feature of this website, the aforementioned calculation of the progress metrics, also turned out as its biggest annoyance over the years. It takes a little while to parse all the big .ASM files in the source tree, once for every push that can affect the average number of removed instructions and unlabeled addresses. And without a cache, we've had to do that every time we re-launch the app server process.
Fundamentally, this is – you might have guessed it – a dependency tracking problem, with two inputs: the .ASM files from the ReC98 repo, and the Golang code that calculates the instruction and PI numbers. Sure, the code has been pretty stable, but what if we do end up extending it one day? I've always disliked manually specified version numbers for use cases like this one, where the problem at hand could be exactly solved with a hashing function, without being prone to human error.

(Sidenote: That's why I never actively supported thcrap mods that affected gameplay while I was still working on that project. We still want to be able to save and share replays made on modded games, but I do not want to subject users to the unacceptable burden of manually remembering which version of which patch stack they've recorded a given replay with. So, we'd somehow need to calculate a hash of everything that defines the gameplay, exclude the things that don't, and only show replays that were recorded on the hash that matches the currently running patch stack. Well, turns out that True Touhou Fans™ quite enjoy watching the games get broken in every possible way. That's the way ZUN intended the games to be experienced, after all. Otherwise, he'd be constantly maintaining the games and shipping bugfix patches… 🤷)

Now, why haven't I been caching the progress numbers all along? Well, parallelizing that parsing process onto all available CPU cores seemed enough in 2019 when this site launched. Back then, the estimates were calculated from slightly over 10 million lines of ASM, which took about 7 seconds to be parsed on my mid-range dev system.
Fast forward to P0142 though, and we have to parse 34.3 million lines of ASM, which takes about 26 seconds on my dev system. That would have only got worse with every new delivery, especially since this production server doesn't have as many cores.

I was thinking about a "doing less" approach for a while: Parsing only the files that had changed between the start and end commit of a push, and keeping those deltas across push boundaries. However, that turned out to be slightly more complex than the few hours I wanted to spend on it. And who knows how well that would have scaled. We've still got a few hundred pushes left to go before we're done here, after all.

So with the tag system, as always, taking longer and consuming more pushes than I had planned, the time had come to finally address the underlying dependency tracking problem.
Initially, this sounded like a nail that was tailor-made for 📝 my favorite hammer, Tup: Move the parser to a separate binary, gather the list of all commits via git rev-list, and run that parser binary on every one of the commits returned. That should end up correctly tracking the relevant parts of .git/ and the new binary as inputs, and cause the commits to be re-parsed if the parser binary changes, right? Too bad that Tup both refuses to track anything inside .git/, and can't track a Golang binary either, due to all of the compiler's unpredictable outputs into its build cache. But can't we at least turn off–

> The build cache is now required as a step toward eliminating $GOPATH/pkg. — Go 1.12 release notes

Oh, wonderful. Hey, I always liked $GOPATH! 🙁

But sure, Golang is too smart anyway to require an external build system. The compiler's build ID is exactly what we need to correctly invalidate the progress number cache. Surely there is a way to retrieve the build ID for any package that makes up a binary at runtime via some kind of reflection, right? Right? …Of course not, in the great Unix tradition, this functionality is only available as a CLI tool that prints its result to stdout. 🙄
But sure, no problem, let's just exec() a separate process on the parser's library package file… oh wait, such a thing doesn't exist anymore, unless you manually install the package. This would have added another complication to the build process, and you'd still have to manually locate the package file, with its version-specific directory name. That might have worked out in the end, but figuring all this out would have probably gone way beyond the budget.

OK, but who cares about packages? We just care about one single file here, anyway. Didn't they put the official Golang source code parser into the standard library? Maybe that can give us something close to the build ID, by hashing the abstract syntax tree of that file. Well, for starters, one does not simply serialize the returned AST. At least into Golang's own, most "native" Gob format, which requires all types from the go/ast package to be manually registered first.
That leaves ast.Fprint() as the only thing close to a ready-made serialization function… and guess what, that one suffers from Golang's typical non-deterministic order when rendering any map to a string. 🤦

Guess there's no way around the simplest, most stupid way of simply calculating any cryptographically secure hash over the ASM parser file. 😶 It's not like we frequently change comments in this file, but still, this could have been so much nicer.
Oh well, at least I did get that issue resolved now, in an acceptable way. If you ever happened to see this website rebuilding: That should now be a matter of seconds, rather than minutes. Next up: Shinki's background animations!

📝 Posted:

2019-09-09 13:11 UTC

🚚 Summary of:

P0016 Website (Price estimate, part 1)
P0017 Website (Price estimate, part 2)

💰 Funded by:

qp

🏷️ Tags:

website

-

Website development time: 12/12

Calculating the average speed of the previous crowdfunded pushes, we arrive at estimated "goals" of…

So, time's up, and I didn't even get to the entire PayPal integration and FAQ parts… 😕 Still got to clarify a couple of legal questions before formally starting this, though. So for now, let's continue with zorg's next 5 TH05 reverse-engineering and decompilation pushes, and watch those prices go down a bit… hopefully quite significantly!

📝 Posted:

2019-09-04 16:14 UTC

🚚 Summary of:

P0013 Website (Design)
P0014 Website (Crowdfunding log + Blog, part 1)
P0015 Website (Blog, part 2)

💰 Funded by:

qp

🏷️ Tags:

website

-

Website development time: 10/12

In order to be able to calculate how many instructions and absolute memory references are actually being removed with each push, we first need the database with the previous pushes from the Discord crowdfunding days. And while I was at it, I also imported the summary posts from back then.

Also, we now got something resembling a web design!

📝 Posted:

2019-08-26 16:21 UTC

🚚 Summary of:

P0012 Website (Memory reference counting)

💰 Funded by:

qp

🏷️ Tags:

website

-

Website development time: 7/12

So yeah, "upper bound" and "probability". In reality it's certainly better than the numbers suggest, but as I keep saying, we can't say much about position independence without having reverse-engineered everything.

Next up: Money goals.

📝 Posted:

2019-08-24 22:10 UTC

🚚 Summary of:

P0011 Website (Completion percentages)

💰 Funded by:

qp

🏷️ Tags:

website

-

Website development time: 6/12

Here we go, overall ReC98 reverse-engineering progress. Now viewable for every commit on the page.

📝 Posted:

2019-08-23 21:43 UTC

🚚 Summary of:

P0010 Website (Golang HTML templating)
P0054 Website (ASM instruction counting, part 1)
P0055 Website (ASM instruction counting, part 2)

💰 Funded by:

DTM, Egor

🏷️ Tags:

website

-

Website development time: 5/12

Now with the number of not yet RE'd x86 instructions the you might have seen in the thpatch Discord. They're a bit smaller now, didn't filter out a couple of directives back then.

Yes, requesting these currently is super slow. That's why I didn't want to have everyone here yet!

Next step: Figuring out the actual total number of game code instructions, for that nice "% done". Also, trying to do the same for position independence.