⮜ Blog

⮜ List of tags

Showing all posts tagged

📝 Posted:
🚚 Summary of:
P0264, P0265
⌨ Commits:
46cd6e7...78728f6, 78728f6...ff19bed
💰 Funded by:
Blue Bolt, [Anonymous], iruleatgames
🏷 Tags:

Oh, it's 2024 already and I didn't even have a delivery for December or January? Yeah… I can only repeat what I said at the end of November, although the finish line is actually in sight now. With 10 pushes across 4 repositories and a blog post that has already reached a word count of 9,240, the Shuusou Gyoku SC-88Pro BGM release is going to break 📝 both the push record set by TH01 Sariel two years ago, and 📝 the blog post length record set by the last Shuusou Gyoku delivery. Until that's done though, let's clear some more PC-98 Touhou pushes out of the backlog, and continue the preparation work for the non-ASCII translation project starting later this year.

But first, we got another free bugfix according to my policy! 📝 Back in April 2022 when I researched the Divide Error crash that can occur in TH04's Stage 4 Marisa fight, I proposed and implemented four possible workarounds and let the community pick one of them for the generally recommended small bugfix mod. I still pushed the others onto individual branches in case the gameplay community ever wants to look more closely into them and maybe pick a different one… except that I accidentally pushed the wrong code for the warp workaround, probably because I got confused with the second warp variant I developed later on.
Fortunately, I still had the intended code for both variants lying around, and used the occasion to merge the current master branch into all of these mod branches. Thanks to wyatt8740 for spotting and reporting this oversight!

  1. The Music Room background masking effect
  2. The GRCG's plane disabling flags
  3. Text color restrictions
  4. The entire messy rest of the Music Room code
  5. TH04's partially consistent congratulation picture on Easy Mode
  6. TH02's boss position and damage variables

As the final piece of code shared in largely identical form between 4 of the 5 games, the Music Rooms were the biggest remaining piece of low-hanging fruit that guaranteed big finalization% gains for comparatively little effort. They seemed to be especially easy because I already decompiled TH02's Music Room together with the rest of that game's OP.EXE back in early 2015, when this project focused on just raw decompilation with little to no research. 9 years of increased standards later though, it turns out that I missed a lot of details, and ended up renaming most variables and functions. Combined with larger-than-expected changes in later games and the usual quality level of ZUN's menu code, this ended up taking noticeably longer than the single push I expected.

The undoubtedly most interesting part about this screen is the animation in the background, with the spinning and falling polygons cutting into a single-color background to reveal a spacey image below. However, the only background image loaded in the Music Room is OP3.PI (TH02/TH03) or MUSIC3.PI (TH04/TH05), which looks like this in a .PI viewer or when converted into another image format with the usual tools:

TH02's Music Room background in its on-disk state TH03's Music Room background in its on-disk state TH04's Music Room background in its on-disk state TH05's Music Room background in its on-disk state
Let's call this "the blank image".

That is definitely the color that appears on top of the polygons, but where is the spacey background? If there is no other .PI file where it could come from, it has to be somewhere in that same file, right? :thonk:
And indeed: This effect is another bitplane/color palette trick, exactly like the 📝 three falling stars in the background of TH04's Stage 5. If we set every bit on the first bitplane and thus change any of the resulting even hardware palette color indices to odd ones, we reveal a full second 8-color sub-image hiding in the same .PI file:

TH02's Music Room background, with all bits in the first bitplane set to reveal the spacey background image, and the full color palette at the bottom TH03's Music Room background, with all bits in the first bitplane set to reveal the spacey background image, and the full color palette at the bottom TH04's Music Room background, with all bits in the first bitplane set to reveal the spacey background image, and the full color palette at the bottom TH05's Music Room background, with all bits in the first bitplane set to reveal the spacey background image, and the full color palette at the bottom
The spacey sub-image. Never before seen!1!! …OK, touhou-memories beat me by a month. Let's add each image's full 16-color palette to deliver some additional value.

On a high level, the first bitplane therefore acts as a stencil buffer that selects between the blank and spacey sub-image for every pixel. The important part here, however, is that the first bitplane of the blank sub-images does not consist entirely of 0 bits, but does have 1 bits at the pixels that represent the caption that's supposed to be overlaid on top of the animation. Since there now are some pixels that should always be taken from the spacey sub-image regardless of whether they're covered by a polygon, the game can no longer just clear the first bitplane at the start of every frame. Instead, it has to keep a separate copy of the first bitplane's original state (called nopoly_B in the code), captured right after it blitted the .PI image to VRAM. Turns out that this copy also comes in quite handy with the text, but more on that later.

Then, the game simply draws polygons onto only the reblitted first bitplane to conditionally set the respective bits. ZUN used master.lib's grcg_polygon_c() function for this, which means that we can entirely thank the uncredited master.lib developers for this iconic animation – if they hadn't included such a function, the Music Rooms would most certainly look completely different.
This is where we get to complete the series on the PC-98 GRCG chip with the last remaining four bits of its mode register. So far, we only needed the highest bit (0x80) to either activate or deactivate it, and the bit below (0x40) to choose between the 📝 RMW and 📝 TCR/📝 TDW modes. But you can also use the lowest four bits to restrict the GRCG's operations to any subset of the four bitplanes, leaving the other ones untouched:

// Enable the GRCG (0x80) in regular RMW mode (0x40). All bitplanes are
// enabled and written according to the contents of the tile register.
outportb(0x7C, 0xC0);

// The same, but limiting writes to the first bitplane by disabling the
// second (0x02), third (0x04), and fourth (0x08) one, as done in the
// PC-98 Touhou Music Rooms.
outportb(0x7C, 0xCE);

// Regular GRCG blitting code to any VRAM segment…
pokeb(0xA8000, offset, …);

// We're done, turn off the GRCG.
outportb(0x7C, 0x00);

This could be used for some unusual effects when writing to two or three of the four planes, but it seems rather pointless for this specific case at first. If we only want to write to a single plane, why not just do so directly, without the GRCG? Using that chip only involves more hardware and is therefore slower by definition, and the blitting code would be the same, right?
This is another one of these questions that would be interesting to benchmark one day, but in this case, the reason is purely practical: All of master.lib's polygon drawing functions expect the GRCG to be running in RMW mode. They write their pixels as bitmasks where 1 and 0 represent pixels that should or should not change, and leave it to the GRCG to combine these masks with its tile register and OR the result into the bitplanes instead of doing so themselves. Since GRCG writes are done via MOV instructions, not using the GRCG would turn these bitmasks into actual dot patterns, overwriting any previous contents of each VRAM byte that gets modified.
Technically, you'd only have to replace a few MOV instructions with OR to build a non-GRCG version of such a function, but why would you do that if you haven't measured polygon drawing to be an actual bottleneck.

Three overlapping Music Room polygons rendered using master.lib's grcg_polygon_c() function with a disabled GRCGThree overlapping Music Room polygons rendered as in the original game, with the GRCG enabled
An example with three polygons drawn from top to bottom. Without the GRCG, edges of later polygons overwrite any previously drawn pixels within the same VRAM byte. Note how treating bitmasks as dot patterns corrupts even those areas where the background image had nonzero bits in its first bitplane.

As far as complexity is concerned though, the worst part is the implicit logic that allows all this text to show up on top of the polygons in the first place. If every single piece of text is only rendered a single time, how can it appear on top of the polygons if those are drawn every frame?
Depending on the game (because of course it's game-specific), the answer involves either the individual bits of the text color index or the actual contents of the palette:

The contents of nopoly_B with each game's first track selected.

Finally, here's a list of all the smaller details that turn the Music Rooms into such a mess:

And that's all the Music Rooms! The OP.EXE binaries of TH04 and especially TH05 are now very close to being 100% RE'd, with only the respective High Score menus and TH04's title animation still missing. As for actual completion though, the finalization% metric is more relevant as it also includes the ZUN Soft logo, which I RE'd on paper but haven't decompiled. I'm 📝 still hoping that this will be the final piece of code I decompile for these two games, and that no one pays to get it done earlier… :onricdennat:

For the rest of the second push, there was a specific goal I wanted to reach for the remaining anything budget, which was blocked by a few functions at the beginning of TH04's and TH05's MAINE.EXE. In another anticlimactic development, this involved yet another way too early decompilation of a main() function…
Generally, this main() function just calls the top-level functions of all other ending-related screens in sequence, but it also handles the TH04-exclusive congratulating All Clear images within itself. After a 1CC, these are an additional reward on top of the Good Ending, showing the player character wearing a different outfit depending on the selected difficulty. On Easy Mode, however, the Good Ending is unattainable because the game always ends after Stage 5 with a Bad Ending, but ZUN still chose to show the EASY ALL CLEAR!! image in this case, regardless of how many continues you used.
While this might seem inconsistent with the other difficulties, it is consistent within Easy Mode itself, as the enforced Bad Ending after Stage 5 also doesn't distinguish between the number of continues. Also, Try to Normal Rank!! could very well be ZUN's roundabout way of implying "because this is how you avoid the Bad Ending".

With that out of the way, I was finally able to separate the VRAM text renderer of TH04 and TH05 into its own assembly unit, 📝 finishing the technical debt repayment project that I couldn't complete in 2021 due to assembly-time code segment label arithmetic in the data segment. This now allows me to translate this undecompilable self-modifying mess of ASM into C++ for the non-ASCII translation project, and thus unify the text renderers of all games and enhance them with support for Unicode characters loaded from a bitmap font. As the final finalized function in the SHARED segment, it also allowed me to remove 143 lines of particularly ugly segmentation workarounds 🙌

The remaining 1/6th of the second push provided the perfect occasion for some light TH02 PI work. The global boss position and damage variables represented some equally low-hanging fruit, being easily identified global variables that aren't part of a larger structure in this game. In an interesting twist, TH02 is the only game that uses an increasing damage value to track boss health rather than decreasing HP, and also doesn't internally distinguish between bosses and midbosses as far as these variables are concerned. Obviously, there's quite a bit of state left to be RE'd, not least because Marisa is doing her own thing with a bunch of redundant copies of her position, but that was too complex to figure out right now.

Also doing their own thing are the Five Magic Stones, which need five positions rather than a single one. Since they don't move, the game doesn't have to keep 📝 separate position variables for both VRAM pages, and can handle their positions in a much simpler way that made for a nice final commit.
And for the first time in a long while, I quite like what ZUN did there! Not only are their positions stored in an array that is indexed with a consistent ID for every stone, but these IDs also follow the order you fight the stones in: The two inner ones use 0 and 1, the two outer ones use 2 and 3, and the one in the center uses 4. This might look like an odd choice at first because it doesn't match their horizontal order on the playfield. But then you notice that ZUN uses this property in the respective phase control functions to iterate over only the subrange of active stones, and you realize how brilliant it actually is.

Screenshot of TH02's Five Magic Stones, with the first two (both internally and in the order you fight them in) alive and activated Screenshot of TH02's Five Magic Stones, with the second two (both internally and in the order you fight them in) alive and activated Screenshot of TH02's Five Magic Stones, with the last one (both internally and in the order you fight them in) alive and activated

This seems like a really basic thing to get excited about, especially since the rest of their data layout sure isn't perfect. Splitting each piece of state and even the individual X and Y coordinates into separate 5-element arrays is still counter-productive because the game ends up paying more memory and CPU cycles to recalculate the element offsets over and over again than this would have ever saved in cache misses on a 486. But that's a minor issue that could be fixed with a few regex replacements, not a misdesigned architecture that would require a full rewrite to clean it up. Compared to the hardcoded and bloated mess that was 📝 YuugenMagan's five eyes, this is definitely an improvement worthy of the good-code tag. The first actual one in two years, and a welcome change after the Music Room!

These three pieces of data alone yielded a whopping 5% of overall TH02 PI in just 1/6th of a push, bringing that game comfortably over the 60% PI mark. MAINE.EXE is guaranteed to reach 100% PI before I start working on the non-ASCII translations, but at this rate, it might even be realistic to go for 100% PI on MAIN.EXE as well? Or at least technical position independence, without the false positives.

Next up: Shuusou Gyoku SC-88Pro BGM. It's going to be wild.

📝 Posted:
🚚 Summary of:
⌨ Commits:
💰 Funded by:
Blue Bolt, [Anonymous]
🏷 Tags:

🎉 After almost 3 years, TH04 finally caught up to TH05 and is now 100% position-independent as well! 🎉

For a refresher on what this means and does not mean, check the announcements from back in 2019 and 2020 when we chased the goal for TH05's 📝 OP.EXE and 📝 the rest of the game. These also feature some demo videos that show off the kind of mods you were able to efficiently code back then. With the occasional reverse-engineering attention it received over the years, TH04's code should now be slightly easier to work with than TH05's was back in the day. Although not by much – TH04 has remained relatively unpopular among backers, and only received more than the funded attention because it shares most of its core code with the more popular TH05. Which, coincidentally, ended up becoming 📝 the reason for getting this done now.
Not that it matters a lot. Ever since we reached 100% PI for TH05, community and backer interest in position independence has dropped to near zero. We just didn't end up seeing the expected large amount of community-made mods that PI was meant to facilitate, and even the 📝 100% decompilation of TH01 changed nothing about that. But that's OK; after all, I do appreciate the business of continually getting commissioned for all the 📝 large-scale mods. Not focusing on PI is also the correct choice for everyone who likes reading these blog posts, as it often means that I can't go that much into detail due to cutting corners and piling up technical debt left and right.

Surprisingly, this only took 1.25 pushes, almost twice as fast as expected. As that's closer to 1 push than it is to 2, I'm OK with releasing it like this – especially since it was originally meant to come out three days ago. 🍋 Unfortunately, it was delayed thanks to surprising website bugs and a certain piece of code that was way more difficult to document than it was to decompile… The next push will have slightly less content in exchange, though.

📝 P0240 and P0241 already covered the final remaining structures, so I only needed to do some superficial RE to prove the remaining numeric literals as either constants or memory addresses. For example, I initially thought I'd have to decompile the dissolve animations in the staff roll, but I only needed to identify a single function pointer type to prove all false positives as screen coordinates there. Now, the TH04 staff roll would be another fast and cheap decompilation, similar to the custom entity types of TH04. (And TH05 as well!)

The one piece of code I did have to decompile was Stage 4's carpet lighting animation, thanks to hex literals that were way too complicated to leave in ASM. And this one probably takes the crown for TH04's worst set of landmines and bloat that still somehow results in no observable bugs or quirks.
This animation starts at frame 1664, roughly 29.5 seconds into the stage, and quickly turns the stage background into a repeated row of dark-red plaid carpet tiles by moving out from the center of the playfield towards the edges. Afterward, the animation repeats with a brighter set of tiles that is then used for the rest of the stage. As I explained 📝 a while ago in the context of TH02, the stage tile and map formats in PC-98 Touhou can't express animations, so all of this needed to be hardcoded in the binary.

A row of the carpet tiles from TH04's Stage 4, at the lowest light levelA row of the carpet tiles from TH04's Stage 4, at the medium light levelA row of the carpet tiles from TH04's Stage 4, at the highest light level
The repeating 384×16 row of carpet tiles at the beginning of TH04's Stage 4 in all three light levels, shown twice for better visibility.

And ZUN did start out making the right decision by only using fully-lit carpet tiles for all tile sections defined in ST03.MAP. This way, the animation can simply disable itself after it completed, letting the rest of the stage render normally and use new tile sections that are only defined for the final light level. This means that the "initial" dark version of the carpet is as much a result of hardcoded tile manipulation as the animation itself.
But then, ZUN proceeded to implement it all by directly manipulating the ring buffer of on-screen tiles. This is the lowest level before the tiles are rendered, and rather detached from the defined content of the 📝 .MAP tile sections. Which leads to a whole lot of problems:

  1. If you decide to do this kind of tile ring modification, it should ideally happen at a very specific point: after scrolling in new tiles into the ring buffer, but before blitting any scrolled or invalidated tiles to VRAM based on the ring buffer. Which is not where ZUN chose to put it, as he placed the call to the stage-specific render function after both of those operations. :zunpet: By the time the function is called, the tile renderer has already blitted a few lines of the fully-lit carpet tiles from the defined .MAP tile section, matching the scroll speed. Fortunately, these are hidden behind the black TRAM cells above and below the playfield…

  2. Still, the code needs to get rid of them before they would become visible. ZUN uses the regular tile invalidation function for this, which will only cause actual redraws on the next frame. Again, the tile rendering call has already happened by the time the Stage 4-specific rendering function gets called.
    But wait, this game also flips VRAM pages between frames to provide a tear-free gameplay experience. This means that the intended redraw of the new tiles actually hits the wrong VRAM page. :tannedcirno: And sure, the code does attempt to invalidate these newly blitted lines every frame – but only relative to the current VRAM Y coordinate that represents the top of the hardware-scrolled screen. Once we're back on the original VRAM page on the next frame, the lines we initially set out to remove could have already scrolled past that point, making it impossible to ever catch up with them in this way.
    The only real "solution": Defining the height of the tile invalidation rectangle at 3× the scroll speed, which ensures that each invalidation call covers 3 frames worth of newly scrolled-in lines. This is not intuitive at all, and requires an understanding of everything I have just written to even arrive at this conclusion. Needless to say that ZUN didn't comprehend it either, and just hardcoded an invalidation height that happened to be enough for the small scroll speeds defined in ST03.STD for the first 30 seconds of the stage.

  3. The effect must consistently modify the tile ring buffer to "fix" any new tiles, overriding them with the intended light level. During the animation, the code not only needs to set the old light level for any tiles that are still waiting to be replaced, but also the new light level for any tiles that were replaced – and ZUN forgot the second part. :zunpet: As a result, newly scrolled-in tiles within the already animated area will "remain" untouched at light level 2 if the scroll speed is fast enough during the transition from light level 0 to 1.

All that means that we only have to raise the scroll speed for the effect to fall apart. Let's try, say, 4 pixels per frame rather than the original 0.25:

By hiding the text RAM layer and revealing what's below the usually opaque black cells above and below the playfield, we can observe all three landmines – 1) and 2) throughout light level 0, and 3) during the transition from level 0 to 1.

All of this could have been so much simpler and actually stable if ZUN applied the tile changes directly onto the .MAP. This is a much more intuitive way of expressing what is supposed to happen to the map, and would have reduced the code to the actually necessary tile changes for the first frame and each individual frame of the animation. It would have still required a way to force these changes into the tile ring buffer, but ZUN could have just used his existing full-playfield redraw functions for that. In any case, there would have been no need for any per-frame tile fixing and redrawing. The CPU cycles saved this way could have then maybe been put towards writing the tile-replacing part of the animation in C++ rather than ASM…

Wow, that was an unreasonable amount of research into a feature that superficially works fine, just because its decompiled code didn't make sense. :onricdennat: To end on a more positive note, here are some minor new discoveries that might actually matter to someone:

Next up: ¾ of a push filled with random boilerplate, finalization, and TH01 code cleanup work, while I finish the preparations for Shuusou Gyoku's OpenGL backend. This month, everything should finally work out as intended: I'll complete both tasks in parallel, ship the former to free up the cap, and then ship the latter once its 5th push is fully funded.

📝 Posted:
🚚 Summary of:
⌨ Commits:
💰 Funded by:
-Tom-, Ember2528
🏷 Tags:

🎉 TH05 is finally fully position-independent! 🎉 To celebrate this milestone, -Tom- coded a little demo, which we recorded on both an emulator and on real PC-98 hardware:

For all the new people who are unfamiliar with PC-98 Touhou internals: Boss behavior is hardcoded into MAIN.EXE, rather than being scriptable via separate .ECL files like in Windows Touhou. That's what makes this kind of a big deal.

What does this mean?

You can now freely add or remove both data and code anywhere in TH05, by editing the ReC98 codebase, writing your mod in ASM or C/C++, and recompiling the code. Since all absolute memory addresses have now been converted to labels, this will work without causing any instability. See the position independence section in the FAQ for a more thorough explanation about why this was a problem.

By extension, this also means that it's now theoretically possible to use a different compiler on the source code. But:

What does this not mean?

The original ZUN code hasn't been completely reverse-engineered yet, let alone decompiled. As the final PC-98 Touhou game, TH05 also happens to have the largest amount of actual ZUN-written ASM that can't ever be decompiled within ReC98's constraints of a legit source code reconstruction. But a lot of the originally-in-C code is also still in ASM, which might make modding a bit inconvenient right now. And while I have decompiled a bunch of functions, I selected them largely because they would help with PI (as requested by the backers), and not because they are particularly relevant to typical modding interests.

As a result, the code might also be a bit confusingly organized. There's quite a conflict between various goals there: On the one hand, I'd like to only have a single instance of every function shared with earlier games, as well as reduce ZUN's code duplication within a single game. On the other hand, this leads to quite a lot of code being scattered all over the place and then #include-pasted back together, except for the places where 📝 this doesn't work, and you'd have to use multiple translation units anyway… I'm only beginning to figure out the best structure here, and some more reverse-engineering attention surely won't hurt.

Also, keep in mind that the code still targets x86 Real Mode. To work effectively in this codebase, you'd need some familiarity with memory segmentation, and how to express it all in code. This tends to make even regular C++ development about an order of magnitude harder, especially once you want to interface with the remaining ASM code. That part made -Tom- struggle quite a bit with implementing his custom scripting language for the demo above. For now, he built that demo on quite a limited foundation – which is why he also chose to release neither the build nor the source publically for the time being.
So yeah, you're definitely going to need the TASM and Borland C++ manuals there.

tl;dr: We now know everything about this game's data, but not quite as much about this game's code.

So, how long until source ports become a realistic project?

You probably want to wait for 100% RE, which is when everything that can be decompiled has been decompiled.

Unless your target system is 16-bit Windows, in which case you could theoretically start right away. 📝 Again, this would be the ideal first system to port PC-98 Touhou to: It would require all the generic portability work to remove the dependency on PC-98 hardware, thus paving the way for a subsequent port to modern systems, yet you could still just drop in any undecompiled ASM.

Porting to IBM-compatible DOS would only be a harder and less universally useful version of that. You'd then simply exchange one architecture, with its idiosyncrasies and limits, for another, with its own set of idiosyncrasies and limits. (Unless, of course, you already happen to be intimately familiar with that architecture.) The fact that master.lib provides DOS/V support would have only mattered if ZUN consistently used it to abstract away PC-98 hardware at every single place in the code, which is definitely not the case.

The list of actually interesting findings in this push is, 📝 again, very short. Probably the most notable discovery: The low-level part of the code that renders Marisa's laser from her TH04 Illusion Laser shot type is still present in TH05. Insert wild mass guessing about potential beta version shot types… Oh, and did you know that the order of background images in the Extra Stage staff roll differs by character?

Next up: Finally driving up the RE% bar again, by decompiling some TH05 main menu code.

📝 Posted:
🚚 Summary of:
⌨ Commits:
💰 Funded by:
🏷 Tags:

Wouldn't it be a bit disappointing to have TH05 completely position-independent, but have it still require hex-editing of the original ZUN.COM to mod its gaiji characters? As in, these custom "text" glyphs, available to the PC-98 text RAM:

TH05 gaiji characters

Especially since we now even have a sprite converter… the lack of which was exactly 📝 what made rebuilding ZUN.COM not that worthwhile before. So, before the big release, let's get all the remaining ZUN.COM sub-binaries of TH04 and TH05 dumped into .ASM files, and re-assembled and linked during the build process.

This is also the moment in which Egor's 2018 reimplementation of O. Morikawa's comcstm finally gets to shine. Back then, I considered it too early to even bother with ZUN.COM and reimplementing the .COM wrapper that ZUN originally used to bundle multiple smaller executables into that single binary. But now that the time is right, it is nice to have that code, as it allowed me to get these rebuilds done in half a push. Otherwise, it would have surely required one or two dedicated ones.

Since we like correctness here, newly dumped ZUN code means that it also has to be included in the RE% baseline calculation. This is why TH04's and TH05's overall RE% bars have gone back a tiny bit… in case you remember how they previously looked like :tannedcirno: After all, I would like to figure out where all that memory allocated during TH04's and TH05's memory check is freed, if at all.

Alright, one half of a push left… Y'know, getting rid of those last few PI false positives is actually one of the most annoying chores in this project, and quite stressful as well: I have to convince myself that the remaining false positives are, in fact, not memory references, but with way too little time for in-depth RE and to denote what they are instead. In that situation, everyone (including myself!) is anticipating that PI goal, and no one is really interested in RE. (Well… that is, until they actually get to developing their mod. But more on that tomorrow. :onricdennat:) Which means that it boils down to quite some hasty, dumb, and superficial RE around those remaining numbers.

So, in the hope of making it less annoying for the other 4 games in the future, let's systematically cover the sources of those remaining false positives in TH05, over all games. I/O port accesses with either the port or the value in registers (and thus, no longer as an immediate argument to the IN or OUT instructions, which the PI counter can clearly ingore), palette color arithmetic, or heck, 0xFF constants that obviously just mean "-1" and are not a reference to offset 0xFF in the data segment. All of this, of course, once again had a way bigger effect on everything but an almost position-independent TH05… but hey, that's the sort of thing you reserve the "anything" pushes for. And that's also how we get some of the single biggest PI% gains we have seen so far, and will be seeing before the 100% PI mark. And yes, those will continue in the next push.

Alright! Big release tomorrow…

📝 Posted:
🚚 Summary of:
⌨ Commits:
💰 Funded by:
🏷 Tags:

🎉 TH01's OP.EXE and FUUIN.EXE are now fully position-independent! 🎉

What does this mean?

You can now add any data or code to TH01's main menu or ending cutscenes, by simply editing the ReC98 source, writing your mod in ASM or C++, and recompiling the code. Since all absolute memory addresses in OP and FUUIN have now been converted to labels, this will work without causing any instability. See the position independence section in the FAQ for a more thorough explanation about why this was a problem.
As an example, the most popular TH01 mod idea, replacing MDRV2 with PMD, could now at least be prototyped and tested in OP.EXE, without having to worry about x86 instruction lengths.
📝 Check the video I made for the TH04/TH05 OP.EXE PI announcement for a basic overview of how to do that.

What does this not mean?

The original ZUN code hasn't been completely decompiled yet. The final high-level parts of both the main menu and the cutscenes are still ASM, which might make modding a bit inconvenient right now.
It's not that much more code though, and could quickly be covered in a few pushes if requested. Due to the plentiful monthly subscriptions, the shop will stay closed for regular orders until the end of June, but backers with outstanding contributions could request that now if they want to – simply drop me a mail. Otherwise, the "generic TH01 RE" money will continue to go towards the main game. That way, we'll have more substance to show once we do decide to decompile the rest of OP.EXE and FUUIN.EXE, and likely get some press coverage as a result.

Then again, we've been building up to this point over the last few pushes, and it only really needed a quick look over the remaining false positives. The majority of the time therefore went towards more PI in REIIDEN.EXE, where the bitplane pointers for .BOS files yielded some quite big gains. Couldn't really find any obvious reason why ZUN used two slighly different variations on loading and blitting those files, though… :onricdennat:

As the final function in this rather random push, we got TH01's hardware-powered scrolling function, used for screen shaking effects and the scrolling backgrounds at the start of the Final Boss stages. And while I tried to document all these I/O writes… it turned out that ZUN actually copied the entire function straight from the PC-9801 Programmers' Bible, with no changes. :zunpet: It's the setgsta() example function on page 150. Which is terribly suboptimal and bloated – all those integer divisions are really not how you'd write such code for a 16-bit compiler from the 90's…

And that gives us 60% PI overall, and 50% PI over all of TH01! Next up: More structures… and classes, even?

📝 Posted:
🚚 Summary of:
⌨ Commits:
💰 Funded by:
Touhou Patch Center
🏷 Tags:

🎉 TH04's and TH05's OP.EXE are now fully position-independent! 🎉

What does this mean?

You can now add any data or code to the main menus of the two games, by simply editing the ReC98 source, writing your mod in ASM or C/C++, and recompiling the code. Since all absolute memory addresses have now been converted to labels, this will work without causing any instability. See the position independence section in the FAQ for a more thorough explanation about why this was a problem.

What does this not mean?

The original ZUN code hasn't been completely reverse-engineered yet, let alone decompiled. Pretty much all of that is still ASM, which might make modding a bit inconvenient right now.

Since this push was otherwise pretty unremarkable, I made a video demonstrating a few basic things you can do with this:

Now, what to do for the last outstanding Touhou Patch Center push? Bullets, or resident structures? :thonk:

📝 Posted:
🚚 Summary of:
⌨ Commits:
💰 Funded by:
Touhou Patch Center
🏷 Tags:

Big gains, as expected, but not much to say about this one. With TH05 Reimu being way too easy to decompile after 📝 the shot control groundwork done in October, there was enough time to give the comprehensive PI false-positive treatment to two other sets of functions present in TH04's and TH05's OP.EXE. One of them, master.lib's super_*() functions, was used a lot in TH02, more than in any other game… I wonder how much more that game will progress without even focusing on it in particular.

Alright then! 100% PI for TH04's and TH05's OP.EXE upcoming… (Edit: Already got funding to cover this!)

📝 Posted:
🚚 Summary of:
⌨ Commits:
💰 Funded by:
[Anonymous], -Tom-
🏷 Tags:

With no feedback to 📝 last week's blog post, I assume you all are fine with how things are going? Alright then, another one towards position independence, with the same approach as before…

Since -Tom- wanted to learn something about how the PC-98 EGC is used in TH04 and TH05, I took a look at master.lib's egc_shift_*() functions. These simply do a hardware-accelerated memmove() of any VRAM region, and are used for screen shaking effects. Hover over the image below for the raw effect:

Demonstration of an egc_shift_left() call

Then, I finally wanted to take a look at the bullet structures, but it required way too much reverse-engineering to even start within ¾ of a position independence push. Even with the help of uth05win – bullet handling was changed quite a bit from TH04 to TH05.

What I ultimately settled on was more raw, "boring" PI work based around an already known set of functions. For this one, I looked at vector construction… and this time, that actually made the games a little bit more position-independent, and wasn't just all about removing false positives from the calculation. This was one of the few sets of functions that would also apply to TH01, and it revealed just how chaotically that game was coded. This one commit shows three ways how ZUN stored regular 2D points in TH01:

… yeah. But in more productive news, this did actually lay the groundwork for TH04 and TH05 bullet structures. Which might even be coming up within the next big, 5-push order from Touhou Patch Center? These are the priorities I got from them, let's see how close I can get!