(tl;dr: ReC98 has switched to Tup for
the 32-bit build. You probably want to get
💾 this build of Tup, and put it somewhere in your
PATH. It's optional, and always will be, but highly
P0001! Reserved for the delivery of the very first financial contribution
I've ever received for ReC98, back in January 2018. GhostPhanom
requested the exact opposite of immediate results, which motivated me to
go on quite a passionate quest for the perfect ReC98 build system. A quest
that went way beyond the crowdfunding…
Makefiles are a decent idea in theory: Specify the targets to generate,
the source files these targets depend on and are generated from, and the
rules to do the generating, with some helpful shorthand syntax. Then, you
have a build dependency graph, and your
make tool of choice
can provide minimal rebuilds of only the targets whose sources changed
since the last
make call. But, uh… wait, this is C/C++ we're
talking about, and doesn't pretty much every source file come with a
second set of dependent source files, namely, every single
#include in the source file itself? Do we really
have to duplicate all these inside the Makefile, and keep it in sync with the source file? 🙄
This fact alone means that Makefiles are inherently unsuited for
any language with an
#include feature… that is, pretty
much every language out there. Not to mention other aspects like changes
to the compilation command lines, or the build rules themselves, all of
which require metadata of the previous build to be persistently stored in
some way. I have no idea why such a trash technology is even touted as a
viable build tool for code.
But wait! Most
make implementations, including Borland's, do
support the notion of auto-dependency information, emitted by the
compiler in a specific format, to provide
make with the
additional list of
#includes. Sure, this should be a basic
feature of any self-respecting build tool, and not something you have to
add as an extension, but let's just set our idealism aside
for a moment. Well, too bad that Borland's implementation
only works if you spell out
both the source➜object and the object➜binary rules, which
loses the performance gained from compiling multiple translation units in
TCC process. And even then, it
tends to break in that DOS VM you're probably using. Not to mention,
again, all the other aspects that still remain unsolved.
So, I decided to just
write my own build system, tailor-made for the needs of ReC98's 16-bit
build process, and combining a number of experimental ideas. Which is
still not quite bug-free and ready for public use, given that the
entire past year has kept me busy with actual tangible RE and PI progress.
What did finally become ready, however, is the improvement for the
32-bit build part, and that's what we've got here.
💭 Now, if only there was a build system that would perfectly track
dependencies of any compiler it calls, by injecting code and
hooking file opening syscalls. It'd be completely unrealistic for it to
also run on DOS (and we probably don't want to traverse a graph database
in a cycle-limited DOSBox), but it would be perfect for our 32-bit build
part, as long as that one still exists.
Turns out Tup is exactly that system.
In practice, its low-level nature as a
make replacement does
limit its general usefulness, which is why you probably haven't seen it
used in a lot of projects. But for something like ReC98 with its reliance
on outdated compilers that aren't supported by any decent high-level tool,
it's exactly the right tool for the job. Also, it's completely beyond me
how Ninja, the
make replacement these days, was inspired by
Tup, yet went a step back to parsing the specific dependency information
produced by gcc, Clang, and Visual Studio, and only those…
Sure, it might seem really minor to worry about not unconditionally
rebuilding all 32-bit
.asm files, which just takes a couple
of seconds anyway. But minimal rebuilds in the 32-bit part also provide
the foundation for minimal rebuilds in the 16-bit part – and those
TLINK invocations do take quite some time after all.
Using Tup for ReC98 was an idea that dated back to January 2017. Back
then, I already opened
the pull request with a fix to allow Tup to work together with 32-bit
TASM. As much as I love Tup though, the fact that it only worked on
64-bit Windows ≥Vista would have meant that we had to exchange perfect
dependency tracking for the ability to build on 32-bit and older Windows
versions at all. For a project that relies on DOS compilers, this
would have been exactly the wrong trade-off to make.
What's worse though:
TLINK fails to run on modern 32-bit
Loader error (0000) : Unrecognized Error.
Therefore, the set of systems that Tup runs on, and the set of systems
that can actually compile ReC98's 16-bit build part natively, would have
been exactly disjoint, with no OS getting to use both at the same time.
So I've kept using Tup for only my own development, but indefinitely
shelved the idea of making it the official build system, due to those
drawbacks. Recently though, it all came together:
As I'm writing this post, the pull request has unfortunately not yet been
merged. So, here's my own custom build instead:
💾 Download Tup for 32-bit Windows
(optimized build at
I've also added it to the DevKit, for any newcomers to ReC98.
After the switch to Tup and the fallback option, I extensively tested
building ReC98 on all operating systems I had lying around. And holy cow,
so much in that build was broken beyond belief. In the end, the solution
involved just fully rebuilding the entire 16-bit part by default.
Which, of course, nullifies any of the
advantages we might have gotten from a Makefile in the first place, due to
just how unreliable they are. If you had problems building ReC98 in the
past, try again now!
And sure, it would certainly be possible to also get Tup working on
Windows ≤XP, or 9x even. But I leave that to all those tinkerers out there
who are actually motivated to keep those OSes alive. My work here is
done – we now have a build process that is optimal on 32-bit
Windows ≧Vista, and still functional and reliable on 64-bit
Windows, Linux, and everything down to Windows 98 SE, and therefore also
real PC-98 hardware. Pretty good, I'd say.
(If it weren't for that weird crash of the 16-bit
that Windows 95 command prompt I've tried it in, it would also work on
that OS. Probably just a misconfiguration on my part?)
Now, it might look like a waste of time to improve a 32-bit build part
that won't even exist anymore once this project is done. However, a fully
16-bit DOS build will only make sense after
And who knows whether this project will get funded for that long. So yeah,
the 32-bit build part will stay with us for quite some more time, and for
all upcoming PI milestones. And with the current build process, it's
pretty much the most minor among all the minor issues I can think of.
Let's all enjoy the performance of a 32-bit build while we can 🙂
- master.lib has been turned into a proper library, linked in by
TLINK rather than
#included in the big .ASM
- This affects all games. If master.lib's data was consistently
placed at the beginning or end of each data segment, this would be no
big deal, but it's placed somewhere else in every binary.
- So, this will only make sense sometime around 90% overall PI, and
maybe ~50% RE in each game. Which is something else than 50%
overall – especially since it includes TH02, the objectively
worst Touhou game, which hasn't received any dedicated funding
- Then, it will probably still require a couple of dedicated pushes
to move all the remaining data to C land.
- Oh, and my 16-bit build system project also needs to be done
before, because, again, Makefiles are trash and we shouldn't rely on
them even more.
Next up: Paying some technical debt while keeping the RE% and PI% in place.