This is another X-Treme Nerd Interlude post. Last time we announced, mercifully briefly, our shiny new blog redesign and if you’re a normal human you should read that, nod thoughtfully, say “looks lovely”, and be on your merry way. The rest of you can frolic deep in the weeds here with Nathan Arthur aka narthur — the mastermind and architect of everything you’re looking at. Also it’s a nice primer on what the heck a Static Site Generator (SSG) is, in case you were curious about that.
WordPress is the de facto tool for building blogs and, as Danny said last time, it served Beeminder well for many years. But now we’re using a static site generator called Astro. Let’s talk about why and how!
We’ll start with the downsides of WordPress. WordPress is a server-rendered content management system. That means that whenever you visit a page on a WordPress website, WordPress retrieves that page’s data from a database, renders it to HTML, and sends it to you.
This architecture is powerful — it allows websites to tailor each page for the current user. And in WordPress’s case it allows non-technical admins to build and customize their sites, no coding required.
The Idea
There’s an alternative — the static site. A static site is simply a set of HTML, CSS, and JavaScript files, served as-is, without waiting for the server to painstakingly construct them before sending them to you.
It would be impractical to build most websites this way. Hand coding HTML files for every post on the Beeminder blog would be painful. And then a change to the footer, say, would require editing every HTML file again.
But what if we did that WordPress-style page construction all at once, one time? Then we’d have a set of static files. We call that one-time painstaking construction the build step.
Whenever you add a new blog post, or change anything in an old blog post, or change anything at all about the website, you trigger a build. The Static Site Generator does its WordPressing, as we’ll call it, to every page and blog post on the whole site again. But, crucially, it’s not doing this in real time with eager blog readers waiting for it. It regenerates the static site and when the new files are ready, it swaps the old ones out and the new ones in. End users are only ever asking for and getting served static pages.
Basically we have the best of both worlds: let the build process do all the HTML generating, putting the same footer on every page and all that, but let end users get served simple static pages. It’s also more secure. WordPress has a whole admin interface to let you add posts and modify the site — a static site needs none of that. And it’s drastically cheaper (or even free, depending on how much traffic it gets) to host a static site. And content delivery networks (CDNs) can cache those files closer to end users which adds to the speed advantage.
The Beeminder blog is particularly well-situated to take advantage of all of those benefits. Its content is primarily static. The non-static bits (like the comments) can use third-party embeds. Beeminder’s staff are technical and not code-shy (so no need for everything to be WYSIWYG). And Beeminder had been spending a lot on blog hosting.
(Danny adds: Does this mean the migration paid for itself? Haha, no. We initially thought it might — we were paying around $1k/year for WordPress hosting — but then decided we didn’t care. We really like this new system! And it might eventually pay off even just money-wise, who knows?)
Objectives & Requirements
Alright, enough why, let’s run through the what and how! We had some specific boxes that we wanted the new blog to tick. First, we needed to preserve what the Beeminder team liked about their workflow. That meant pulling in external markdown source files for each post, and being agnostic about where Beeminder may decide to store that markdown. And builds needed to be fast enough that fixing a typo wouldn’t be excruciating.
We also wanted the blog to be nice and maintainable — easy for future developers to improve the blog. And of course not increasing Beeminder’s security exposure or other costs. Like I said, security and cost should both be improved a lot by a static site generator.
In terms of what users see when read the blog, we needed to Pareto-dominate the old blog. So of course all the old posts should stay the same, the RSS feed shouldn’t break. Beeminder also has a bunch of fancy custom markdown augmentations like footnotes and LaTeX-style references that needed to keep functioning.
And as icing on the cake, we wanted the blog to feel nicer for end users — faster and more modern and more in line with Beeminder’s visual identity.
Choosing a Static Site Generator
We needed to take our backlog of content and use a static site generator to convert that content into static HTML, CSS, and JavaScript files.
Many options exist. I didn’t want to use those that I’d used previously. Jekyll — not my preferred stack. Gatsby — too unintuitive. Next.js — too many unneeded features. Also Brent Yorgey recommended Hakyll which seems cool but he recommended it too late and it would have been hard since I’m bad at Haskell.
One I hadn’t used previously was Astro. I watched a video of someone building a simple site in Astro, and I was impressed. It seemed like our use case was exactly Astro’s happy path, which made me feel much more comfortable that I could spend more time executing and less time fighting with the framework. I also appreciated that I could use my preferred stack — node, pnpm, typescript, vitest. I liked how the default Astro templating language is JSX-like, and how it co-locates server JS, client JS, HTML, and CSS in each component file. I also really liked that it comes with folder structure routing out of the box. And I liked how easy it makes using external data in static generation, just using the standard fetch API.
Astro had everything we wanted. It has file-based routing with dynamic route segments and simple route-based data fetching. It’s written in JavaScript and supports TypeScript out of the box. It uses file-based components. The Astro templating format is close enough to JSX to make a React developer feel at home, while sidestepping all the complexity that comes from using React. It’s built on Vite, making for good build performance. And it has an RSS plugin which made setting up our feed quite simple.
Additionally Astro proved beneficial in several other ways that we didn’t think to look for. It has built-in link prefetching. It includes experimental support for image optimization (still working on setting that up!) and static redirects. And it has a hot-reloading local dev server.
Choosing a Host
Danny suggested Render.com. I hadn’t used it before, but I’m glad we did. It was a great experience. I’ve never enjoyed deploying to AWS or Google Cloud. In contrast, Render.com integrates with GitHub, providing painless git-triggered deployments. Also blueprints.
Initially I planned to have GitHub Actions run our builds. I planned to take the custom PHP code from Beeminder’s WordPress plugin and run it as a stand-alone PHP proxy in front of Etherpad inside the build action. That way I could avoid reimplementing the custom functionality from the old blog for things like footnotes. The Continuous Integration (CI) action would then deploy the built blog to whatever static host we settled on.
However Render.com has its own Netlify-style static site builds, so no need for GitHub Actions. Instead I reimplemented the custom logic in TypeScript. While this meant more effort in reproducing the features from the old blog, it kept the deploy pipeline much simpler and kept everything in the main TypeScript codebase, which I think was the right call for long-term maintainability.
Migrating the Data
Beeminder hosts the raw markdown behind the blog posts in an Etherpad instance separate from WordPress, so we didn’t need to worry about exporting post content from WordPress. But we still needed to export post metadata, including the Etherpad source URLs, as well as some info about the authors, and then use that data in the builds for old posts that aren’t using the new markdown frontmatter.
I used two WordPress plugins to export the data — one for posts and one for users. I then took this data, imported it into Google Sheets, and exported just the columns we needed into CSV files which are stored in the blog repository.
At build time the code uses these CSV files to look up information about posts that were written before the blog rewrite. For new posts, this meta information will be stored in frontmatter at the top of the markdown files.
Build Optimization
I spent too much time optimizing build performance. It’s a fun puzzle.
I memoized liberally. You don’t want to repeat any expensive computations. In addition I cached the raw markdown we requested from Etherpad. In a production build, this is cached in-memory. Locally (i.e., in development) it’s cached in the filesystem to avoid needing to wait for a bunch of requests to finish before seeing the result of a test build.
There are some fancy markdown features that I needed to use a virtual DOM library to implement. I started by using jsdom, but this was very slow. I ended up switching to happy-dom and it was much faster.
When I first started fetching markdown from Etherpad, I made all the requests at once. This flooded Beeminder’s Etherpad instance and caused our builds to fail. I switched to doing these requests synchronously, which allowed the builds to complete, but made builds take a long time. Finally I switched to using p-limit to send the requests as fast as possible while limiting the number of simultaneously-active requests. I tested that limit, timing different values, until I found one that was optimal for build speed and reliability.
In conclusion… voila? Goodbye WordPress, hello Astro. And it’s being served by Render.com at zero monthly cost. Here’s the full final stack in case you got bored partway through the above and are jumping here for the punchline:
- Pnpm - package manager
- Zod - data schemas and transforms
- Astro - static site generator
- Vitest - testing framework
- Happy-dom - virtual dom
- Render.com - hosting and builds
- GitHub - version control and ci
- TypeScript - static types
- Node - build runtime
- Vite - Astro’s underlying build tool
Danny would also like me to remind everyone that there’s a $10 honey money bounty for any bugs or typos you may find on the blog. You can report those in the comments or in the forum!