← Back to Here To Slay

Here To Slay, part 8: going live

From a game that only ran on my machine to an executable that connects to a 1 vCPU, 2 GB server, the load test that found a lobby bug and what the first matches with other people showed.

#tecnico#kafka

With the new engine from part 7, the rules were in place. But Here To Slay still only really existed on my machine, with three Kafka brokers and a bunch of replicas running in Docker, and every match was me against myself in two tabs. To play with friends, three things were missing: a way for people to open the game, a server for it to run on and a game that could handle real people on the other side.

Before inviting anyone

Playing alone in two tabs hides a lot. Nobody takes long to load, nobody drops mid-match and nobody tries to join two tables at once. Before inviting anyone, I went after those cases:

  • Loading screen. I wasn’t sure how real games handle this, and the answer is almost always the same: the match waits for everyone to connect before it starts. Now it waits up to 30 seconds and starts with whoever made it.
  • Dropped connections. If someone’s connection drops mid-match, only that person is disconnected. The chat tells the table, the match waits 60 seconds for them to come back and then carries on without them. Anyone who wants to leave for good has a surrender button.
  • One table per player. You could join two matches at the same time, with very strange results. Now the lobby knows who is already playing and blocks the second one.
  • Everyone sees the choice. When an effect asks for a card or a player to be chosen, everyone else used to stare at a frozen table. Now they all see the same selection screen as the person choosing, without being able to touch it.

Blocking a second table changed the role of a topic. In part 3, match-state-store was meant to hold the whole state of each match, and I admitted that recovery didn’t really work. Today it holds something else: the list of players in each match (the roster), who is still active and whether the match has ended. Each engine republishes that list every five minutes, as a sign that the match is still alive. The lobby reads the topic to know who is already playing, and the gateway uses the same list to deliver monitoring alerts.

Same code, two sizes

The server is small: 1 vCPU and 2 GB of RAM, with Nginx in front handling HTTPS and forwarding the WebSocket to the gateway. From there in, the architecture is the same as in part 3: gateway, lobby, engine and monitoring talking only through Kafka topics.

Three brokers and seven replicas don’t fit there, so there’s a second, leaner Compose file: one broker, one container per service and JVMs capped to fit in memory.

Local with 3 brokers and several replicas, production with 1 broker and 1 container per service

gateway:
  environment:
    JAVA_TOOL_OPTIONS: "-Xmx160m -Xss512k -XX:+UseSerialGC"
kafka-broker-1:
  environment:
    KAFKA_HEAP_OPTS: "-Xms256m -Xmx320m"

That was an unexpected benefit of splitting everything by topics: the code is the same in both environments. Only the number of copies of each thing changes.

An executable

You can play in the browser, but I wanted my friends to download a file, open it and land straight in the game, with nothing to set up. So I packaged the React client with Electron, which builds an executable already pointing at the server. I chose to ship only the portable version, for Windows and Mac: a file that opens without installing.

The first builds of the executable brought problems the browser didn’t have:

  • The class symbols didn’t show up. They came from a font loaded from the internet, and Electron’s security policy blocked it. The font now ships with the app.
  • Closing and reopening went back to the login screen, while the old account stayed stuck in the match. Now the client keeps the session and, when it opens, resumes the same account and goes back to the table or match it was in.
  • The file is heavy. The portable is about 159 MB, and almost none of that is the game. Electron bundles a whole Chromium inside the app, and the background music, an hour of MP3, takes 58 MB on its own. The game code is under 1 MB.

How much the server can take

With the game live, the next question was how many people fit on a server that size. To find out, I wrote a Node script that opens WebSocket connections as if it were the client: each match had two simulated players that logged in, created or joined a table, readied up and played at a human pace, with one action every 1.5 to 3 seconds. I raised the number of simultaneous matches step by step, holding each level for a little over a minute, while measuring the response time of each action and the server’s CPU and memory.

Load test: about 100 simultaneous matches, 200 ms median response, CPU saturating near 130 matches and engine memory going from 95 to 124 MB

Matches (players) Median response p95 Average CPU
10 (20) 165 ms 210 ms 26%
30 (60) 168 ms 324 ms 50%
60 (120) 184 ms 351 ms 78%
100 (200) ~200 ms up to 549 ms ~85%
130 (260) 276 ms 909 ms 98%

The server holds about 100 matches at once, or 200 players. The limit is the single processor, which saturates near 130 matches; the engine’s memory only went from 95 to 124 MB. For a card game among friends, that’s plenty.

The bug the test found

The most useful result of the test wasn’t in the table. Every now and then, a table wouldn’t start: one of the players had joined and readied up, but the lobby acted as if they never had. With dozens of tables being created at the same time, it happened all the time.

The cause was in how the lobby uses Kafka. Every change to a table is written to the compacted lobby-state-store topic, and every lobby instance reads that topic in the background to keep the replicas up to date. But the instance that wrote also read its own write, and sometimes it arrived late: an old snapshot of the table overwrote, in memory, a join or a “ready” that had just happened. A second problem made it worse: each table’s id was the time in milliseconds, so two tables created in the same millisecond got the same id.

The fix was to tag each write with the instance that made it, in a Kafka header, and skip its own writes in the background read. Ids still follow the clock, but never repeat:

long now = System.currentTimeMillis();
long uniqueId = lastLobbyId.updateAndGet(prev -> Math.max(prev + 1, now));

That’s the kind of bug a two-tab test would never show. It only appears when a lot happens at once, and it could have happened to real players.

What the test matches showed

After the load test came the real matches, with other people playing on the executable. Every match ended with a list of notes, almost all about timing and visibility, things no automated test would catch:

  • Dice results have to stay on screen. The dice stopped and the match moved on right away, and nobody could see what had been rolled, especially after modifiers. Now each result stays visible for a few seconds, even when nobody plays a modifier.
  • The roll bar follows the modifiers. The number on top changed, but the 2-to-12 bar kept showing the original value. Now both show the same total.
  • Everyone can react. The modifier and challenge windows now open for every player and wait the full time, and anyone who passed can take it back while the window is open.
  • A choice is always a choice. Even when there’s only one option, like destroying a hero when the opponent only has one, the player has to pick. Resolving it automatically left people not understanding what had happened.
  • The chat became a drawer that can be collapsed without missing new messages, and the victory text only shows up after the last move’s animations finish.
  • A challenged card flies in from whoever played it, so it’s clear where it came from.
  • Reloading the page doesn’t kick anyone out of the match. The client reconnects and goes back to the table where it was.

And some rules were implemented differently from the rulebook. Turn order didn’t come from the highest initiative roll, modifiers didn’t apply to rolls against monsters and Fist of Reason didn’t count on every challenge roll. All three were fixed.

The game is still being tested with a few players, and every match with new people brings a new list of tweaks. Going live showed me that making the game work on my machine was only half the job: the other half shows up once other people start playing.