[Eon Altar] Bottled Awesome--Reconnecting Mobile Devices to the Game

One of the things that makes Eon Altar unique is we use mobile devices--phones and tablets--as controllers to the primary game. Of course, this kind of connectivity means that some state needs to be communicated to the mobile devices as you play. But even more concerning is how to get all that data back to the mobile device when it crashes, or you answer a text and iOS kills the app due to memory pressure, or your battery runs out.

Flying Helmet Games playing a test build on their phones.

Interestingly enough, we went for a very long time without a robust reconnect feature, which actually spoke pretty highly of the quality of our controller builds, but now that we have one, I definitely regret leaving it as long as we did.


The First Solution

The hardest part about reconnecting, of course, is what I hinted to in the first paragraph: ensuring your state on the controller is correct. The easiest way of doing this, without having to think of any special logic is to copy/paste the state from the main game onto the controller over the network. In programmer terms, serialize the state, and deserialize it on the other end. Given our time/budget constraints, we thought this was a pretty good idea.

However, once we implemented it, it turned out to be terrible in practice. The performance was abysmal, primarily because of how we organize our data on the main game. When we serialized the state, we also ended up picking up character models, animations, textures, and so on. And not just for your character, but for all player characters. We had reconnect times upwards of 8 minutes per controller when playing with 4 players. That was...not so hot.

We left the problem for a couple months, though, because we had more pressing needs in keeping the designers unblocked (which is another interesting discussion entirely). We'd lived that long without reconnect, what was a few more months?

Once we got back around to it, it was also clear from a design perspective that what we were doing wasn't going to hack it. Sending all of that data to the controllers was far too much on 512 MB devices. We were also using the same code to save our game to a file, and that was problematic due to the fact that patching the character or level data after the fact was pretty well impossible, not to mention save files bordering on 1GB+ of data.


The Final Solution

So serializing the state naively wasn't going to work, which meant that we were going to have to dig in and actually reason about what data we needed to send across (or save to disk!). Turns out it wasn't too bad at all.

Thanks to Unity's RPC framework guaranteeing delivery and order, it "simply" became a matter of writing a new network model on top of Unity's to decouple how they instantiated objects over a network, and then reinstantiate the player actor we needed on the controller. Since we knew what we started with baseline, it then became an exercise in determining how that state changed from the baseline. What abilities do you have unlocked? Gear and recipes? Treasure? Renown? Current attributes and status effects? Skills? Lore? Are we in the middle of a dialogue?

It was slightly more complex than that in the end, because we have a bunch of other synchronized tracking variables that had to be transferred as well, but most of our "state" in the previous paragraph are just prefab objects, so the majority of controller reconnect rehydration is telling the controller, "Hey, go instantiate this actor, and then go grab prefabs A, B, and C for lore, and unlock skills 1, 2, and 6. Also, update these synchronized variables." It's simple, and quite robust.

Unity Project Setup

What also helps us here is the fact that our Controller app and our main game executable actually share resources and code. Unity is (mostly) smart enough to only pull in assets based on what scenes we include, so in the picture above, when we build the game executable, Unity pulls in a bunch of data to spit out for the build, and if we build the controller app, same deal, unity pulls in the correct data.

I say mostly because the Resources folder is special in Unity. We had to do some...shenanigans around Resources folders to get this system to work, and it's not quite perfect. but in terms of reconnect, it's great because when reconnecting, as long as the builds are compatible, I can be guaranteed that the correct data will be there on the controller. And if not, oh well. Think of it as the equivalent of updating tooltips in WoW. The backend server data might change, but the tooltips are local client data, and may get out of date. A patch will fix that right up later.

There are definite cons to this setup as well, like requiring patching the controllers more often, and the Resources folder shenanigans means sometimes extraneous data gets pulled into the controller build. But it's also super fast from a CPU time and networking perspective. We're not serializing tonnes of data to send over the (local) wire, so the reconnect now takes about 1 - 3 seconds rather than 8 minutes. Actually, the grand majority of the reconnect time is actually just loading the controller UI textures, the same as a normal load rather than anything reconnect specific.


Bottled Awesome

The best part about the latest version of reconnect? It's ultra solid. Our lead designer told me it was "bottled awesome". Pretty much any controller app bug we get can be worked around by killing the controller app and reconnecting, which has been an immense boon now that we're in Early Access. 

We haven't seen many bugs come out of it either, which makes me pretty proud of the quality of the feature as a whole, too. The lack of bugs isn't for a lack of testing, either. We use reconnect all the time. I actually really regret not having a robust reconnect completed much earlier in the project. Mind you, part of why it's so robust is probably because it was done in a single 2-week period after most of the game was already complete. It's "easy" to build something like this and get it right in a single go rather than finish something nascent and update it as you add other systems.

We probably spent a good 5 - 6 weeks at a minimum between two senior engineers to get this feature correct. When we're talking about only having around 36 engineering months across the whole team, eating up 3% of total engineering project time on a single feature is pretty significant. Granted, it could have been a lot more, but when you look at the list of features as a whole, it's a bit scary how much of our schedule this took. We definitely underestimated as a team when we made the schedule.

Overall, this is a pretty interesting look at how many things are interconnected: reconnect and save rely on the same code; reconnect relies partly on how our builds are generated; reconnect and other systems such as dialogue, skills, powers, and so on rely on each other. Not to mention how the feature affects the technical requirements, such as build size, performance, and RAM usage. A bug in reconnect can cause display bugs on the controller, or bugs where the controller and the game get out of sync which would likely stall the game.

I'm glad it's done and it works like a charm, and it was a super interesting feature to work on. We've a few more rough edges to iron out around the reconnect experience, but otherwise, it's been great!
#EonAltar, #GameDevelopment, #IndieDev