[Eon Altar] Bottled Awesome--Reconnecting Mobile Devices to the Game

One of the things that makes Eon Altar unique is we use mobile devices--phones and tablets--as controllers to the primary game. Of course, this kind of connectivity means that some state needs to be communicated to the mobile devices as you play. But even more concerning is how to get all that data back to the mobile device when it crashes, or you answer a text and iOS kills the app due to memory pressure, or your battery runs out.

Flying Helmet Games playing a test build on their phones.

Interestingly enough, we went for a very long time without a robust reconnect feature, which actually spoke pretty highly of the quality of our controller builds, but now that we have one, I definitely regret leaving it as long as we did.


The First Solution

The hardest part about reconnecting, of course, is what I hinted to in the first paragraph: ensuring your state on the controller is correct. The easiest way of doing this, without having to think of any special logic is to copy/paste the state from the main game onto the controller over the network. In programmer terms, serialize the state, and deserialize it on the other end. Given our time/budget constraints, we thought this was a pretty good idea.

However, once we implemented it, it turned out to be terrible in practice. The performance was abysmal, primarily because of how we organize our data on the main game. When we serialized the state, we also ended up picking up character models, animations, textures, and so on. And not just for your character, but for all player characters. We had reconnect times upwards of 8 minutes per controller when playing with 4 players. That was...not so hot.

We left the problem for a couple months, though, because we had more pressing needs in keeping the designers unblocked (which is another interesting discussion entirely). We'd lived that long without reconnect, what was a few more months?

Once we got back around to it, it was also clear from a design perspective that what we were doing wasn't going to hack it. Sending all of that data to the controllers was far too much on 512 MB devices. We were also using the same code to save our game to a file, and that was problematic due to the fact that patching the character or level data after the fact was pretty well impossible, not to mention save files bordering on 1GB+ of data.


The Final Solution

So serializing the state naively wasn't going to work, which meant that we were going to have to dig in and actually reason about what data we needed to send across (or save to disk!). Turns out it wasn't too bad at all.

Thanks to Unity's RPC framework guaranteeing delivery and order, it "simply" became a matter of writing a new network model on top of Unity's to decouple how they instantiated objects over a network, and then reinstantiate the player actor we needed on the controller. Since we knew what we started with baseline, it then became an exercise in determining how that state changed from the baseline. What abilities do you have unlocked? Gear and recipes? Treasure? Renown? Current attributes and status effects? Skills? Lore? Are we in the middle of a dialogue?

It was slightly more complex than that in the end, because we have a bunch of other synchronized tracking variables that had to be transferred as well, but most of our "state" in the previous paragraph are just prefab objects, so the majority of controller reconnect rehydration is telling the controller, "Hey, go instantiate this actor, and then go grab prefabs A, B, and C for lore, and unlock skills 1, 2, and 6. Also, update these synchronized variables." It's simple, and quite robust.

Unity Project Setup

What also helps us here is the fact that our Controller app and our main game executable actually share resources and code. Unity is (mostly) smart enough to only pull in assets based on what scenes we include, so in the picture above, when we build the game executable, Unity pulls in a bunch of data to spit out for the build, and if we build the controller app, same deal, unity pulls in the correct data.

I say mostly because the Resources folder is special in Unity. We had to do some...shenanigans around Resources folders to get this system to work, and it's not quite perfect. but in terms of reconnect, it's great because when reconnecting, as long as the builds are compatible, I can be guaranteed that the correct data will be there on the controller. And if not, oh well. Think of it as the equivalent of updating tooltips in WoW. The backend server data might change, but the tooltips are local client data, and may get out of date. A patch will fix that right up later.

There are definite cons to this setup as well, like requiring patching the controllers more often, and the Resources folder shenanigans means sometimes extraneous data gets pulled into the controller build. But it's also super fast from a CPU time and networking perspective. We're not serializing tonnes of data to send over the (local) wire, so the reconnect now takes about 1 - 3 seconds rather than 8 minutes. Actually, the grand majority of the reconnect time is actually just loading the controller UI textures, the same as a normal load rather than anything reconnect specific.


Bottled Awesome

The best part about the latest version of reconnect? It's ultra solid. Our lead designer told me it was "bottled awesome". Pretty much any controller app bug we get can be worked around by killing the controller app and reconnecting, which has been an immense boon now that we're in Early Access. 

We haven't seen many bugs come out of it either, which makes me pretty proud of the quality of the feature as a whole, too. The lack of bugs isn't for a lack of testing, either. We use reconnect all the time. I actually really regret not having a robust reconnect completed much earlier in the project. Mind you, part of why it's so robust is probably because it was done in a single 2-week period after most of the game was already complete. It's "easy" to build something like this and get it right in a single go rather than finish something nascent and update it as you add other systems.

We probably spent a good 5 - 6 weeks at a minimum between two senior engineers to get this feature correct. When we're talking about only having around 36 engineering months across the whole team, eating up 3% of total engineering project time on a single feature is pretty significant. Granted, it could have been a lot more, but when you look at the list of features as a whole, it's a bit scary how much of our schedule this took. We definitely underestimated as a team when we made the schedule.

Overall, this is a pretty interesting look at how many things are interconnected: reconnect and save rely on the same code; reconnect relies partly on how our builds are generated; reconnect and other systems such as dialogue, skills, powers, and so on rely on each other. Not to mention how the feature affects the technical requirements, such as build size, performance, and RAM usage. A bug in reconnect can cause display bugs on the controller, or bugs where the controller and the game get out of sync which would likely stall the game.

I'm glad it's done and it works like a charm, and it was a super interesting feature to work on. We've a few more rough edges to iron out around the reconnect experience, but otherwise, it's been great!
#EonAltar, #GameDevelopment, #IndieDev

[WoW] The Show Must Go On -- Agile Development vs. Content Pipelines vs. Social Media

Alternative Chat has an interesting blog post where she talks about the tension between what some devs call "open development", and the fact that Blizzard is basically the Titanic as far as steering the product goes.

For those not in the know, #RaveholdtOrRiot is a hashtag used by the WoW Rogue community who're upset that the Rogue Class Hall was said to be in the sewers of Dalaran. See, Ravenholdt was the defacto Rogue central in Vanilla, and that was reinforced by the legendary daggers quest at the end of Cataclysm largely taking place in and around Ravenholdt itself. When you have Blizzard stating that Paladins are going to get Light's Hope Chapel in the Plaguelands, and Shaman are going to get a cavern system by the Maelstrom, of course Rogues will want something that's tethered in their history as a class, and not in the basement of the floating Mage city.

@Muffinus @FlyingWorgens @WatcherDev To be fair Muffinus, historically, sometimes waiting that long for changes, results in it not happening

— Lesliechow (@lesliech0w) September 15, 2015

 
Timelines

Of course, as Alt points out--and Blizzard has confirmed multiple times--Legion has been in development for well over a year now already. Since before Warlords was even released. And despite Blizzard's insistence for folks to wait for the full reveal, as the tweet above indicates, waiting often means it's too late. By the time it's polished and ready for reveal, there's no going back. In fact, it's honestly probably already too late today.

I hinted at this a year and a half ago with my post Should Monolithic Expansions Be Obsoleted?. There's just way too much pipeline in a giant expansion in a giant game like WoW to switch directions. A timeline might actually look similar to this:

• 2012 -- Figure out if Warlords or Legion should come next, begin planning Warlords
• Sep. 2012 -- Mists Released
• 2012 - 2013 -- Heavy work on Warlords, Begin Planning Legion
• Nov. 2013 -- Warlords announced
• 2014 -- Heavy work on Legion, Presumably planning next expac
• Nov. 2014 -- Warlords released
• Jul. 2015 -- Legion announced

The groundwork for Legion started 3 years ago, probably slightly more. Based on my experiences as a game developer and as a software engineer, plus the data they gave us via the linked interview above, the timeline for Legion might look something like:

High Level Planning (2012)

• Basic Concept
• Basic Story/Continent/Raison d'être, Concept Art
• ~1% of staff

Mid-Level Planning (2012 - 2013)

• Zone flow, dungeon concepts, story flow
• Feature planning (ie: artifact weapons, class halls, PvP talents)
• Concept art for zones/dungeon/major story scenes
• Engineers start prototyping features
• Ramp to ~25% of staff by next phase

Low-Level Planning, Level Design, Feature Coding (Late 2013 - 2014)

• Start fleshing out individual zone design, quest hubs
• Fleshing out dungeon/raid designs
• Blocking out the above in rough
• Start coding features and new tech for server/client
• Artists start turning concept art into in-game assets
• Ramp to ~50% of staff by next phase

Full Production (2014 - 2015)

• Zones get filled (interview linked above states they had 2 zones in production by 2014 Gamescom) with terrain, art assets, quests, quest hubs
• Features get design iteration with engineers
• Dungeons get similar treatment to zones
• Classes get design attention for new abilities/talents, may need engineering help
• Artists churning out artworks/music/sound effects for design to use/by design request
• Probably ~75% of staff

Beta Release (Late 2015)

• Most zones, class halls, artifact weapons will likely be mostly done, aside from certain design tweaks.
• Dungeons and raids may be mostly complete, but missing balance passes, or some encounters, loot, etc.
• Ramping down to ~25% of staff again.
• Artists and engineers are likely already working near full time on next expansion by this point, if not a bit before, leaving mostly designers (quest, encounter, class, systems)

I'm probably off a little bit between phases, and likely pretty off with the staff percentages but eh, I'm more interested in magnitudes than actual percentages. Also keep in mind their production phase was probably side-lined by doubling their staff in the 2013 - 2014 era.

I don't know precisely how Blizzard pipelines their work, but it most definitely is pipelined. Interestingly, graphics cards and CPUs may provide a good analogy to this. The deeper your pipeline, the more work you can parallelize. But if you have to do a context switch? Dumping the entirely of your multiple parallel pipelines sucks hardcore.


Agile Development and Content-Heavy Games

So to get more content out faster, Blizzard has probably built a very deep pipeline. They'd have to, really. There's only so much in a single expansion you can parallelize since you'd have designers waiting on engineers, artists waiting on designers, artists waiting on engineers, engineers waiting on designers, and so on. Having a deep pipeline means less downtime due to waiting on others; you're almost always busy, which is good from a financial and throughput perspective. But having a deep pipeline also means you're very much not agile.

That being said, consumers generally don't care about the why something is not responsive, even if that something is as huge as World of Warcraft. Consumers see an issue (lack of flying, rogue class hall being in the "wrong" place), and they speak up. If Blizzard can't or won't change in response to that--whether it's for practicality of development reasons, or because they want to stick to their guns--then as Alternative Chat suggests, perhaps they shouldn't be interacting with the community on these issues.

Maybe discretion is the better part of valour here, and despite the fact that they listened to the community and changed their mind on flight--something they allowed themselves to do because they developed Warlords with the possibility of flight in mind in the first place and probably pulled a bunch of QA and designers from Legion to pull it off--a class hall may have a bunch of art assets, quests, level design, and the like already done likely a year or more in advance is not able to be changed without throwing out a bunch of work and starting from near-scratch. It's like Karabor vs. Stormshield in Warlords all over again, really.

Using my game, Eon Altar, as an example, having to radically redesign one of our stages is probably outside our current capabilities as a studio. Tweaks to improve quest flow, or encounter design, sure. But re-creating an entire chunk of the level? That'd take our current staff an immense amount of time--time that would be taken from other features/levels/encounters. Time we literally cannot afford. And we have maybe at most 1/30th of the raw content WoW throws into an expansion.

Agile development can be powerful, especially for programming. But for heavy content games, I'm not sure agile is really possible at the content level, at least not for monolithic expansions the way Blizzard does them. For one, you'd think the film and television industry would've figured something out on that by now, and they haven't really in nearly 100 years, and they're a decent analogue to the content portion of game development. For a small team, with a game that's more features than content, like a match-3 game, or Geometry Wars? Yeah, agile works well to change things up really fast. But WoW? Not so much. Even FFXIV's content pipeline is probably pretty deep. It'd have to be to put out patches like clockwork the way they do.


Social Media

Blizzard has a choice to make here: try to make agile work for content and keep responding to content-based feedback as if it matters for the present, rather than as data points for future content; or just...stop responding to that kind of feedback, because right now the halfway point they're hitting feels more frustrating than helpful. People think they have the developers' collective ear, and then folks are told to sit tight, and the devs are clearly frustrated with consumers banging on their window since said consumers don't have the full picture yet (which, to be fair, is Blizzard's fault for giving only a piece of the puzzle).

Something you learn in music theory very quickly is that silence can say as much as any note. Blizzard may need to learn to exercise that when dealing with feedback that cannot be actionable, rather than being defensive. Even if it hurts to see people upset about your work.
#WoW, #GameDevelopment

Party Progression Problems

Recently I've been playing some Diablo III, and it was a lot of fun for a bit of time. A friend and I played together a fair bit, and running some rifts/bounties was a good time, especially once we figured we could bump up the difficulty a bit. Playing with friends usually makes the game better.

I had the unhappy event of having to turn a different friend down who wanted to play with me though. See, my first friend and I were about Paragon level 20, and just peeking over the Torment II boundary. This 2nd friend was Paragon level 580, and trounced Torment VI+ with ease. We literally could not keep up with him. He ran by monsters, and they exploded, and his movement boosts put him well out of our reach. The game devolved into us chasing him around and picking up leavings, which frankly wasn't fun, at all.

I don't blame the 2nd friend at all, and he understood when I asked him to leave that I want to play the game--I'm not interested in power leveling at all. I like to play it on my own terms, which includes friends who are at a similar point in the power curve.

An approximation of Diablo III's power curve.

Diablo III suffers from this issue pretty massively because just how fast power goes up, especially once you're at end-game levels. Honestly, if I had just bit the bullet and let myself be power-leveled, I'm sure I could've gotten close enough to 2nd friend's power level in a few hours, but again, that's not fun. That's a chore.

Interestingly enough, most MMOs suffer from this issue as well. WoW and FFXIV are a couple of the more active games that exhibit this behaviour, but in a sense even a game like Minecraft has this problem. One of my friends joined my Minecraft server, and he's a helper, but I already had everything I ever needed, so him helping was basically useless. He showed up, and we basically had tamed the wilderness, even providing a train stop for his base.


Possible Solutions?

WoW solves it to an extent by basically resetting progression every raid tier, let alone every expansion. You get catch-up mechanics (Timeless Isle, Tanaan Jungle, Isle of Thunder, End of Time Cataclysm heroic 5-mans) to bring you to the power level you need to perform in current content.

This has the problem that it invalidates older content, and has the player base playing only the most recent content with any sort of ardor.


Another option some MMOs use--Guild Wars 2 and FFXIV come to mind--is mentoring of sorts. They drop your power level down to the one that's appropriate for lower level content.

Mentoring works quite well in FFXIV in my experience, though I find losing half my keybinds when I enter a dungeon that's 40 levels below mine really frustrating. Or having to keep Stone I on my keybinds because Stone II might not be available due to the level down system. Guild Wars 2 felt worse in my opinion. Everything you did leveled you down, to the point where it didn't really feel like levels were worth anything. What's the point of having levels at all if you never get to use that power? There are better gating mechanisms than using a number to prevent players from skipping content.


Or as Diablo III shows, you can just have players start over. Seasons performs this admirably--and as a bonus aside, I really enjoy the new task list they have for Seasons so it's not just you wandering aimlessly. Minecraft and Terraria also have this option, where starting over in a new world is easy, and often fun.

For a persistent MMO, starting over isn't a great option, because as mentioned above most of the progression in the game is invalidated by the the resets every tier, and the population doesn't exist to support the game at lower tiers of play. And even games that support starting over well, as 2nd Friend showed, you can still easily find a gulf in your powers that is too large to be bridged short of asking the other person to stop playing while you spend hours to catch up.


Others?

Those are three techniques I could think of off-hand to allow parties to play together without ruining the others' experiences because of a gross power disparity. Are there others out there? Or is this a mostly intractable problem with RPG progression mechanics that we'll only mitigate, but never overcome? I don't really believe it's impossible to overcome, and mentoring comes close to a good solution, but with a few kinks--to the point that WoW has started using it with Timewalking dungeons to great effect as well.

But overall, I wish there was something because it really sucks having to tell your friend, "Sorry, I don't want to play with you."
#GameDesign, #DiabloIII, #Progression

[Eon Altar] Post-PAX, Post-Release, and Developer Empathy

I promise that soon I'll talk about other games (I've been playing some Diablo III and Armello in my spare time), but I did want to touch on how our PAX weekend ended up.


PAX and Early Access Release

Four days is a long time to be standing and talking with folks. I haven't done that much hawking my wares at people since I worked retail back in University like a decade ago. While I was extremely excited all weekend, by day 3 I was pretty strung out, as the Aggrochat podcast mentioned. Day 4 I ended up doing an interview on video with some lovely folks, and the only thing I remember from it is that I'm pretty sure I came off as a little manic and under slept. I don't think it's up yet for perusal, though.

I'm in the upper-left here, demoing our build. This was a relatively quiet period.

I'll be honest, when my coworkers suggested that we release the game on Early Access Steam just before PAX Prime, I thought they were nuts. We're all at PAX, who's going to be on support if things explode in a bad way? Thankfully, the launch went quite smoothly, excepting a crazy bug I had in my server discovery code. The good news is thanks partly to Fierydemise, I figured it out quite quickly and had a workaround up for folks, and then we patched it on Android immediately. Still waiting for Apple to say 'yes' on the build on iOS though.

We ended up playing support in the evening when we weren't on the floor, and it worked out pretty well. Being able to say to folks at PAX that "hey, we're on Early Access right now for $5 US," was a pretty decent draw for folks, so I think it was worth it in the end, even if I was more than a bit frazzled from the attention split by the end of the weekend. Add to that the emergency patch for the IP Address issues the Tuesday after PAX.

Slightly different angle here, getting more of the convention in frame.

I didn't really get to enjoy much of PAX at all as a con-goer, though. I spent 90% of my time at our booth. I got like 45 minutes to wander the floor and get a quick look at the other indie games (which there were some pretty sweet ones there). I also did a lunch with a bunch of other blogger friends from Twitter, some of who are on the aforementioned Aggrochat podcast. And I ended up taking an evening to myself and hung out with Fierydemise, which was a total blast just talking about various things, from eSports to philosophy to programming.


Developer Empathy

If folks follow me on Twitter, or read my blog, you'll know that I'm usually a strong developer advocate. Communication, bugs, game design, etc. I find myself defending devs more often than loud consumers, but now that I'm in the hot seat, I feel I can defend both sides a bit better.

Eon Altar is basically multi-platform in a way nobody else is. You need to have iOS/Android mobile devices to play the PC game. It's a bit weird, and trying to message that hasn't been easy. We've certainly goofed on the message before.

So when we started getting forum posts from people angry about our game requirements, I suddenly got why WoW devs often mention that level of passion might be a good thing, somewhat. These are folks who want to play our game, which is great, but are frustrated that they can't because they don't have the prerequisite equipment, which isn't so great. To be fair, one could post a game that requires the Oculus Rift and get the same sort of response, but the mobile/multiple screen thing really does throw folks for a loop. Again, comes back to maybe we're not communicating why we use mobile devices well enough.

Once I figured out that, it totally changed how I approached people at PAX. I'd very quickly dive in to not only what made our game unique, but why we used multiple screens. I found that turned around the interest factor significantly on the PAX floor, even for folks who don't normally like RPGs or games like D&D.


Active Development

We're back to fixing up the game, and incorporating some feedback we got from PAX itself, as well as folks who've written about or played the game already in Open Access. The feedback has been absolutely fantastic. Some things we need to improve, and a few places where we've nailed the experience. The next month promises to be busy, but I can say that manning the booth at PAX, and doing Early Access on Steam? Both immensely worth it already. #EonAltar, #PAXPrime, #EarlyAccess