[IndieDev] Checkpoint Saves: Ugh, Why? And How, Part 2

Last week I chatted about the start of Eon Altar's save system, why it didn't work, and how we fixed it. This week I'll go in-depth about Eon Altar's Checkpoint Save system. 

Fast forward nearly a year from our new save system implemention--Aug/Sept 2015--when we finally entered Early Access. The game was probably about 80% functionally complete and 60% content complete. As I like to say, the last 20% of your game will take about 80% of your time, and Eon Altar was no different. We spent 10 months in Early Access, and initially, the biggest point of feedback we got was, "How can I save my game mid-session?". Our sessions were about 30 minutes to 4 hours depending on the players, and in 2015 shipping an RPG without the ability to save mid-session was, well, pretty bad. So began the process to create a checkpoint save system, and retrofit our levels to save data correctly.


Checkpoint Saves: Less Complex?

Why checkpoint saves, though? Why not save anywhere the player wanted? The answer to that is largely to reduce potential complexity. If a player can save anywhere and anytime they want, it means you have effectively an infinite number of states, and good luck testing that. A specific example of this would be Myrth's Court in Episode 1: The Prelude.

Myrth's Court

That "moment" as a whole had the following:

• A check to see which player characters were available.
• A dialogue based on that to posit a vote.
• A vote to decide which character's solution to use.
• The actual moment where the party implements aforementioned solution.
• Potentially a combat as a result of the solution.

If players could save at any point in that process, that would significantly increase the testing complexity around that moment. What happens if you reload with different characters mid-moment? What happens if you have fewer characters? More characters? By only allowing saves to occur at specific points in the level, we can avoid having to test those mid-moment saves.

By using checkpoint saves, we could tie them to an existing checkpoint mechanic we had in the game already--Destiny Markers/Stones. Again, not having to worry about partial encounters is a huge complexity save, but also not worrying about how to turn on/off saving in certain locations. What if we had a bug that prevented save from being turned back on? Or a bug that allowed saving in the midst of a complex moment? Also, how do we communicate if we can save or not to players? And what would the save UI look like? By tying it to an existing checkpoint mechanic, it made it very easy to communicate and very easy for players to grok. No special rules or explanations necessary.

So while checkpoint saves aren't as convenient for players, the reduced complexity was enough to make checkpoint saves doable with our small team and budget.


How to Train Your Save System

We already had a method to quickly save data to disk, and the checkpoint save system would continue using it. The questions then became, where do we store that information at runtime so designers could access it, and how do we design it in such a way that required as little designer input/time as possible?

First we had to determine what we would have to save:

1. Enemy spawner state: were they dead or alive?
2. Game object state: was it enabled or disabled?
3. "Usable" object state: was it waiting or already used?
4. Finite State Machine (FSM) state: what state was it left in?
5. Specialized game object state: what is the game object's transform (position, rotation)?
6. Specialized spawner state: what is the enemy's transform (position, rotation)? What is the enemy's AI settings (aggressive, passive, patrolling; allied to players, or enemies; patrol state)

With those 6 items, we could literally save anything and everything in our levels.

I created specialized game components that could track those states and report them to the save subsystem as they changed, so we wouldn't have to trawl through level data to extract information--remember, we wanted to ensure the save system was fast. All the designers had to do was add them to an object they wanted to save that particular state out for, and give it a unique ID (well, my code autopopulated the ID based on a random GUID and the name of the object in the hierarchy, but the designers could override that if they chose).

This worked extremely well. Design quickly retrofitted our existing levels. The vast majority of our save data is items 1, 2, and 3. FSM save data is rarely used unless the FSM is long-lived (our Destiny Markers are the primary users of this tech). Most FSMs would trigger and finish in one go, or at least in one encounter so we'd not have to worry about partial FSM execution by the time we got to hit a save point (yay checkpoint saves!). 5 was almost never used outside of redirecting patrol nodes for NPCs, and 6 was generally only used on super special NPCs: ones that changed their AI based on designer scripts, or NPCs that were used for escort quests.

Wild Checkpoint Data draws near!

The code took about a week to create/test/deploy for design. The lion's share of the time (and bugs) was designers retrofitting levels. I think it was easily a full man-month of time to get the levels up to snuff, and the amount of testing required was still absolutely immense, despite the reduced complexity of checkpoints.


The Bugs
 
A pitfall of this--and I'm not sure there's an easy way to solve this pitfall, I don't believe it's specific to this solution--is when designers forgot to put save components in levels, or they chained components in such a way that would create a problem on game load.

A specific example of this is a door in Episode 2, Session 1. Level design logic had the door with the following states: unopenable, locked, unlocked, open. Depending on the quests you did in the level, it could become locked, unlocked, or open. However, if you saved and quit and reloaded later, then the door would be unopenable because the door wasn't actually saving its state out, and players would become blocked.

Now, when we ran into those issues, we would add the save component in the level data, and then use code that ran on save data load to modify the data before it got applied to the level itself. Basically, we could determine based on what other quests were complete and save object states if the door should be locked, unlocked, or open, and set that state in the upgrade code.

Today we have 10 such save file upgrades that potentially run on a save file to give you an idea of how often we've had to use this, and the lion's share of them are for Episode 2 Session 1. Enough to make me glad we implemented it, but just how different E2S1 was from the rest of the levels really showed how easy it is to screw up save state if you're not careful thinking about it holistically.


The Future: SPARK: Resistance

SPARK won't have need of checkpoint saves, as sessions won't last more than 10-15 minutes at a maximum. Rather, any save data will be related to your "character". Unlocks, experience, statistics, etc. Thankfully, I'll be able to take our save system nearly wholesale from Eon Altar and apply it here, minus the checkpoint stuff.

A Randomly Generated Map and Associated Data

The in-level checkpoint stuff wouldn't work in SPARK anyhow, as the level structures are fairly different to start with thanks to both the procedural nature of the levels as opposed to hand-crafted, and the fact that the levels are networked right from the start, which is very different from a local multiplayer game.


Conclusion

The current save system in Eon Altar is robust, extremely fast, legible, easy to modify, and minimalistic in data requirements aside from the fact that it is XML, but the actual data output is all essential. It requires as little designer input as I could possibly get away with (even most checkpoint save data is attached to prefabs and autopopulates all IDs in the scene at the click of a single button). 

Yes, it took a fair amount of engineering work altogether, but I think that's a result of you just cannot skimp on engineering for a system like this. You get what you pay for, and if you're not willing to put the engineering time in, you're not going to get a great system on the other end. And as mentioned at the beginning, persistence is extremely important to games. A game can't afford to skimp on their persistence systems in my personal opinion.
#IndieDev, #EonAltar