Thursday, January 7, 2016

[IndieDev] The Omniscient Programmer

We've been in Early Access for Eon Altar for a good 4.5 months now, and as we get closer to releasing, it hasn't been an easy transition ramping down. I've been the Lead Programmer since April 2015, and recently I've been the only professional programmer on the team. Even when we had a team of 3 programmers, it necessitated a very generalist approach as we just haven't had the number of staff to allow for deep specialization. It's helped that some of our non-programmers know some programming (our lead designer and our animator specifically), but they've not been entirely self-sufficient in that arena--and nor would I expect them to be any more than I'd be expected to be self-sufficient building an animation.

However, today, as the primary technical person, I am expected to know, well, everything technical.

It's a crazy challenge, but an awesome one as well, especially on a game as relatively complex as Eon Altar is. Compare that to my time at Microsoft where I could afford to go really deep on a single problem at a time, like designing and implementing a replacement for an old, inefficient proprietary database; or syncing two file sync engines to the same cloud endpoint. Granted, those are both incredibly complex and nuanced areas, but I had months, even years, to really get everything correct. Eon Altar, our timeline, feature set, and low staff numbers didn't allow for that amount of depth or complexity.

With almost no one aside from the Internet to fall back on at this point excepting a few very specific cases, I need to understand the entire codebase, and be able to make changes within it with confidence.


Being a Programmer

At the lowest layer, this coincides with the sort of things most professional programmers should know at some level: coding and debugging, memory management, algorithmic complexity, language features (C++ vs. C# vs. Javascript, for example), performance profiling, optimization (space and/or time), multi-threading, networking, testing. If one were to compare programming to another job, like a civil engineer, this would basically be "how to build a basic building with infrastructure that won't fall down under the stresses of the natural world".

Nothing too special, though having extra knowledge of how code looks and performs when it's compiled; how different hardware such as graphics cards, processors, RAM, and hard disks changes performance characteristics; how the operating system you're working in handles memory and disk writes; how your garbage collector works if you have one; and so on can help make your code even better--and part of why coding for consoles which largely have set hardware capabilities is actually pretty appealing.


Being a Unity3D Game Programmer

To be a game programmer specifically, and being the only programmer, it means I must understand how every aspect of games works, including rendering, sound, physics, time, animation, AI, pathfinding, cameras, and runtime asset management, just to name a few major systems. We can't have one programmer working on rendering and animation and a different programmer working on AI and pathfinding.

That's where a system like Unity3D or Unreal Engine both come into play, because all of those things are written for you. But you're still building on top of the tools provided, which means you still need to understand how they work, and fairly intimately in many cases--especially when someone decides that the built-in pathfinding tech just isn't good enough.

Some basic camera math using characters on-screen and the camera frustum to calculate the distance the camera needs to be to include all characters on the screen.
An example here would be Unity3D gives us basic camera tech. We don't need to write the code that tells Unity3D how to render a screen based on a camera and a bunch of polygons and textures that the camera can see. But I do need to understand how that works to be able to debug rendering issues, or build a camera controller system, or use cameras to create projectors.

Using our civil engineer example from before, this might be analogous to, "How to build a skyscraper specifically, and all the extra aspects that go into it," like ensuring it doesn't collapse under its own weight, can handle winds at high altitudes, and keeping upper floors cool because heat rises.


Being an Eon Altar Programmer

Then for Eon Altar specific features, I need to understand the level design tools we've created or imported from the asset store and how they interact with everything else, like state machine scripts which control many in-game sequences; character data; dialogue; level loading; UI; Controller-Central Game networking; layered music system; saving and loading games; logging and collecting crash data; analytics; game versioning; combat/action systems; tutorials; lore; camera cage to keep players on screen; combat and power AI; platform-specific features; and I'm probably missing a few other systems.

And then a lot of those systems interact with each other. Save games requires knowledge of player and level state, as does reconnecting controller devices. Game versioning is important to do robustly so we can prevent bad builds or old builds of the game and controllers from connecting to each other. Combat and power AI hooks into character data and the combat/action systems. Camera cage and camera controller both know about player movement and can guide/control it at occasions. Level design tools can inform pretty much everything on the list.

Interestingly enough, being the programmer makes some of that easier to an extent. Despite not writing the original version of some of those features--I didn't write the dialogue system or UI, or the underpinning of the combat/actions, or the sound systems, and while I've had a strong hand in many other systems, I wasn't always the lone programmer on them--because I have intimate knowledge now due to iterations and bug-fixing, writing other features such as save games and reconnect actually went pretty smoothly as I already have all the details I need to know. In a bigger team, I'd have to rely on others understanding what I want to do and getting their feedback on how to integrate with their features.

I'm stretching the civil engineer analogy pretty hard at this point, but this might be analogous to, "How to design extra specialized infrastructure systems within the skyscraper." Maybe I should've picked a different job to use as an example...


Being the Technical Person

Of course, it doesn't end with that. I am the technology gatekeeper, which means I'm also in charge of creating, developing, and maintaining the technological infrastructure surrounding the game, including automating build systems for all platforms; unit testing; patch scheduling; deploying to Steam, iOS, and Google Play; debugging Windows, OSX, iOS, and Android; keeping abreast of changes to the Unity3D platform and keeping it up to date; code repository management, including creating release branches, development branches, and bailing out coworkers who get in trouble with it; differentiating asset management for the exe based on platform (as both our Controller and Central game use the same codebase/project); and of course, technical support for our customers.

Technical support so far involves a lot of debugging customer networks from afar. Are all their devices actually on the same network? Can their iPhone or Droid even see their PC in any app, not just ours? Is something like a firewall eating UDP packets? Why are Zenfone 2s specifically freezing?

We've had a few other bugs that our logging has helped us nail, along with good descriptions from the folks in the forums as well. Add to that automated crash/unhandled exception logging for iOS, Android, and PC (and I had to write something cloud-side for the PC from scratch, which was an interesting exercise), and a lot of what I do today is trying to suss out wtf went wrong at any given moment.

For our civil engineer, this might be equivalent to organizing delivery of materials and workers, scheduling when parts of the building would go up, orchestrating repairs, and answering the phone when someone's toilet got clogged.


Bigger Teams Than Ours

If you were to look at a bigger team, like at EA, the build engineering and deployment would generally be another department entirely, and you'd have specialists on your team for handling platform specifics, or at least mobile vs. desktop. Tech support would still be done by those who owned the breaking features.

You'd probably have different programmers for game logic versus designer tools versus your engine programmers, who'd be handling rendering/sound/asset management/networking, and often would specialize in one or two of those sub-areas. And the more programmers you can afford, the more complex those features could be. A lot of our features are pretty simple, because we literally did not have the manpower to ship anything more complex.

While programmers make up a small portion of most teams (content creators generally being the lion's share), you're still looking at anywhere from 6 - 18 programmers for a major game like an RPG, and usually for many years. Mass Effect, a triple-A RPG, had about 45 in-house programmers over the course of it's development according to the credits. Probably not all at the same time, mind, and Mass Effect is certainly far more complex than Eon Altar (especially since we didn't have to create the engine from scratch), but that gives an idea of team size. Tales of Zestiria seemed to have 10 in-house programmers, and 10 from a contracted studio based on their credit scene. Heck, Super Mario RPG had 5 programmers.

Which all is to say we definitely would never have been able to make Eon Altar without an engine like Unity3D, but even with an engine like Frostbite, Dragon Age: Inquisition still has well over 60 programmers listed in their credits.


Holy Moly So Much Stuff

So how does one keep all this stuff in their head? Well, to be fair, you kinda don't. I hinted at it earlier, but the Internet is a great resource to allow you to make it once you've faked it to a degree. You really do need your fundamentals down pat and largely internalized, but once you start getting into systems such as physics, you kinda just need to understand vaguely how it likes to store information, and how it works on a frame-to-frame basis, and you can usually either puzzle the rest out when you need it, or look up the specifics. Game specific features, thankfully, are documented and/or in code.

A lot of tech support is going, "Oh, crap, why is it broken? Okay, let's try a few general tests to see if it's a problem I understand. Nope? To the Internet!" Build engineering? I knew I had to link a bunch of stuff together, and knew that I could do it in a bash script, so it became a question of, "Let's go learn bash scripting for OSX to tie all this together!" One day I'd like to learn more about Jenkins if I'm going to be doing more automation, as that seems to be the industry goto for that sort of thing based on further research.


My university degree provided me a mathematics background as well as some classes in graphics, physics, and cameras on top of a lot of the "being a programmer" stuff, but Unity3D itself I learned on the job. I've learned a fair bit about software lifecycle management from my time at Microsoft, as well as a lot of other engineering skills that you just don't learn in school. Everything else I've picked up effectively by osmosis, or research when required. And so much research, all the time!

What makes programming so wonderful and terrifying is that every day you're pretty much learning something new: about the tech you're using; about some pattern or technique you've "discovered" or read about; about some security flaw that you now have to patch in your own code; about how to perform some task that you didn't even realize you needed to perform the day before; about some new tech you need to integrate into your product.

The danger in generalizing, of course, is not being able to dive deep enough when a large organization requires more specialized expertise. It also means that it's quite possible for you to know just enough to get yourself into trouble, but not out of it. But so far that hasn't happened to me, knock on wood and all that.

Eon Altar has been a wonderful project to work on, and I've learned so much doing it, both from my fellow programmers when we still had them, and a lot just by going through the motions of building features and debugging issues. We're not done yet, but it's definitely been worthwhile.
#IndieDev, #EonAltar, #Programming

No comments:

Post a Comment