Monday, July 27, 2015

Blizzard and the Mythical Man Month; Or Why WoD is a Transitional Expansion

Alternative Chat had an interesting tweet the other day which spurred a conversation as they are wont to do, trying to figure out "what's wrong with Warlords." It's no secret that a vocal portion of the player base is pretty unhappy with the dearth of max level solo content in WoW currently.


The first half of Mists of Pandaria can be seen as sort of the closest to what the WoW team want to achieve: rapid content releases. Having a non-raid patch leap-frogged with a raid patch worked really well, not to mention that each patch was pretty meaty. 5.1 brought in a slew of new dailies, story missions, and scenarios. 5.2, Isle of Thunder, Dinosaur Island, and Throne of Thunder. 5.3 was happy invasion land in the Barrens, and while a little thinner content-wise, was still engaging (plus the Brawler's Guild). 5.4 brought Siege of Orgrimmar and Timeless Isle. Content-wise, by most accounts it almost came too quickly, leaving us with over a year of empty space between Mists and Warlords.

Compare that to Draenor, and post-leveling content has been extremely sparse. Twitter integration, and a selfie camera which "was a fun side project for one designer and one programmer, coming in on the weekends to make a cool toy to go with Twitter integration." Tanaan Jungle, which I actually quite enjoy and has provided a good framework over Timeless Isle in my humble opinion. Timewalking, which most of the tech existed back in Mists. Finally, Hellfire Citadel, which--aside from problematic story moments (as usual)--has been an excellent raid.

So what's the deal? Why is Warlords flailing where Mists actually turned out quite well (ignoring the Lost Year)?


The Mythical Man Month

I postulate that this is the transition expansion. Blizzard is ramping content creation up: they mentioned that they increased the team size by 40% at the end of Mists, and were working on another 40% increase in size from there. Basically, they've nearly doubled the size of their team. I'll talk mostly from the programming angle, but to a large extent this still applies to other disciplines.

At my old job, we tripled the size of our team overnight. Went from about 20 developers to about 60. I was in charge of educating them and bringing them up to speed, so I can tell you with first-hand experience that this is an immense task, and comes at (an extreme) cost to general productivity for the team as a whole. And it has little to do with how good or bad the developers are in general. In fact, almost every developer I worked with when we brought on the new folks were incredibly smart, passionate, and knowledgeable people.

No, the issue with exploding your team size is much more basic than strictly skill sets, and yet far more nuanced than mathematics can describe.

There's a book, sometimes known as the "Bible of Software Engineering," called The Mythical Man-Month. This book is basically required reading for project managers (and if it's not, it should be). The basic premise of the book is the following:
Adding manpower to a late software product will make it later. -- Brooks' Law
The book itself goes into the myriad reasons why this holds true, but the primary reasons are it takes time for new employees to get up to speed, and more employees means an exponential increase in lines of communication


Onboarding New or Transferred Employees

By far, the biggest short term cost is getting all these new folks up to speed. Each team has their own processes, their own tools, their own hierarchy, their own codebase. Established employees know the history of the product, and they know the bugs, quirks, and odd spots in the code. An established programmer might know intuitively that if they change something in the animation engine, for example, that they may need to double check the sound because of how those two systems might be intertwined in that particular product.

Think of it this way: let's say you were brought on to write a book set in the World of Warcraft universe just after the events of Warcraft III, but you knew little about it aside from, "I've heard of Warcraft." Sure, you know how to write a decent story, you know English, and are decent at self-editing. So you have the skill set, but you don't have the context.

You'd need to research what lore already existed around the subject, you'd need to make sure what you wrote could sit in the timeline correctly without accidentally introducing minor bugs (like getting Jaina's eye colour wrong), or worse, accidentally introducing things where you'd have to retcon entire chunks of the story to make fit. The Warcraft lore universe is massive. And you don't really know your way around.

So instead, you get the basics from someone who knows everything inside and out, and you pester some folks for questions because you're still not sure if you got the details correct. The experienced employees take time out of their day to fix your mistakes and guide you through your work, so not only are you really slow to start with, but your work impacts the efficiency of the experienced employees.

This is absolutely true for programming, art, or design, among other disciplines as well. For programming in particular, one spends a lot of time coaching the new/transferred programmers. That's not to say the time is wasted. Eventually those newbies will become productive team members on their own, but adding them to the team can and will reduce that team's throughput in the short term.

And note, short term can mean anywhere from 3 months to a year. Depends on the complexity of the product. Even experienced, senior programmers coming into an existing codebase can take upwards of 3 months before they're productive enough to work without someone established looking over their shoulder (or bailing them out of trouble). It's not that they aren't smart or accomplished; it's that they just don't know the source material well enough yet to not accidentally goof up without realizing it yet.


Doing the Math

If we assume that for each new employee, it reduces existing team throughput for an established employee for an average of 50% for the first 3 months (which in practice is a conservative estimate, from what I've seen), then having a team increase in size by 40% means you're looking at reducing overall individual throughput by about 20%. And no, these new employees aren't contributing enough to overcome that throughput loss, at least not in the first few months. They're often making mistakes and bugs that the more established employees need to compensate for (aka negative contributions).

And of course, from there it's not like it spikes to 100% efficiency, either. You'll have a gradual ramp back to 100% efficiency, but ideally you'll be at greater than 100% relative to the old team size eventually.

But it's never that easy. First of all, it's only the most experienced transferred employees who'll ramp up in only 3 months. But often, they're pretty independent by that time. No, generally for the average programmer it seemed that about 5 - 6 months was required before they could do work largely independently (assuming they're doing stuff that isn't isolated). For really fresh out of school employees (brand spanking new), ramp-up takes about a year, usually. Sure, they'll make features, but they'll be pretty bug-ridden and still need a lot of guidance.

The other gotcha here is that often the established employees doing the education will be the ones with the most experience in the code and/or product, which means that you're actually taking a bigger than 20% overall hit.

Let's say a programmer with 10 years of experience is about four times as productive as a programmer with 2 years of experience. Interestingly, Brooks mused that a good programmer was 5 to 10 times more productive than a mediocre programmer, and that fits in my experiences as well. If you're using your super productive people to teach the new folks incoming, then that 20% hit might actually be closer to 40% because not every employee is equal.

So by adding a bunch of folks, Blizzard would have tanked their individual throughput anywhere from 20% to 40%, depending. And they did it twice by doing two major hiring waves (because chances are, the people who did onboarding to begin with probably were tapped to do it again the second time).

That all doesn't take into account the re-structures Blizzard would've had to perform--that many new/transferred people means new managers, new communication structures, and so on. Those alone generally tank productivity as the rank and file try to figure out what's going on and educating their new leads on what they've been doing. It also doesn't take into account the standard "forming; storming; norming" that would occur when introducing that many new people onto the team, which would also exacerbate the ramp-up time.


The Lost Year of Mists

So when can we as consumers expect the fruits of Blizzard's labours here? It takes a while for content to work through the pipeline. While it looks like a bunch of stuff happened last second on the PTR just before Warlords was released, you can bet most of that work was planned months in advance (and then iterated on later). Art assets had to be completed, game systems programmed, server tech vetted, the entire thing QA'd. We knew they were working on Tanaan as "early" as before Warlords launched, so what we have today was started probably at least 8 months before the patch was released.

The lost year of Mists was quite likely half a write-off from a production perspective. I don't know when they'd have done the second surge of new employees, but it would've reset the clock on the productivity dip.

Which would explain why Warlords felt so anemic. Leveling was amazing, Garrisons was a lot of "content", but most of it is random/programmatic. It likely was more programmers and designers than art--which makes it pretty attractive as the programmers probably needed something to do while everyone else was building the Draenor continent.

But the daily hubs? Even Celestalon Watcher said they were half-assed. Flight wasn't a thing, so that'd have reduced the load on the art/design team for building the continent (which they're paying the deferred cost now, hence why flight is delayed). No scenarios. Proving Grounds literally just got adapted to the scaling tech.
Edit: Whoops, Watcher == Ion, Celestalon == Chadd.

The Next Expansion

But we know they were already working on the next expansion even before Warlords shipped, so I think we'll probably see an expansion announcement at Gamescom. We'll also likely see a beta server ready to go by Blizzcon in my opinion.

Honestly, software takes a long time to make, and organizations take a long time to readjust--especially ones as large as the World of Warcraft team. So while people may be getting antsy about Blizzard's ideals of an annual expansion at Gamescom in 2013, they're probably only now getting to the point where the hiring and training they've done will become visible to us consumers.

Blizzard lost a lot of ground, and Warlords likely suffered because of these huge organizational changes. But the engine should be primed and I'm looking forward to seeing what the next expansion brings. Blizzard has a lot to (re)prove, and hopefully they'll be up to the task. #WoW, #GameDesign, #Expansions

34 comments:

  1. I wish Blizzard and World of Warcraft and the game's truer fans luck in whatever the future holds. Maybe it'll be roses and sunshine. Maybe it won't. But I feel like I got fleeced by this expansion, overall, and I regret ever being excited for it, purchasing it, or following its development to see if things turn around.

    No use crying over spilt milk or failed expansions though. Wipe it up and move on!

    ReplyDelete
    Replies
    1. Yup, I don't disagree with this at all. Blizzard done screwed up this expansion. A lot of folks voted with their dollars and unsubscribed. It'll be interesting to see the next quarter's subscriber numbers.

      I also would fully expect consumers to be wary. Fleeced is a good term for what I see a lot of folks feeling. They bought into something and that something hasn't really delivered. I'd expect folks to be wary when the next expansion comes out; to the point where I don't think we'll ever see a singular jump like what we saw at the beginning of Warlords ever again.

      Delete
    2. I have to wonder if Blizzard permanently damaged relationships with certain people with Warlords. It will be very interesting indeed to see the next few months to see what they intend to reveal, because if they don't announce an expansion until Blizzcon and then the beta doesn't start immediately afterwards... we're back to the 12 months of the same content problem.

      Delete
  2. I'd love to believe this article but at this point, I just don't. I dont trust that blizzard has some master plan or that this was all by design. The percentage of features cut this expansion must be close to 80%. They seem to figure it out (wotlk), then screw it up (cata), then fix the mistakes (mop), and then take their success for granted (SoO 14 months, wod). Even if the next expansion is good, I can't trust that they won't screw up halfway through, or at the end, or with the expansion after that. The damage is done for a lot of ppl and I'm not sure they will ever see 10 mil again. The anger over this expansion is all over their forums.

    I'm not saying wow is dead... It isn't. But it seems like no mmo will ever be the "wow killer". It appears the eventual wow killer will be blizzard

    ReplyDelete
    Replies
    1. I dunno, FFXIV is coming up pretty hot behind WoW these days, subscriber numbers wise, anyhow.

      They'll screw it up again one day. No doubts about it. I'm a game dev and I know I'll screw it up, too. Mistakes are made all the time.

      Unfortunately for WoW, being the market leader and an organization that swings around like the Titanic, mistakes take some time to work through their pipeline, as do solutions.

      That being said, I don't blame folks for being cautious. Heck, as a consumer, *I'm* cautious at this point. But the other thing to remember is that there's the organization: engineers, designers, etc. and then there's the game: story, raids, dungeons, etc. They're not entirely separate, but I think they can have a master plan for their organization even if they don't have a master plan for say, the story of WoW.

      Delete
    2. They've just dropped the ball on too many things. Flying STILL isn't allowed. I don't care about flying but good lord... What happened to keeping a promise? This is where the mistrust comes from. They can't even stick to promises they made "weeks" ago.

      Delete
    3. Honestly this expansion has been a massive failure in so many areas. Their beta testing, which I was apart of, was an utter disaster. So little feedback was listened to and communication was nonexistent. Obviously not all feedback is correct, but the degree to which they ignored(?) the testers was astonishing. Total breakdown of communication and then terrible results.

      They also made some really bad decisions. The messed with the only content they really have (raiding) by creating mythic 20 but Ion refuses to believe it's a problem. They also cut literally almost everything. We went so many months without content and then we get a raid and a crappy timeless Isle.

      It's pretty clear they completely abandoned this expansion very soon after launch. Development died because they moved onto the next expansion.

      If that next expansion is another swing and miss... Wow won't recover

      Delete
    4. Flight is on the PTR as we speak? And they can't just flip a switch and turn it on, they need to test and fix geography for flight. That takes time.

      That being said, I've heard the beta testing feedback getting ignored from several folks now, which lends further credence to the idea that they were effectively understaffed for this expansion. It's not an excuse; rather, it's an indictment of their upper management for screwing the pooch.

      I agree, however, that if the next expansion is a swing and a miss, they're probably in big trouble as a franchise.

      Delete
    5. Being "on the ptr" means very very little. Who knows how long a PTR can last...assuming flight is even on the ptr atm. There was actually a decent forum post I saw that brought up some good points. And it makes for some valid concerns. http://us.battle.net/wow/en/forum/topic/18300227144

      TL:DR: "The new expansion had it's groundwork laid while the devs were doing pretty much everything wrong, at a time when REVERTS were the most celebrated form of "new content" and as such, I'm setting the bar very low for expectations, with the hope I will be pleasantly surprised." - quote from the OP in the link.

      Delete
    6. "The messed with the only content they really have (raiding) by creating mythic 20 but Ion refuses to believe it's a problem."

      I'm a Mythic raider and I don't see how it's a problem. Raiding seems better than ever right now for all difficulties. Flexible Normal/Heroic that's cross server and tightly tuned Mythic without 10v25 problems.

      Delete
    7. Youre a mythic raider and you dont see a problem...well no offense but of course you wouldnt see a problem. The problem is other guilds who are unable to find enough ppl to mythic raid. Its competitive out there and nobody wants to join middling-level guilds. The problem with Mythic is the requirement for 20 ppl. If that were reduced to 15 we would see a LOT more guilds raiding mythic. Maybe not to the end bosses...but at least 4-5 mythic bosses per tier.

      Delete
    8. Just I understand your argument...

      "The messed with the only content they really have (raiding) by creating mythic 20 but Ion refuses to believe it's a problem."

      is a true statement.

      "The messed with the only content they really have (raiding) by creating mythic 15 but Ion refuses to believe it's a problem."

      is a false statement (if Mythic had been 15 people instead of 20).

      Is that correct?

      I mean, part of the whole point to flexible Normal/Heroic is to let raid groups organically grow up to Mythic over time. Don't have to sit anyone and can keep recruiting.

      I'm not even sure I'd really care about Mythic being 15 vs 20 man overall but I'm honestly wondering why you think it makes a big difference. I'm guessing that if Mythic was 15 man then we'd have people complaining about how it's not 10 man and how hard it is to grow up to 15.

      Overall, the percentage of people (like me) willing lead/organize Mythic guilds is an extremely small percentage of the raiding population -- and from that perspective it makes a lot of sense to have Mythic be larger as that increases the ratio of raiders:raid leaders.

      Delete
    9. Mythic 15 vs 20 is an enormous difference. Im not going to waste my time typing up the reasons why because its obvious. Look at how many heroic 10 man guilds there were before WoD vs 25 man. Setting mythic to 20 was stupid considering how many more 10 man guilds there were. Expanding an 50% (10 -> 15) is realistic for most guilds. Expanding 100% is not.

      Delete
    10. "Im not going to waste my time typing up the reasons why because its obvious."

      I'm a Mythic raid leader who formed and led a 10 man Heroic guild at the start of Cata and transitioned to 25 after killing Heroic Garrosh in preparation for Mythic. It's not obvious to me.

      "Look at how many heroic 10 man guilds there were before WoD vs 25 man."

      What ratio do *you* think it was at? I mean, keep in mind that you need 2.5 times as many 10 mans as 25 mans just in order to break *even* on population. That's an absolute minimum.

      "Expanding an 50% (10 -> 15) is realistic for most guilds. Expanding 100% is not."

      How is it not realistic with the flexible system?

      Also think of it in reverse -- if you're a 25 man then shifting down to 20 man is realistic, but shifting from 25 to 15...damn. Lot more friends you'd have to cut.

      Delete
    11. Its not realistic for most because they cant get the ppl. This has been argued over and over. If you cant see it then nothing I will say will convince you.

      Your guild accomplished it and now suddenly you cant see why its a problem. Come on...

      Delete
  3. "...so I think we'll probably see an expansion announcement at Gamescom"

    We have a winner. Nice call.

    ReplyDelete
    Replies
    1. I figure if I keep saying they'll announce an expansion, I'll get it right eventually :P

      Delete
    2. The announcement coming so soon and outside of blizzcon is a huge signal of how bad the quarter two subscriber numbers will be. I can almost guarantee it.

      Delete
    3. That's preposterous. If you think any company can spin on a dime like that for an announcement that big, you're extremely mistaken. This announcement likely would have been planned a few months in advance at least.

      They need to build media--like logos, icons, screenshots, possibly teaser trailer videos, etc. They need to have trademarks in place, CMs ready to answer questions on forums, etc.

      There's no way this could be in a knee-jerk reaction to the previous quarter's subscribers.

      Delete
    4. Not even the "slightest" preposterous. You're assuming they don't know the subscriber numbers day to day. I'm positive they do. I'm guessing they noticed a severe drop in subs about 4-6 months ago and ramped up their efforts to make the expansion asap.

      This isn't knee jerk at all. This was planned my this and months ago and explains why so many features were cut.

      Delete
    5. "This was planned months ago"*

      Delete
    6. Or perhaps if they wanted to make sure they could drop a new expansion in just over a year, Gamescom or PAX Prime would fit the timing for the announcement perfectly.

      That's not to say that they haven't necessarily changed directions a little after they saw how WoD turned out as your other comment suggested, but without looking behind the curtain we'll never know that.

      At this point either of us have any hard evidence for our theories, just the public data, and I believe that the data fits my theory more soundly, especially with my own experience in large software companies. Basically, I'll have to agree to disagree with you here.

      Delete
    7. Nothing to disagree with. Its all interconnected. They saw low sub numbers, estimated how long it would take to make a proper announcement for an expansion and targeted Gamescom or PAX. They knew that waiting for blizzcon to announce their expansion would be way too long. It doesnt require experience in anything to be able to pull back the curtain on decision making.

      Delete
    8. Blizzard has been saying they've been working on trying to get yearly expansions for a long time now. Your guess would make a lot more sense if Blizzard said they wanted an expansion every two years and suddenly changed that to one year presumably due to the state of WoD.

      Delete
    9. Not really. They couldnt switch gears that quickly. Is it any wonder that subscriber numbers will be announced on 8/4 and gamescom starts 8/5? Not coincidence.

      Delete
    10. Sub numbers down to 1.5 million. Now sit at 5.6 million total. Like I said, this announcement on Thursday was planned LONG ago.

      Delete
    11. "Like I said, this announcement on Thursday was planned LONG ago."

      Neither Talarian nor I disagrees with that statement.

      Delete
    12. "Like I said, this announcement on Thursday was planned LONG ago."
      Of course it was. I'm just saying it was planned longer ago than even you were saying.

      Delete
    13. It wasnt. Looking at how half-assed it was...it wasnt. It was basically demon hunters, artifacts and garrisons 2.0.

      Delete
  4. Thank you, Talarian.

    I admit I've been getting really frustrated lately ABOUT the negativity I've been seeing. Especially since most of it comes down to people effectively saying "I refuse to be even semi-social in an MMORPG and therefore THERE IS NO CONTENT."

    And that's not even counting the people who are proclaiming that the Wrath model of running the same five man faceroll dungeons for two years to get (increasingly better) raid quality gear was the BEST THING EVER.

    I mean, I get it, some people want more solo content. And I liked 5.1 during MoP, to be clear. But the way a lot of people are talking WoD must have shipped with one leveling zone, two dungeons, five max level quests, and three raid bosses.

    ReplyDelete
    Replies
    1. If we compare WoD to Mists or FFXIV, WoD's content--aside from raids specifically--is anemic. I'm not going to pull punches on that particular diagnosis. 5-mans got shafted by poor reward design, which Myhtic 5-mans have attempted to corroborate, but then he devs failed to allow for rando-queuing, putting that content outside the casual reach of many players. That and Mythic 5-mans are over-tuned for the average player (but oh my god, I love them so much, completely serious!)

      Most of the issue with WoD's content is execution. Ashram, a dearth of storyline outside raids, the aforementioned reward longevity of 5-mans, daily zones, and so on. Not enough breadcrumbs leading to those activities.

      I think it was GC that mentioned that fighting the playerbase's mindset for extreme efficiency is difficult. If it doesn't reward something, it may as well not exist for most people for better or worse.

      Delete
    2. Though raiding has honestly never been better in my mind. Well, aside from some of the fights getting a wee bit too complex for normal raids. Iskar's hot potato can go screw itself.

      Delete
    3. *ameliorate, not corroborate. Also, so many typos.

      Delete
    4. "Most of the issue with WoD's content is execution."

      Aye. I mean, the game launched with *62* solo quest achievements for doing solo stuff in the outdoor world. But because they didn't give shiny gear, GC's statement meant the people complaining about a lack of solo content ignored them.

      "Iskar's hot potato can go screw itself."

      You can literally ignore the wind mechanic on normal and not pass it at all >.> Just sayin'!

      Delete