Sunday, December 19, 2010

End of Week 5 Update : A GTA Style Story, or a Counterstrike Style Match?

Games modes - who needs em?

Let's talk a moment about the fundamental concept of APB, and how it is NOT like a multi-player Grand Theft Auto game (aka how it is NOT "GTA Online"), at least not yet.

When the team designed APB back in the day it appears one of the goals for the game was truly to create a Grand Theft Auto "Online" inspired game with two opposing sides duking it out for supremacy. No problem with that in theory. Except that a multi-player GTA clone that preserves any semblance of story progression (or Red Dead Redemption clone for that matter) is a really hard thing to pull off for one basic reason; people on the "opposing" side will never do anything that resembles "story-line-reenactment," and more fundamentally, the whole GTA concept, while open world-ish in nature, is still at its core a story-driven progression with a logical beginning and end. The great stories themselves is really what make the GTA games brilliant on their own (after all, how many old ladies can you run over before you want to put the rampage behind you and start experiencing some story progression?). As an aside, I am still a bit confused about the story-value of the Zombies in RDR's Zombie update after the brilliant first act of RDR - but that's a whole different discussion.

So to put it in perspective, imagine yourself being one of the NPC characters in one of the GTA games, ie - not playing a protagonist like Niko Bellic in GTA 4 or Claude Speed in GTA 2, but instead playing the guy who works with Governor Johnny Rotten, or working for Jerkov in the Russian Mob. If you in fact PLAYED that character, you would basically sit around and do nothing 99% of the time, while waiting for your "scene" to occur, at which point you would add your cool flavor to the game and then get shot. No sane gamer would actually want to act out that role (it would be like a really boring version of a Civil War reenactment). But that kind of single-player progression design is what makes those games brilliant from the protagonist's point of view (and duds for the NPCs). One of the complaints about APB was the fact that sometimes one side in a conflict were basically asked to act as story progression for the other side, and that didn't always seem to work out very well.

On the opposite side of story driven games are the hard-core Player vs. Player games that basically throw away the story. Counterstrike, Team Fortress 2, Unreal Tournament 2004 even our own War Rock and to some extent Knight Online. Is there a story in any of those games in their primary game modes (ie the mode that 99% of players use daily while whacking away at each other with guns or swords)? Neh. Not really. Sure there are some allusions to a world out there (what's the Tournament thing in UT anyway?, and why is the Spy in TF2 French? and what's the "War Rock" meteor that apparently crashed to earth from the heavens? - who cares?), so basically the storyline is almost completely irrelevant to the core game play and the core fun. But those are all still brilliant games, thanks to tight level design and the ability for players to have a completely new game experience with each respawn and by focusing on fast-action strategy, the game becomes more akin to chess on steroids (you know the board, but you don't know what your opponent will do), rather than a cinematic story progression.

That creates a dilemma for the current incarnation of APB. Because it tried combining what is really two separate types of games into one game mode (story plus PvP; where PVE style NPCs give you orders to do stuff that lead to PvP confrontations), the game originally felt like it hadn't quite made up it's mind yet on what type of gamer interaction it expected (either the ultra-detailed player-and-team-control demanded by PvP, or the progression driven feel of story-focused PvE?) .

APB actually has the beginnings of a lot of the elements that will make it a really successful game in the long term, and could in theory exploit (maybe "complete") both types of interactions; there is ACTUALLY a giant back-story to the entire city (100+ pages of it), there are deep anthologies of the various factions in the city, and there is even the foundation for a single-player mission mode, which would in essence turn one of the training missions into something more akin to GTA. And if we were to go further in that direction, then there is really nothing stopping us from going "Law and Order" on the whole thing (ie - episodic story-driven content meant to be experienced either by an individual in single-player mode, OR by groups of people in a more traditional PVE/collaborative mode where teams or clans go "raiding" together and battle AI opponents that do more things than just run away like the current inhabitants in the city).

On the other hand, there is no reason the PvP fighting shouldn't also be extended to it's logical conclusion, and we are considering starting a new mode where we hand back direct control to the player, a la Counterstrike. Potentially we would call that "Session Mode," "Clan Mode" or "Team Mode" or something similar. Of course this type of mode risks breaking the design-continuity between the Social Districts and Action Districts, but fundamentally, there is no good reason you should not be able to have more of a lobby-type function where your clan can take on a competing clan (even Crims on Crims), and have a show-down in a new (presumably smaller) map until one clan or team stands victorious. It would be like challenging someone to a duel at dawn, posse vs posse. And if you don't yet have a clan, what better method to find future clan mates than by permitting a more user-controlled joining process that enables short, fast paced game instances that can be completed in ten to twenty minutes.

There is even a third design philosophy; PvE collaborative mode using fast-paced (minimal-story) PvP mechanics. The closest here would be a Zombie style mode (hah - didn't I just question Zombies earlier?). The idea would be that you and some buddies should be able to show up at a building in the Social District, and upon entering you descend into a Resident Evil style nightmare clearing Zombies (and not a whole lot of story progression beyond shooting). This third mode is really just a PvE event with minimal story line, but would be something that provides more variety for the gamers, since it's a really great "introductory" collaborative mode mechanic (everyone is a winner, except the dead zombies who are all losers).

Our end goal is clear - we want people to become so engaged in the activities we will make available in San Paro that gamers want to "hang out" in some part of the city almost all the time. There should be a huge variety of interactions (PvP, PvE, collaborative, single-player, storylines, social interactions etc.) and  players should really drive what modes we spend our time expanding and increasing in size (and what modes we abandon).

For the first release we will clearly not have had a chance yet to implement much, if any, of the above philosophies, however, this type of expanded game play is what we are planning out for the next 12 months.

Next week there will be no Blog Update (since it is Christmas week after all), but I will pen a short New Years update, and then we are planning to publicly release the first official launch schedule during the first week of January, as well as giving you more details of what will be in the various CB, OB and Live releases.


Sunday, December 12, 2010

End of Week 4 Update : UK Studio Strategy and "Hackers and Cheaters"

First things first; the Scottish snow really sucked this past week (link). Being stuck in transit between London and Scotland made me really appreciate some of the lovely California weather we have come to expect. But once I made it up there, I do have to say it was worth the very painful trip getting there.

The talent we have been able to find in the UK and have been contracting with has been really great, so our next step is now to set up a new game studio in the UK in February 2011. Right now we are just debating WHERE to put such a studio as an extension to our US operation. The clear front-runners at the moment appear to be Edinburgh (near Weaverly Station which would put us right next to Rockstar North of all companies) or Cambridge (near JagEx and others). Other options include Dundee, Liverpool, Newcastle, Birmingham, Manchester or London/Guildford. Each of these offer various advantages (and in some cases various incentives), and we are spending the next four weeks determining the right location both for us and our staff. I will keep this blog updated with our progress as we evaluate all our options (and I might even invite people to make suggestions) before we pull the trigger.

Hackers and Cheaters

Now on to the main issue this week. Hackers and Cheaters. This is a tricky topic to talk about in anything except "hushed tones" and using code-words and only partial information. But I will give it a shot, so that during the rest of the relaunch process we have a basic framework for the discussion when the topic comes back up again.

First off, let me define two separate terms; we usually refer to people that do bad things in-game as cheaters, spammers or griefers. We really try hard not calling them "hackers" - only because hackers tend to connote much more sinister stuff like DDoS and Malware attacks. But sometimes we slip, and we call the cheaters "hackers." But I digress.

So what can we do about Cheaters in APB? Well there are a ton of "cheat-busting"solutions out there such as NProtect, HackShield, PunkBuster, XTrap etc. All those have various advantages and disadvantages, though the most annoying disadvantage is of course that none of them are perfect in catching all cheats, some of them are easily circumvented, and some tend to interfere with normal gameplay to lesser or greater degrees. But everyone pretty much agrees that using some form of execution protection is a must, and the newest types of protection are now much more custom and game-specific, rather than generic. That's generally the type of protection we are now moving toward.

First the good news; as a mostly server-driven game APB is naturally much more resilient to cheaters than most other F2P MMO's. There is a good reason F2P games usually don't use server driven architectures; they cost a lot more per CCU to operate than P2P communication solutions, so in APB we are going to run a giant experiment to basically determine if hardware costs/specs have progressed far enough to make F2P server-driven games financially viable. Our current calculations seem to support that it will in fact work. By avoiding using P2P to transmit action data, the ability to attack or hack the core APB functions drop significantly. This is true for all types of hack attacks except one key category of cheats; aim-bots. Because aiming relies on visual data that by necessity HAS to exist on the client (after all - if you didn't have this data, you wouldn't be able to shoot anyone, since you wouldn't be able to see them) it makes them the trickiest category to protect against.

But even here there are some good news; before APB was shut down most aim-bots were actually actively being detected. Unfortunately (or fortunately) the data was not being acted on for various reasons (analysis was on of the reasons).

There is another issue - most people who lose any match in any shooter game tend to presume that they lost to a cheater, even when they lost fair and square. Part of human nature I suppose, which means cheating is actually over-reported. When we have run live in-person tournaments we basically end up concluding that there ARE many people out there who are basically MACHINES, and play at a level many many steps above where everyone else is.

On to hackers. Well we have been particularly aggressive trying to deal with hackers, since in some cases they have made our lives a living hell. The most interesting case being the one in 2007 when an employee of an unnamed telco-company decided to DDoS us, and basically the DDoS lasted during the hours he was on the job every day. Eventually the particular perpetrator was tracked down.

More recently we were hit by a string of DDoS attacks starting on December 25, 2009 (thanks for that Christmas present). That was actually the start of more than 50 attacks over the last 12 months alone.

To combat DDoS, which we now seem to be pretty good at doing, let's just say that "we do things," and in some cases "we use some outside providers" and internally "we have some good stuff," and we finally collaborate with some pretty mean people to help us out bringing people to justice. Just two days ago the newspapers in Turkey released this story: (here is the somewhat cryptic Turkish to English Google Translation). These guys were part of the group that were specifically targeting Knight Online.

DDoS attacks are clearly annoying (and very destructive to the business), but they are not as dangerous as the targeted malware attempts by Chinese goldfarmers that have become prevalent toward game companies (think of Stuxnet, but aiming for game companies instead). Sometimes it boggles the mind that there are people out there with an actual business model that requires some form of hacking to improve the margins of their operation.

That's a quick overview of the situation. Apologies for not being more detailed given that we have to keep things quiet in order to be effective. But once the game gets back up and running I am happy to share more specific details about specific items as they arise when we are live.

Cheers, Bjorn / TechMech

Sunday, December 5, 2010

End of Week 3 Update : Patch 143 (Closed Beta) Status
In other news : Download Size Matters, PC Gamer's unclear headline, Anti-Spam Explanations and “Stay the Frick Away from Private Servers…”

Patch 143 (Closed Beta) Status
Alright – so now we are getting closer to defining the final shape of the first Closed Beta build. When RTW closed up shop, there was already a build known as “build 142” in the works which further refined some of the balance-issues and dealt with a whole range of fixes. For closed beta we are now working toward defining an official “build 143” which will be the foundation of the game going forward. This closed beta candidate will clearly do some key initial things (like integrating G1 credits, create a framework for the premium accounts etc.), but very likely will not yet cover all the items we want to have before we truly go live. Therefore we are very likely to launch a closed beta version WITHOUT the original character data, and then wipe the databases at the end of Closed Beta after we have been able to determine the impact of the various game changes, and then look to restore some of the original character data for the Open Beta start.

Now this is all still hypothetical, since the balance changes and code changes we are performing may cause issues with the original character data and we are not yet sure the original data will actually fit with the final version of the game. However, much of it probably will, and if so we will clearly try to restore a lot of it as part of the Open Beta launch. The details of how this will be done are still TBD, but I will keep this blog updated on the progress, and will share a lot more details as we get closer to Closed Beta.

Download size matters
One other critical issue we are working to solve long-term is the issue of initial download size. Because the game uses Unreal 3, the way content and data is packaged is very monolithic. It’s a great engine, but it was never really built to be friendly to streaming partial content or to easily allow lower content levels that can later be dynamically updated since it presumes someone out there will just pop in a DVD and be done. Therefore the current game (ie the game at patch level 141) is just above 7GB in size. That’s clearly a problem. When we look at our other games’ download completion statistics, we see great completion rates at below 1GB (around 91%), so-so rates at 4GB (about 70%) and pretty lousy rates at 8GB (50%). It’s also very driven by market (for example the US and former West Germany markets don’t suffer the same download drop for 8GB clients, but for other parts of the world this is a huge issue, especially in countries that still have metered ISP tariffs).

The solutions we are working toward include researching procedural texture generation (which reduces initial download size pretty well, but makes install times much longer), removing a large chunk of the pre-downloaded music, creating a package of lower quality assets for a “starter” edition of the game without forking the game, and researching going the route of WoW which has an initial 50MB streaming client. Unfortunately some of these solutions will take a long time to perfect, and it’s very likely we will be going in to Closed Beta with a client that is only marginally smaller than the current version. However for Open Beta we aim to have a reasonable first installer size sorted out, with further improvements then coming down the line over the next 12 months. Chances are we will use combinations of all the techniques above, plus other things we have not yet considered.

Now to some quick comments, and some housekeeping items:

PC Gamer accidentally reports on my last post using a misleading headline?

It’s surprising how much “framing” or just putting a new headline on things can accidentally skew an issue (just read the comments on this blog and compare them with the wildly conspiratorial comments on the PC Gamer blog and you’ll see what I mean). Social psychologists tell us that once you have read a headline, you are going to read everything after it with a strong bias that aligns with the headline.

So what happened? Well – PC Gamer created this headline that made it sound (probably accidentally) as if we will charge for customizations. At least judging from the PC Gamer user comments, that’s what they took away from the PC Gamer headline. To be fair, when you parse the grammar (“[Reloaded] will charge for customisation options”) it could be argued that PC Gamer actually meant to say that charges only applies to “some” customizations (ie some options), but to any casual reader of the magazine that headline screams that you will now have to pay each time you customize anything. So let me clarify – NO that’s actually not what I said last week (nor is it really what PC Gamer probably meant, though their readers seem to have taken it that way).

As I am sure you have read in my last entry (Update Week 2) it clearly states that everyone will be able to customize FOR FREE (!). Now isn’t that a REALLY nice upgrade from what used to be the case – much different from how things were back when you had to pony up 50 bucks or 40 quid just to buy the APB box, and thus there were NO free customizations at all! - BUT (and of course there has to be a “but”) those devoted players in the Free2Play model that want to share COMPLEX customizations will now indeed have to have a premium account. Why? It would simply kill us if we tried giving everyone the ability to spam everyone else with complex stuff. Also – our presumption is actually that players will upgrade to Premium accounts, NOT primarily because of the customization expansions, but rather because of all the other goodies that Premium Account holders will get.  

Now I should be the first to point out that I am really grateful that PC Gamer is ready to pick up the cause of APB and give it some coverage during its metamorphosis from Box to Free2Play (and we all should be happy for coverage since it will help bring more players in to what we believe will be a fantastic game). But it would be nice if they didn’t accidentally scare people in the process of doing so. Next week I expect PC Gamer to run a story that starts with “PC Gamer prints unclear headline, of course APB’s customization will be free for 80% of players, and will no longer cost anyone 50 bucks to get started.” I am just not counting on that particular headline appearing anytime soon.

Auto-blog-spam detection

Some people that have left comments have been complaining that their comments keep disappearing. That’s actually not intentional at all, but entirely a function of Blogger’s automatic anti-spam filter. Quite frankly it beats me exactly what triggers the blog-spam detection system since it has no settings or parameters, but presumably anything you write that has a lot of external links, or uses a lot of run-on sentences (bad grammar actually seems to trigger the spam filter), or creating several copies of the same comment as a response to several different blog entries, all of these activities make it more likely your entry ends up in the auto-spam-trap. This blog is actually technically “un-moderated” and your comments should pop up right away, but I do try to check the spam folder every day if it has caught something odd. And – for the love of God – please don’t create blog posts entirely in ALL-CAPS. If that happens again I WILL delete them. I am ok with capping things for emphasis. I do it all the time. But ALL-CAPPING is the same as SCREAMING. Don’t do it. I need mental earplugs when I read those entries.

Don’t go private on our asses, so we have to go tough on yours!

Having said the bit about spam – there appears to be some people who are trying to create a private APB server out there – and those "promos" and comments will in fact be removed as spam as soon as they are seen.

I will probably try to reach out to the folks engaged in this private server endeavor in the next few weeks as a goodwill measure to see if we can have a little chat about what they are doing. Technically given the layers of protection that are baked in to the code for the action district communications, any attempt in this direction falls squarely under DMCA and also violates our IP rights in a lot of different ways, and I would not recommend doing it.

But, being a good sport, I will at least start a dialogue with some of these enthusiastic coders. After all – we are always on the lookout for great developers, so potentially some of these enthusiasts would like to work on the real thing instead of doing something shady they probably should not be doing? Coincidentally here is a PR Release from Joe Rush from two months ago (our Director of Game Operations) titled “GamersFirst deploys task force against illegal private servers” highlighting our success in literally raiding datacenters. Pretty brutal stuff. Joe gets mean when he is mad, so I wouldn’t want to cross that guy. Then again, he is just protecting all the people in our company who are putting their livelihoods on the line to bring you these great games, so we are all clearly strongly supporting the effort to discourage private server creation.

Also on an interesting side-note, I am normally no fan of DMCA, but it turns out to be one of several effective weapons against private servers in various jurisdictions.

Next Week
Right now I am en-route back to Dundee (via our offices in Bangalore of all places) to meet with even more folks in Scotland this week. We have not yet completely determined the next step, but we are now seriously considering setting up a permanent UK studio with full-time staff in 2011 to work with our teams in Southern California, Bangalore, Istanbul and Sao Paulo. We might still consider Hamburg for a European presence since that’s where a lot of our PR and marketing work is being done, but Scotland (or possibly some other UK locations) now certainly look like interesting alternatives for sure on the studio side.

Stay tuned and I will update as we progress through all this fun stuff.