Wednesday, June 29, 2011

Waking back up?

Alright. We have had an incredibly intense three weeks, so really sorry for the long posting gap (while of course the team has kept updating Facebook and Forums regularly). This will just be a quick preview of what's coming up, and I hope to follow this up with much more detail at 'some point™' when some of the critical items we are working on have gotten sorted out, and at that point also post a brand new roadmap.

We just released patch tonight, which fixes various issues, and adds more server-side controls for certain items to reduce some specific client exploits. That's the good news. Keep in mind patch 96 is a specific hotfix patch, and does not contain the large items the team has been working incredibly hard on for version 1.5.2 of the game.

The Next Big Thing - version 1.5.2

1.5.2 introduces many new items outlined in the roadmap, most importantly district recommendation and skill-rating based matchmaking and it ALSO introduces a lot of items we had not yet talked about; meta-grouping for better PuG-ing, action grouping relaxation (fancy way of saying you will be able to shoot right as you stop sprinting instead of being blocked from shooting), 70+ different mission fixes, new clothing, new items and a boat-load of various bug fixes and improvements based on much of the forum and tester feedback we have had. So there is much more to talk about once that version is ready to go live.

The plan had been to get the 1.5.2 build out by June 30 (per the roadmap), but right now it seems we will go past that by 'some period™', partly because of the massive amount of changes requiring a lot of testing. In the next few days we will also decide if we will use this build as the first one to go up on the Public Test World. We have recruited and selected about 500 volunteers (who do not yet know who they are I should add), and people with those accounts will be able to test builds going forward before going live to the public. The next few days will determine if the PTW system will in fact go live before 1.5.2 or vice versa.

We have also completed the networking upgrades that were scheduled on the roadmap. Now players in Europe will also be able to connect through Telia, which should give customers a drastically reduced ping when going to the servers especially from Northern and Eastern EU. We are in the next few weeks also adding TiNet (an Italian telecom) which has improved network coverage for Southern EU, for areas such as Spain, Italy, France and Greece.

Some Currently Known Issues in the Beta

Finally - we are fully aware of several annoying issues we are tracking and hoping to address shortly. There are a lot of specific or individual items that are on our list - but there are two in particular that are more general and requires a special mention here - Latency spike and 32-bit issues.

There is a latency issue that kicks in every few minutes or up to every 10 minutes for several players, where suddenly latency spikes to 800ms in the game. However, if you simultaneously happen to be running a long-running ping in the background to (our ping probe in the EU) or (ping probe in US-East) or (ping probe in US-West) - you'd see that the network ping/latency actually stays low during the spike, and server side and client side frame-rates also stay normal during the latency spike. So from what we can tell the current spike is not a direct networking, datacenter or server issue and instead appears to be a client side issue, at least the most likely culprit at the moment. We are working to sort out what other dependencies trigger the random client latency spike, but because of the complex interactions of all the components it will take a bit of time to isolate the root cause. We will update as soon as we have a patch that addresses this particular issue.

And the biggest issue of them all continues being 32-bit optimization for those who do not yet have 64-bit systems. APB LOVES memory. The districts are VERY large (compared to other high end games APB's districts can even be described as HUGE), and therefore the game demands 1.8GB of memory or more (and especially after a few district switches and encountering 100 other players with their customizations, that usage easily hits 2GB plus). We are working to optimize this, but at the moment the game will continue running close or at the 2GB limit. Unfortunately 32-bit windows by default only addresses 2GB of RAM, so this can lead to crashes on Vista-32 and Win7-32 in particular (XP-32 tends to fare a little bit better).

For MOST Vista-32 and Win 7-32 systems, especially if you have 4GB of Ram we continue to highly recommend TRYING (at your own risk) the bcdedit /set IncreaseUserVa 3072 command (here is a great reference from Autodesk which uses the same trick for AutoCad and 3DS Max on 32-bit systems). This command will increase addressable application space from 2048MB (default) to 3072MB, minus any memory mapped devices (such as graphics cards). Unfortunately it also means if you have a large memory mapped device (such as 1GB graphics card) this setting could cause conflicts. Some players have reported success when using a slightly smaller amount - in particular bcdedit /set IncreaseUserVa 2560, which sets the user-mode (program) address memory to 2560MB, which often seems to be enough to keep APB and it's very large world happy for a few hours. Here is the Microsoft reference on the bcdedit /set command. Again - try all these commands at your own risk (or go the proven route and upgrade to 64-bit Windows 7, which is clearly the preferred route).

There is a pretty good collection of known workarounds by moderator nXe here, which we have not fully vetted - but certainly feel free to try these as well in case you have compatibility issues.

Expanding the Team; more C++ game coders needed!

Also - as a result of the great response the game has had in Beta, we are right now expanding the development team in Edinburgh, UK. If you (or someone you know) is a fantastic C++ coder interested in either game engine development or game play development you can send in your cover letter (describing  why you would be great for a critical job on the APB team), and your CV/resume to careersUK /a t / GamersFirst / d o t / com (split apart just to help with our spammier friends - so your first test is to see if you can figure out what email to send your resume too :) ). 

There is some additional info on this page (click on "UK" to see the description): Our team is located right at the heart of Edinburgh, one of the coolest cities in Europe, with many great direct connections to other key cities on the continent. If you or someone you know are up for an amazing job and an amazing challenge where you will have a lot of impact, certainly drop us a line and try to convince us why you would be a great team member. Only friendly, well adjusted, optimistic and nice people need apply :) . And we are pretty serious about that requirement, since the team requires incredibly tight collaboration between all team members around the planet at all times.

In Closing

So - in closing - this preview is intended to get everyone up to speed, and prepared for the next major build, which really is the foundation for the more complete game that we had envisioned when we set out on this journey many months ago. A huge thanks to everyone who have played, tested and engaged in the world of San Paro for the last three months. We expect the game will be that much better thanks to all those efforts.

See you in San Paro!


Monday, June 6, 2011

The Development Road Map

In our effort to "hyper-communicate" about what we are working on, today we have released the Development Road Map (also reachable as a tab on top of this blog page).

At the moment it only has the high level public items we are working on for the next 30 days (and a lot of low level details which are not yet published). However, later this week we are going to expand  include all the cool long-term features that we are working on, such as Clan Warfare, Asylum, Racing etc. which really will open the game up to a whole new set of experiences.

Instead of answering the same questions (and quite frankly - zany conspiracy theories) over and over again on forums and on the blogs, we figured we'd rather publish all the ongoing work in one place, so if you have any concerns or constructive suggestions, hopefully the most common issues (lag, queues etc.) are addressed as ToDo items in the Road Map; for example the upcoming Firewall rebuild happening this week in the EU, the new ISP peer and IP provider we are adding next week to improve latency for EU players, the launch of the CBAK (Closed Beta Appreciation Kit), the launch of car kits, the new matchmaking and the new district recommendation items.

 So - while the current list is just a taste of what we are hoping to share, at minimum our aspiration is to NOT promise things "soon" Wow-style, nor on "Valve-time"  but instead the plan is simply to share things that we feel comfortable promising within some reasonable time frame. Will we ever miss a date? You'becha. And occasionally we will beat a date or two by getting there a tad early. But the general development direction should certainly be very clear from the roadmap itself.

Friday, June 3, 2011

Pearlescent paint bug patch today, and introducing the TMC.

Alert: Emergency patch v.91 today. Estimated to be released at 1:00pm PST / 21:00 BST / 22:00 CEST. In patch 90 a bug was introduced when customized items with Pearlescent paint appears in very crowded areas, causing a GPF crash. We are patching to fix this asap.

More specifically it's the type of bug that can only be tracked down with 100 live players (a crowd) and a full district where people have a great variety of different items (given the trillions of combinations of items players create). Even a static load test has a hard time catching this particular issue (and required some creative load testing for the development team just to reproduce the issue).

But - to catch these issues more reliably in the future, the next step is the launch of our public test world and the TMC (our Test Monkey Crew).

We will start taking applications for how to participate in the TMC after the E3 show is over on Monday the 13th of June, and Neume will send out more details on how it will work (and how you apply). Some of you will literally be asked to participate (since we can "see" what you have been doing in the game), and for some slots we will simply take applications.

(I am also working on a bigger blog post right now about our upcoming roadmap, so stay tuned)


update: June 6 - Scratch the additional blog post with info. Instead we'll convert (and condense) our formal "Roadmap" page instead and make it public for everyone to read. That way you will get far more details about what's going on in the game for both the next few weeks, as well as for the next several months than what you'd get from the blog posts (and commentary).

Also - it's still Beta. For 80% of players it's in fact very close to production quality and an amazingly great game to play. For some however it is not at that level for various reasons (hardware compatibility, latency to their ISP or routing issues, queues, SLI comparability etc.). And - yes - it's a commercialized beta, since part of this open beta is to test what items actually sell and for how much. Wouldn't be much of a test if we didn't do it "for reals." So - please hold any additional commentary until we release the roadmap.