Things to do in San Paro when you're Dead

We all know that San Paro can be a pretty volatile place to live, but when you're dying and coming back within thirty seconds, death really is more of a speed bump than anything worth writing home about. We figured it'd be pretty great if you could pick where you re-spawn, so over the last couple of months we've been working on a feature called Elective Spawning, which lets you pick your spawn location. We spoke to APB Reloaded's Lead Designer, Qwentle for more information about this latest revolution in resurrection...

Current System

The spawn system in APB: Reloaded has long been a bone of contention for players. It doesn't garner a mention when it runs okay, but when it screws up it can lose you missions and cause massive frustration.

When you die, the current system grabs all the spawn zones in the district, then starts a process of elimination to find the best one for you. First, it removes everything too far away from your death location or teammate locations (we don't want players suiciding to teleport large distances). We then remove any objectives too close to enemies. Once we have this initial list of valid points, we run a formula to determine which of these is the 'best' for you. This weighs positively towards living teammates and objectives, and negatively against enemy players, your death location, and the death locations of teammates. The objective of this system is thus to get you as close to your teammates and objectives as possible, and at the same time far away from enemies and direct death locations.

This can fail in one of three ways. First of all, the system has to use direct distances when determining this, as it would be much too heavy on the server to determine likely foot patterns between the objectives / teammates and the hundreds of potential spawn zones (see below for every spawn zone in the game):

This can lead to situations where it picks a location that's best as the bird flies, but leaves you stranded behind a block or otherwise out of the way; for instance, the piers in Waterfront can be quite bad for this. Here's an image of Waterfront spawn points:

Secondly, there is a potential that the constraints fail to find any valid spawn zones (this mostly happens during large matches in edges of the map). In this case, the system will revert to the original spawn system, which will simply pick the nearest spawn zone out-with 70m of your death location. This fall-back system doesn't take enemy location into account, so may result in you spawning on an enemy. Finally, you can be in a situation where the current spawn zone is the best as far as the system is concerned, due to lots of live teammates nearby and objectives, but you keep dying and it sends you back to the same spot each time. Not ideal...

New System

We've been working on an Elective Spawning System which will allow players to choose where to spawn themselves. At its core, this uses the same set of rules that our current spawning system does, but instead of choosing the best one, it chooses the top five. We have also added an additional rule to remove spawn points close to chosen ones, so once it chooses the first spawn point, it won't pick the one beside it as well.

Now, when you die, you will be returned to a spawn map with a selection of spawn zones. The system still picks a favorite (and it will eventually spawn you here if you don't pick yourself), but if you feel that one of the others is better for you, you can use that instead. During the spawn time, you are free to change where you are about to spawn, and when the timer is up it will spawn you at your chosen point. In addition, you can see where any other dead teammates are about to spawn, allowing you to make tactical decisions about where you want to be.

There is one caveat here. If an enemy gets too close to one of your spawn zones, that spawn zone will be rendered invalid as long as they are close. Any players that have chosen that spawn zone will need to select another.

On top of this system, we have also added Spawn Waves. If you die shortly after a teammate, you will inherit his spawn timer and his choice of spawn points. This will especially help fights in public groups, where players often attack piecemeal. Now, if you all die close to each other, you'll spawn together, which will help increase group cohesion and provide more opportunities for teamwork.

Finally, we've added a fairly major feature to this new system, which is a modification for spawning within vehicles. Any group with these mods in their vehicle can choose to spawn directly into it instead of at one of the other spawn points, as long as the vehicle isn't too close to an enemy, on fire, and has available seats (and to avoid some exploiting, is within bounds). This is a huge change. It potentially opens the game up to be much more heavily based around player vehicles, and hunting them down / protecting them. As such, we need to open this functionality to everyone, so this modification will be unlocked to all players by default, and will also be available on the default starting vehicle.

Next Time

Next time we're going to ask StuartReloaded to explain a little more about the technical side of Elective Spawning. He programmed it and - well, you'll have to wait until next week because things are going to get in depth and they're going to get weird... Trust us, it's gonna be great.

Songs of the City

We sat down with our very own in-house DJ, music producer, and audio engineer Farfletched for a Q&A session about the APB Reloaded Artist Program, and also to find out a little bit more about this most elusive of developers. The setting for this impromptu interview was Farfletched’s desk at the Edinburgh office of Reloaded Productions.


Can you tell us a little bit about yourself?

My first job in the games industry was with Rockstar North. I managed to bag five days of testing and ended up staying for a year and a half. I had prior knowledge of audio DAWS (going back as Scream Tracker, for those of you in the know) and was producing music at the time. I kind of latched onto the audio department and pitched as much material at them as I could. It was a really good learning experience. After Rockstar came Realtime Worlds. I fell into a QA role on APB: All Points Bulletin before becoming an associate audio engineer. When that ended, I got picked up by Reloaded Productions and hired as a Community Manager. When they realized that they 1) had no audio team, and 2) that I shouldn’t really be allowed near the forums, I became the sole audio engineer on APB Reloaded.

Any other claims to fame?

Mostly from DJ'ing and Producing. Through the years I’ve had the privilege of playing alongside some ridiculous acts: Tiesto, Pendulum, Faithless, Mylo, Underworld, Pete Tong. More recently I’ve released tracks on FUTURECOP! Compilations, and record labels alongside my idols. I’ve also done work on TV adverts and some neat indie iPad games.

What inspires you musically?

An extremely diverse set of places - often it's just an individual sound. I remember watching “The Running Man”, and there was a sound as a camera panned down over the first arena. As soon as I heard it, I knew I had to emulate it and expand upon it. Musically, The Labyrinth Soundtrack / Bowie in general, Giorgio Moroder, Harold Faltermeyer, Retro Chipsounds, and non-generic movie soundtracks like “The Life Aquatic”, “Explorers”, “Mad Max”, “Robocop” etc. I also have a mild case of Synesthesia (seeing notes as colors), so often I feel like making some “Green” music...is that weird?

Can you tell us a little bit about some music you listen to outside of work?

I can do better - here's a YouTube playlist.

What’s the best concert you’ve ever been to?

At the height of the Trance scene I had some amazing nights in Ibiza, Paul Oakenfold at Amnesia spamming the ice/smoke machine. However, the best gig I have ever been to would probably have to be Green Day at the Barrowlands in Glasgow. Crazy crowd, and I spent most of the evening crowd-surfing to the front, being booted about a bit by security, and then I would run to the back and start all over again.

If you could have anyone make a song for APB, who would you have make that song?

A three-way collision between Kavinsky, David Bowie, and Haezer.

Can you tell us a little bit about the Artist Program? What inspired you to give this a try?

The Artist Program is something of an ongoing project at the moment. Player content is the heart of APB:R, and we’re always trying to find new ways to bring really cool player content into the game from our end. Speedball plays a bit part of this, for example. The Artist Program is one of those things that’s as old as time itself. Before we approached bigger bands and record labels, we wanted to try and open the game to bedroom producers and garage bands and give them a shot at the limelight. I’m glad we did, as some of the content we’ve received is nothing short of amazing. All chosen artists will get a mention on our Facebook page and have their tracks appear in the music player in APB Reloaded. It’s all about giving them publicity. Some of them will even get a bit of extra attention when they're chosen as a featured artist.

What exactly is a featured artist?

With each Artist Program update, we would choose one artist to be a “Featured Artist.” Aside from being featured on our Facebook page, the Featured Artist will get plastered on billboards around the game.

So, you’re turning APB:R into an advertisement for someone’s new album?

I don’t think it’s quite like that. It’ll help keep the backdrop of the game fresh, with billboards that get updated regularly. We got a really good reaction from people when we changed up the billboards to feature Mr. Bunnyface and Speedball.

How do you choose the featured artists?

The featured artists will simply be the artist with the best all-around track in each update. Whichever artist has the track that really shines in the game, along with fantastic quality and style, will be chosen as the featured artist.

Music in the background has always been a part of APB Reloaded. Do you create all the music in-game?

Yeah. All the new tracks that have appeared since Reloaded launched were produced by myself. Check out the new sounds at the Joker Store, Gaijin, Myst, Celestial, Rimbaud and Asylum. We’ve got a nice treat in store for people that remember that weird piano conservatory music from Asylum…

Is the Artist Program still taking submissions? What kind of music are you looking for?

Yeah, it’s still open! Give us all your music! All genres! Check out details here!

In a nutshell, all you need to do is submit your track as a Soundcloud link via email to artistprogram@gamersfirst.com. You can only submit one track, but it must be a quality recording, and it must be a completely new creation (no copyright materials allowed). We accept all genres, but as you can imagine some styles have more chance than others…..Drum and Bass > Accordions , Metal > Boy Bands,  Electro House > Latin Ballroom.

If chosen, you will enter into a non-exclusive agreement with GamersFirst, which basically means that your track still belongs to you, but you are giving us permission to use the track in APB:R and affiliated products.

What’s your guilty pleasure track? (C’mon, we all really love Britney, so just admit it…)

Munamies – Pomppufilis (the hook at 28 secs in makes me smile EVERY time) http://www.youtube.com/watch?v=ZHReqKRvonE
Also, I wouldn’t pass up a little bit of Perfume – Electro World (when no one is watching) http://www.youtube.com/watch?v=Ewhfe19B25E


So there you have it. A little look into the mind of Farfletched. He's a pretty cool guy and someone that everyone should give their music to.

A Picture is Worth a Thousand Words

Today we're going to officially reveal all the new stuff that's coming out in the next release. It's content - BIG content - so get excited!

Shake that asset

Customization is one of the core features of APB Reloaded, and it's also one of the main reasons that our game is so unique. There's no other game that allows you to customize virtually every aspect of the game in as much detail as APB:R, and every time we speak to someone about what they like about the game, customization is one of the first things mentioned. We have a dedicated art team that are always hustling to bring more clothing, weapons and vehicles to the game, and today we’re going to look at some of the fruits of their labor.

The average clothing asset takes one of our artists about three weeks to create. Every item needs to be prototyped, modeled in high poly, normal mapped, modeled in a low poly version next, and then have color masks created for it - all of this before going into testing. There are other steps, but we won't go into the details of that just now.

We asked Tim, one of our Senior Artists, about his time working on the clothing in APB Reloaded. "Each item presents us with a different series of challenges. One of the strangest difficulties is being able to find a way to make an item of clothing that looks unique and very cool, but also generic enough that you can customize and do plenty of different things with it," said Tim. His favorite clothing item he's created for APB:R is the jumpsuits for the new Bishada packs, and his favorite item ever created by our team is the Desert Scarf (which was created by another Senior Artist).

Speaking of which, here's another image for your viewing pleasure:

Something that’s become fairly critical to our team is trying to get a stronger sense of parity between the genders and factions in APB:R. At the moment, male characters have far more clothing items than female characters, so some of the old assets that weren’t released are awaiting creation of female counterparts. We’re finally getting through some of the backlog of old items though, which means that the Desert Scarf is, at long last, on the list for the next release!

Where we’re going, we don’t need roads

A while back, we showed everyone a piece of concept art that showed the new, cross-faction Bishada Rapier. Since then, our vehicle team has been hard at work turning that concept into a reality, and we’re really excited with the final product. So, ladies and gentlemen, the Reloaded Team is proud to present to you the first official images of the new Bishada Kissaki and Bishada Mako!

When we design a new vehicle pack like this, we tend to think about it in terms of how the two sides play off each other.We like to think of the Enforcers being better funded than their Criminal counterparts – Criminals tend to have to work their equipment out of gear that’s believably available on the open market. Enforcers should be capable of getting their hands on more military spec and specialized equipment. While this isn’t a rule that we treat as gospel, it’s something that we run with when possible, and we feel that the hybrid we managed to achieve with the Criminal Bishada is great. A lot of the styling for the vehicle was provided by our Senior Vehicle Artist as something akin to his dream car.

The latest addition to APB Reloaded’s arsenal – The Nekrova N-SSW, which we mentioned in a recent post, will also be released in an upcoming update. We’ll be bringing two versions of this gun to the game: one version that’s obtainable through the Armas Marketplace and another version that’s obtainable through Joker Tickets. 

We’re shaking things up a bit in our next post, with a Q&A session where we lovingly interrogate our Audio Engineer about the Artist Program (and his guilty music pleasures - Prefab Sprout anyone? Google it if you don't already know...), so stay tuned!

From Your Mouth to Our Ears

It's crunch time once again at Reloaded Games. It’s been a pretty hectic week with "Settle the Score" getting ready to make its way out the door, not to mention the recent announcement of APB Vendetta. You should check out the Vendetta Kickstarter (if you haven't already), to find out more about the exciting game in the making, inspired by the APB Reloaded universe.

But first stick around!

We're going to talk about the process of how our Community team reports issues and ideas from our forums, twitter, Facebook and blog back to our Development team. We're also officially introducing you to the new Contact, Speedball, who will be joining us in the version 11 update.

Manning the Floodgates

One of the things that we're really working on this year is the quality and frequency of communication that goes on between Reloaded Games and our community. We're constantly striving for new and better ways to communicate with you guys, and it’s always good to see feedback on our efforts, be it positive, negative or (in some cases) outright weird.

Trouble is, there’s a lot of information coming in, so the community team needs to filter through a lot of the noise and try and get the strongest signal to pass along to the development team. What are the real concerns or ideas that the majority cares about the most, and how do we tackle those ideas?

We have a page on our Reloaded internal wiki that details the top ten issues in the community with links on how we intend to actually fix those issues. We're aiming to start crossing off more and more of these throughout the year. Some of the issues that we've got plans for are (in no particular order): server performance, new content, mission balancing, and in-mission spawning.

Cool Community Stats

We keep track of a lot of stats in APB Reloaded – just about everything that players can do in the game is tracked and collated into spreadsheets that we access and use to analyze player behavior. Recently we compared a list of the most lethal weapons over a week long period on Joker NA East before the weapon balance patch, and then again after the weapon balance patch when we saw a similar number of players on.

Over a 24 hour period on NA East we saw 203,055 kills; that’s 141 player deaths per minute. The most lethal weapon in the game was (prior to the weapon balance patch) the NTEC-5 with the STAR 556 coming in a close second. Shotguns accounted for slightly more kills after the balance patch than they did before, and the STAR and the NTEC had swapped places. Keep in mind that these stats don't reflect the lethality of particular weapons, rather they reflect their popularity. These stats also didn't include kills caused by vehicle explosions that were triggered from weapon damage.

Also, the total number of Easter Eggs delivered this year was a staggering 10,543,233 - that's over 8 eggs per second for the entire duration of the Easter Event! The person that delivered the most eggs was Colby’s DonCostello. Well done DonCostello, you had some tough competition!

The New Dealer

Who's this new Contact hanging out in Social? His name is Speedball and he works for the new faction, Joker Associates.

Speedball is a character that we're very proud of because everything about him has come from the community! The appearance of the character was taken from the winner of the Almost Famous contest that was run on the APB Reloaded forums for Reloaded Day 2013. His look and biography was created by Darrell Kayne from Obeya, his voice was provided by Bumbleton from Joker, and the outfits that he sells at the moment were created by players from all over the game.

Speedball will be selling player made content for various upcoming Joker Associates contests, as well as stuff that we find in the community that we think the rest of the world deserves to see and use. If you think you make stuff that other players would love to have, then keep an eye on the APB Reloaded forums for more Joker Associates contests in the near future for your chance to win. Together we can all help Speedball build a decent stockpile!

Winners of Joker Associates contests will get listed as the creator of the item in-game, and they'll be sent the new, super-exclusive Joker Associates weapon skin (props to Vivianne from the APB Reloaded forums for the inspiration on this skin!), as modeled by Gumball below. So far only twelve people in the entire game have this weapon skin, so it really is as exclusive as it gets!

That’s everything from the Community team for now. We hope this has given you some insight on what we do and how we work, as well as helping to bring you up to date on the latest in 1.11. Our next blog post will be up soon and should have details about a much requested item that’s finally going to see a release. Some sort of… dessert scarf? How the heck would you wear a dessert sca - 

Oh... right. Desert Scarf.

Settle the Score - Version 11

Earlier last month we announced we were working on "Settle the Score" - the Version 11 update (a.k.a update 1.11). We've been a bit coy about the fine details on this update, mostly because there were a few things still being finalised for release. We know everyone's been waiting in anticipation for more info so today we're going to show you some of the things that are absolutely pinned down and ready to go!

To start - there's a whole new Scoring System and two new weapons – the LMG and a shiny new secondary weapon. There’s plenty more going into this update, but here's the scoop of what's been finalized so far to get your appetite started.

New Weapons

This is the Nekrova Squad Support Weapon, a brand new LMG. It’s Nekrova’s first entry into the field of Light Machine Guns and it’s a hybrid of the ALIG 762, SHAW 556, and the N-TEC. Less damage than the ALIG and a slower rate of fire than the SHAW, but a much faster setup time. Other LMG’s suppress from a stationary position and are really effective, but this weapon’s more about mobility – it’s an LMG you can sprint with to get into position quickly before setting up and pouring the pain on your enemies.

The Flare Gun has been a concept that we've been throwing around for a while now. It’s a secondary weapon that we think every team is going to want one of, and it's going to be a total game changer during missions! It carries four shots at the moment, but these bullets aren't like anything else in the game – they bounce and explode causing very light damage (don't worry, it’s less damage than a perc.)

The real beauty of this weapon is that the flare tags every enemy in line of sight of the explosion on your radar and HUD. It’s an AOE tagger and can change the flow of any match by highlighting your enemies for the rest of your team. It’s the ideal support weapon and a must have for San Paro's elite groups.

New Scoring System

(Edited Section on 4/5/13.)

Ever since the start of APB Reloaded’s development, we've wanted to make player performance more of a factor in how you get matched up. One of our earliest tasks in the bug database (the 19th task that was brought into the database, in fact) was to simplify the scoring system and expose what it did to the player.

Version 11, introduces a whole new version of the score mechanic – everything you do while on Mission will count towards your score and your score will count towards your Threat. As you can guess, this feature inspired the name for the update - "Settle the Score."

Here’s the current system (this is a really, really simplified version):
You ready up, you get into a match and you win the match. Your threat goes up a little bit. If you lose your threat level goes down.

This is an ‘okay’ system. It works fine, but it leaves a lot to be desired in terms of edge cases. What if you lose because someone called for back-up with a minute left on the clock? What if you stomp on the other team but lose at the last second? What if the other guy on your team does really badly? And why should that guy get to be gold if they're getting carried there?

While some of those edge cases are covered (players called into a mission late only receive a fraction of the penalty) we can do better. These are the sorts of edge cases that we really want to address with the new system.

From Version 11 onward, when you kill an enemy player you get 100 score. Objective completion scores vary depending on the task. When you're near an objective getting completed you get 75 score for providing cover to your team mates. You get more score for kill streaks, blitzkriegs, arrests, stuns, freeing teammates, and vehicular kills. You get a major chunk of score for winning a match and a lot of score even if you lose.

At the end of each Mission the system will rank everyone according to their Score (people that win matches get a bonus to their score that averages around the same amount of Score as about five kills). The Threat system looks at everyone’s score and then ranks them against each other as if it were a free-for-all match. While it’s likely that the team that won will have the most score, it’s not always the case. We've seen occasions on OTW where the losing side had higher individual scores but these were when matches were incredibly close calls.

If you're below a certain threshold in score events or participation then your score and rewards are nulled. If you AFK through a Mission you gain zero cash, zero standing and zero threat. If you join a Mission very late but don’t have time to interact with anything (even things that don't score itself, like damaging other players), you won't gain or lose any threat at the end of the Mission.

Score also factors into your rewards after a Mission. If you play really hard, then you get a bigger cash reward – in fact, you get a percentage of your score as a cash reward up to a Mission-determined maximum. Getting a significant amount of score in one match compared to everyone else won't move your threat any more than a very close win but it will give you a significant amount of cash as a bonus.


So there you have it folks - the first round of feature and content details on "Settle the Score" Version 11 update! Don't forget to stay tuned for an upcoming post where our Community Team will share some cool APB Reloaded statistics we’ve gathered, and chat about the process of how community feedback gets filtered through to Development.

In the meantime, here’s a picture of the new Joker contact. Enjoy!

The First Annual San Paro Egg Hunt

(Warning: Egg puns in the following blog post)

It’s Easter, which means it’s time for an egg-straordinary and egg-citing San Paro event! This year we’re pulling out all the stops to try and give you the most egg-cellent Easter we can, in a two week egg-stravaganza from March 20th to April 3rd. We’re giving every player the chance to win a customizable Animal Suit and Rabbit Head hat in our in-game Egg Hunt Event, and we're adding eggs-essively brilliant Easter content to ARMAS in the way of an Easter Egg Weapon Skin and an eggs-travagant Chicken Costume!

The Chicken Head mask (bundled with the Animal Suit) will only be available from ARMAS, so you can finally be the Baddest Motherclucker in the city!

 

All you need to do is log into the game and go into either Financial or Waterfront. Then find one of the Bunnies walking around the city, mug them by pressing your interact key (usually F), and take their eggs to your nearest Faction Contact. You’ll hit five stars as soon as you finish ‘interacting’ with Mr. Bunnyface, so you’ve gotta run if you’re going to get to your contact before someone takes you down and steals your eggs! It’s sure to be egg-splosive! Killing Mr. Bunnyface doesn’t make him drop his eggs though, so you’ve got to make sure to go up to him and press F. And remember! Enforcers can mug him too so everyone gets to play!

Level 1 of the new role gets you the Easter Bunny Suit. Level 2 gets you the Easter Bunny Head and level 3 gets you the ‘Easter Bunny’ Title!

Now, if you’ll egg-scuse us, we're off to compile an egg-saustive list of egg puns for future reference.

Welcome to the Future of APB Reloaded

At the beginning of 2013, we promised we’d take a lot more time to communicate what’s happening with APB Reloaded and our plans for the future. Starting March 2013, we’re going to make “Reloaded Day” the start of our Reloaded calendar year. In this blog post we’re going to share an overview of our plans for the year.

2012 was a massive year for Reloaded. We had a newly-launched game and all the associated teething issues that needed to be ironed out that we attacked with zeal - things like compatibility with thousands of different PC set-ups, launching localized versions in Russia and Brazil, and creating and managing in-game events for Valentine’s, Halloween, and Christmas to name a few.

For us, 2012 was the year of Consolidation – making sure everything worked, that the game was fun and that we were listening to your comments, suggestions and bug reports from the community. 2013 is the year of Delivery. With the game and our development systems improved and more advanced, we can look to delivering more new features and content; things that we all want to see, and that you've been asking for.

Everyone knows that game development is a very fluid process – new software and hardware is released all the time, new ideas are born, new, unforeseen challenges appear. Regardless of the challenges that all passionate game development teams face, our goal for 2013 is to do everything possible to deliver on planned updates, so long as they are highly polished and ready for our players.

New & Improved Features for 2013

Our ‘To Do’ list  for this year is pretty ambitious. Some projects are only in the design stage, but others are being coded as you read this. We’ll also be constantly working on performance, anti-hack measures, and bug fixes throughout the year.

So on to the good stuff! (Please note the list below is not in order of release or priority.)

Our major goals for 2013 are:

• Score system update
• New tutorial
• Brand new front-end design
• Mission re-vamp in mission districts
• Elective spawning
• New  contacts
• Asylum map with new game mode
• Threat segregation
• Joker Ticket activities in mission districts
• Smart district spin-up
• Re-designed HUD

“Settling the Score” (Version 11)

We’ll be releasing a few major updates throughout the year, the first of these being called “Settling the Score” (Version 11), set for an April release. This update will feature an improved District instancing mechanic, a new scoring system that balances arrests, kills and mission objectives, and a new way for players to choose their own spawn point after death. We’ll write a blog post with full details just before release.

ARMAS Marketplace Content

We have a small but talented team of artists constantly working on new content for the game. 

As we've already revealed in a concept teaser we’re working on an Enforcer Performance Kit for the Bishada, and of course we’re working on the Bishada Criminal Performance Pack too. There will also be some new clothing elements for both packs, and these will be revealed a short while before release. This is all a few months away, so consider this a very early sneak peek.

Special Events

We’ve had a lot of positive feedback about our in-game special events because they allow us to create some fun seasonal gameplay and rewards. We’ll be doing major events for the usual big holidays - Valentine’s, Easter, Halloween and Christmas – and there will be plenty of other stuff in between. Our next major holiday is Easter, and while we don’t want to reveal it all yet, let’s just say that you can expect to find a brand new character walking around the streets of San Paro.

We’ll continue to work on improving online game performance, update current anti-hack measures, pursue new possible anti-hack measures and destroy bug that may pop up throughout the year.

A Bright Future

Improving APB Reloaded is a continuing conversation with our community. We appreciate your feedback and we read every Forum, Facebook, and Twitter suggestion. We’ll continue to work on improving online game performance, update current anti-hack measures, pursue new possible anti-hack measures and destroy bug that may pop upthroughout the year.

So 2013 is the year of major updates, more content, and more communication. We appreciate your support, we hope you like what you've heard so far, and we look forward to seeing you in the game.

The History of Reloaded Productions

Before we take you on a journey through the history of our game development studio – Reloaded Productions, we would like to apologize for publishing this blog post a day late. We have been in crunch mode trying to get everything ready for Reloaded Day, especially all of the celebration gifts we mentioned in our previous blog. So without further ado, please join us for a stroll down memory lane …

________________________________________________________

Back at the end of 2011, we finished up APB Reloaded’s open beta period and went live with the game. It was a pretty eventful time for us, and we’d like to share some stuff about what we’ve done in the last year and what that start-up was like for us.

While the initial development of APB happened in a huge multi-floor building in Dundee, the continued development of APB Reloaded has been taking place in a single floor building located in the bustling city center of Edinburgh. It started as a fairly spacious location, but in the last six months, as we’ve brought on more staff from around the world, we’ve found that it’s actually starting to get a little cramped! The pictures below show what the room looked like before we’d really moved in, and what the same space looks like right now.

You can see our audio, production, design and community departments. Qwentle, Scottomon, Jotunblut, Revoemag and Farfletched all share this area and it’s typically the loudest part of the office. To the right of the picture is where our artists are kept, which probably explains the sheer number of action figures that there are loitering in the area (‘they’re for anatomical research reasons only, honestly’). We keep the programmers in the background of the picture. Aphadon, StuartReloaded, Hezeki and Androvald are just a few of the programmers that we have on site. Over on the right behind the glass wall is the QA area where Funkadefunk, A3 Paper and Digitalbeeps keep an eye on features to make sure everything’s ticking as it should.

The last two years of APB Reloaded’s development could be described as a kind of ‘organized chaos.’ 

We’ve really gotten to grips with a lot of the unusual methods that were employed during APB’s original development. The last eighteen months have been all about us learning the framework of the game so the next year can be all about more fixes, more content and faster development. 

We’ve added a lot of content (six new Contacts, Fight Club, server performance fixes, environment changes, new Weapons, new Mods, new Vehicle Kits, more clothes), all while coming to terms with a huge game and codebase. Now that we’ve had time to get to know the systems that exist in the game , we have some very cool plans for the 2013 that we will tell you all about in our next blog post.

We’re really proud of what we’ve achieved with APB Reloaded. We have to make a lot of difficult decisions every week – which feature gets pushed to the next release, which assets to include in the next update, sugar or no sugar – but we feel strongly that we’ve done the best that we can and we’re in a fantastic position for the road ahead.

We’re really excited about the next twelve months and beyond and hope that you all are too. It’s great to see so many people that are so passionate about the game and it’s always heartening for a developer to have a dedicated and lively fan base. Thanks for reading and remember: tap fire the n-tec for greatest damage!

We asked our developers what their favorite moments of the last twelve months were: 

“I could tell you, but then I’d have to kill you.” – Senior Systems Developer, U.N.C.L.E.

“Killing pumpkins! The squeaky laugh you got every time you killed one was great. The whole Asylum preview was really fun for me.” – QA Tester A3Paper.

“I think the highlight of my year was getting sent muffins in the mail from the SPCT. Muffins and brownies.” ‘So what you’re saying is that free food is the highpoint of your year?’ “Kind of, yeah.” – Androvald and Alistair, Game Programmer and Hosting Programmer respectively. 

“Does the time someone kicked a hole in the wall count? Okay then, how about when we went live on Steam?” – Revoemag.

“Staffing up with people from back in the day.” – Tim, Clothing and Character Artist.

“The CS ticket explaining in great detail what a camel-toe is, and then asking for one for their character.” – M, Head of Intelligence and Office Manager.

“Getting our clothing pipeline actually fixed so we can start turning out more clothing faster.” – SoleSurvivor, Lead Artist.

“Getting visual studio 2012 up and running. Performance boosts all round.” – Aphadon, Game Programmer.

“When the folks at Effigy News sent me a real life Joker box for Christmas. That was pretty much the best.” – Jotunblut, Community Manager.

“I have to say my favorite moment was basically everything that went on in the ‘Monte and Kogey’s Stupid Competition’ thread. There is no prouder moment for a developer than watching folk sacrifice themselves in formation just for the fun of it and because you are an egomaniac with too much time on your hands in the evening, a keyboard, and a varying list of whims with which to formulate your demands. It was fun, and the community availed themselves magnificently.” – MonteCristo.

“Late night totally sober community interaction.” –Qwentle, Lead Designer.

“My favorite moments of the last twelve months are not suitable for public consumption.” – Michael Boniface, Studio Manager.

Get Ready for Reloaded Day!

Hello from sunny Scotland to all our Reloaded fans! It’s time to kick-off the month of March with style AND substance as we share our plans for the year ahead and reward our loyal players.

March 7th is the birthday of the APB: Reloaded team and so we’ll be celebrating this as “Reloaded Day” with a whole week of free stuff and the best contests we've ever launched. We’ll also be blogging about the history of APB: Reloaded and what we’ll be doing in the future.

We’ll be giving away over 400,000 G1 Credits in total prizes with Kill and Customization Contests. The ‘Almost Famous’ customization contest gives you the chance to see your design in the game and featured on billboards across San Paro as the new Joker Industries spokes-model!

Other giveaways include an in-game party hat & streamer and FREE 3-DAY PREMIUM for everyone – you get Bonus XP, Bonus APB$, and 20% off purchases from the ARMAS Marketplace!

We wanted to come up with something to reward our long-term players so we've created totally unique Titles for Closed Beta Test and Open Beta Test players. We truly appreciate all the help in getting Reloaded started.

Thanks for all your support,

The Reloaded Team

________________________________________________________________________

Upcoming Blog Posts

•  4^th March – “History of Reloaded” blog post.
• 7^th March – “Future of APB: Reloaded” blog post.

Reloaded Day ‘Kill-a-thon’

Duration: 00:01 hrs UTC Monday 4^th March 2013 to 23:59 hrs UTC Sunday 10^th March 2013.

This is a simple “Most kills wins” event. The top three players with the most unique kills in each world will win a prize (Mission and/or Fight Club Districts).

Top 3 players per World win:

• 1st Place – 40,000 G1 Credits
• 2nd Place – 20,000 G1 Credits
• 3rd Place – 10,000 G1 Credits

Winners will be announced week commencing 11^th March.

‘Almost Famous’ NPC Contest

Duration: 00:01 hrs UTC Monday 4th March 2013 to 23:59 hrs UTC Sunday 10th March 2013.

Design a new Joker Industries “Spokes-model” contact! Your character will be added to the game and will also appear on Joker Industries billboard ads throughout the game.

Design a character on your account and submit their image, character name and up to 750 word back story to the forum post. One entry per account, please see forum post for full rules.

The Top three creations as judged by the Reloaded Productions team will win:

• 1st Place – 40,000 G1 Credits + Character appears in game!
• 2nd Place – 20,000 G1 Credits.
• 3rd Place – 10,000 G1 Credits.

Winners will be announced week commencing 18th March.

Celebration Gifts

Free ‘Reloaded Day’ Party Hat and Streamer

Log-in to the game and automatically receive these two customizable in-game items for free.

Duration: 00:01 hrs UTC Thursday 7th March 2013 to 23:59 hrs UTC Sunday 10th March 2013.

Free 3 days Premium Account

Log-in to the APB: Reloaded ARMAS store during the offer period and enter the following code to receive a free 3-day duration Premium account:

ReloadedDay2013

Duration: 00:01 hrs UTC Thursday 7th March 2013 to 23:59 hrs UTC Sunday 10th March 2013. (3 days Premium begins when you enter the code during this period. Even if you enter the code on Sunday, you will receive the next three days as Premium).

Get the following benefits in-game:

• 20% discount in the Armas Marketplace.
• Up to a 90% bonus to all Contact Standing (XP) gains.
• 125% bonus to all APB$ (in-game currency) gains.
• 50% reduced cool down on activated abilities.
• Advanced access to customization suites (up to 100/50/50).
• Extended duration for free Armas trials.

Free Titles

Free in-game titles for players who have been with us since the Closed Beta and Open Beta days.

Duration: A promo code will be automatically delivered to qualifying players on Thursday 7th March, and is available to redeem until the end of this month. The code can be claimed by logging into Armas and going to the 'Redeem Code' page.

• “CBT Veteran” Character Title Unlock (Account Lifetime) 
• Closed Beta Players (First logged in before 23rd May 2011)
• “OBT Veteran” Character Title Unlock (Account Lifetime)
• Open Beta Players (First logged in 23rd May 2011 - 6th December 2011)

Many Cheers and Happy Birthday!

Optimization Magic - Part 2

(link to Part 1 of Optimization Magic)
(a slightly circular reference to forum discussion)

First; barring any unforeseen issues we are launching 1.10.2 globally in a few hours (of course with the caveat that anything could be rolled back). We expect 1.10.2 to result in a 15% server-side performance improvement across the board on the existing hardware. So while that does NOT completely address the current performance spikes, it should generally improve the experience and allow the game to run more stable at its current 25 FPS server-side tick rate. Also 1.10.2 is the first of a series of optimization focused updates.

SPCT and trashing the new hardware.

Second; a big thanks to the SPCT (San Paro City Testers) who jumped in and tried trashing the new software and new server hardware on Saturday as well as some invited guests (and for those who don't know, you can join the SPCT program to participate in the public tests of new builds by applying through our forums. Keep in mind that part of your entrance exam is figuring out HOW to apply...)

It was a pretty fun event since we basically put up the new hardware, upped the CCU to 100 per server core, then unlocked the FPS-lock on the new software, and asked everyone to show up in the same spot and "shoot and holler" and basically perform as many close encounter actions as they could (which in turn trashed client FPS for most players, since rendering 100 other players within 90m actively shooting at you and setting off explosions turns out to have its own client FPS issues, though that's separate from the server effects).

In short we tried creating the worst possible scenario for the game (server wise) and we think we pulled that off pretty well.

The good news is that on the new hardware running 1.10.2, we saw a stable server side FPS around 40 FPS (basically frame ticks stayed around 25ms), with the biggest spikes at 28ms (which is a server side FPS of 35FPS).

This means we should be able to comfortable return to 100 CCU and set the server tick to 35FPS using the new hardware combined with the upcoming software version of 1.10.2.

You can see the video of the carnage here:

Here is the resulting performance graph from the test above:

In fact, as an added bonus, during the 1.10.2 maintenance we are going to put SOME of the new hardware into production for SOME (not all) of the Financial Districts on US-West.

Due to some quirks in the configuration system, we will not be able to change the 25FPS limit until ALL the districts have new hardware, but at minimum US-West should start seeing more and more stable performance in Financial as a result. After all the US-West Financial have been upgraded, then we will move on to Waterfront.

Next steps

Next step is to evaluate 1.10.2 after the software patch, find other obvious performance issues, and then review what other optimizations we will focus on next. After that we will monitor and measure the effects of the OverKill hardware being put in US-West in production, and hope it doesn't physically explode or blow up, or catch on fire in our datacenter given it's 'pushed' pretty hard. 

Presuming all goes well, we will expand the roll-out of the new hardware and progressively replace Financial and Waterfront in each of the worlds, while at the same time continuing software optimization.

One of the challenges on the client side remains the giant particle-effect slow-downs. Unreal has a known issue where unlit particles cause large FPS hits. Epic addressed some that in the 2009 build of the engine with the introduction of the Light Mass system, but unfortunately APB Reloaded pre-dates that particular engine build, and does not have access to that lighting system.

We have various ideas of how to upgrade or replace parts of the engine over the next few months, and some are more radical than others (maybe just creating a whole new APB game using the new Unreal engine instead?.... :) ), but the basic direction is to continue with optimizations in any way shape or form we can think of, and to continue keeping you informed via the blog and forums as we roll out these changes.

See you in San Paro!

/TechMech

Optimization Magic

(a slightly circular reference: link to forum discussion)

One of the hardest things for a small team like ours to fit in to the product cycle on a regular basis is optimization.

Optimization is a little bit like making sure we are eating our vegetables every week. Every time we add or change a feature in the game, in theory we should also refactor everything around the code that just changed. Since that's usually not practical, large scale optimization really tends to happen in chunks when the team is finally given some time to focus on the task.

Now that we are about to release newly optimized code, I wanted to take some time to lift the hood on the work being done, and to share some insight into issues we have to contend with - from the software engine to server hardware.

We are really proud that APB Reloaded has remained a consistent top-5 (out of 100+) in Steam's Free2Play category since its December 2011 launch (as an aside; the game actually gets the vast majority of its traffic directly through http://www.gamersfirst.com/ rather than through Steam, but Steam provides a convenient benchmark to compare against other games), so we are planning for this game to provide many more years of entertainment for all its fans. But this tension between short term and long term goals (survive today, but plan for a 5+ year lifespan) means that every day we struggle with what we should focus on next; features, maps (Asylum anyone?), game content, security or optimization? All of these are pressing needs for one reason or another.

Given that turning off the game and entering an optimization-only cycle is not possible, we instead attempt  the next best thing; we optimize while running full speed. It's a little bit like changing the oil in your car while travelling at 65 MPH down the highway. What could possibly go wrong?

APB - a server resource hog?

Few Unreal engine based games have attempted to throw 100 fully customized players (how many FPS games ever give you 50 v 50 in a single map?) in a twitch FPS/TPS setting on a single area, where each player can have a fully customized character, car and skinned weapon, which means at least 200 fully custom player items (cars and characters), up to 850 autonomous NPC pedestrians and 350 autonomous NPC cars driving around in a district, in addition to everything that's movable and destructible (traffic lights, dumpsters, billboards etc.).

This means roughly 18,000 dynamic actors for the server to track while running in a single shard, and in fact on a single server core (and read more on the 'single-core' issue down in the hardware section).

Granted newer games using other engines like Planetside 2/Forelight have used a very different 'continent', 'distance' and 'mission' optimization system to allow much larger factions on a continent (though not necessarily in the same firefight), and even Fallen Earth uses a system of dynamic shards to allow 10,000 players in an area. But neither is an Unreal game.

Technically speaking customization has some impact on the server side performance (mostly due to large amounts of asset streaming), but customization has a larger negative impact on the game client, and tends to drive client-side frame rate lower than the expected frame rate from someone's gaming rig if customizations had not been such a central part of the game.

While there are clearly other FPS/TPS' that perform amazing graphical feats on older hardware (CoD-MW series, Crysis series, Gears of War series, Far Cry etc.), they rarely allow this many complex humans and AI actors in a single battle area, or when they do, the participants are streamlined and unified and do not permit nearly APB's level of insane customization (or city-wide destruction). Or they behave more like RPG or RTS games and generally have much lower requirements for hit registration, server tick rate and movement prediction. The amazing feat in APB is that this game still actually works on a lot of pre-2009-era hardware given the extreme computational complexity of the game.

Server FPS vs Client FPS

As a general rule, we want the server to perform a full pass of computations for all the 100 players and 18,000+ district actors 30 times per second (giving each CPU core at most 33ms to complete all computations in that one frame). If we achieve 30 FPS on the server, then connected game clients can easily run at 2X-3X the server tick rate (60FPS - 90FPS) without any noticeable loss in accuracy. At 1:2 or 1:3 server-to-client ratio movement prediction and frame-interpolation provide a very smooth game experience.

Unfortunately during the last few updates we have had to temporarily lower the server tick rate to 25 FPS and reduce the max CCU per core, so it's high time to perform another full optimization pass.

Software Optimizations and Server Side Computation times.

Below is a graph of what version 1.10.1 server-side computations look like under ideal test circumstances AND using our new test hardware (more details on this new hardware at the bottom of this post).

In the current 1.10.1 build the server completes 1 full frame (moving those thousands of actors around) on 1 core in 1 full district using the new hardware type in 19.2ms. In 'theory' this means the server on the new hardware is capable of running at 52FPS tick rate (!).

This is to be compared with the 'current gen' hardware, where we have only been able to run a 'safe' server tick rate of 25FPS in the current 1.10.1 build.

The lower part of the graph shows version 1.10.2 with the new software optimizations.

From the synthetic test it appears the team has been able to squeeze a 16% performance improvement in software alone (which amounts to about a 10 FPS improvement on the server). This improvement drops the per-frame processing time to 16.1ms, which means a theoretical 62FPS server tick rate (again on the new hardware).

This 'should' mean that software optimization alone (the 16% improvement) will let us go from 25FPS back to the original 30FPS serverside tickrate on the current hardware as part of the 1.10.2 update (to be determined after the game is live).

You can read these graphs from the bottom up, starting with receiving network packets from all connected actors, updating game elements and physics, updating cameras and streaming, and ending with sending data back to all clients. What's rather surprising is that almost 50% of the entire server processing time consist entirely of receiving/parsing and serializing/sending network traffic. The actual game updates (players, objects, physics etc.) take only 50% of available CPU time.

From the above graphics you can see that the team has been able to really squeeze and optimize the "Receive Network Traffic" and the "Update Game Objects" steps. We expect to continue optimizing all the steps in the system, but presuming QA signs off on the upcoming patch, we will measure the real-world impact of these improvements in the coming week.

The Single-Core Engine Conundrum

First a disclaimer. The Unreal Engine has served us (and thousands of other games and companies) incredibly well. It's a great engine and a fantastic rendering system. Now the engine has certain design choices that create certain hard-to-overcome limits (as all engines do).

The biggest one for large scale games is Unreal's monolithic and (almost) single-threaded server-client-response system. The philosophy behind Epic making that design choice back in the era of Unreal Tournament / Gears of War makes perfect sense, given the engine's focus on small-scale lobby based FPS/TPS games or even single player or co-op games. Some Unreal based RPG's (for example Blade and Soul) have clearly adopted the engine as a renderer, and then created an entirely proprietary server system to handle RPG style updates and connection loads (which usually requires 2000-3000 players per shard but only a server tick-rate of 10FPS or less in RPG mode).

APB Reloaded uses a hybrid of standard Unreal server code (originally we used Unreal version 2008, so the engine is getting a little aged at this point) and its own proprietary TCP message stack coordinating the communications between worlds and districts, as well as a very proprietary customization system. But the general actor-to-actor interaction relies on a system that's very close to the original Unreal system. Mostly handled in a single game update loop.

This means all the processing in a single district happens on a single core and in a single thread.

One way to think of this is that the engine fundamentally works like a turn-based game where each actor has  33ms to move per turn. Within the scope of a single server core/thread, the process gives each actor one chance to make a move (or combination of moves). When all actors have signaled their move, everyone is told of everyone else's  updated moves, and the game now proceeds to the next move (though from the chart above you can see that we actually only spend about 7.5ms moving stuff around, the rest of the time is spent sharing that information).

Human reaction time (or as it's called Mental Chronometry) is around 160ms, so processing everything at 33ms on the server, plus the packet roundtrip time (ideally less than 40-80ms) for a total of about sub 120ms of processing delay, should give us sufficient headroom to provide a good player experience.

However, even just a slight improvement in server side processing will actually enhance the fluidity of the game. We humans are very good at processing sequential frames of information and can easily spot the visual difference between film at 24fps and video at 30fps (or as the case may be "the Hobbit" at 48fps for those of you who now hate Peter Jackson). This means we will notice visual processing hiccups long before we react to new on-screen events.

Why does all this single-threaded-ness matter to us? Well, it turns out that most of the performance gains in recent years in server processors from Intel and AMD have NOT come from performing more computations on a single core, but rather to have many parallel cores performing parallel tasks.

Sadly for APB Reloaded, that type of parallel task division does not improve individual district performance... But... there is hope...

New OTW Hardware Test World going live: OverKill

In the near future we are about to release a new OTW (Open Test World) called OverKill. OverKill is actually an apt name and is the result of a lot of hardware experimentation by our IT team (and the above computation tests were run in this hardware as well).

The current generation APB servers consist of Intel Xeon X5570 "Nehalem" based processors (operating in 3.2Ghz Turbo Mode) with 4 cores x 2 processors each. We use Dell M610 blades like these (just recently pulled from our datacenter).

The benefits of blade servers are that we can increase the density of the hosting operation, since we can fit 16 servers in 10 "rack units." The drawbacks - the types of processors supported by blade servers and the inability to overclock those processors - have caused us some serious problems in optimizing the hardware for the game.

For quite some time we have been looking for a new processor solution specifically to handle Financial and Waterfront (and eventually Asylum) districts. Something that can live in our three datacenters, but at the same time give us a cost effective solution to run at much higher single-core clock speeds, while also taking advantage of the newer "Sandy Bridge" and "Ivy Bridge" Intel processor architectures.

After much playing around with various combinations of server chips, it turns out that  server boards and server chips really don't like or even permit overclocking and they are almost never engineered to optimize single-threaded performance (other than the incidental improvements that come from larger L2 and L3 cache systems), and we also need at least 6 cores to be able to perform these calculations in a cost effective manner (which let's us run 3 fully loaded districts on a single server) which left us with a conundrum.

After experimentation we have settled on having a public test using a custom solution that uses a high end desktop board (ASUS Rampage 4 Extreme) combined with an unlocked Intel i7-3930K 6-core processor that in a datacenter settings (with lots of cold air) easily runs stable at 4.25Ghz (technically we can push it to 5Ghz, but we are starting small).

Will it work once we throw real APB district computations into these systems? The synthetic test indicates it will indeed work. Will I/O performance hold up (given the strangle-hold that network I/O has on server CPU)? That's much harder to test, so we will find out as soon as we start running the  OTW tests.

In a synthetic benchmark the i7-3930K OC (compared to the stock X5570) shows raw gains of nearly 70% in single-threaded performance (!). We do lose two cores per server, but the extra expense (more servers) seem worth the vast performance gain.

CPU Bench – Single Threaded:
[ORIGINAL X5570] – 1349
[EXPERIMENTAL 3930K] – 2284

If we can capture some of these performance gains in the real world, and translate it into improved Action District performance, then our longterm goal is not only to ensure a stable 30 fps server tick-rate, but gradually be able to raise the CCU in each district as well.

From the graphs on software optimization, you can see that the new hardware with the new software 'COULD' run a theoretical server-side tickrate of 62 FPS, which is 206% more than we actually require for the 30 FPS tick target rate.

Our plan is to use the extra performance (again once we have run the real world tests) to ensure we can increase CCU in a single district. Since CCU taxes the server in a non-linear fashion, we expect to only increase CCU 25%-50% before dragging the server back down to 30 FPS tick. Of course this is still speculation, and is still to be determined during live testing.

Higher district CCU would mean better matchmaking (but THAT is a whole other blog entry, though needless to say 80 people in a district means 20 teams with potentially 10 ongoing matchups whereas 120 people in a district means 30 teams with 15 ongoing matchups, resulting in 50% improvement in match availability. Of course it's not quite that simple - but you get the gist). More players = better matchmaking.

OUR hardware, software and network vs. YOUR hardware, software and network.

In this post we have only talked about server side processing and optimization, and have not touched the OTHER things that also affect performance  First and foremost - you need a good gaming rig to play APB. We always recommend having 8GB of RAM and using 64-bit Windows 7. Anything less is asking for trouble. In particular using 64-bit Windows is critical. Also client-side FPS in most Unreal games tends to drop dramatically during very large semi-transparent VFX events (i.e. very big explosions where the player does NOT die - something APB of course has a lot of) so only higher end graphics cards tend to perform ok during those big VFX events (and to optimize that part of the engine code is a whole other ball of wax, far beyond the scope of this current post).

Of course network connectivity, and your latency to our core datacenters are critical as well (Los Angeles, Washington DC and Frankfurt) or to the datacenters managed by our Russian (Moscow) and our Brazilian (São Paulo) publishing partners.

I hope this article has shed some light on the optimization work currently being done. If you are one of our OTW testers, then expect to see the "OverKill" world come online in the next two weeks. And for everyone else we expect to release 1.10.2 very soon, which should have some immediate performance improvements.

Til Next Time!
/TechMech