Pages

Tuesday, February 26, 2013

Optimization Magic - Part 2


(link to Part 1 of Optimization Magic)
(a slightly circular reference to forum discussion)

First; barring any unforeseen issues we are launching 1.10.2 globally in a few hours (of course with the caveat that anything could be rolled back). We expect 1.10.2 to result in a 15% server-side performance improvement across the board on the existing hardware. So while that does NOT completely address the current performance spikes, it should generally improve the experience and allow the game to run more stable at its current 25 FPS server-side tick rate. Also 1.10.2 is the first of a series of optimization focused updates.

SPCT and trashing the new hardware.

Second; a big thanks to the SPCT (San Paro City Testers) who jumped in and tried trashing the new software and new server hardware on Saturday as well as some invited guests (and for those who don't know, you can join the SPCT program to participate in the public tests of new builds by applying through our forums. Keep in mind that part of your entrance exam is figuring out HOW to apply...)


It was a pretty fun event since we basically put up the new hardware, upped the CCU to 100 per server core, then unlocked the FPS-lock on the new software, and asked everyone to show up in the same spot and "shoot and holler" and basically perform as many close encounter actions as they could (which in turn trashed client FPS for most players, since rendering 100 other players within 90m actively shooting at you and setting off explosions turns out to have its own client FPS issues, though that's separate from the server effects).

In short we tried creating the worst possible scenario for the game (server wise) and we think we pulled that off pretty well.

The good news is that on the new hardware running 1.10.2, we saw a stable server side FPS around 40 FPS (basically frame ticks stayed around 25ms), with the biggest spikes at 28ms (which is a server side FPS of 35FPS).

This means we should be able to comfortable return to 100 CCU and set the server tick to 35FPS using the new hardware combined with the upcoming software version of 1.10.2.

You can see the video of the carnage here:

Here is the resulting performance graph from the test above:


In fact, as an added bonus, during the 1.10.2 maintenance we are going to put SOME of the new hardware into production for SOME (not all) of the Financial Districts on US-West.

Due to some quirks in the configuration system, we will not be able to change the 25FPS limit until ALL the districts have new hardware, but at minimum US-West should start seeing more and more stable performance in Financial as a result. After all the US-West Financial have been upgraded, then we will move on to Waterfront.

Next steps

Next step is to evaluate 1.10.2 after the software patch, find other obvious performance issues, and then review what other optimizations we will focus on next. After that we will monitor and measure the effects of the OverKill hardware being put in US-West in production, and hope it doesn't physically explode or blow up, or catch on fire in our datacenter given it's 'pushed' pretty hard. 

Presuming all goes well, we will expand the roll-out of the new hardware and progressively replace Financial and Waterfront in each of the worlds, while at the same time continuing software optimization.

One of the challenges on the client side remains the giant particle-effect slow-downs. Unreal has a known issue where unlit particles cause large FPS hits. Epic addressed some that in the 2009 build of the engine with the introduction of the Light Mass system, but unfortunately APB Reloaded pre-dates that particular engine build, and does not have access to that lighting system.

We have various ideas of how to upgrade or replace parts of the engine over the next few months, and some are more radical than others (maybe just creating a whole new APB game using the new Unreal engine instead?.... :) ), but the basic direction is to continue with optimizations in any way shape or form we can think of, and to continue keeping you informed via the blog and forums as we roll out these changes.

See you in San Paro!

/TechMech


26 comments:

  1. if you guys gonna try to modify the latest UE3 into APB:R then the new UI is farther than we think

    unless the new UI will be added twice.

    wish you guys had done Waterfront first but its a start

    ReplyDelete
    Replies
    1. Actually Scaleform is one of the big updates coming in a future build (potentially as soon as in the next version). And yes, it is a bit tedious and a lot of work to retro-fit all these upgrades to APB:R given its massive customization does not permit a simple code-import from Epic. But the team is soldiering on like the brave troopers they are!

      Delete
  2. "maybe just creating a whole new APB game using the new Unreal engine instead?"

    Now you have my interest. APB 2 on Unreal Engine 4 or at least a updated version 3...would be very nice :).

    ReplyDelete
    Replies
    1. Well - maybe not an entire APB2... but something... Stay tuned :)

      Delete
    2. They can't make an APB 2 yet!!! I still haven't gotten a desert scarf! LOL

      Delete
    3. And yet Asylum has taken how long?

      Delete
    4. In related to an new or upgraded APB. Is this another dream that you have and will not tell us any more just in case it takes too long or is taken off the table. Come on guys, you need to open the comms line more/again.

      Delete
  3. Would love to see this update! my 660 ti keeps crashing when i running this game (luckily there is a temporary fix in the forums) I know this game has a old engine and i have hope in you guys, this is one of the best games i have. With over 800 hours, this is one of the games i play all the time until i got my 660 ti:/
    P.S. i love you guys keep up the work.

    ReplyDelete
    Replies
    1. Disable the cuda cores under the nvidia drivers, look for the apb profile. this should help a bit.

      Delete
  4. Can't wait! Keep up the work, though that idea of using the new Unreal Engine would really be interesting.

    ReplyDelete
  5. APB 2 should be saved for U4, until then, a port of APB:R to the latest engine should SERIOUSLY be considered. I've been playing this old thing since.. Since.. Unreal Tournament 3 O.o

    ReplyDelete
  6. These blog posts are really very interesting and I love reading about developments and improvements of games I enjoy playing. It makes me very excited for the future of APB. To the team behind it I would like to thank you for your hard work and dedication.

    ReplyDelete
  7. Exciting news, I would love to see APB:R on a later unreal build! Would such an event mean that the game would run better on weaker CPU's?

    ReplyDelete
  8. It can't just be as simple as a port though? Surely entire libraries an assets have to be built from the ground up? Obviously that would be the best idea, but isn't it too costly and time consuming?

    ReplyDelete
  9. Will you be able to increase the amount of traffic and add multiple versions of traffic cars to drive around all at once? It is a city :P

    But it's great to hear that better things are coming :)

    ReplyDelete
  10. Who's the sexy person running up to you @ 0:40 ? :D

    ReplyDelete
  11. Heya, glad to see you're working hard! I'm personally running the game flawless on 120fps apart from a few server side lag spikes, however many of my friends experience problems which makes them and through them me less motivated to play, regarding a new APB that sounds amazing, a new fresh engine with lotsa new stuff, only thing I worry about is losing all the armas stuff I bought ^_^

    Thanks for your hard work

    ReplyDelete
  12. Bjorn any chance you guys will unlock the client fps to 150-200 for everyone instead of 100 and 120 for people with 120hz monitor?


    time to test the new server :)

    ReplyDelete
  13. After playing a bit on the financial servers in Colby, there is a noticeable difference in hitreg and server perfomance

    good job and thanks

    ReplyDelete
  14. A remake would be so cool. I have a whole list of things (like faction appropriate weapons, location damage and avatar responses to location damage, better driving mechanics so I don't hit those pesky civis, more civilian car types, etc)that could be done, but yeah whatever you guys want XD

    ReplyDelete
  15. Unreal Engine has come a VERY long away, spending a long time updating the game into a new build of the engine would go a VERY long way. But it may take a while, while being said, take your time just be sure to work on it slowly but surely.

    ReplyDelete
  16. This comment has been removed by the author.

    ReplyDelete
  17. Dear Developers.
    I am posting this in hopes that some of what I say is addressed. I have played your game for a very long time. I also have contributed a fair amount of my own money to it in hopes that things would get better. However there are so many issues that have been there for so long that I have to at least try and get something done here. The biggest issue for me is hitboxes. I cannot tell you how many times I have been killed around a corner or run over by a car that passed beside me. They aren't hitting me they are hitting my hitbox. It is either far to large or something of that nature. Its incredibly frustrating to be able to see there be no contact from the vehicle or rounds and still die despite that. Then on the flip side of that to hit someone at full speed and have them be pushed on top of the car or just pushed along. These aren't new issues and have been there for as long as I have played. I can see your working on the server so I'm going to let that one go. At least your trying to make it better. So I have respect for that. This next issue is much more personal. In the time that I have played I have seen numerous kits come out for the game nearly every car has some body mods available for it. However despite it being available in the old version of this game AND in your own code. I know it was given out by mistake by gm's or whatever the case was. Why not release the Fresno kit? You get constant posts on all your forums people know its there and want it. It makes no sense at all NOT to release it if you want your consumers money. Instead kit after kit is released for the popular cars. Not every player wants to drive those some of us enjoy the road less traveled. Would it really take that much effort to release it and gain something from it? Hackers I've seen addressed a few times so again I'm going to leave it at that. However the last thing that bothers me is the near constant use of racially and constant uses of slurs against others as well in the chat. Its so common now that I cannot log on once without seeing such things. I know we are all adults but some of us don't act like it. Those things aren't "cool" to constantly be throwing around its offensive. It wouldn't be tolerated in real life and it shouldn't be online. One minor one that I doubt bothers anyone but me, why does my gas can expand and glitch out when I'm burning things this is silly. Please don't ignore this post. DO something about these issues.

    ReplyDelete
  18. Thank You Reloaded for give us the privilege of play your game and be part of the community! You are doing a incredible good job sharing your upadtes briefings and we are all very excited with the future of APB! H5 BjorK! Hi5 G1 and Reloaded!

    ReplyDelete
  19. Thank You very much for taking time to write these posts.

    Over 48% of the time in The Loop is spent sending and receiving data from the network. Question: couldn't You use the other core to handle things such as network I/O, parsing and serializing? Core1 would read (valid!) structures prepared for casting from shared memory, convenience of Core2. Core1 would put "data to be sent to clients" in shared memory and Core2 would take care of it while Core1 is computing a new frame. Have You thought about that?

    ReplyDelete

Please keep comments on-topic and reasonably civilized. The moderators reserve the right to remove any off topic, uncivilized or troll-ish comments.