Pages

Monday, July 21, 2014

Fighting Fair in APB Reloaded with FairFight

...and the sad naming and shaming of 200+ players.


I bet many of you know “FairFight” as the anti-cheat system currently in use in several top tier games, including Battlefield 4 and Titanfall.

But I bet you don’t know that the Reloaded team has been working closely with the FairFight team for the past several months to integrate and optimize FairFight in APB.

When were able to combine our existing tools and systems with the FairFight system, we were suddenly able to review a vast range of information and get much better visibility into what players were doing in the game.

Some of these new insights made us very happy; for example it turns out that the numbers of actual cheaters in APB is much lower than the community would believe. But some insights made us terribly sad; players with as many as 8,000 hours of playtime were caught red-handed cheating.

What kind of cheating did we catch? The current combination of tools can now catch a wide range of activities using almost every conceivable method of detection. The FairFight system has been great, since it has acted as a “key” to let us pull together a lot of disparate the data into one single effort. As a result we found lots of players who actively had a cheat running, but, sort-of didn’t use it much, and often appeared to be running mostly legit, while relying on their purchased cheat tool “to cheat just a little bit.”

Of course any cheat is a gross violation of our terms of service, and just as when pregnant; there is no “a-little-bit-of-pregnant” category in the human gestational cycle, just as there is no “a-little-bit-of-cheating” category when playing a fair game.

The sad naming and shaming of 200+ players

Today as we formally move out of “stealth mode” to ensure a fair APB game going forward we are also publicly listing a sampling of 212 people whom were detected cheating sometime in the last few weeks or months. The list of names can be found here. It’s not comprehensive (the total number of bans are several times this list), and it includes players caught using several different detection methods. A few were even tagged by the “trusty” old Punkbuster as well, though FairFight enabled us to truly be efficient in tracking down all these miscreants.

In the “name and shame” section we also opted to not only list the character names of those who cheated, but decided to take it a big step further and list the time spent playing the game, player rank, server, clan and the range of real cash each player had spent in the game before getting terminated.

APB Cheat Stats

There is some good news that came out of our intense cheat review; on a daily basis, the number of cheaters who log in to the game stayed pretty constant around 1.2% of all players (some of which were re-rolls). Inversely 98.8% of players in APB played on a daily basis without any trace of any cheat (which we are very thankful to see)! So as usual, it’s the 1% who is messing up the world for the rest of humanity.

For some even more startling facts; 60% of the players banned in this sample were paid players. That’s surprising on several levels. For comparison, about 10% of the monthly unique players in APB make a new purchase in the game in any given month. On a cumulative basis about 30% of all players playing the game have paid money for something at some point during their account history.  So in simple terms, paid players turned out to be 2X over-represented in the initial ban sample, compared to the APB population as a whole.

Another startling fact; paid players who were banned had on average 260 hours of play time in the game, had played for over 1 year, and had on average spent $180 on virtual goods.

Now clearly they got their monies worth already (in comparison they could have chosen to pay $59.99 for 20-40 hours of play time in a traditional single-player game, or pay for 12 months of MMO subscription time for about that amount of money) and it's crazy that they chose to have themselves eliminated from the "sport of APB" by cheating.

A Few Bad Apples Spoiling All the Fun

So why has there been such a giant perception that cheaters are rampant in APB, if the ratio of cheaters, even before FairFight went live, was only around 1% (when we were just observing what was going on during stealth-mode)?

Well, there are really two explanations.

First, yes the few cheaters that exist have a disproportionately negative impact on the matches (snapping, appearance of cheating etc.) partly BECAUSE they also tend to be active players. So they do affect a much larger number of matches than 1%.

Second, it also turns out that APB is a REALLY HARD GAME to master and play. In spite of what everyone says, APB takes some serious skill (much more so than money). Locational awareness, amazing twitch-skill, strategic planning, and a flawless group to play with are critical to success in the game. And in fact we found ourselves analyzing some matches where EVERYONE accused everyone else of cheating, but no one was actually cheating!

A few of the players in those matches were in fact just THAT much better at the game than everyone else. It’s a really tough feeling when you realize you got crushed in some inexplicable way, and a very common immediate reaction is to accuse everyone of cheating. Statistically there is a very good chance they actually were not cheating the last time you played them. They probably (though not always of course) were just better players, a better group etc.

Cheating “just a little bit”?

Even more fascinating, as we go through the list of cheaters, we realize that often they did NOT use a cheat to crank all their stats (well of course, some idiots did), and our only conclusion must be that some of them have been using cheats as a ‘crutch,’ probably because they thought the other people around them were beating the crap out of them (even if statistically they were likely NOT to be cheating).

Notice that the overall K/D ratio of the cheaters in the sample name and shame list is around 1.7 – meaning for every 17 kills the cheaters died 10 times. That’s around 70% ahead of break-even, but not massively better than players with equivalent time spent in game.

And finally, we also found accounts where it appears the players have been playing legit in the account for a long time, and then for some reason or another decide “to just try a little cheating.” That’s sort of like “just trying a little meth.” Your mom and your local DEA officials always tell you “don’t do it” to the meth. And we say the same about cheats. Don’t do it. One day you will be caught.

Should we enable Purgatory?

So while we are very glad to see some of our community members go due to these flagrant violations of our terms of service, one idea that we keep kicking around is to potentially enable a “Purgatory” District.

The idea is that cheaters don’t get banned right away (which of course doesn’t happen since the system runs on various delays), and instead freshly detected cheaters get kicked into special cheaters-only districts, where they can continue playing against other cheaters without initially realizing they have been tagged (or whole groups get thrown in there as long as they have a cheater on their team). Titanfall has implemented a similar system, and it seems to work reasonably well for them.  The main reason for us to consider doing this is purely for research. We benefit from continuing to capture data from those who cheat.

But we also realize this is a sensitive item for all players. We’d be curious to hear your feedback on this type of district (maybe it becomes a ‘mass-spectate’ thing where you can view them all cheating each other)? No final decision has been made so far about this type of new district type.

In fact – this is NOT free!

In jest I had added a post-script to last week’s ‘teaser’ post about making an announcement that said “hint: it’s free.”

Make no mistake. Spending THIS many resources to combat cheating, even with an excellent partner like FairFight (who have a great team, but of course needs to charge for their services), combined with all the other related internal and external tools involved, it all takes a massive amount of resources away from the Reloaded team. And again puts us in a position where we have to prioritize anti-cheat efforts which directly takes away critical resources that could be better spent on new shiny features and new systems in the game.

But in this case we feel the investment will be worth it in the long-term. And we are hopeful we will continue getting great support from our devoted players who spend countless hours in the game every month.

Final words

The other big project that’s ongoing is of course the engine upgrade, which continues moving forward at a steady pace. However, what started as one project has now morphed into two related projects that have much larger teams attached. The details of which I cannot release “next week,” but I promise that it will be “sooner than SOON™.” And I am personally super-thrilled about what our teams are already working on, and again, I hope to issue a really detailed update as soon as I am able to disclose all the details.

In the meantime, we continue being incredibly thankful for all your ongoing support, and hope to share more exacting news as it becomes available.

And just to cap it off – as an APB player, did you know that APB Reloaded is in fact the 19th most played game on Steam (out of 2750 total games distributed) of ALL TIME, just ahead of Borderlands2 and behind Call of Duty: Modern Warfare 2? At least according to this Ars Technica article (and which, from our internal numbers, we can certainly corroborate, if not ponder that Ars may have under-counted some stuff). That makes APB more played than 99.3% of all other PC games (!). There is some additional data in this Ars Technica update as well.

In fact turns out Reloaded Games have 2 of the games in the Top-100 out of 2750 games: APB Reloaded at No. 19 and Fallen Earth at No. 94.

In short, we are very much looking forward to when we can share our next set of exciting updates.

In the meantime – play fair! And if you want to see the conversations that this has sparked – then you can go here to check out threads like these in our social district forums.

/TechMech



52 comments:

  1. For many years iv reported a few of theses guys and finally to day i stand tall good to see guys good to see ill be back on Wednesday with my MONEYS so you best have sumthing good on the store

    ReplyDelete
  2. Still dont work very good. My friend was instant banned when he was using steam,ts3,razer boost and gaming mouse program that comes with mouse itself. Funny thing that he cant kill 5 ppl. without death and that he spend 2000$, And after 4h I seen cheater in Asylum FC. with 67 kills and only 3 deaths. and he didint get ban.

    ReplyDelete
    Replies
    1. The system mixes insta-bans with delay bans. On purpose it records as much data as it can, and then often bans the cheater after a random time.

      Delete
    2. ofc. its nice to see that some one doing something....
      I just hope i wont get BAN for NVIDIA Geforce Experiance, Steam, Teamspeak3, skype or TuneUp Utilities 2014.

      Delete
  3. I hope a full list of all caught cheaters gets made public at some point if possible. I'm having so much fun crossing off names that I know from my list :)

    ReplyDelete
  4. THIS TIME DON'T GIVE PARDON, A SECOND CHANCE, AMNESTY, ETC. THIS TIME. I remember TechMech, I bloody remember those times.

    ReplyDelete
    Replies
    1. "Should we enable Purgatory?"

      No, TechMech, you guys literally gave them PLENTY OF TIME TO GRIEF THIS GAME TO A LOWER POPULATION and STILL gave them second, third, and fourth chances to come clean.

      Keep funding FairFight, they can use APB as there #1 example in success and so can you.

      Delete
  5. Congratulations !! soon APB will be full of players ^^

    ReplyDelete
    Replies
    1. Nope Full destricts are coming again O.o

      Delete
  6. This is probably the best news in APB this far! I have an important question that I know many players are thinking about: Has the introduction of FairFight pushed anything previously belonging to the grey area in to the black area? I am referring mainly to the use of macros, which many gaming mice are delivered both in software and hardware. The main reason for this question is that we are already seeing a start of a trend of banned cheaters blaming their ban on that FairFight detected macro use, when they where in fact actually just using plain old paid hacks and are trying to make it look more innocent than it was. It would be great to get a clear understanding of this.

    ReplyDelete
  7. This is actually really good news. Great to see these steps being taken to protect the product.

    ReplyDelete
  8. Kudos G1 and RP, much respect. I can see a lot of Armas purchases and a thriving player base of new and veteran players coming in the near future. I personally thank you

    ReplyDelete
  9. What about wallhackers or trigers? How can FairFight detect them? What about macrousers?

    ReplyDelete
  10. Great job, this is a great day and I hope to see how this works in the future, I'm sure there is still plenty of self-coded cheats still around, but this is a great day!!!

    Also I'd like to say that the idea of a separated server may worth to give it a try. I remember GTA IV where they allowed you to host the free roam match with or without cheats enabled so everyone else would follow the same rules, it was something useful to use the trainer for different purposes such as spawning vehicles and objects or ramps. Another game that I can recall is Command & Conquer Renegade, where the servers were privately hosted and they had a few "cheats-free" servers while the rest were 100% cheat-proof.

    The point with all this is that if you simply ban every cheater around they'll end up so mad that many of them will try to re-roll and rage-hack harder. But if you let them play on a isolated environment, then they'll be able to get rid of their frustration without break the game for anyone else instead. That's how I think it would work.

    All my praises for you and the team for this Björn!

    Best regards RP guys!

    ReplyDelete
  11. Well peoples of San Paro, you can stop calling me a cheater now after 3+ years thanks...

    ReplyDelete
  12. Re: purgatory: Diablo II's Realm vs Open comes to mind, as well. Seems like a good way to make everyone happy.

    * Legitimate users happily play with legitimate users.
    * Tool-assisted users happily continue playing the all-tactics omniscient no-aim game they bought into.
    * G1 gets to continue happily monetizing tool-assisted users.
    * Fairfight and Punkbuster happily get lots of clear heuristic priors.
    * Gamersfirst gains a community of hackers passionate enough to dissect memory structures and network protocols and user interface models. Hackers that can experiment and give feedback without reprecussion. Hackers who can be incentivized to report game-breaking protocol exploits and server vulnerabilities. Hackers who become a first line of detection and defense against other hackers.

    Seems like it's worth a shot to try it long enough to get at least a vague cost/benefit enumeration.

    Remember: if everybody is fighting to destroy values and motivations they can't internalize, then nobody is working to actualize whatever values and motivations they have internalized. All we need are clear personal boundaries.

    ReplyDelete
  13. I always figured it'd be nice to let them end up in their own playspace pit against each other. It'd be nice if it could be hidden from them, but I doubt it'd stay that way for long, given the way premades tend to work.

    Even so, the opportunity to get a closer look at how the cheats work - without having to let them run wild in legitimate instances - would be ideal.

    I suppose it would indeed have the added benefit of continuing to make money off of them while not having them in the mix with the regular players. Not banning cheaters outright from all competitive portions of the game (ie, their own action districts/fight clubs) and allowing them to hold onto their ARMAS toys in their own instances does also carry the added bonus of giving them less incentive to reroll, seeing as their sunk costs would not be unusable.

    ReplyDelete
  14. Your next course of action is to take the name & shame list and file charges for obstruction of business against every one of them. Go the Japanese way and take their money away. I can promise you this, there is A LOT of potential players who turned their backs (and by extension money) away from APB due to the cheating. Now's your chance to bleed some money of that cheating scum. Getting a ban probably sucks, but being forced to pay 30.000$ on top of that through a law suit would make those cheaters really cry.

    Do it while the going is good and the media is warm about the Japanese trials.

    ReplyDelete
    Replies
    1. Haha, wow. Dial back the intensity there, champ. It's just a game.

      Delete
  15. "And finally, we also found accounts where it appears the players have been playing legit in the account for a long time, and then for some reason or another decide “to just try a little cheating.” That’s sort of like “just trying a little meth.” Your mom and your local DEA officials always tell you “don’t do it” to the meth. And we say the same about cheats. Don’t do it. One day you will be caught."

    2 years ago, I tried the cheats, I played with them half the day and I got the ban permanent proof. In this account I have given more than $ 500 a little over 1600 hours of playing.
    After reading this paragraph, you say, I would not get unban?

    ReplyDelete
    Replies
    1. Serves you right being banned for cheating.

      Should have thought about that 2 years ago, shouldn't you?

      Zero tolerance for cheaters. So glad your account is gone.

      It took 2 years, which is 2 years too long, but still... now you got slammed in the face, and I hope it hurts.

      Good riddance.

      Delete
  16. Yes, yes and yes!!! Make cheater district cos you get money, game is longer alive *insert evil face here*, and other (we) will be able to finally after 1,5 years play again vs legit players. Good job guys.

    P.S. Keep up the good work :D

    ReplyDelete
  17. Wow! Miracles do happen! I haven't played regularly for a year if not longer for the sole purpose of there not being any mention of anticheat in so long. Since the beta days I have only spent $10 on the used-to-be exclusive skateboard backpack, and maybe $5 worth of premium status once. I want to support you guys for this I am telling you right now that I WILL spend money on this game one way or another! I loved this game since the start and I am really looking forward to playing regularly again. There is something about it that tugs on my strings. I really hope that things will only improve and that many players who stopped playing will come back after knowing this. (I will do my part to tell former players about this). Thanks to the APB:R team and keep up the great work!

    ReplyDelete
  18. Noooo no one will call me cheater again for 5 years+ I love being called cheater cuz it means I am as good as a hacker (I am not a hacker omg do not ban me xD)

    ReplyDelete
  19. *sigh* A breath of fresh air.
    I remember having quite a discussion with one of the banned players who was trying to tell me that it was his gaming rig with ultra low latency superfast connection that made him get the edge every time, and that the lag spikes I faced every time I got close to him were down to his extensive customisation rather than possibly other unseen elements my client was struggling to keep up with.
    8000 hours of his life lost due to cheating. The irony was that at one stage he had a young lad in his clan who was banned because he bought cheats in an effort to 'keep up' with his elder mentors who it turns out were themselves corrupt.

    Thank you Bjorn.

    ReplyDelete
  20. Thank you! Kinda feel it is time to start playing regularly again :D

    ReplyDelete
  21. Couldn't you like off load some of the engine upgrade work off on some students from universities that teach the game art and design stuff? I know a few who go to leeds uni and do this. I'm sure they'd love the chance to get some actual work experience on this kinda thing, also it would give them something for their port folio/resume.

    ReplyDelete
  22. You finally did something about this! Finally!! Was about time!

    ReplyDelete
  23. Wha?

    "But some insights made us terribly sad; players with as many as 8,000 hours of playtime were caught red-handed cheating."

    "So while we are very glad to see some of our community members go due to these flagrant violations of our terms of service, one idea that we keep kicking around is to potentially enable a “Purgatory” District.

    The idea is that cheaters don’t get banned right away (which of course doesn’t happen since the system runs on various delays), and instead freshly detected cheaters get kicked into special cheaters-only districts, where they can continue playing against other cheaters without initially realizing they have been tagged (or whole groups get thrown in there as long as they have a cheater on their team). Titanfall has implemented a similar system, and it seems to work reasonably well for them. The main reason for us to consider doing this is purely for research. We benefit from continuing to capture data from those who cheat.

    But we also realize this is a sensitive item for all players. We’d be curious to hear your feedback on this type of district (maybe it becomes a ‘mass-spectate’ thing where you can view them all cheating each other)? No final decision has been made so far about this type of new district type."

    cynicism at best.

    what G1 wants to say is:

    "Cheaters left more money than legal players"

    ReplyDelete
  24. In my opinion, they should all hackers get a second chance. It's the truth that in this game they put their money to support the development of this game! Without these people, this was not a game!
    Surely they should get ban, which will be either limited or having to write in support.
    Should it happen again, he would have gotten a ban forever or would have deleted the entire account.
    People make mistakes, and these mistakes, some can learn, so why deny them this opportunity? You do not have rather lead people to not cheat?
    By that someone catch you and ban him for cheating, so most people figure it out and you will not cheat.

    Or maybe you think that the people who you send money to support this game, are not entitled to another chance?
    Or are you as a company GamersFirst and APB Reloaded game players did not give another chance? When servers lag and it still lag? And still send you the money that you have you money?

    Above this would be your thought.

    Sincerely,

    Petr Hnátek
    Czech Republic

    Patriot EU2
    Fakerko (Criminal) (Banned 2 years ago)
    Taradinha (Enforcer) (Banned 2 ago)
    Ladyzinha (Criminal) (Current character)

    ReplyDelete
    Replies
    1. "I promise not to sexually assault children from the local primary school again your honour... after all I did donate some cash into the police benevolent fund in exchange for some stuff.... let me out of gaol and I promise I'll be good. Just let me hang around the school playgrounds again and I may even donate some more money!"

      No. Unless they can prove a false ban (difficult to do) they should stay gone. If 'Purgatory' is brought into play it should be for those who have not been kicked out thus far. Those who have cheated in the past have the opportunity to create a new account I guess and play legit - just like you seem to have after your first ban.

      Delete
    2. Yes, sexual child abuse and cheating at free-to-play shooter games are practically the same thing.

      Delete
    3. The argument are worlds apart in degrees of importance but the principles behind them are the same. I have just taken the same argument and applied it to a different situation that even a cheating piece of scum finds abhorrent. You seem to feel that one is acceptable but the other is wholly unacceptable. My stance is the NEITHER are acceptable.

      Delete
  25. Thanks you, now we can start enjoy game after few years of cheaters... Maybe it will lure new players nice.

    ReplyDelete
    Replies
    1. I hope this was life time bans and we will never see these people or most of them never more. Block IP, delete account is best option how to make stop people use this cheats

      Delete
  26. If the player plays fair and at it more than 8000 hours of time, him banned?

    ReplyDelete
    Replies
    1. He got banned because he DID NOT play fair. More information came out from some players who he teamed with. From what I hear he mostly only cheated when he got mad or in a fit of petulance and petty revenge when someone he felt he should have beaten killed him.

      Delete
  27. I loved your game and actually did pretty well at it. My girlfriend, meh, not so much. God love her, she tried. She wanted me to think she was improving so she went out and got a cheat program and mostly used it for the map functionality - to see where enemies where and if she was about to get ambushed. She was an example of those "little bit of cheating" cheaters. No excuse for her, she knew it was wrong and she did it anyway. She got banned (some time ago) and she deserved it. Unfortunately, I got banned with her and have never EVER used a cheat program in my life and, at the time, had no idea she was even using one.

    I have been so irritated about losing all my progress and everything I put into APB that I've never come back (although you still show up on my FB feed, that's how I found this article). While I think this is great, I hope it is better than PunkBuster that apparently considers people that group with cheaters to be just as bannable as actual cheaters.

    ReplyDelete
    Replies
    1. I guess you got banned too through hardware association.
      It sucks, but I'm sure you understand G1's position here.

      Theoretically someone's 'girlfriend' could actually be just an alt account that was both shared and a 'cheating' account used to cheat while trying to protect the main 'legit' account.

      I have no personal reason to doubt the truth of your story but there are a whole host of similar excuses used by people who are less honest than yourself.

      Delete
  28. i doubt any legits are getting banned from fairfight, it should only combat rage hackers

    ReplyDelete
  29. lol if i had kept playing at the same rate i could prolly had beat the #1 guy with kills that got banned

    pathetic

    ReplyDelete
  30. should make a list of closed beta ppl who still havent been banned.. it would be easier

    specially those who achieved max tl back before it was hidden XD or have achieved high # of kills

    ppl in the ban list have weak number of kills... pathetic

    ReplyDelete
  31. Ug, stupid thing ate my post. Too lazy to retype it, tl;dr:
    Purgatory: No. The game performs poorly enough, there's no need to waste server resources to have a war of the aimbots.

    Fairfight: About time, tbh. The internal anti-cheat was a novel ideal, but it clearly wasn't up to par, and it fits the same niche as fairfight. Better to just pay someone with a history to do it properly, even if it's more time and money.

    ReplyDelete
  32. Getting FairFight is great but it's two years too late. You lost a lot of players because of cheaters. Purgatory? No, how about you ban cheaters and not let them take up server resources. Why would you want to let cheaters continue to play the game? Oh I see, you still want to make money from cheaters huh?

    ReplyDelete
  33. Cheating in a game is like cheating in Vegas. You just get thrown out, all money you made is lost and your banned from entering that Casino. There is no Casino that has Purgatory, or a compulsory cheating lobby where you keep playing, but with Monopoly money (and free cocktails are only tap water with a sprinkle of Kool-Aid).

    If you are going to make a Purgatory, make it fair. Throw 100 cheating imbeciles in a district with 10 bots. Make the bots 3x faster, no recoil, macro firing assisted and with snapping, and make the cheaters play 3x slower with ridiculous recoil. I'd buy that for a dollar.

    ReplyDelete
  34. What about mouse drivers and some other stuff which may appear suspicious (like VisualStudio debugger, procmon etc)?

    ReplyDelete

Please keep comments on-topic and reasonably civilized. The moderators reserve the right to remove any off topic, uncivilized or troll-ish comments.