Cheating

[Memfis]

Let's talk about cheating (i.e. trying to pass off TAS demos as live runs).

 

Do you suspect that there is at least some cheating going on in the current scene? You don't have to name anyone specific (in fact it's probably better not to do so because most likely you can't prove anything anyway), but if you have a healthy dose of skepticism I think it would be good and interesting if you expressed some of your doubts. Yes, it's a tricky subject, but we don't have to always be completely silent about it, right?

 

Are you satisfied with the situation where cheating can be trivially done by anyone and everything depends on simple human trust? Or would you prefer if the community was actively fighting cheating by creating new ports, developing cheat detecting tools, talking about possible signs of cheating such as certain kinds of mouse turns, etc? Of course trying to strongly oppose cheating might inspire some people to come up with more advanced cheating techniques and it can became an endless ugly cycle, so I'll understand if you say that you simply don't want to go there at all.

 

Is it realistic to expect some changes in this field in the future? Do you think at some point we will get an anti-cheating port made specifically for speedrunning that will have online confirmation and other methods of ensuring that everyone plays fair? How difficult would it be to achieve something like that? Is there enough interest\knowledge in the community to create an anti-cheating port that won't look like a total joke?

[Vorpal]

I've seen majorly obvious as well as subtle and clever cheating in other, much less populous, competition communities. It's out there, people do it, and they won't stop (perhaps even spurred on by the challenge of "fooling" everyone). 

 

I just let it go, because it's not worth worrying about. Have fun competing with the folks you like/respect, and ignore the ones you suspect aren't legit.

 

To combat cheating is a sort of neverending arms race, with the end conclusion of inconveniencing legitimate players or even violating their privacy/security (e.g. processes that monitor data on their hdd and report to some server). Commercial games with such "anti cheat" measures are more of a hazard imo. I would say a simple lan party style competition with over-the-shoulder judges is the most reasonable sort of policing that can be done.

[GoneAway]

I've personally never cared whether somebody is cheating or not.  Maybe if it was really blatant it would be disappointing, but I generally take people at their word with demos.  Almost every demo, even if the person cheated, is beatable if you're willing to put in the effort, so I don't think it massively has an impact on competition.  Also, there's rarely much competition anyway, so there's probably not much worth worrying about.  If someone started posting a bunch of tas level iwad runs, maybe it'd be worth looking into, because of the "sanctity" of those runs.

[Linguica]

I keep wanting to mess around with Tensorflow and train a neural net where you feed in a demo file and it evaluates if it appears to be TAS or not. I feel like it should be pretty doable...

[Fonze]

9 minutes ago, Linguica said:

I keep wanting to mess around with Tensorflow and train a neural net where you feed in a demo file and it evaluates if it appears to be TAS or not. I feel like it should be pretty doable...

I feel like an interesting side-effect of this would be a "find out how lucky you got in that run" tool. At least on the presumption that this would be looking at how the RNG is played. I'm sure particular movements would be looked at, but it would seem to me that part of the biggest things with TASs is playing that RNG to get beatable odds.

[dew]

There's an interesting side-effect of Doom's longevity: even speedrunners hang around for years. If you're a cheater, you either gotta maintain firmly business-like attitude, or have a pretty thick skin to lie to your friends for years and years. Given how hard it is to detect tool usage, the honour system is our main line of defence. Cheating can gain you some quick respect, but community is made of personalities, not numbers in a table; and you'll have a body buried under the foundations of your house for as long as you live in our town.

 

That said, Ling's pipe dream sounds helluva cool. It would only target specific types of cheating, of course, but I like the spicy taste of modern science.

[baja blast rd.]

25 minutes ago, Linguica said:

I keep wanting to mess around with Tensorflow and train a neural net where you feed in a demo file and it evaluates if it appears to be TAS or not. I feel like it should be pretty doable...

 

The true danger of cheating is less overt TAS-like cheating (e.g. slow-mo, RNG manipulation) and more stuff like:

 

- That any good but not top-tier player can basically obliterate the human efforts of a top-tier player with much less effort by just grinding well placed segments. It wouldn't be a 'trivial' task -- one still needs to route stuff, come up with strategies, and grind out the segments -- but it's still a lot less effort. The lower bound of skill level required to pass this off is probably 'good' (or at least 'knowledgeable'), I think, because it'd be obvious when someone has no real idea what they are doing wrt tactics and strategy but still manages to turn out god-like runs.

 

- A cheater who doesn't go as far as the above can casually use segmenting every one in a while to 'save' a failed run that they feel they 'could have got' and pass it off as normal. In a practical sense, that allows them to record above a level their skill and time/energy reserves would ordinarily allow, even if theoretically they could have done it anyway. 

 

It's kind of unfortunate that it's possible. Maybe this shouldn't be posted so no would-be cheater gets inspired.

 

Edited April 30, 2017 by rdwpa

[dew]

1 minute ago, rdwpa said:

It's kind of unfortunate that it's possible. Maybe this shouldn't posted so no would-be cheater gets inspired.

Meh, I wouldn't worry about it too much. I mean, your post is 100% accurate, but it's not exactly a divine revelation to anyone who already spent a bunch of hours grinding a speedrun, heh. And if we get a horde of new zeromasters all of the sudden, because your post becomes the top google search result for "how do i cheat in the old and shitty doom because i want to screw over their speedrunner community", at least we know who to blame. Seriously though, treat it like sex talk in a decent company. Everyone knows it exists and everyone knows how to do it (and everyone here is a virgin), just don't describe all the nasty details of it.

[j4rio]

I keep telling everyone here I'm a cheater for years but nobody seems to believe me.

[Rayziik]

Keyboards are a tool, so technically all runs are TAS.

[4shockblast]

It's not unlikely that it happens, but I haven't really seen any recent demos from anyone where there was serious reason to make a cheating accusation, nor am I aware of any particularly active players who seem likely candidates for doing it (other than j4rio, of course). There's no money in this, and for most runs there isn't that much fame either, so most of the time, it seems silly that someone would bother cheating on something. Maybe there are cheated IWAD records, since those are more coveted, and there is more competition there, but there's not that much IWAD running anymore these days probably because there are better runs to try that are not as hard to beat. Even out of IWAD runs, most will not receive much attention outside of the Doom community, unless it's something like Doom 2 D2ALL UV-Speed, though for that it seems no run got more attention there than ZM's first 30uv2303, which has been beaten numerous times by several players since making the fame incentive low still :D.

Edited April 30, 2017 by 4shockblast

[Memfis]

1 hour ago, dew said:

If you're a cheater, you either gotta maintain firmly business-like attitude, or have a pretty thick skin to lie to your friends for years and years.

Well, I'm not sure about "large scale" cheating like literally posting cheated stuff every week, but I know it's totally possible to convince yourself that it's almost okay to rerecord once in a blue moon when you fail in a particularly unfair way that you "didn't deserve". I've done it once and then there was the Kimo Xvirus story with 6 cheated demos. It seems almost naive to think that we two were the only players to ever do it (this is not intended to sound like an excuse for what I did).

 

[Grain of Salt]

Abraham Lincoln once said "if you're a cheater, I will attack you with the North".

[GoneAway]

I don't think creating a machine learning tool to detect would work.  I mean you'd have to train it on known cheated or tased runs, and that would assume people will cheat in the same way.  Or you train it on expected non-cheated runs, but everyone moves differently.  If you look for things like pattern of turning movement and such, it wouldn't work on segmenting anyway.  If you learn certain patterns, someone can inject an obfuscating noise into their cheated movement to counteract your detection system.  If you want to just look at the luck factor, that obviously doesn't need a neural net :P

 

A machine learning algorithm that can guess who made a demo, that's definitely possible and something I plan on doing for fun in the future.

Edited April 30, 2017 by kraflab

[dew]

15 minutes ago, kraflab said:

I mean you'd have to train it on known cheated or tased runs, and that would assume people will cheat in the same way.

How about I or any other experienced runner produces a bunch of private cheated runs to feed into the bot's maw?

 

And yes, this won't catch segmenting, but that was already said. :P

[rehelekretep]

ignorance is bliss.

now j4rio said something im watching the start of sf12-30 with some suspicion!!

[Kotzugi]

9 hours ago, dew said:

And yes, this won't catch segmenting, but that was already said. :P

IIRC there are ways to check the binary if the game was paused during playback. Why is this not possible for detecting segments? For example, would it be possible to program a source port to produce some kind of output whenever the join button is used?

 

PS I would actually consider it the highest form of flattery if somebody suspected me of cheating ;)

[Ryback]

It's been quite a while since I watched a demo and thought, this is not on the level. Actually I'd say Twitch and the rise of speedrun broadcasting has alleviated some of this problem. If 20 people are watching a run attempted in real time you don't have to rely on trust anymore. Five years ago trust was all anyone had.

 

A general purpose TAS detector is probably too hard to write, but looking for slowmotion in a demo is easier - you could extrapolate from the length of keypresses, and maybe modify a prboom-plus port so it was able to come up with a measure for reaction time, and see if it falls into an acceptable human range. Demos recorded with slowmotion have a particular distinct look to them so there must be some way to quantify that.

 

In theory you could also come up with some kind of randomness test for demo connection. But in practice if a particular cheated demo has only 3-4 connection points amidst a 5 minute run in which the RNG is called tens of thousands of times there'd be no way to test for significance.

 

As an aside I wonder if anyone has a copy of the old LMPCHECK program Adam Hegyi used to verify incoming Compet-N submissions. Would be interesting to see what it actually tested...

[Da Werecat]

2 hours ago, Kotzugi said:

For example, would it be possible to program a source port to produce some kind of output whenever the join button is used?

You mean, deliberately leaving a clue in the demo lump?

 

A closed-source one maybe. What's stopping a speedrunner from removing the code otherwise?

[Kotzugi]

4 hours ago, Da Werecat said:

You mean, deliberately leaving a clue in the demo lump?

 

A closed-source one maybe. What's stopping a speedrunner from removing the code otherwise?

Yes. Could one limit the closed-sourcedness of the code to that particular function / set of functions? And if there is, would it be a good or bad idea in general?

Edited May 1, 2017 by Kotzugi

[Linguica]

Not gonna happen. First and foremost, Doom is open source and you can't just add closed-source code to it without breaking the terms of the GPL and having everyone get on your case. Secondly, changing the demo format in any way would mean that, by definition, you were requiring people to use a new special source port to record and play back all demos, and good luck with that one.

[BigDickBzzrak]

18 minutes ago, Linguica said:

Doom is open source and you can't just add closed-source code to it

Isn't Zandronum closed-source? How does that work?

(I'm not too familiar with all that licence stuff.)

Edited May 1, 2017 by bzzrak

[Da Werecat]

ZDaemon is.

 

It uses Doom Source License which permits it.

[4shockblast]

Yeah, creating a separate source port with anti-cheating measures to force people to move to is just not a thing that's likely to work whether or not the demo format changes (see CNDoom). The current standards are heavily entrenched, and most people simply will not care enough to use something else, especially for the vast majority of runs that simply aren't important enough that many people would even bother watching them, let alone make any cheating accusations. Forcing something like web authentication is also problematic because that forces people to have a stable Internet connection, something not everyone can afford all the time, especially in countries with generally not good Internet like Australia, and an Internet glitch could happen to anyone.

 

I also agree that machine learning probably would have limited scope and capabilities to detect cheating; maybe extreme slowmo could be detected, but differences in how people cheat at Doom and how people play Doom normally would probably make it really hard to determine anything substantial about most demos as there are too many variables affecting the results. Unnatural turning, use of strafe50, movement, etc. could probably happen to anyone in any demo, would probably happen more or less depending on the player, the wad, the player's environment, the player's experience at the game making false positives and negatives probably very likely for such a tool.

[termrork]

On 30.4.2017 at 9:28 PM, dew said:

There's an interesting side-effect of Doom's longevity: even speedrunners hang around for years. If you're a cheater, you either gotta maintain firmly business-like attitude, or have a pretty thick skin to lie to your friends for years and years. Given how hard it is to detect tool usage, the honour system is our main line of defence. Cheating can gain you some quick respect, but community is made of personalities, not numbers in a table; and you'll have a body buried under the foundations of your house for as long as you live in our town.

very nicely said!

Ontopic: honestly, if I do not know a speedrunner and he shows a really nice performance a little voice always tells me: "he might cheat".

also an easy way nobody will think you cheat is by simply playing badly. This is of course the only reason why all my records are slow as hell 8-)

[Looper]

On 1.5.2017 at 11:55 AM, Kotzugi said:

PS I would actually consider it the highest form of flattery if somebody suspected me of cheating ;)

What I've seen is that cheaters get angry/aggressive/uncomfortable quite often when being called a cheater. Legit players take it more often as a compliment, which shouldn't be that big of a surprise.