Rating, in the Elo sense which seems to be popular today, is the gold standard for measuring skill because it’s both meaningful and objective.

You can assign any old number you want to a performance, but a proper rating has predictive power. If we know the ratings of the players in a match, we can convert that information into an accurate probability of winning or losing the match. That’s what a rating is. You can assign a rating number however you want, as long as there’s a rule for how to convert it into probability-of-winning.

I’ll stay on this point because it’s important. Someone can assign a rating by sticking a wet finger into the air, or by an advanced, statistical, computationally heavy iterative formula. It doesn’t matter. What matters is being able to translate it into a probability of winning a specific match. The rating that yields the best win predictions is the best rating, no matter how it was constructed.

This is what is meant by meaningful: a rating directly measures skill through chance of winning. And it’s objective because the best rating system is determined by real-world outcomes rather than someone’s opinion. Thanks to its objectivity, a rating system can be calibrated over time to increase our faith in its meaningfulness.

The problem with ratings is that to yield good predictions, they necessarily have to update slowly. If they update too quickly, they will overfit to noise, and noisy predictions are bad predictions.4⁴ I think the Counter-Strike 2 Premier rating actually suffers from this. My rating wiggles up and down a lot – by almost 15 % of the available range! A rating does an excellent job of measuring someone’s skill over the long term.

For improving our game5⁵ And crafting handy decision rules such as taking fights that are 54 % in our favour. we want faster feedback. That’s where per-match metrics like the Leetify rating we saw in the previous article comes into play. It asks not “how good is this player” but “how much of their team’s win in that specific match was caused by this player?”

We can’t know how good a player is based on one match alone, because there’s a lot of noise in there. But we do have a remote chance of figuring out how much of the outcome can be attributed to the player, because we’re now asking about something concrete that have just happened, rather than an abstract quality inherent to a player.

There’s a danger of conflating the two questions. Someone contributing much to a win does not necessarily mean they are good. They could just have been lucky. But in constructing and computing this per-match metric, we have to figure out what it actually means to contribute to a win, and that’s a valuable question to ask!

When asking that question, it is tempting to get subjective. Here’s what one service, tracker.gg, says about their system:

[…] we’ve picked specific factors that we think are strong performance indicators. […] We understand deciding what stats are chosen for performance rating is controversial. Our choices for factors were made based on a combination of intuition about the games we love, and a more boring detailed analysis. With that said, we believe the community has great intuition, and if we find that a change is needed to how or what we’re scoring, we’re ready to make adjustments.

It sounds as if they are measuring performance based off of their own vibes, and/or community vibes. Of course that is going to be controversial! When evaluating people, choosing subjective criteria is inviting controversy.

But it doesn’t have to be this way. Statistically inferring causality based on past events can be difficult and hairy, but it is possible.6⁶ Causality: Models, Reasoning, and Inference; Pearl; Cambridge University Press; 2009. We have the data. Instead of picking factors someone thinks are important performance indicators, we can figure out what are important performance indicators. We can figure out how important they are relative to each other, and use that to weigh them, rather than pick weights based on what someone thinks sounds good. This can be really simple to start out, and then expanded and deepened as more factors are discovered with time. That’s what I like about Leetify – it appears they are doing exactly that.7⁷ I have also had recommended another alternative service, scope.gg. From what I can tell, their rating is similar to the hltv 2.0 rating, i.e. it decomposes into kills, deaths, or proxies thereof and does not try to capture the nuance of the game the way the Leetify rating does. When they attribute +12 rating to a player, it’s because that player caused an event in game that improved their team’s win probability by 12 %. Over large timescales, this is verifiable!

I think the reason many people prefer not to do this is that it often gives a boring and uninteresting result. Counter-Strike matches are really noisy. Many of the factors people want to matter are probably inconsequential enough in the big scheme of things that adding them to a model mainly serves to increase noise, rather than improve its predictive power.

A concrete example of this from the previous article is the number of assists a player has gotten per round. An assist means a player did non-trivial damage to an opponent that was later killed by another teammate. An assist ought to be a good thing. It ought to matter. It speaks to the part of our brain that does logic. Obviously an assist is good. But when we chuck it into the model, we learn that it’s not important at all. That kind of discovery annoys people to no end. Statistical reasoning often ends up standing in opposition to logical reasoning, and we have been taught to revere logic above all else.8⁸ This rant bleeds into another article on statistical literacy I’ve been intending to write, so I’ll try to keep it brief.

Recently, Leetify seems to have made a similar discovery around smoke grenades. Sure, they are good. They are important. But when Leetify had finally nailed down an exact definition of what should count as a good smoke grenade, it didn’t appear to be correlated with winning at all, so they had to take that definition back to the drawing table. I think most of the detailed definitions my readers’ could think of for what constitutes “a good smoke grenade” would end up not correlating at all with victory.

We can start by answering the first question. I have downloaded data from my performance in something like 60 of my last matches.

Among these, there are 26 wins, yielding a win rate of 45 %. My Leetify rating in these matches is distributed thusly:

We learned in the previous article, and can confirm here, that my Leetify rating is negative on average. This coincides with the win rate which is below half.

Now, we wanted to know if the Leetify rating is a good predictor for match outcome. We can tell immediately from a scatterplot (and an +0.6 correlation) that it is some sort of indicator.

This is actually a little surprising. Remember this is just the Leetify rating of one player on the team. That player could have the game of their life but be paired up with bad teammates. Or they have an off day but happen to be paired up with good teammates. Despite all those unknown variables, the performance of one player looks like it’s a meaningful indicator of how the game went.9⁹ If you guessed victory when that one player had a positive rating and loss otherwise, you’d be right more than 75 % of the time! This is an argument against the so-called “Elo hell”: if you consistently play well, you’ll have a 75 % win rate on average, even if you sometimes get matched up with a bad team.

If we do a logistic regression of Leetify rating against match outcome, we get a fitted curve like this, which translates a Leetify rating into probability of winning. Note that it does this based on my recent match history – this is not the definition of the Leetify rating, it’s just a consequence of the outcomes of my recent matches.

How do we know if this is a good fit? We can compute its forecasting error with cross-validation. The Leetify rating gets a Brier score of 0.17 when trying to predict the outcome of my matches. I would consider that to be fairly good – if I had that score in a forecasting tournament I would give myself a pat on the back.

It’s not perfect. We knew that ahead of time. I don’t think there’s any way to use the statistics from a single player to determine match outcome perfectly, because it is going to depend on everyone else on the team as well! But it’s fairly good.

We could construct a similar model based on e.g. the excess kills per round, which is what we’ll call the difference in kill rate and death rate.

This would have been similarly surprising, if we hadn’t discovered in the previous article that the Leetify rating is to a large extent made up of just excess kill rate.

With the same logistic regression, we get a very similar curve.

This model, based purely on kills and deaths, how well does it predict match outcome compared to the Leetify rating? Turns out it’s very close: Brier score of 0.18. Comparing the two models’ forecasting error distributions, we can also see that they are practically equivalent.

My conclusion here is that yes, the Leetify rating is a good match outcome predictor – but it’s not much better than the excess kill rate. Since the excess kill rate is much easier to compute10¹⁰ The Leetify rating is somewhat of a trade secret, after all, so I wouldn’t be able to replicate it. Anyone can replicate the excess kill rate!, I actually prefer that model.

The next thing we might want to do, then, is figure out what players can do to improve their chances of getting kills. Here are some statistics easily available from Leetify that might help get kills:

• Average flick angular distance
• Body shot accuracy
• Head shot accuracy
• Spray accuracy
• Counter-strafing
• Time to damage
• Enemies flashed
• Average blind time
• Grenade damage
• Fire damage
• Unused utility
• Shots fired
• Trade kill opportunities
• Trade kill attempts

In my – very small – data set, only one of these variables is correlated with excess kill rate.

It wasn’t obvious to me at first which one that would be: number of shots fired. In other words, players that shoot their gun more also get more kills before they die. My best guess for why this relationship comes up is that it’s an inverse causation: people who happen to stay alive for longer will both get more kills, and get more opportunities to fire their gun.

This means it’s sort of another dead-end from the perspective of getting actionable analytics out of the enterprise. It’s highly likely these other variables do help getting kills, but with the limited data at my disposal, we just can’t tell signal from noise.11¹¹ If only Leetify had a public api that could be used to get more data!

So the question of why we don’t look at more actionable analysis is simply that I do not have the data for it. I have many ideas of things that could be explored by downloading a large amount of match demo files and parsing them programmatically, but it is not important enough for me to spend more time on. Sorry!