A Scathing Review of FIFA’s new ELO Rankings

We’re not trying to burn bridges, but we understand if that’s the case. Despite being a website heavily driven by rankings, we don’t talk about our rankings (or ELO Rankings) all that often. FIFA has shown no interest in what we’ve had to say previously.

“over two years of reviews and studies of different alternatives and a comprehensive consultation process with all confederations, the FIFA administration put forward to the FIFA Council an overhauled formula to calculate the FIFA/Coca-Cola World Ranking” and “a group of sports specialists and statisticians developed a formula based on the Elo method of calculation” – FIFA

We were not involved. Probably because we’ve posted a lot of stuff like the below:

 

The FIFA Rankings were ripe for manipulation. And let’s be honest… teams took advantage. Romania hired one of the leading experts on the FIFA Rankings to get them into Pot 1 for the 2018 World Cup draw. The specifics are out there on the internet. We also have spoken to numerous federations in the past regarding ways to ascend the FIFA Rankings at the perfect time to maximize your Pot for a draw. It happens

We have a Consulting page on our site!  And you are certainly welcome to contact us to discuss rankings and strategy at any time. You should always have a strategy… and the longest view in the room.

 

FIFA published a new ranking system over the weekend, which converts to an “Elo” system. Elo Rankings are a zero-sum game in which one team gains points depending upon the outcome of a match and another team loses an equal number of points. The number of points typically are based on the outcome of the match versus expectation, adjusted with various factors to account for margin of victory, competition, and location. The new FIFA Rankings do not capture all of these aspects as they should.

We like the idea of switching to these Elo Rankings, but the execution is really, really bad.

 

The Good

FIFA has eliminated the absolute joke that was the “Confederation Coefficient”. Based on results at previous World Cups, FIFA docked teams not in UEFA or CONMEBOL by 15% on every single match. If you were in CONMEBOL and earned 100 points from a match, you got 100 points. If you were in CAF, you got 85 points. For no other reason than simply being in CAF, you were shoved down the rankings.

On this, FIFA said:

The Confederation weighting variables in the current formula, which make rising in the standings more challenging for teams from Confederations other than Europe or South America, will be removed. Teams will now have an equal chance to improve in the standings irrespective of their regional affiliation

This is embarrassing. The fact that this Confederation Coefficient even existed was offensive. Of course we questioned this too. We’re glad this nonsense is gone.

 

Additionally, the elimination of the points average and movement to this style system of ELO Ratings is a GOOD thing. We support it! Avoiding friendlies to climb the rankings is a bad system that needed to be corrected. We’re glad FIFA decided to make this change.

That’s about it for the good.

 

The Bad

One of the main advantages of SUM is that it allows for a smooth transition from the current ranking formula to the new one without displacement of teams in the existing ranking table. The current FIFA / Coca-Cola World Ranking will be replaced seamlessly by the new SUM formula without gain or loss of member association rank positions.

Depending on your interpretation, FIFA indicates that the new system will pick up where the old left off. We know that the order will not change, but given all of the information we have, it appears the starting point for the new rankings will be teams’ current points in the FIFA Rankings. This is a MASSIVE problem which we will get to in due time.

Let’s get into what these rankings do NOT have:

  1. Any adjustment for margin of victory whatsoever
  2. Any adjustment for location of the match

This is nothing new. We criticized the rankings previously (of course) all over the place for not factoring in these two critical components. Every test we’ve ever run on significance of variables has shown that location and margin of victory are hugely important in ranking teams.

 

This begs the question… what are these new FIFA ELO Rankings trying to do? In our rankings explanation, we clearly lay out the purpose of our rankings… we want to rank teams to best predict future performance. That’s the purpose of our rankings. It’s not to say who was the best over the past year, who has the most points, etc. We are trying to predict future performance based on past results. That’s it.

We understand that FIFA has a bit of a different obligation. FIFA mandates that their rankings are used to seed teams for the World Cup draw. Are the rankings intended then to create fair groups in qualifying? FIFA is clearly pissed at Romania and Wales. Ireland and the Faroe Islands were in the same Pot for cryin out loud! FIFA has drawn a line in the sand and said that they are “eliminating the potential for ranking manipulation”. This is a reactionary measure which doesn’t seem to truly address whether the rankings achieve their purpose, which is still unknown.

 

The Ugly

So what in the world is this graphic? The rankings on the left are the June FIFA Rankings. The rankings on the right are the new ELO Rankings…. after almost 500 games. We simulated the results for the World Cup, Nations League, Asian Cup, AFCON Qualifiers, Friendlies. Everything that is scheduled as of this date.

TEAMS DO NOT MOVE

We can’t overstate this at all. Teams do not move in the rankings. This is 500 games!! Yes, this could be considered a “small” sample size, but come on! This is almost all matches that will be played over the next calendar year. It’s going to take teams 5 years to climb the rankings at this rate!!!

 

So What Happened?!?!?

FIFA made a massive, massive mistake by not factoring in goal difference whatsoever. Margin of victory is one of the most important pieces of information you can use in rankings. Beating a team by 1 goal is not the same as beating a team by 8 goals. It’s not. It’s an undeniable fact that beating a team by 8 goals should be worth more than beating that same team by 1 goal.

Because there is no multiplier like in the real ELO Rankings, the amount of points teams can earn in a match are mitigated. Because points are mitigated, it makes it that much harder for teams to move up and down the standings. Teams are going to be picking up 5-10 points frequently in matches. There’s a 750 point gap between #1 and #28. Slovakia would have to beat Germany 20 times in a row just to pass them in the rankings. And that’s just if it’s a World Cup Qualifier. If it’s in friendlies, Slovakia would have to beat Germany 50 times to pass them. Slovakia could beat Germany by 10 goals on the road 40 times in a row and Germany would be ranked higher. Think about that. How does that make any sense whatsoever?

There are two huge problems with the rankings:

  1. Using current points as a jumpoff point is wrong. It just is. There is way too much spread among teams as it stands for there to be significant movement.
  2. Matches are not worth enough. We get that devaluing friendlies is important to avoid gaming, (BTW we don’t devalue friendlies in our rankings and we think our rankings look just fine) but there just aren’t enough points available to move the needle.

In conjunction, these two problems keep the rankings in a stable state for years and years at a time. There will be no movement in the FIFA Rankings for at least 3 years.

One big caveat here. If FIFA chooses to use a modified number of points as a jumpoff point, and not the June points for each team, it could work a bit better. The problems still exist, but it would be a little bit easier for teams to jockey in the rankings if FIFA decides to, say, cut the points in half across the board.

And here’s an unbelievable stat (thanks @Rog_nye)

 

What This Means for Football

FIFA has to fix this. Maybe they will not start with the existing FIFA points (they absolutely should not). They had to have run simulations to see what would happen over time, right? The new ranking formula is nothing short of disastrous. It renders the formula, and the rankings in general, utterly useless. How can these rankings be used for anything when they’re static year after year after year?

ELO states “Ratings tend to converge on a team’s true strength relative to its competitors after about 30 matches. Ratings for teams with fewer than 30 matches should be considered provisional.”

But that’s without the two huge problems listed above. With limited points and a wider spread of rankings, it’s going to take something like 40-50 matches, if not more. That’ll take 5+ years for many teams. Minnows shouldn’t even bother. They have no chance to climb the rankings, ever. Given that the rankings are used for World Cup draws, this decision, and this formula, makes it that much harder for a smaller nation to ever qualify for the World Cup. Toiling in lower Pots for eternity and having to claw through round after round after preliminary competition takes its toll.

FIFA should be embarrassed by this formula and the harm it will cause to smaller nations. If you’re one of the lucky teams to be at the top in the June 2018 rankings, congratulations. You’ll be reaping the financial benefits from FIFA’s decision for many, many, many years to come.

This is the hurtful truth about this rankings decision. Federations which reach the World Cup benefit financially. Being higher up in the rankings gives you a better chance to qualify by being in a higher Pot. And this makes it infinitely harder for a smaller team to actually get to the World Cup. This favors the big teams in a big way.

 

8 thoughts on “A Scathing Review of FIFA’s new ELO Rankings

  • June 13 at 5:17 am
    Permalink

    I never liked too mutch the ELO formula because of the slowness in movements, but it could be an improvement respect the previous horrific system if well done. Of course this Fifa decisions are the worst. But the bigger problem in my opinion aren’t the inizial points.
    In the current World Football Elo Ratings ranking, the difference between Brazil (top of the ranking) and the last Fifa nation, American Samoa, is 1678 points.
    In the June Fifa ranking the difference between the first and the last is 1522.
    Is true that differences are too large for nations at the top of the ranking, but in my opinion the big problem is not that Fifa decides to use these starting points, but the k-factor that is too small.
    The maximum value, 60 for the World Cup, is the same for both systems. But in the WFER ranking it is increased by 1,5 if you win a match by two goals, and 1,75 if you win by three. So you can reach 90 or 105. When Germany beaten Brazil 7-1, they reached 127,5. Fifa decided to not use goal difference, and this is not a big problem, but they failed to adjust consequently the K-factor. It is like every match is winned by the minimum result 1-0, and the ranking moves too slow. I think it would be enough to double the k-factor decided by Fifa, to have a decent ranking.

    Reply
  • June 17 at 4:26 pm
    Permalink

    To be fair, that FIFA statement did not say that teams points’ tallies would remain the same, only that their ranking positions would. Perhaps they have some formula to convert the old tallies to the new system. I suspect FIFA doesn’t want Italy, for example, to miss out on another World Cup, if only for self-interested financial reasons. I don’t think they’ll do what you described above, though I could be wrong.

    Reply
    • June 17 at 6:13 pm
      Permalink

      And we fully acknowledge that. Just come out and say it. “Transitions seamlessly” is not the same as modifying what’s in place. They definitely don’t want the big teams to miss out.

      Reply
    • June 20 at 5:27 am
      Permalink

      I agree with mister X, I don’t think Fifa will use the same points. There is a different version of the Fifa document, published on 14 june (the other was published on 10 june) where the I constant are from 0.5 to 6.0, instead 5 to 60. It’s clear that using these constants with the actual points will lock the ranking as of June 2018.

      Reply
      • June 20 at 5:54 am
        Permalink

        We actually assumed 5 to 60 were the I constants in our projection (the other just doesn’t make sense).

        They still have not clarified what the starting points are. Starting to think it’s a bit more intentional. To release a brand new ranking system and have the winner of the World Cup not be #1 is a bad look.

        Reply
        • June 20 at 7:15 am
          Permalink

          0.5 to 6.0 with a scaling constant of 600 won’t make any sense regardless of the initial points, so one of them *has* to be wrong.

          You might be right that the initial points will only be decided after the World Cup results are in. Anyone other than Germany or Brazil winning should yield a ranking system that’s better than the old one, in that case.

          Have you or has anyone asked the FIFA directly what the starting points are? An evasive answer would be a pretty big clue that they can’t answer the question *yet*.

          Reply
  • August 6 at 3:11 pm
    Permalink

    And France will still be 7th and Croatia 20th? Lol

    Reply

Please Comment Below!