A Scathing Review of FIFA’s new ELO Rankings

We’re not trying to burn bridges, but we understand if that’s the case. Despite being a website heavily driven by rankings, we don’t talk about our rankings (or ELO Rankings) all that often. FIFA has shown no interest in what we’ve had to say previously.

“over two years of reviews and studies of different alternatives and a comprehensive consultation process with all confederations, the FIFA administration put forward to the FIFA Council an overhauled formula to calculate the FIFA/Coca-Cola World Ranking” and “a group of sports specialists and statisticians developed a formula based on the Elo method of calculation” – FIFA

We were not involved. Probably because we’ve posted a lot of stuff like the below:

• We were among the first to jump aboard the “schedule a friendly smartly” train all the way back in 2013.
• We tried to help teams who didn’t play enough matches.
• We had no problem calling out FIFA for their CAF draw disaster
• We offered an opportunity to teams eliminated way too soon from World Cup 2018 Qualifying

 

The FIFA Rankings were ripe for manipulation. And let’s be honest… teams took advantage. Romania hired one of the leading experts on the FIFA Rankings to get them into Pot 1 for the 2018 World Cup draw. The specifics are out there on the internet. We also have spoken to numerous federations in the past regarding ways to ascend the FIFA Rankings at the perfect time to maximize your Pot for a draw. It happens

We have a Consulting page on our site!  And you are certainly welcome to contact us to discuss rankings and strategy at any time. You should always have a strategy… and the longest view in the room.

 

FIFA published a new ranking system over the weekend, which converts to an “Elo” system. Elo Rankings are a zero-sum game in which one team gains points depending upon the outcome of a match and another team loses an equal number of points. The number of points typically are based on the outcome of the match versus expectation, adjusted with various factors to account for margin of victory, competition, and location. The new FIFA Rankings do not capture all of these aspects as they should.

We like the idea of switching to these Elo Rankings, but the execution is really, really bad.

 

The Good

FIFA has eliminated the absolute joke that was the “Confederation Coefficient”. Based on results at previous World Cups, FIFA docked teams not in UEFA or CONMEBOL by 15% on every single match. If you were in CONMEBOL and earned 100 points from a match, you got 100 points. If you were in CAF, you got 85 points. For no other reason than simply being in CAF, you were shoved down the rankings.

On this, FIFA said:

The Confederation weighting variables in the current formula, which make rising in the standings more challenging for teams from Confederations other than Europe or South America, will be removed. Teams will now have an equal chance to improve in the standings irrespective of their regional affiliation

This is embarrassing. The fact that this Confederation Coefficient even existed was offensive. Of course we questioned this too. We’re glad this nonsense is gone.

 

Additionally, the elimination of the points average and movement to this style system of ELO Ratings is a GOOD thing. We support it! Avoiding friendlies to climb the rankings is a bad system that needed to be corrected. We’re glad FIFA decided to make this change.

That’s about it for the good.

 

The Bad

One of the main advantages of SUM is that it allows for a smooth transition from the current ranking formula to the new one without displacement of teams in the existing ranking table. The current FIFA / Coca-Cola World Ranking will be replaced seamlessly by the new SUM formula without gain or loss of member association rank positions.

Depending on your interpretation, FIFA indicates that the new system will pick up where the old left off. We know that the order will not change, but given all of the information we have, it appears the starting point for the new rankings will be teams’ current points in the FIFA Rankings. This is a MASSIVE problem which we will get to in due time.

Let’s get into what these rankings do NOT have:

1. Any adjustment for margin of victory whatsoever
2. Any adjustment for location of the match

This is nothing new. We criticized the rankings previously (of course) all over the place for not factoring in these two critical components. Every test we’ve ever run on significance of variables has shown that location and margin of victory are hugely important in ranking teams.

 

This begs the question… what are these new FIFA ELO Rankings trying to do? In our rankings explanation, we clearly lay out the purpose of our rankings… we want to rank teams to best predict future performance. That’s the purpose of our rankings. It’s not to say who was the best over the past year, who has the most points, etc. We are trying to predict future performance based on past results. That’s it.

We understand that FIFA has a bit of a different obligation. FIFA mandates that their rankings are used to seed teams for the World Cup draw. Are the rankings intended then to create fair groups in qualifying? FIFA is clearly pissed at Romania and Wales. Ireland and the Faroe Islands were in the same Pot for cryin out loud! FIFA has drawn a line in the sand and said that they are “eliminating the potential for ranking manipulation”. This is a reactionary measure which doesn’t seem to truly address whether the rankings achieve their purpose, which is still unknown.

 

The Ugly

So what in the world is this graphic? The rankings on the left are the June FIFA Rankings. The rankings on the right are the new ELO Rankings…. after almost 500 games. We simulated the results for the World Cup, Nations League, Asian Cup, AFCON Qualifiers, Friendlies. Everything that is scheduled as of this date.

TEAMS DO NOT MOVE

We can’t overstate this at all. Teams do not move in the rankings. This is 500 games!! Yes, this could be considered a “small” sample size, but come on! This is almost all matches that will be played over the next calendar year. It’s going to take teams 5 years to climb the rankings at this rate!!!

 

So What Happened?!?!?

FIFA made a massive, massive mistake by not factoring in goal difference whatsoever. Margin of victory is one of the most important pieces of information you can use in rankings. Beating a team by 1 goal is not the same as beating a team by 8 goals. It’s not. It’s an undeniable fact that beating a team by 8 goals should be worth more than beating that same team by 1 goal.

Because there is no multiplier like in the real ELO Rankings, the amount of points teams can earn in a match are mitigated. Because points are mitigated, it makes it that much harder for teams to move up and down the standings. Teams are going to be picking up 5-10 points frequently in matches. There’s a 750 point gap between #1 and #28. Slovakia would have to beat Germany 20 times in a row just to pass them in the rankings. And that’s just if it’s a World Cup Qualifier. If it’s in friendlies, Slovakia would have to beat Germany 50 times to pass them. Slovakia could beat Germany by 10 goals on the road 40 times in a row and Germany would be ranked higher. Think about that. How does that make any sense whatsoever?

There are two huge problems with the rankings:

1. Using current points as a jumpoff point is wrong. It just is. There is way too much spread among teams as it stands for there to be significant movement.
2. Matches are not worth enough. We get that devaluing friendlies is important to avoid gaming, (BTW we don’t devalue friendlies in our rankings and we think our rankings look just fine) but there just aren’t enough points available to move the needle.

In conjunction, these two problems keep the rankings in a stable state for years and years at a time. There will be no movement in the FIFA Rankings for at least 3 years.

One big caveat here. If FIFA chooses to use a modified number of points as a jumpoff point, and not the June points for each team, it could work a bit better. The problems still exist, but it would be a little bit easier for teams to jockey in the rankings if FIFA decides to, say, cut the points in half across the board.

And here’s an unbelievable stat (thanks @Rog_nye)

 

I reworked the rankings as if new formula came in June 2014. Nothing changed from 2014 to now. Top 5 would still be the same as it was. Only 1 change in the Top 20, Russia go from 19to 25, Costa Rica from 28 to 19. Unless the starting values change the rankings will never move.

— ROGNYE (@Rog_nye) June 12, 2018

What This Means for Football

FIFA has to fix this. Maybe they will not start with the existing FIFA points (they absolutely should not). They had to have run simulations to see what would happen over time, right? The new ranking formula is nothing short of disastrous. It renders the formula, and the rankings in general, utterly useless. How can these rankings be used for anything when they’re static year after year after year?

ELO states “Ratings tend to converge on a team’s true strength relative to its competitors after about 30 matches. Ratings for teams with fewer than 30 matches should be considered provisional.”

But that’s without the two huge problems listed above. With limited points and a wider spread of rankings, it’s going to take something like 40-50 matches, if not more. That’ll take 5+ years for many teams. Minnows shouldn’t even bother. They have no chance to climb the rankings, ever. Given that the rankings are used for World Cup draws, this decision, and this formula, makes it that much harder for a smaller nation to ever qualify for the World Cup. Toiling in lower Pots for eternity and having to claw through round after round after preliminary competition takes its toll.

FIFA should be embarrassed by this formula and the harm it will cause to smaller nations. If you’re one of the lucky teams to be at the top in the June 2018 rankings, congratulations. You’ll be reaping the financial benefits from FIFA’s decision for many, many, many years to come.

This is the hurtful truth about this rankings decision. Federations which reach the World Cup benefit financially. Being higher up in the rankings gives you a better chance to qualify by being in a higher Pot. And this makes it infinitely harder for a smaller team to actually get to the World Cup. This favors the big teams in a big way.