Ranking Algorithms for Competitive Ultimate

1. Produce a ranking

Of course, the ranking algorithm should produce an overall ranking of teams. Everybody agrees with this.

2. Consider the structure of tournaments (not every game worth the same)

To me, the principle weakness of the actual USAU ranking algorithm is that it doesn’t see the structure of tournaments. The algorithm can be great for a league or whatever sport where games are worth the same, but not for competitive ultimate where games are played during tournaments. For the algorithm used by USAU, a tournament is just a bunch of random games. The algorithm does not see the difference between the Saturday morning game and the Sunday afternoon final (apart from the one day difference which won’t affect much).

The following paradox illustrates how the results of games in a tournament are completely independent of the outcome of the tournament itself. Four teams (A, B, C and D) play in a tournament. The format is a round robin followed by semi-finals, final and 3rd place game: 10 games all in all. The results are (in no particular order):

• game #0: (15) A – C (7)
• game #1: (15) B – D (6)
• game #2: (15) A – D (8)
• game #3: (15) B – C (10)
• game #4: (15) A – B (12)
• game #5: (15) C – D (9)
• game #6: (12) A – D (15)
• game #7: (7) B – C (15)
• game #8: (8) A – B (15)
• game #9: (9) C – D (15)

From this information only, can you decide which team won the tournament? The answer is NO. If games #0, #1, #2, #3, #4, #5 are pool play games, then the seeding out of pool play is A(3-0), B(2-1), C(1-2), D(0-3). Games #6 and #7 are the semis, game #8 is the 3rd place game and game #9 is the final. Team D wins the tournament. Final standings are D-C-B-A.

If games #0,#1,#6,#7,#8,#9 are played in the pool, then the pool outcome becomes B(2-1), D(2-1), A(1-2), C(1-2). Teams B and C play in semi-final game #3 while Teams A and D play their semi in game #2. The final game is game #4. The 3rd place is game #5. Team A wins the tournament. Final standings are A-B-C-D.

After the tournament, we can discuss about what should be the proper ranking of those four teams based on the results of the 10 games they played. But, there are good chances that we are just wrong as the above paradox shows. In a tournament, all games are not equal. It is all about the context. A good ranking algorithm should consider the format of tournaments and accept that some games are worth more than others. How should it do this? The actual algorithm is already very complicated. How should it put more value on particular games? It may possible, but it may be complicated as well.

I believe there is an easy solution to this. The algorithm must take a step back. Looking at games is like looking into a microscope: we don’t see the global picture. Which team gets the trophy at the end of the tournament in front of everyone without contestation from anybody? The trophy is given at the end of the tournament to the team who reached the final and won it. This is the best team. The best team is not the one who got the best overall game results. This is why I believe ranking algorithms in competitive ultimate should be based on the outcome of tournaments and not on the outcome of games.

3. Be predictable

In a text by Lou Burruss about rankings published in 2012, the author speaks about the recursive aspect of the actual ranking algorithm making it unpredictable:

“In conversation with Sholom, he made clear there is still some mystery to this very complex process. Even after 20 years of official rankings and simulations, it is impossible to predict exactly how things will behave.”

On the contrary, if the ranking system is predictable, then teams know in advance that winning a certain game will give them exactly 200 points, let’s say, and if this is the amount they need to become #1 in the country, then it will be one more motivation to win the game.

This is when rankings are no longer only about rankings. Rankings can also be a source of motivation for a team to improve. But rankings will do this at its best when it is predictable. This is why I believe USA Ultimate should look into this if they want to enhance the level of competition as they claim:

“In the ongoing effort to enhance the level of competition and achieve organizational goals, USA Ultimate continues to look to improve the ranking algorithm […].”

4. Reward what is the most valuable: winning when it counts

In my opinion, rankings should reward what is the most valuable to the community. This should correspond to what is the most difficult to achieve. Again, this point is a necessary condition for the ranking to be used as a source of motivation for a team to improve.

If a ranking values something else, then people won’t value the ranking and a team won’t try to improve themselves to get a better ranking. For example, think about a ranking that would only consider the best differential. We all know that the best differential in a tournament can be reached by the team G who finishes 7th out of 16 teams. The team G will think they are the best in the country and they won’t try to win a tournament: too much effort and useless. The team A who put all the effort to get to the final and win it will wonder if they are optimizing the good thing. They will be less motivated to win another tournament.

Naturally, the community values teams getting in the final and winning it. This is what I call the natural attractor. The ranking algorithm chosen also defines a ranking attractor. As we have seen in the last paragraph, the two attractors can be different. But, I strongly believe the best choice of ranking algorithm is when syzygy (picture here) happens; that is, when the ranking attractor equals the natural attractor.

“The syzygy produces the more powerful spring tide due to the enhanced gravitational effect of the Sun added to the Moon’s gravitational pull.”

In this situation, the natural attractors and the ranking attractor double their forces and team then really wants to win tournaments because they get the natural recognition from the community plus they get an high rank in the official ranking. This is the best way to optimize the improvement of teams.

It is known that the actual ranking algorithm encourages teams to play a bit differently than they would if it was just about winning tournaments. In Maximizing Your Team’s Ranking: Strength of Schedule or Margin of Victory? (Dec. 2013), Scott Schriner confirms this. He wrote:

“To increase your game performance score, you will want to win more games — but also win each game by as large of a margin as possible. This might include playing your starters longer than necessary, or loading up your D Line with an O Line handler even if you don’t need that break. Of course that decision carries with it other tradeoffs: you may increase injury risk or stall the development of younger players on your team.”

I believe it is a mark of respect for tournament structure that the ranking attractor corresponds to the the natural attractor. Moreover, this system is trusting teams and letting them have all the liberty they need to win a tournament. Good teams will peak at the good moment and will try new strategies or give more playing time to rookies at other moments. Teams that try to optimize every point of every game do not improve and do not push the game further.”

A ranking system for competitive ultimate

There is an important observation to be made when creating overall ranking in competitive ultimate. The output of a tournament is a ranking of teams. Why not build on this to create an overall ranking? A natural system is to give points to teams according to the final ranking. In such a system, teams need to win tournaments to get the maximal amount of points.

I think a system like tennis ATP and WTA rankings is a good inspiration for competitive ultimate. Tennis players go to tournaments they choose. Some tournaments are worth more (Grand Slams) and some less (ATP World Tour Masters 1000, ATP 500, ATP 250). The number of points for a tennis player is the sum of his or her best 18 results.

The problem with tennis compared to ultimate is that there are no placement games. The outcome of a tennis tournament is that you either win the final, make the final, make the semis, make the quarters, make the round of 16, and so on. In ultimate, there are placement games and finishing 9th is better than finishing 16th. Also, 18 tournaments is a lot. We need to consider fewer tournaments.

In 2007 in Quebec, we adapted the ATP ranking to ultimate tournaments with placement games. In 2011, the series became very popular. We needed to adjust the system (88 teams took part in the Mars Attaque 2011 but only 32 got points). Since 2011, 1000 points is given to the winner of a Grand Slam event, 938 points is given to the finalist, 884 points is given to the 3rd place, 835 points to the 4th place and this goes until the team finishing in 50th place obtains 1 point. The number of points are obtained from the integral of a logarithmic function. I am going to write another text on my blog where I explain the details and the math behind the system.

Example: ranking of USA Open Club teams during 2013 regular season

As an example, I am considering the 2013 USA Club Open seaon. I selected the following list of 14 tournaments. Those tournaments are divided into 2 groups (7 tournaments of 500 points, 7 elite tournaments of 1000 points).

Below is the resulting ranking made from the 2013 USA Club Open season with the parameters chosen above. Comments are welcome.

Suppose such a ranking was used for bid allocation in USA. The next table compares the number of bids that were allocated per region for the USAU Championship with the number of teams per region in the top 16 of the above ranking.

Of course, the relative number of points for tournaments is important. It must be well studied and tested to reach the good equilibrium. For example, maybe 400 or 600 points is best suited for second level tournaments instead of 500.