Ranking Algorithms for Competitive Ultimate

by | April 17, 2014, 8:00am 0

Ranking teams is a great challenge. There can be a lot of mathematics and complicated computations involved. The ranking algorithm used by USA Ultimate is not an exception. Taking as huge an input as the results of thousands of games played during a season, it outputs a ranking of teams. Recently, USA Ultimate announced modifications to their ranking algorithm for the 2014 Season, which I see as local changes keeping the same global approach. I quickly shared my thoughts on Ultiworld recently. Then, I was asked to explain my ideas more deeply, which I do in this text.

I believe the actual USAU ranking algorithm is not too bad at fullfilling its objective: decide bids per region. Once the number of bids are set, the season results and the USAU ranking has no impact anymore on deciding which team is the best. The best team will be the team who won Nationals after qualifying through Sectionals and Regionals. But depending on the objective to achieve, the choice of ranking algorithm can be more important as its role is more determinant:

Role A. Rank the teams at all times of the year (like tennis ATP and WTA rankings)
Role B. Choose the season champion (like in Formula 1)
Role C. Select teams for a championship (Quebec Ultimate 4 on 4 Circuit)
Role D. Decide bids per region (USAU)

Depending on the role of the ranking, the chosen ranking algorithm might be more or less suited. In this text, I explain my point of view on ranking algorithms for competitive ultimate. I consider the case where the season is made of many tournaments and where the ranking has a role in deciding the national champion, that is like role B or C above. It may also apply for deciding bids per region (role D). But whatever its role, to me, a good ranking algorithm for competitive ultimate must:

  1. Produce a ranking
  2. Consider the structure of tournaments (not every game worth the same)
  3. Be predictable
  4. Reward what is the most valuable (winning when it counts)

Below, I explain each of the above conditions. I also propose a ranking algorithm for competitive ultimate based on results of tournaments. This text ends with an example on the 2013 Club Open season to see how my proposed ranking would work for bid allocation in USA Ultimate. In general, I hope my text can stimulate a reflection about what algorithm is best suited for competitive ultimate.

1. Produce a ranking

Of course, the ranking algorithm should produce an overall ranking of teams. Everybody agrees with this.

2. Consider the structure of tournaments (not every game worth the same)

To me, the principle weakness of the actual USAU ranking algorithm is that it doesn’t see the structure of tournaments. The algorithm can be great for a league or whatever sport where games are worth the same, but not for competitive ultimate where games are played during tournaments. For the algorithm used by USAU, a tournament is just a bunch of random games. The algorithm does not see the difference between the Saturday morning game and the Sunday afternoon final (apart from the one day difference which won’t affect much).

The following paradox illustrates how the results of games in a tournament are completely independent of the outcome of the tournament itself. Four teams (A, B, C and D) play in a tournament. The format is a round robin followed by semi-finals, final and 3rd place game: 10 games all in all. The results are (in no particular order):

  • game #0: (15) A – C (7)
  • game #1: (15) B – D (6)
  • game #2: (15) A – D (8)
  • game #3: (15) B – C (10)
  • game #4: (15) A – B (12)
  • game #5: (15) C – D (9)
  • game #6: (12) A – D (15)
  • game #7: (7) B – C (15)
  • game #8: (8) A – B (15)
  • game #9: (9) C – D (15)

From this information only, can you decide which team won the tournament? The answer is NO. If games #0, #1, #2, #3, #4, #5 are pool play games, then the seeding out of pool play is A(3-0), B(2-1), C(1-2), D(0-3). Games #6 and #7 are the semis, game #8 is the 3rd place game and game #9 is the final. Team D wins the tournament. Final standings are D-C-B-A.

If games #0,#1,#6,#7,#8,#9 are played in the pool, then the pool outcome becomes B(2-1), D(2-1), A(1-2), C(1-2). Teams B and C play in semi-final game #3 while Teams A and D play their semi in game #2. The final game is game #4. The 3rd place is game #5. Team A wins the tournament. Final standings are A-B-C-D.

After the tournament, we can discuss about what should be the proper ranking of those four teams based on the results of the 10 games they played. But, there are good chances that we are just wrong as the above paradox shows. In a tournament, all games are not equal. It is all about the context. A good ranking algorithm should consider the format of tournaments and accept that some games are worth more than others. How should it do this? The actual algorithm is already very complicated. How should it put more value on particular games? It may possible, but it may be complicated as well.

I believe there is an easy solution to this. The algorithm must take a step back. Looking at games is like looking into a microscope: we don’t see the global picture. Which team gets the trophy at the end of the tournament in front of everyone without contestation from anybody? The trophy is given at the end of the tournament to the team who reached the final and won it. This is the best team. The best team is not the one who got the best overall game results. This is why I believe ranking algorithms in competitive ultimate should be based on the outcome of tournaments and not on the outcome of games.

3. Be predictable

In a text by Lou Burruss about rankings published in 2012, the author speaks about the recursive aspect of the actual ranking algorithm making it unpredictable:

“In conversation with Sholom, he made clear there is still some mystery to this very complex process. Even after 20 years of official rankings and simulations, it is impossible to predict exactly how things will behave.”

On the contrary, if the ranking system is predictable, then teams know in advance that winning a certain game will give them exactly 200 points, let’s say, and if this is the amount they need to become #1 in the country, then it will be one more motivation to win the game.

This is when rankings are no longer only about rankings. Rankings can also be a source of motivation for a team to improve. But rankings will do this at its best when it is predictable. This is why I believe USA Ultimate should look into this if they want to enhance the level of competition as they claim:

“In the ongoing effort to enhance the level of competition and achieve organizational goals, USA Ultimate continues to look to improve the ranking algorithm […].”

4. Reward what is the most valuable: winning when it counts

In my opinion, rankings should reward what is the most valuable to the community. This should correspond to what is the most difficult to achieve. Again, this point is a necessary condition for the ranking to be used as a source of motivation for a team to improve.

If a ranking values something else, then people won’t value the ranking and a team won’t try to improve themselves to get a better ranking. For example, think about a ranking that would only consider the best differential. We all know that the best differential in a tournament can be reached by the team G who finishes 7th out of 16 teams. The team G will think they are the best in the country and they won’t try to win a tournament: too much effort and useless. The team A who put all the effort to get to the final and win it will wonder if they are optimizing the good thing. They will be less motivated to win another tournament.

Naturally, the community values teams getting in the final and winning it. This is what I call the natural attractor. The ranking algorithm chosen also defines a ranking attractor. As we have seen in the last paragraph, the two attractors can be different. But, I strongly believe the best choice of ranking algorithm is when syzygy (picture here) happens; that is, when the ranking attractor equals the natural attractor.

“The syzygy produces the more powerful spring tide due to the enhanced gravitational effect of the Sun added to the Moon’s gravitational pull.”

In this situation, the natural attractors and the ranking attractor double their forces and team then really wants to win tournaments because they get the natural recognition from the community plus they get an high rank in the official ranking. This is the best way to optimize the improvement of teams.

It is known that the actual ranking algorithm encourages teams to play a bit differently than they would if it was just about winning tournaments. In Maximizing Your Team’s Ranking: Strength of Schedule or Margin of Victory? (Dec. 2013), Scott Schriner confirms this. He wrote:

“To increase your game performance score, you will want to win more games — but also win each game by as large of a margin as possible. This might include playing your starters longer than necessary, or loading up your D Line with an O Line handler even if you don’t need that break. Of course that decision carries with it other tradeoffs: you may increase injury risk or stall the development of younger players on your team.”

I believe it is a mark of respect for tournament structure that the ranking attractor corresponds to the the natural attractor. Moreover, this system is trusting teams and letting them have all the liberty they need to win a tournament. Good teams will peak at the good moment and will try new strategies or give more playing time to rookies at other moments. Teams that try to optimize every point of every game do not improve and do not push the game further.”

A ranking system for competitive ultimate

There is an important observation to be made when creating overall ranking in competitive ultimate. The output of a tournament is a ranking of teams. Why not build on this to create an overall ranking? A natural system is to give points to teams according to the final ranking. In such a system, teams need to win tournaments to get the maximal amount of points.

I think a system like tennis ATP and WTA rankings is a good inspiration for competitive ultimate. Tennis players go to tournaments they choose. Some tournaments are worth more (Grand Slams) and some less (ATP World Tour Masters 1000, ATP 500, ATP 250). The number of points for a tennis player is the sum of his or her best 18 results.

The problem with tennis compared to ultimate is that there are no placement games. The outcome of a tennis tournament is that you either win the final, make the final, make the semis, make the quarters, make the round of 16, and so on. In ultimate, there are placement games and finishing 9th is better than finishing 16th. Also, 18 tournaments is a lot. We need to consider fewer tournaments.

In 2007 in Quebec, we adapted the ATP ranking to ultimate tournaments with placement games. In 2011, the series became very popular. We needed to adjust the system (88 teams took part in the Mars Attaque 2011 but only 32 got points). Since 2011, 1000 points is given to the winner of a Grand Slam event, 938 points is given to the finalist, 884 points is given to the 3rd place, 835 points to the 4th place and this goes until the team finishing in 50th place obtains 1 point. The number of points are obtained from the integral of a logarithmic function. I am going to write another text on my blog where I explain the details and the math behind the system.

Example: ranking of USA Open Club teams during 2013 regular season

As an example, I am considering the 2013 USA Club Open seaon. I selected the following list of 14 tournaments. Those tournaments are divided into 2 groups (7 tournaments of 500 points, 7 elite tournaments of 1000 points).

Tournaments considered Points for champion
Cal State 500
Cazenovia 500
No Surf 500
Old Dominion Q. 500
Mot. Throwdown 500
San Diego Slammer 500
Col. Cup noTCT 500
Chesapeake Invite 1000
Club Terminus 1000
Colorado Cup 1000
Heavyweights 1000
Pro Flight Finale 1000
US Open 1000
West Coast Cup 1000

 

Below is the resulting ranking made from the 2013 USA Club Open season with the parameters chosen above. Comments are welcome.

PosPointsTeam NameRegionBest2nd Best3rd Best
12746DoublewideSC1 (1000) Colorado Cup2 (911) Pro Flight Finale3 (835) US Open
22670PoNYNE1 (1000) Club Terminus3 (835) Colorado Cup3 (835) Chesapeake Invite
32619RevolverSW1 (1000) US Open2 (911) West Coast Cup5 (708) Pro Flight Finale
42603SockeyeNW1 (1000) West Coast Cup3 (835) Club Terminus4 (768) Pro Flight Finale
52603MachineGL1 (1000) Heavyweights3 (835) Pro Flight Finale4 (768) Club Terminus
62387Johnny BravoSC2 (911) Colorado Cup4 (768) West Coast Cup5 (708) Club Terminus
72377IronsideNE2 (911) US Open2 (911) Chesapeake Invite8 (555) Pro Flight Finale
82361GOATNE1 (1000) Pro Flight Finale5 (708) Chesapeake Invite6 (653) Club Terminus
92166Chain LightningSE2 (911) Club Terminus6 (653) Chesapeake Invite7 (602) Pro Flight Finale
102140Sub ZeroNC1 (1000) Chesapeake Invite5 (708) Heavyweights11 (432) Club Terminus
112043Ring of FireSE3 (835) US Open6 (653) Pro Flight Finale8 (555) Chesapeake Invite
121937High FiveGL3 (835) Heavyweights7 (602) Chesapeake Invite1 (500) No Surf
131927RhinoNW5 (708) Colorado Cup5 (708) West Coast Cup9 (511) Club Terminus
141910Madison ClubNC2 (911) Heavyweights7 (602) Colorado Cup12 (397) Club Terminus
151881Truck StopMA4 (768) Colorado Cup7 (602) Club Terminus9 (511) Chesapeake Invite
161685InceptionSC4 (768) Heavyweights1 (500) Col. Cup noTCT3 (417) San Diego Slammer
171492StreetgangSW6 (653) Heavyweights2 (455) San Diego Slammer4 (384) Cal State
181455Chicago ClubGL5 (708) US Open4 (384) Mot. Throwdown13 (363) Heavyweights
191340LA RenegadeSW7 (602) Heavyweights4 (384) Cal State5 (354) San Diego Slammer
201302CondorsSW1 (500) Cal State1 (500) San Diego Slammer15 (302) Colorado Cup
211284MadcowGL2 (455) Mot. Throwdown11 (432) Colorado Cup12 (397) Chesapeake Invite
221252Cash CropSE10 (471) Chesapeake Invite12 (397) Colorado Cup4 (384) Old Dominion Q.
231087Dire WolfMA2 (455) No Surf4 (384) Cazenovia17 (248) Heavyweights
241085Prairie FireNC6 (653) Colorado Cup11 (432) Heavyweights
251011Florida UnitedSE9 (511) Colorado Cup1 (500) Old Dominion Q.
261010Boost MobileSW8 (555) Club Terminus2 (455) Cal State
271007Medicine MenMA12 (397) Heavyweights5 (354) Old Dominion Q.9 (256) Cazenovia
28986MephistoNE7 (602) US Open4 (384) Cazenovia
29888VoodooNW10 (471) Colorado Cup3 (417) San Diego Slammer
30865SprawlSW9 (511) Heavyweights5 (354) San Diego Slammer
31835Buzz Bullets3 (835) West Coast Cup
32833GarudaNE8 (555) Heavyweights8 (278) Cazenovia
33768Clapham4 (768) Chesapeake Invite
34686OaksSW5 (354) Cal State14 (332) Heavyweights
35656CastleNC5 (354) Mot. Throwdown15 (302) Heavyweights
36653Euforia6 (653) US Open
37632Sheet Metal5 (354) No Surf8 (278) Cazenovia
38594Beachfront PropertyGL3 (417) Mot. Throwdown20 (177) Heavyweights
39575Space City IgniteSC7 (301) San Diego Slammer16 (274) Heavyweights
40562Chico6 (326) Cal State10 (236) San Diego Slammer
41555PlexSC8 (555) Colorado Cup
42555Ragnarok8 (555) US Open
43534GridlockSW8 (278) Cal State9 (256) San Diego Slammer
44500Ulysse1 (500) Cazenovia
45471BrickyardGL10 (471) Heavyweights
46471Furious GeorgeNW10 (471) Club Terminus
47467Midnight Meat TrainGL7 (301) No Surf14 (166) Mot. Throwdown
48458Lake EffectGL7 (301) Mot. Throwdown21 (157) Heavyweights
49455TanasiSE2 (455) Old Dominion Q.
50455PowderhogsNW2 (455) Col. Cup noTCT
51453Madador10 (236) No Surf11 (217) Mot. Throwdown
52435BD Air ShowSW10 (236) Cal State12 (199) San Diego Slammer
53432OaklandMA11 (432) Chesapeake Invite
54417Boneyard3 (417) Old Dominion Q.
55417Choice City Hops3 (417) Col. Cup noTCT
56417Madcow Y3 (417) No Surf
57405Inception-Red4 (384) Col. Cup noTCT30 (21) Heavyweights
58399CAKti11 (217) No Surf13 (182) Mot. Throwdown
59384Madcow X4 (384) No Surf
60363PhoenixNE13 (363) Colorado Cup
61354Sweet RollSC5 (354) Col. Cup noTCT
62354The Nights WatchNE5 (354) Cazenovia
63326Grand Trunk6 (326) No Surf
64326Vanier Wildcats6 (326) Cazenovia
65326Floodwall6 (326) Old Dominion Q.
66326EnigmaGL6 (326) Mot. Throwdown
67301The Ghosts7 (301) San Diego Slammer
68301Swell7 (301) Old Dominion Q.
69278Old Growth8 (278) Cal State
70278Burnside8 (278) Old Dominion Q.
71278Jurassic SharkGL8 (278) Mot. Throwdown
72278Maverick8 (278) No Surf
73274LancasterMA16 (274) Colorado Cup
74256Journeymen9 (256) Cal State
75256Grantham U.9 (256) No Surf
76256VAlhalla9 (256) Old Dominion Q.
77256Impulse9 (256) Mot. Throwdown
78237Hustle10 (236) Mot. Throwdown32 (1) Heavyweights
79236Triforce10 (236) Old Dominion Q.
80236Jester10 (236) Cazenovia
81231Spoiler12 (199) Mot. Throwdown29 (32) Heavyweights
82223HaymakerGL18 (223) Heavyweights
83217BrawlSW11 (217) San Diego Slammer
84199ROY12 (199) No Surf
85199Centretown Gunners12 (199) Cazenovia
86199Warriors of Rad12 (199) Cazenovia
87199GnarwhalNC19 (199) Heavyweights
88194INfamous16 (137) Mot. Throwdown27 (57) Heavyweights
89182StonefishNE13 (182) Cazenovia
90166YoungbloodsNE14 (166) Cazenovia
91151Flying Pig15 (151) Mot. Throwdown
92137Freaks Uv NatureSE22 (137) Heavyweights
93137Firebird16 (137) Cazenovia
94137Throw'n Together16 (137) Cazenovia
95134IllusionNC17 (124) Mot. Throwdown31 (10) Heavyweights
96119Mufasa23 (119) Heavyweights
97112MicroMachines18 (112) Mot. Throwdown
98102Mad Men24 (102) Heavyweights
99100Rust Belt War Bonds19 (100) Mot. Throwdown
10089Salvage 320 (89) Mot. Throwdown
10186H1N1NC25 (86) Heavyweights
10271yogosbo26 (71) Heavyweights
10344City Park UltimateNC28 (44) Heavyweights

Suppose such a ranking was used for bid allocation in USA. The next table compares the number of bids that were allocated per region for the USAU Championship with the number of teams per region in the top 16 of the above ranking.

Region Total bid allocated Teams in top 16
GL 2 2
MA 1 1
NC 1 2 (+1)
NE 3 3
NW 2 2
SC 2 3 (+1)
SE 3 2 (-1)
SW 2 1 (-1)

 

Of course, the relative number of points for tournaments is important. It must be well studied and tested to reach the good equilibrium. For example, maybe 400 or 600 points is best suited for second level tournaments instead of 500.

Comments Policy: At Skyd, we value all legitimate contributions to the discussion of ultimate. However, please ensure your input is respectful. Hateful, slanderous, or disrespectful comments will be deleted. For grammatical, factual, and typographic errors, instead of leaving a comment, please e-mail our editors directly at editors [at] skydmagazine.com.