Rankings: Implications

This is part 5 in a multi-part series on the USAU ranking system and bid allocation.  Here are 1, 2, 3 and 4.

In the short two seasons the rankings have been tied to bid allocation, we have already seen some massive changes in the structure of the college season.  We will continue to see big changes for the next few seasons, but slowly things will settle into a routine.  Today, I want to look at those changes and some of the hidden implications and assumptions about the rankings.

Uneasy

Far and away the biggest implication of tying the bid allocation to the rankings is to make everyone very, very uneasy.  Suddenly, every game counts.  And in weird ways.  Does playing this team hurt our ranking?  Does it matter how badly we got blown out?  Consolation games matter!?  Initially, a big part of that uneasiness came from not necessarily understanding how the rankings worked.  The black box effect caused people a lot of worry.  Here was a computer making decisions about who would and wouldn’t go to Nationals and no one really knew how it worked.  As we begin to peel back the veil, the nature of the unease has shifted toward frustration (read comments from article 1 or the rsd discussion) and calculation (read Adam and Bryan’s articles).  Even Sholom Simon, the author of the algorithm, is uneasy: “I’m not entirely comfortable.  I conceived of the Top 20 with the idea that it would generate publicity and add to the fun.  I never envisioned that the results would have such a major impact on something as important as wild cards.”

How do we deal with this sense of unease?  As I coach, I deal with it the same way I deal with anything I can’t really control, like weather.  I make a game plan that puts my team in the best position for success and then I forget about it.  No point agonizing about something you don’t control.  Yes, there are some problems with the rankings and bid allocation, but one of the biggest is the feeling of unease itself.

There is no right answer

One of the major sources of frustration is the sense that there is a ‘right’ way to rank the teams #1-50 and a better algorithm or a system of informed voters or something could make the rankings and the bid allocation ‘fair’ and ‘right’.  There isn’t.  Unlike a professional sport with a closed, complete season that produces a clear set of rankings based on wins and losses, the ultimate season is open and diffuse.  In most cases, there is no clear way to delineate who is better than who particularly when you move into the 16-24 range of teams.  These teams are typically sitting on a bunch of wins and a bunch of losses, creating a contradictory set of data.  Compare ultimate to NCAA basketball or football.  The seasons of those teams are much more structured than an ultimate season, the resources the NCAA can allocate to a ranking algorithm or selection committee are several orders of magnitude greater than USAU’s and there are still major difficulties associated with making rankings and bid decisions.

One of the really great things to come out of this discussion about the rankings are some clear and implementable adjustments to the rankings that will make them work a little better and more justly. But the second thing I hope will emerge from all this is a change in how teams work their season and also some changes in how USAU uses the rankings to make bid allocations.  More on that tomorrow; today is focused on the problems and implications.

Professionalism

The implementation of a ‘regular season’ has had a very profound and positive effect on teams’ professionalism, particularly in the middle section of the rankings.  Historically, there was a huge gulf between the top programs in terms of resources, coaching and commitment.  The rankings and bid allocation has pushed the mid-level teams to get organized and get thinking about their seasons earlier.  They are traveling more and taking tournaments more seriously.  In short, they are acting like top-shelf teams.

Rookies, playing time and development

Lindsey Hack, coach of UNC Pleides, says it best: “If these girls had their flights, jerseys, fees paid for I would have no problem letting my less experienced players ride the bench despite them giving similar levels of effort as my more skilled players.  But they put in as much work during the season, put in as much money and deserve some field time.  Enough field time to make their trip worth it.  I do believe that playing time is earned…but the pressure to win now is so great that even the games that I could get these girls playing time in (consolation) I cannot anymore.  And it is incredibly frustrating.”

The need to win now creates a really novel pressure for ultimate teams.  Traditionally, the entire preseason could be looked at as one huge practice session and a time to develop new players, help established players grow into new roles and build your team identity.  Losses, while always painful, had no long-term, tangible effect.  Now they do.  Balancing win-now, team development and playing time has become one more thing for coaches and captains to factor in.

The Corvegas Problem

This year, with a new and very young team, Oregon Fugue decided to go to Corvegas.  We knew going into the tournament that it would likely have a drag on our ranking because of the low rankings of the other teams attending, but we went anyway.  Framing this decision in the context of the discussion above, we felt team development was more important than the ranking.

Oregon’s current ranking is 1721.  At Corvegas, we played Western Washington (1101) twice, Pacific Lutheran (894) twice, Lewis and Clark (505) and Puget Sound(estimated at ~511; <5 games).  Oregon 606ed in every game and so the game rankings were 1707, 1707, 1500, 1500, 1111 and 1117.  This averages out to a tournament average of 1440, well below the team’s current ranking.

It seems unjust that a team could lose points no matter how well they play.  It also seems unjust that a team could gain points no matter how poorly they play.  One suggestion that has been made is to drop any game that falls into this category.  The immediate effect would be to elevate the top teams’ rankings and depress the lower teams’ rankings.  A rough calculation (it is really impossible to actually figure it out without running a simulation) for dropping the Corvegas scores from Oregon’s ranking would elevate it to ~1850.

Removing scores from the algorithm potentially has some very, very weird mathematical effects.  One thing that would happen is that as you removed some blow-outs, teams’ scores would change and potentially you would have to remove even more.  This change to the algorithm would have to be studied quite carefully before it would be possible.

The Spencer Diamond problem

Spencer Diamond.  Spencer Diamond.  Spencer Diamond.  Hey, did you know that Spencer Diamond of Dartmouth is overseas this quarter and is coming back for the series?  Now that the name dropping is out of the way, let’s look at the issue here.  The problem is that the teams that earn bids aren’t necessarily the ones that enjoy a trip to Nationals.  Who knows where Dartmouth will end up ranked this season; if they maintain the level of excellence they’ve displayed so far they will certainly earn a second (or first) bid for the NE.  But should they falter, suddenly Tufts is forced to share a single spot with a team that is much better with the addition of a two-way starter.

In a real regular season like professional sports use, your regular season gets you into the post-season.  In ultimate, the regular season defines the shape of the post-season.  Odd.  To make matters odder, the teams that defined the post-season by winning still have to defend what they built at Regionals.  Right now, there is nothing preventing a team from going 30-1 and not making Nationals.  That’s scary.

$5 a gallon

The ranking system works well enough when it gets enough data.  That is why the early season results are so often so weird; there just isn’t enough of a connection between teams to pull them together into a coherent whole.  Early on, you are often looking at two or three independent or quasi-independent sets of rankings.

Bridge teams play a huge role in tying things together.  This season on the women’s side Michigan (Michigan Indoor, Pres Day, Easterns, Music City and Centex) and UNC (Queen City Qualifier, Queen City, Easterns, Stanford, Centex) are playing a lot of tournaments and covering a lot of ground.  Their results help bind together regions (particularly East to West) that otherwise might stay distinct.  On the men’s side, the top teams have stayed in much more segregated tournaments (like Warm-Up) which is why the men’s rankings are still a little nonsensical right now.   Easterns and Centex should clarify the picture a bit, but it is still worrisome that it will take until March 19th to get a good picture.

There is a real long-term danger in this system: gas prices.  Three of the top 6 women’s teams: Washington, Sonoma and Oregon aren’t going to Centex and the reason is money.  Centex is a $500 – $800 outlay per player under current gas prices.  Should gas prices rise to $5 per gallon or higher, how many teams will we see willing to get in an airplane in February or March?  As the amount of cross country travel decreases, the accuracy of the algorithm will collapse.  There simply won’t be enough links between different subsets of data.  This won’t be a problem this year, but it has the chance to blow up in everyone’s face without much warning.

Collusion

There have been some pretty serious accusations of collusion made in this series and on rsd.  I’m not going to comment specifically about those accusations because I don’t know a thing about them first- or even second-hand, but the possibility of collusion is very real.

Bryan and Adam’s excellent analysis has the benefit of hindsight, but collusion requires foresight, (if we do this, then we’ll get…) which in almost all cases is lacking.  Teams just don’t know enough about how the season is going to play out to be able to game the system with regularity.

That said, USAU should land on this with both feet.  The two most obvious solutions are voiding results and banning teams from competition.  I lived through the era of iffy-eligibility and it wasn’t until teams started losing entire seasons that people really started cleaning up.

Feature photo by Andrew Davis