How to Build a Championship Ultimate Team – Part 3: Facts & Data

I know, I know, this was supposed to be an article about game strategy, but as I was writing on that topic, it became increasingly clear to me that so many strategic decisions depend on having an understanding of what works and what doesn’t work for a given team.  The best way to understand how a team performs is to be able to track the things that are important to team performance.  But what stats are important?

Everybody loves fantasy stats, I know I do.  It’s fun to track who gets the monster Ds, who scores big, and who gets the most assists. But are those things really that important?  They might be if the rules of the game were changed to where every throw has to go for a score.  But the game isn’t played that way.  Ultimate centers around a complicated thing called teamwork and the stats must measure that. As it turns out, that monster D might be the result of an outrageous mark, that assist might be the last of 57 consecutive passes, and that highlight catch might be the result of a brilliantly conceived offensive play.  The fantasy stats are for the spectators.  It’s a different set of stats that give the best indicators of team performance.

Keep in mind, though, that overall stats alone don’t measure contribution.  I know of players with huge D numbers who I feel actually hurt their team.  Under closer inspection, the players in question generate their huge numbers against weak teams where they can cheat off their defensive assignments and use their athleticism to either make crowd pleasing plays or recover from being out of position.  This works well against the weaker teams, but against a skilled team they get beaten often and badly.  These players will get their teams to the big game, but they will ensure a loss when it counts.   I know of other D line players who don’t get the same impressive numbers but instead put their team responsibilities first. Their marks are effective, they deny the continuation, and they generally just make things difficult for their opponent.  These tight, well-coordinated defensive efforts tend to spread the stats around evenly without any one player standing out.  The same can be said for the O-line players.

Just what stats should you keep then?   Well, that’s not a fair question.  The fair question is: what stats can you keep?  To answer that we’ll first talk about the two different kinds of stats needed.  You’ll need stats for game time decisions and you’ll need stats for post game analysis & future adjustments.

Good Stats, Bad Stats

For game time use, it’s always good to track the score, time outs, and playing time & player combinations.  The captains go to great effort to balance individual playing time against the requirements of the game.  They must adjust playing time based on many factors:  Who’s hot, who’s tired, who’s injured, who needs some reps, who gets the team pumped up, and who calms them down. Some teams track O & D-line sets as well

Other game specific information can be tracked, it just depends what the team needs and the number of people available to track the information.  The amount of data that can be tracked by one person as the game is played is limited.  Typically, a prepared sheet is carried by the stat keeper who checks off players, keeps score, and logs time outs.  If the team wishes to track individual players on catches and passes, it will require two stat keepers, one to call the game out loud and one to record the information.  There are some stat gathering S/W apps in existence and some in development.  It will be interesting to see how they evolve.

Below is an example of the type of stats I record during a game.  The data is used during the course of the game and then for post-game analysis.

 

Preparing the score sheet before the game saves a lot of game time effort.  If you’re following a specific team you can preprint the form with the player names and numbers, otherwise you can add the player information as they enter the game.  It makes for easier data collection if you organize the player list by O & D-lines.  I put the mostly O-line players at the top, the mostly D-line players, at the bottom, and the swing players in the middle.  I do this because the time between a score and the following pull is extremely limited and the better organized you are the less the likelihood of missing a lineup.  It’s easy to miss a lineup because some teams will show 10 players on the line and then remove 3 of them just before the pull.  This makes for a good strategy, but can make tracking the line information tough. Another thing that happens is that the team captains can ask to see the stat sheet between points.  Making line up corrections while the teams are sprinting down the field is tricky.  Did I mention using pencils to write the data?

You see some additional information that I include.  I include the date of the game, the opponent, and the name of tournament (or league).  Another piece of information I keep is the call of the pregame disc flip.  You can see I’ve put a little X to the left of the team name to record that the Brain Rottweilers won the disc flip.

For the first point you can see several things. For the players in the game, I put a 1 in the box to the right of their name.  The 1s indicate that this is the first time these players are in the game.  The next time one of these players is in the game, I’ll put a 2 in the box associated with their name.  From the fact that the O-line is on the field, you can infer that they received.  For the first point, the Rottweillers had 19 consecutive catches with the last catch being for a score.  Why do I track catches and not passes?  It’s mostly to take into account both dropped pulls and stall counts.  You can track passes or catches; but they each have certain game situations where you run into problems recording the information.  After the point is scored, I write the current score in the two top rows.  I lightly shade the box for the team that scored.  It just makes it easier to view the scoring history during the game.

In Ultimate, a lot of passes get called back.  I count and include every catch that must be made to avoid a turnover.  For instance, I count the catches following a travel call but not the drops following a foul.

The second point shown is a bit trickier.  The player indication is simple enough; it’s the same recording technique as for the previous point. The difference is that when I record a D-line point, I start recording the catches one line down than if the O-line was on the field.  This way one team’s passes are easy to distinguish from the others.  If I’m using a sheet that doesn’t have the gray stripes running across it, I will circle my teams catch count.  The second point reads as 4 O-line catches, a turnover, and then the D-line scores on their second catch.

The third point could be a dropped pull and then a 7 pass D-line scoring effort from the goal line, or letting the pull hit the ground then a turnover on the first pass by the O-line, followed by 7 passes working it up the field and a score for the D-line.

At the half, I quickly tally all the player stats into a single column and hand the sheet off to the team captains.  The captains review the information and adjust their line strategies for the second half.

After the game, I tally the game totals and make the sheet available to the captains for post-game review.

A Heuristic Approach

But that isn’t the end of things.  There’s much more that can be done.  I collect the data on paper sheets so my next step is to transfer the data to a computer spreadsheet.  In my case it’s MS Excel on a netbook PC.

Whenever I can, I follow the team and collect the game stats.  A tourney has 5 to 8 games and over the duration of a season there are about 5 tourneys.  Of course this is typical of club teams seeking a championship.  It may be different for league teams.  Let’s add this all up.  You’ve got maybe 30 logged games a year and about 25 combined points in each game.  That makes for about 750 O & D-line opportunities shared between 25 players.  The team brain can look through all this data and assess what works and what doesn’t.  Evaluating player combinations is difficult.  Do you know that for a 25 player roster, there are some 480,700 unique combinations of 7 players?  How do you know which player combination is the best?

Fortunately, player combinations are not quite that random.  There is a technique most teams will use to set lineups.  This oft used technique is called a heuristic approach.  Heuristic is a neat little word, it means “strategies using readily accessible, though loosely applicable, information to control problem solving in human beings and machines.”  It’s the accepted academic word for “My best guess.”  Once the various combinations are defined, the team leaders can pore through the game data and assess the relative performance of each combination, fine tuning aspects accordingly.

Still though, it’s a lot of data to sift through to tabulate the performance of any particular combination, which is why I wrote a little program to assemble the performance of different player combinations.  There are a couple caveats: First, the software compiles team performance and not individual performance.  The second thing is about a saying I heard years ago which goes “Players win games, coaches lose them.”  This means that the coaches maximize the chances of the team winning by putting the best combination of players on the field.  Also, by the time you get to this point in data collection, you’re working to maximize the performance of the players you already have.  It is the coach/captain’s responsibility to organize, select, motivate, cajole and admonish as required to get that maximum performance.  Now that all the disclaimers are out of the way, let’s give this software a test drive.

For this article, I’ve evaluated two MS Excel workbooks; data collected at the 2010 USA Ultimate NW Open Club Regionals & Nationals.  The software I’ve written scans through the two workbooks and tallies all the recorded information.

 

You can see all the Sockeye players and all the opponents Sockeye faced.  Along the bottom are the cumulative stats for Sockeye’s O & D-lines.

The cumulative team stats breakdown as follows:

Total O-lines       self explanatory

Scores(%)            the number of scores by the O-line and the percentage

Passes(%)           the total O-line pass count and the completion percentage

Turns(%)              the total number of O-line turnovers and the percentage

Broken(%)          the number of times the other team scored off O-line turnovers

Defends(%)        the number of passes by the opponent’s D-line

Gotbacks(%)      the number of times the O-line got the disc back following a turnover

 

Total D-lines       self explanatory

Breaks(%)           the number of scores by the D-line

Passes(%)           the number of total passes by the D-line

Turns(%)              the number of times the D-line turned the disc over after gaining possession

Beaten(%)          the number of times the D-line gave up a score

Defends(%)        the number of passes thrown by the opponents

Ds(%)                    the number of turnovers forced by the D-line

 

This initial screen gives the baseline performance of the team in the selected workbooks.  We can now compare the various player combinations against this baseline.  Let’s look at a specific combination of O-line players.  How about a look at the all-star lineup of Ben, Adam, Dave, MC and Moses?

 

You can see I’ve selected the five guys and the stats have adjusted accordingly.  With this combination as part of an O-line, the team performance has changed.  The analyst can repeatedly modify the selected combinations and assess how each performs.  Another wish is to quickly see how the team performs when this combination isn’t in.  This happens by selecting the “inv” button (way up by the “Players” caption).  When pressed, the “inv” button displays the combined performance of all other player combinations.

 

The same sort of player selection technique works for the D-Line as well.  Let’s pick the Matt R, Matt K, Ray & Sebby combination and see how they perform when part of the D-line.  The data are low because this combination was only possible at Nationals.

 

Now, this is how the D-Line performs when compared with all the other player combinations.

 

The point here isn’t so much that I feel that these are good stats to compile.  The point is that if you compile stats, you need to be able to access them quickly and in a meaningful way.  Taking game stats that end up sitting in the equipment bag until they crumble away really isn’t all that useful.

Here’s why I feel tracking pass completion percentage is important.  It’s easy for one person to do, you can do it without looking away from the action, and it’s useful information.  Here’s why I find it useful.

Sabermetric/Ockham’s Razor

I’m a fan of the Sabermetrics methodology and of Ockham’s razor.  Combining the two will give you a good way of looking at your team’s data.

Sabermetrics does not relate directly to Ultimate.  It was created to better understand which baseball stats were objective and which ones weren’t.  Baseball is full of interesting stats and some people felt that many of them are not as useful as popular thought had made them out to be.  So a bunch of people started pouring through a hundred plus years of baseball stats looking for meaningful and objective relationships.  I certainly don’t mean to oversimplify their study results, but they found out that the team that goes the longest without making an out tends to win.

Ockham’s (Occam’s) razor is a somewhat misunderstood principle that says if there are two explanations for something; the one that has the fewest assumptions has the better chance of being correct.

Keeping with the spirit of Sabermetrics and Ockham’s razor, a person can study the game of Ultimate.  In Ultimate it turns out that the team with the fewest turnovers wins.  The statement is both objective (provable) and makes no assumptions.  When you collect stats, they should yield information that you can use to identify where turnovers occur.  The data collected may not tell you what causes the turnovers; it tells you where the turnovers occur.  The coaches will need to do some root cause analysis to uncover the underlying reasons.  This is generally true of any statistic.

Final Data

Here’s a chart of data extracted using my software.  These games feature a veritable who’s who of elite open club teams playing against Sockeye.  The chart contains data from games with Chain, Doublewide, ECU, Furious, Ironside, PoNY, Revolver, and Ring among others.  These were some real top shelf games played under good weather conditions and with each team’s tournament future on the line.  The data is from 22 games.  In these games, there were 6,704 passes thrown, 544 points scored and 471 turnovers.

 

This chart shows an aspect of game strategy that intrigues me.  It cross-plots the team scoring percentages vs. the team pass completion percentages.  Ray (Sockeye #99) and I go back and forth on whether it is better to only throw low risk passes or to be more aggressive in putting the disc downfield.  I’m under the opinion that if you don’t turn the disc over, you will eventually score.  Ray counters with the observation that the other team will pick up on the short game, crowd underneath and jump the passing lanes for the D.  Also that the probability of 1 long lower % throw can be equal to many short high % throws. Modern defenses are quite astute in compiling the offensive tendencies of their opponents.   Ray suggests that an occasional lower percentage deep throw will keep the defense honest.  I believe that the need for an occasional deep throw is an artifact of using the horizontal stack offense and is not true for some other offensive strategies. His point is well taken, but then so is mine.  So, here’s some data that provides some insight into that question.  I’ve extracted from the game data, passes thrown & caught and whether a particular sequence of passes resulted in a score or a turnover.  There is a data point for the O-line of each team and there is a data point for the D-line of each team.  The D-line data points represent how well the D-line performs after getting the disc on a turnover.  I was expecting to see similar passing performances between O-lines and D-lines at the elite level.  I was surprised by the large differences between the two lines.

 

If we isolate O-line performances, we can see that the winning O-line will score on 85% of their opportunities while completing over 95% of their passes.  On the other hand, if the O-line scored on less than 70% of each opportunity, they lost.  What exactly does this mean?  Does it support Ray’s or my contention?  Let’s look at the D-line performance while on offense first.

 

This would seem to support another of my contentions; most O-lines have better defense than most D-lines have offense.  As an analyst, I’d suggest that teams can achieve greater success by increasing the offensive capability of their D lines.  Being able to exploit turnovers to generate points is the area with the greatest potential for tangible performance gains.

Next, I went back into the data looking for when game changing events happen.   I extracted the data with a different sort of organization.  I divided the data into two groups; winners and losers.  In this way I could see if there’s a difference between successful and unsuccessful teams in any particular game.  For all the winning teams I recorded what happened on the first throw, the second throw, the third throw, and so on all the way through to the tenth throw.  Each throw has three possible results; a completion, a score or a turnover.

There were 1012 first passes, 899 second passes, 807 third, 688 fourth, 581 fifth, 495 sixth, 410 seventh, 334 eighth, 287 ninth and 235 tenth passes.

So…. Here’s a plot of the data.

 

This is perhaps the single least usable graph I have ever created.  It’s difficult to understand and any information it presents is completely unintelligible.  The only reason it’s here is because it took a week to create the data and then by golly, something that takes that long to do, deserves to be seen.

Let’s look at the data a bit differently.

 

This is a really neat little graph.  It gives some insight into how most teams implement their offensive strategies.  Let’s take a look at what it says.  We’ll start with the obvious.  Winning teams score at a greater rate than losing teams and that rate continues to increase with each throw.  The increase in scoring rate is understandable as each pass gets your team closer to the end zone and the closer to the end zone; the easier it is to score.

The next thing the data shows is that scoring happens in threes. If you pay attention to games, you can pick this up.  It’s kind of the dump, swing, strike philosophy repeated over and over again until a score or turn over.  The data isn’t quite on the three pass cycle as some teams are a three pass cycle team and others are a four pass cycle team.  The ultimate culture found in the Pacific Northwest is different from the Bay Area disc is a different style of play, as is play in New England, Middle America, Europe, Asia and Australia.  Still, NW teams are on both sides of the win-loss line.

Just as interesting is what the losing teams are doing.  They score at a lesser rate then the winning teams but even more interesting is that the three pass cycle has been broken.  It would seem that breaking the three pass cycle will throw an offense out of sync and reduce their scoring opportunities.

Let’s now look at turnovers.

 

Great stuff here.  The most interesting thing to me is the winning teams have the most turnovers on the fourth pass.  This is immediately following the third pass being the highest scoring pass.  Defensive coordinators would be wise to take advantage of this.  Teams that end up losing seem to fall apart on the sixth and seventh passes.

The two graphs can be combined to create an effectiveness measure of team performance.

 

This graph gives some measure of team effectiveness for each throw in a sequence.  The data is smooth and predictable with the exceptions of pass four for the winning team and pass six for the losing team.  In general, the first throw is dangerous for both teams, but the winning teams seem to get their offensive flow working quickly and gain the upper hand.

Note that this is data averaged across many teams and does not represent any single team.  If the data is restricted to a particular team, the tendencies of that team are readily apparent.   Since this article is general in nature, that information is intentionally missing.  I see no compelling reason to make it available.

Data Collection in Ultimate

There’s a lot more data that can be mined from the information I’ve recorded.  When time permits, I’d like to look at the scoring percentages if the pull lands in or out of bounds.  I’d like to look at the relative performance between points with no turnovers and points containing turnovers.  There are some interesting things that happen with the team choice of whether to pull or receive to begin the game.

As for other game metrics, there are a number of people out there who are saying that tracking the distance of each pass might be a good metric.  I’m thinking distance from the end zone may be valuable as well.  I’ve got a hunch that the amount of time the disc is in the air is a strong factor as well.

To bring this whole article to a close, I can say a couple of things.  Collecting information about game performance is important for any team that wishes to be successful.  Meaningful information can be used to identify areas of concern and where opportunities for improvement exist.  The information collected can be used to identify opponent tendencies.  Being able to quickly and effectively access the information is key, otherwise the effort is mostly wasted.

How to Build a Championship Ultimate Team series:

Part 1: Development

Part 2: Team Makeup

Photo by Andrew Davis