Home > Sports > Grading Draft Analysts (2007 Edition)

Grading Draft Analysts (2007 Edition)

Mel-Kiper-JrThe Situation

A few weeks ago, I wrote an article on Football Outsiders summarizing the “draft gurus'” grades of the recently completed draft. Obviously, trying to attribute grades to draft classes just days after the draft commences is folly, and most will admit to such. Of course, since these articles are incredibly popular from a readership standpoint, editors demand their existence. The NFL has become a 24/7/365 league, with every fanboy/girl up to date on their team’s latest moves, thoughts and feelings (If you don’t believe me, read the comments on my ESPN article roasting me for not knowing Denver was leaning towards Nate Irving, not Joe Mays). Michael Wilbon summed it up best, saying it’s “a plague on the world of sports.”

Something that many of the readers of that draft article wanted to see was how well these draft analysts really were at assessing these picks. It goes without saying that trying to answer that question is an inexact science. To do so, we first have to come up with a way to measure a draft pick’s performance. Luckily, Pro-Football-Reference.com features a tool called “Approximate Value” that works much in the way that Wins Above Replacement (WAR) works in the baseball arena.  This provides a single number to all players, based on their performance. For some context, Tom Brady’s record-setting 2007 season had an “AV” of 24 and Aaron Rodgers’ 2011 MVP season had a 23. By comparison, Brandon Weeden’s sub-par rookie year earned an AV of 8. High numbers for non-quarterbacks can also be earned. J.J. Watt’s transcendent 2012 season had an AV of 20. Is there room to scrutinize this number? Of course, but since it’s the best we’ve got to work with, we’re using it.

Now, to address both arguments about analyst grading the picks as they help the team (value brought to the team), versus the haul of NFL talent the team brought into the league (career value), we’ll look and rank both of those.

Because I don’t stray far from Football Outsiders, I’m using their concept of waiting six years to analyze a draft. Therefore, the data we’ll be pulling will be from the 2007 NFL draft, or as I call it, JaMarcageddon. For the uninitiated, this is the infamous draft that saw JaMarcus Russell shoot up the draft boards thanks to his towering stature and ability to throw the football 80 yards from a knee. Russell went on to make $39 million, start just 25 games and throw more interceptions than touchdowns.

In 2007, Football Outsiders used thirteen different draft report cards to tally their results. Thanks to websites deleting old articles to preserve space, only seven of these remain available: Mel Kiper, Steve Silverman, Pete Prisco, ESPN’s SportsNation, Tom Weir, Jason Cole and Charles Robinson and the infamous Dr. Z of Sports Illustrated.

The Process

With the data sources chosen, the analysis can begin. First, I created a pivot table from the draft, summing the different AVs for each team’s draft, then ranked them accordingly. The winners of the draft were undoubtedly the San Francisco 49ers, who came away with Patrick Willis, Joe Staley, Ray McDonald, Dashon Goldson and others. The losers were easily the New England Patriots, who’s DrAV (Drafted Approximate Value) was 23% lower than the 31st ranked team. This, however, has an asterisk that we’ll address later.

Next, I had to translate these rankings into grades to create an apples-to-apples comparison with the analysts. So I figured out the average “GPA” for the 7 analysts and 32 teams (2.58) and tried to recreate it by evenly distributing grades according to rankings. I came the closest (2.61) by giving 2 “A+” scores and 3 of every other grade (i.e. A, A-, B+…) down through D. I then compared the analysts grades to the grades the rankings had created and noted all the instances in which there was a difference of one letter-grade or more (ex: B+ vs C+). I also noted anytime there was a two letter-grade difference, which indicted a severe disparity between perception and reality.

The Results

In the table below, you’ll see how each team ranked by both the DrAV and CarAV.

Team Sum of DrAV Sum of CarAV DrAV Rank CarAV Rank
ARI 63 86 22 20
ATL 127 158 2 2
BAL 82 104 12 11
BUF 63 121 23 8
CAR 119 130 5 7
CHI 55 75 28 24
CIN 63 79 24 23
CLE 89 104 9 12
DAL 75 75 14 25
DEN 33 44 30 30
DET 72 84 15 21
GNB 89 102 10 14
HOU 70 84 17 22
IND 68 75 19 26
JAX 77 121 13 9
KAN 59 72 27 27
MIA 61 90 26 17
MIN 117 139 7 3
NOR 68 96 20 15
NWE 23 33 32 32
NYG 123 133 4 5
NYJ 125 132 3 6
OAK 85 109 11 10
PHI 63 72 25 28
PIT 118 139 6 4
SDG 71 93 16 16
SEA 67 90 21 18
SFO 203 212 1 1
STL 30 54 31 29
TAM 94 103 8 13
TEN 69 87 18 19
WAS 35 43 29 31

Looking at how the analysts did at grading the return the teams got (DrAV), most were predominately average. Everybody but the Cole-Charles combination and Steve Silverman had 13 to 15 differences noted above. Dr. Z was the only one of those who didn’t have a two-grade difference. Cole and Charles were the clear winners of the draft. They had only six one-grade differences and three two-grade difference. The worst was Silverman, who have 17 total differences and 5 of them being of the two-grade variety. His worst was easily his New England grade, which he gave an A-. The Patriots claimed everybody as victims, though, and here’s where the asterisk comes into play. While most analysts thought the actual picks were decent to good (they had high hopes for Brandon Meriweather), most attributed their high grades (New England had the second-rated draft) to the Randy Moss trade, and some even brought up the Wes Welker trade.

To see if these analysts theories of the Moss trade hold up, I’ve added his AV from his 3+ years in New England (45) and re-ranked the teams. Because Wes Welker was fairly unknown at the time, I’m not adding in his value, as it likely didn’t play a part in most of the analysts’ decisions. This brings the Patriots DrAV rank up to 20th, good enough for 2.333 points. You can see the changes the shuffling brought below.

DrAv (without Moss)
 Grade Differences Kiper Silverman Prisco Weir Cole Dr. Z SportsNation
1 13 12 11 10 6 13 9
2 2 5 3 4 3 0 5
DrAv (with Moss)
  Grade Differences Kiper Silverman Prisco Weir Cole Dr. Z SportsNation
1 14 14 12 10 7 14 11
2 1 4 2 3 3 0 4

If you’re looking at it from the perspective that draft grades are given on the NFL potential of each team’s draft picks, you’ll find that analysts did even worse, for the most part. Kiper and Dr. Z both improved while the rest took a collective nosedive.

CarAv (without Moss)
Grade Diff. Kiper Silverman Prisco Weir Cole Dr. Z SportsNation
1 11 15 14 16 13 12 15
2 2 4 2 2 2 0 3
CarAv (with Moss)
Grade Diff. Kiper Silverman Prisco Weir Cole Dr. Z SportsNation
1 12 15 13 18 14 12 14
2 1 4 3 1 2 0 4

The Conclusion

Most did about as well as expected, knowing only half as much as they claimed they did. Some of the more respected analysts, such as Dr. Z, performed well, circumstances provided, while others, like Steve Silverman, were purely outclassed (this would explain why he now writes for BleacherReport. Bazinga!). Those that fared better in the CarAv category are likely those that study film more often and can see strengths and weaknesses. Those that don’t probably just read daily blogs on major news sites like you and me.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: