Super Lawyers's Not-So-Super Law School Rankings

Last week, Super Lawyers released its “2010 Super Lawyers U.S. Law School Rankings,” its first-ever ranking of law schools.  The premise of its law school rankings is simple:  rank law schools based on the number graduates each produced that received a “Super Lawyer” designation in 2009.  Super simple?  Sure!  Super ranking? Nah.

Super Lawyers obviously knew its methodology was flawed before it even started receiving criticism here and here.  Mr. Bill White, Super Lawyer’s publisher, authored a detailed blog post that announced its 2010 law school rankings and preemptively tried to explain away some of the most obvious (and obviously legitimate) criticisms.

Before analyzing those criticisms, though, let’s compliment Super Lawyers for suggesting a somewhat novel insight in the law school rankings space – that it may be interesting and relevant to rank law schools based on their ability to create successful, ethical lawyers.  Or, as White puts it:  “Our approach is similar to the way baseball crowns a homerun king based on total homeruns without employing a weighted average based on plate appearances.”

Actually, let’s start with White’s baseball analogy, because it shows why the Super Lawyers ranking is so obviously flawed.  Despite what White says, the Super Lawyers ranking is not at all like crowning a home run king based solely on total home runs a player hits.  Instead, the Super Lawyers approach is more akin to crowning a whole baseball team (i.e., a law school) the home-run king based on counting the number players on each team that hit a certain threshold of homers (i.e., reached Super-Lawyer status).

That wouldn’t be such a bad rankings system if each baseball team (law school) had the same number of players on its roster (i.e., class size over the past few decades).  Yet we know that these baseball teams (law schools) have DRAMATICALLY different roster sizes.  Harvard’s “team” has 556 “players” on its Class of 2011 roster, while Yale’s team has only 189.  It’s one thing when the Yankees and Red Sox tell the rest of the baseball world that money doesn’t buy championships.  It would be quite another if they said that batting three times as many players didn’t improve their runs scored!  But that’s EXACTLY what Super Lawyers does in its rankings.

Super Lawyer’s response to this flaw?   White replies:  “The quality of graduates, not the size of the school, is what ultimately determines where schools land on our list.”  But that’s provably false.

But wait, it gets worse, because Super Lawyers designates “Super Lawyers” by region (division), not nationwide (all 30 MLB teams).  The problem with this approach is that it rewards law schools that are regional powerhouses and rewards law schools in regions where comparatively few graduates of elite schools go to practice law.

Plus it rewards historically strong law schools (and punishes newer law schools) because, as a practical matter, lawyers rarely achieve “Super Lawyer” status until they’ve practiced 10 or 20 or 30 years by building a reputation of excellence in the legal community.  That’s fine if you’re looking at graduates of Stanford or Columbia, because those schools have been around forever, and so they’re likely to have plenty of graduates who are eligible to achieve “Super Lawyer” status.  But what if you’re looking at graduates of young, “farm team” law schools, like Chapman, which have jumped up traditional law school rankings systems only recently (going from not ranked, to 4th Tier to 3rd Tier in the last decade of USNEWS Rankings alone)?  They’re penalized for no other reason than that they haven’t been around for very long.

Even the designation “Super Lawyer” is the byproduct of a series of subjective judgments.  When Albert Pujols hits a home run we know it, because the ball he bats has to travel over a fence somewhere.  That’s a simple, measurable act.  Whatever you may think of the methodology that produces a “Super Lawyer” designation of a given attorney, it’s based on a series of highly subjective judgments.  And whatever you may think of law school rankings like Leiter’s Student Quality Rankings that are based on LSAT and GPA scores, they use methodologies based entirely on measurable facts (just like home runs). 

In the end, Super Lawyers should have called their 2010 rankings the “Super Lawyers 10 to 40-Year-Old Ranking of Law Schools Based Largely on Class Size and The Propensity of Lawyers In A Region To Attend A Local Regional Law School and The Propensity Of Lawyers In A Region To Graduate (Or Not Graduate) From The Most Competitive National Law Schools.”  Now THAT’s a great name for a law school ranking!

Back To About Us