February 25, 2009

Ruminations on data from last year's NAAC

Well, the regional rounds of the ABA National Appellate Advocacy Competition kick off tomorrow, so for the next three weekends, 187 teams from 110 schools (five teams who had signed up did not submit briefs) will start the long march to Chicago. Of course, I'm always interested in competition results, but I'm particularly geeked up to see how things shake out this year after examining some data from last year's competition.

Professor Kent Streseman's (Chicago-Kent College of Law) e-mail comment to me from the National Moot Court Competition about the "air of arbitrariness" that exists at tournaments got me thinking: Just how reliable are the results that all of us, as law schools, trumpet around as "proof" that our advocacy programs are better than the next guy's? Because it's one of the few two-stage (regionals plus nationals) moot court competitions that gives us information on the top finishers at each stage, I dissected last year's results from the NAAC to see if there was anything consistent between the two. I didn't find a whole lot that made sense, so I'm issuing the following advice to coaches and students:

1. Disappointed with your brief score at regionals? Don't fret. Of the 25 teams that advanced to last year's national finals, only 7 won any type of brief recognition at their regional competitions. In other words, nearly 75 percent of teams that advanced to the national finals did not even place within the top five of briefs at their regionals. That's not particularly surprising to me; because a team's brief score is not factored in past the preliminary rounds (a rule change instituted two years ago), teams with strong brief scores don't really hold an appreciable advantage at the NAAC regionals.

2. Had a good brief score at regionals and advanced to Chicago? Don't get cocky.
Here's what did surprise me: Of the seven advancing teams that did win regional brief awards, just two won brief awards at nationals. Otherwise put, of the five briefs that won national awards, the majority had not garnered even top-five recognition in their respective regions. How can that be? Remember--the briefs get re-scored by new graders in Chicago.

Last year, the national best brief (Michigan State University College of Law) had only been the fourth best brief in its region. The top two brief scores in that region both came from Regent University School of Law, which did advance both of its teams to nationals. But neither Regent brief--both of which outscored Michigan State at regionals--failed to even place in the top five at nationals.

3. Oralists not getting high scores? Worry not. Sixty one people (one region had a tie for 10th place) received top-ten speaker recognition at regionals. But only 19 of them were on on teams that advanced to nationals. In other words, only 30-or-so percent of those students who were named as Best Advocates in their respective regions went on to compete at the national level.

4. Won a speaker award at regionals and advanced to Chicago? You may not be so lucky at nationals. Of the 19 nationally advancing advocates who did garner speaker recognition at the local level, only 4 won any type of national award. Otherwise put: Of the 10 students who won national speaker awards, 6 of them hadn't even placed in the top ten of speakers if their regions.

5. 2008's National Best Advocate was pretty good. Of all the data I looked at, only one thing made sense: The first-place advocate at the national rounds was kind of awesome. That distinction went to Jennifer Bellott of the University of Memphis Cecil C. Humphreys School of Law. At the regional competition, she ranked second. Who was first? Elizabeth Barchas of Harvard Law School, whose team ultimately defeated Bellott and Memphis in the national championship game. Aside from that fact, I saw Ms. Bellott myself in Chicago at a preliminary round against my Texas Wesleyan team, and she was extraordinarily good. So, in short, the National Best Advocate also ranked as the number-two best advocate at her regional, where the number-one speaker was on the team that went on to win the whole shebang.

So what can we learn from this? Aside from the lessons above, maybe not a whole lot. Although I think the NAAC is one of the best (if not THE best) competitions in the country, the nature of the activity lends itself to arbitrary results. How can a brief dominate at the national level but stink at the local level? I don't know, but as anyone who's been doing this long enough will tell you, it seems to happen more often than not.

As for the oralist discrepancy, that's probably easier to explain. Most oralist awards (ABA included) are calculated by averaging speaker points from the first few rounds. The problem, of course, is that different judges have different standards--an 85, for example, may be an outstanding score in one judge's mind, while just a mediocre score in another's. So when you combine that fact with the relatively small sample size of three or four rounds, you wind up with results that can be all over the board.

Anyway, it will be interesting to see how this year's NAAC results compare to last year's. Scoring-wise, there's only one rule change: In the post-preliminary round at both regionals and nationals, the winning team will be the one who garners a majority of ballots, as opposed to a majority of points. That will take care of the problem of a team winning only one judge's ballot, but winning it by a bazillion points and therefore stealing the round.

By the way--one interesting note: Harvard, last year's national champion, isn't even competing this year. Anyone else find it odd that it wouldn't want to defend its championship?

2 comments:

Anonymous said...

Thanks for this rundown, Robert. I'm going to pass it on to our ABA teams (which I'm advising) as they're in the home stretch of their preparation. Very enlightening information!

Jim Dimitri
IU School of Law - Indianapolis

Ed Telfeyan said...

The Harvard team that won last year was put together by my son, Phil Telfeyan. Phil had done the same thing the previous years (each year with a different partner). Last year, he struck gold with Elizabeth. He never received any meaningful support from the faculty or administration at Harvard, which is why I am not at all surprised the school is not defending its title this year.

-Ed Telfeyan
Moot Court Advisor
McGeorge School of Law
University of the Pacific