Philosopher's Annual vs. Philosophical Review
Brian Leiter just revealed this year's selections for the Philosopher's Annual "attempt to pick the ten best papers of the year". I'm led to wonder: How successful are such attempts? Selection for the Philosopher's Annual is certainly prestigious -- but to what extent does selection reflect durable quality as opposed to other factors?
There is of course no straightforward way to measure philosophical quality. But here's my thought: If an article is still being cited thirty years after publication, that's at least a sign of influence. [Insert here all necessary caveats, qualifications, hesistations, and hem-haws about the relationship between quality and citation rates.]
I compared citation rates of the 30 articles appearing in the first three volumes of Philosopher's Annual (articles published 1977-1979) with the citation rates of the first ten articles published in Philosophical Review during the each of those same years. (I only included citations of the articles' original appearances, not citations of later reprints of those articles, since the latter are much harder to track. I see no reason to think this would bias the results.) To the extent the Philosopher's Annual selection committee adds valuable expertise to the process, they ought to be able to beat a dumb procedure like just selecting the first ten articles each year in a leading journal.
Evidently, however, they don't. Or at least they didn't. (Time will tell if this year's committee did any better.)
The median total citation rate in the ISI citation database was 14 for Philosopher's Annual and 18 for Philosophical Review -- that's total citations in indexed journals over the course of 30+ years (excluding self-citations), less than half a citation per year on average. (The difference in median is not statistically significant by the Mann-Whitney test, p = .72.) The median total citations since 2001 is 2.5 for Philosopher's Annual and 3.5 for Philosophical Review (again not significantly different, p = .62).
But the medians might not tell the whole story here. Look at the distribution of citation rates in the following graph.
Although the median is about the same, it looks like Philosophical Review has more articles near the median, while Philosopher's Annual has more barely-cited articles and much-cited articles. It's hard to know this for sure, though, since we're dealing with small numbers subject to chance fluctuation: Only three articles, all with Philosopher's Annual, had 100 or more citations. (My measure of the difference in spread is statistically marginal: Levene's test for equal variances on a log(x+1) transform of the data, p = .09.)
The three articles with at least 100 citations? Walton's "Fearing Fictions" (118 cites), Lewis's "Attitudes De Dicto and De Se" (224 cites), and Perry's "The Problem of the Essential Indexical" (301 cites) -- all good choices for influential articles from the late 1970s. The most cited article from the Philosophical Review list was Watson's "Skepticism about Weakness of Will" (73 cites). Worth noting: Lewis's "Attitudes De Dicto and De Se", though counted toward PA rather than PR, was actually published in Philosophical Review -- just not among the first ten articles its year. Also, skimming forward through the mid-1980s, my impressionistic sense is that it is not the case that 10% of the PA articles are as influential as the three just mentioned. So possibly the apparent difference is chance after all, at least on the upside of the curve.
Maybe those people who selected articles for Philosopher's Annual in the late 1970s were more likely both to pick swiftly-forgotten duds and to pick home-run articles, compared to an arbitrary sample from Philosophical Review. The selection process did not appear to favor quality, as measured by influence over three decades, but possibly the selection procedure added variance in quality, both on the upside and the downside. The normative implications of all this, I leave to you.
UPDATE, August 22:
Given the uncertainty about whether Philosopher's Annual shows greater variance, I couldn't resist looking at the next three years (which, it turns out, are articles published in 1980, 1981, and 1983, skipping 1982). The trend toward similar median and greater variance appears to be confirmed. On the high side, PA had three articles with 100 or more cites (R. Dworkin "What is Equality: Part 2" [426 cites], Lewis "New Work for a Theory of Universals [262], and Shoemaker "Causality and Properties" [106]), while PR had only one article in that group, the least-cited of the four (Dupre, "Natural Kinds and Biological Taxa" [104]). On the low side, PA had 6 articles cited 3 times or fewer, while PR had only 3. Here is a new graph of the spread, merging all the data from 1977-1983:
The medians of the merged data remain tied: 15.5 for Philosopher's Annual vs. 16.0 for Philosophical Review (Mann-Whitney, p = .75). And Levene's test for equal variance (on the log(x+1) transform) now shows statistical significance (p = .009).