Ranking law journals and the limits of journal citation reports.
Eisenberg, Theodore ; Wells, Martin T.
I. INTRODUCTION
Departments, universities, and countries use rankings to inform
decisions. School ranks depend in substantial part on their
scholars' ranks. Scholar ranks depend on the quality of the
journals in which they publish. Independently of rankings' use by
third parties, journals use them to self-promote, which can lead to the
self-fulfilling prophecy of highly ranked journals continuing to be
highly ranked. "Publishing in journals with a high impact factor as
measured by citations, and in journals that are used as source journals
by (the Institute for Scientific Information (ISI) now Journal Citation
Reports [JCR]), has become an independent measure of scientific
quality" (Wouters 2000). The methodology, validity, and consistency
of ranking journals are clearly important.
Efforts to rank journals can be extensive. A country with
centralized research funding may have a national list ranking journals
to help inform decision making. Australia's 2012 journal list
includes 22,414 journals (ARC 2012), though the ARC has abandoned its
formal journal ranking system (Moosa 2011). Thomson Reuters ranks
journals by citation counts as part of a commercial product. JCR and
other entities rank journals by citation counts for informational
purposes (Washington and Lee Law School 2012). Journals may be ranked by
peer assessment surveys (Currie and Pandher 2011) in addition to
citation counts. This article analyzes citation count rankings of
refereed law journals.
The Washington and Lee Law Library (W&L) and JCR rankings of
refereed law journals are surprisingly inconsistent, with no
statistically significant correlation. The main sources of inconsistency
are likely the different representation of journals in the two databases
and W&L's greater institutional knowledge of law, law journals,
and law schools. JCR overrepresents economics and other journals
compared to law journals and contains hundreds fewer law journals than
other sources. JCR evidences less institutional knowledge of law
journals and law than does W&L. JCR does not enable users to account
easily for the dominance of student-edited journals in the United States
and fails to provide measures that account for journals' efforts to
address nonacademic constituencies, an important function of legal
publishing. Aside from statistical inconsistency across many journals,
we illustrate the databases' different treatment of journals
through case studies of three elite journals, the Journal of Law and
Economics (JLE), Supreme Court Review (SCR), and the American Law and
Economics Review (ALER). Our analysis suggests that JCR's methods,
divorced from the context of the fields it ranks, produce results that
should be regarded with extreme caution.
II. BACKGROUND AND DATA SOURCES
A. Background
Eugene Garfield, a founder of bibliometrics, was inspired in part
by the U.S. legal profession's Shephard's Citations, which
began in 1873 as a way to track citations to cases. In a legal system
that honors and relies on precedent, tracking citations is a core
research task. Indeed, Hebrew law-related citation indices have been
used for about 700 years (Wouters 2000). Garfield proposed a scientific
literature citation system that allows similar tracking of scientific
findings rather than legal precedents. He wrote in Science over 50 years
ago:
... I propose a bibliographic system for science literature that
can eliminate the uncritical citation of fraudulent, incomplete, or
obsolete data by making it possible for the conscientious scholar to be
aware of criticisms of earlier papers. It is too much to expect a
research worker to spend an inordinate amount of time searching for the
bibliographic descendants of antecedent papers. It would not be
excessive to demand that the thorough scholar check all papers that have
cited or criticized such papers, if they could be located quickly. The
citation index makes this check practicable. [Garfield 1955],
The ability to track the development of ideas through citations is
so widely accepted that Garfield's vision is now likely
underappreciated (Cronin and Atkins 2000). Garfield's idea was
implemented through the ISI that he founded in 1960, which is now owned
by Thomson Reuters.
Garfield's original purpose differed from the ranking function
citations now often perform, an expansion fostered by Garfield himself.
He recognized that the citations could be used to evaluate journals. He
reported in the journal Science in 1972 that ISI "decided to
undertake a systematic analysis of journal citation patterns across the
whole of science and technology" (Garfield 1972, 472). He
illustrated the methodology, published an impact factor ("average
citations per published item," p. 474), and offered three
applications beyond the "most important"--studying science
policy and research evaluation. The applications were to use citation
frequency and impact factor to help manage library collections, to help
scientists select journals to read and keep, and to help journal editors
evaluate their editorial selection policies. The use of impact factors
and other measures to evaluate scholars, departments, universities, and
countries came later.
Garfield recognized that problems of journal inclusion existed. He
noted that his product was "less likely to cover a journal that
presents problems of transliteration ... and translation than one that
does not," which could adversely influence the ranking of foreign
journals (Garfield 1972, 473). As we shall show, problems of journal
inclusion and exclusion beyond those linked to language can produce
questionable results.
B. Data Sources
Garfield's concept has expanded to include a system in which
JCR ranks thousands of science and social science journals in many
disciplines, including the discipline that contained the
citation-indexing precedent that inspired Garfield, law. JCR claims to
be "the only source of citation data on journals." (JCR 201
la). With respect to law journals, that claim is incorrect because
W&L not only ranks law journals but has much broader journal
coverage than does JCR. We first describe the W&L and JCR measures
that we use to compare their journal rankings.
Washington and Lee Law Library. W&L contains five journal
performance measures: Currency-Factor, Impact-Factor, Journal Citations,
Case Citations, and Combined-Score. (1) All are measures of citation
counts or a combination of citations counts and the counts are limited
to citations to journal volumes published in the preceding 8 years. We
use the Currency Factor in this article as it is closest to JCR's
most widely cited measure, JCR's Impact Factor. The W&L
measures are highly correlated with one another and the choice of
measure is not critical to our results.
W&L bases its citation counts on journals that are included in
Westlaw's Journals and Law Reviews ("JLR") database.
Westlaw is a widely used online research service that includes journals
and cases covering U.S. law and materials from a few non-U.S.
jurisdictions. Westlaw describes the JLR database as "containing
documents from law reviews, Continuing Legal Education (CLE) course
materials, and bar journals" (Westlaw Summary 2012). A document is
an article, a note, a symposium contribution, or other item published in
a periodical in the database. As of March 27, 2012, the JLR database
indicates that it included full or part coverage of 985 journals. (2)
The JLR description states that it contains documents from U.S. and
Canadian-based publications but this is not a full description. JLR
includes, for example, the Melbourne University Law Review and the
Oxford Journal of Legal Studies. JLR appears to be limited to English
language publications. The breadth of JLR is impressive in some
respects. It includes 14 journals published by Columbia Law School, 12
journals published by Yale Law School, including two online versions of
print journals, and 16 journals published by Harvard Law School. It
includes journals such as Wyoming Lawyer, a journal that presumably is
primarily of interest to lawyers in one of the United States' least
populous states. Many of the JLR journals are run by temporary student
editors with 1-year tenures. In the W&L database, one can limit
searches to refereed or peer-edited journals.
W&L's Currency Factor compares journals based on how
rapidly their articles become cited. As described on the W&L
website, which illustrates the Currency Factor calculation for the
2002-2009 survey period:
currency-factor is the number of articles added to
Westlaw's JLR database in the three-year period of
2002-2004 that cite to volumes of a journal dated
during those same three years, divided by the number
of items published by the journal during those same
years. (3)
So the W&L Currency Factor would equal two for a journal that
published 20 articles in the period 2002 to 2004, and for which 40
articles were added to Westlaw during that period citing the 20
articles. As is the case for all the W&L citation measures, it
appears that multiple citations to a journal article within the same
document do not increase the citation count because the count is based
on the number of articles citing the journal, not the actual number of
citations to the journal. For 2011, for refereed journals, the ten most
highly ranked journals by Currency Factor had factors ranging from 3.41
to 1.8. The lowest ranked journals had Currency Factors of zero. But
many of the zeros are due to foreign journals being ranked based on a
database that contains mostly U.S. journals. For 2010, the year we
analyze to compare W&L with JCR, the ten most highly ranked
journals' Currency Factors ranged from 3.42 to 1.41.
The Currency Factor requires a denominator to account for the
number of articles a journal published in the relevant time period.
W&L notes that no fully satisfactory and automated method exists for
determining this number. It reports that information about the number of
articles was obtained from the WilsonWeb Index to Legal Periodicals (4)
when possible and, in other cases, from various sources. (5)
Journal Citation Reports. While W&L specializes in law journal
metrics, JCR covers a broad range of subject areas. The JCR Social
Sciences Edition purports to offer "a systematic, objective means
to critically evaluate the world's leading journals, with
quantifiable, statistical information based on citation data"
(Thomson Reuters 2012). It covers more than 2,600 social science
journals, though it is likely that it includes only a small fraction of
the universe of social science journals. Braun, Glanzel, and Schubert
(2000) estimate that the associated Science Citation Index includes
about 10% of the science and technology journals.
JCR reports five journal influence measures: Total Cites, Impact
Factor, Immediacy Index, Eigenfactor Score, and Article Influence Score.
(6) The measures are highly correlated. We use the Impact Factor in this
article because, like W&L's Currency Factor, it is a measure of
citations to recently published articles. It is also the most frequently
invoked JCR citation measure.
JCR organizes it measures by what it terms the "JCR
year." Each JCR year contains one year of citation data. JCR
searches are year-specific so a search of the 2010 data yields the
number of citations to a journal in 2010. As of this writing, 2010 is
the most recent year for which JCR citation counts are available. Not
all items published in journals are included in JCR calculations.
Editorials, letters, news items, and meeting abstracts are not counted
"because they are not generally cited" (JCR 201 lb).
A journal's Impact Factor is the average number of times
articles from the journal published in the past 2 years have been cited
in the JCR year. The number of citations is divided by the number of
articles published by the journal in the previous 2 years. So the 2010
Impact Factor equals the citations in 2010 to articles published in a
journal in 2008 or 2009, where the citations are in journals included in
JCR, divided by the number of articles published by the journal in 2008
and 2009. For example, an Impact Factor of 1.0 means that the average
number of citations to the articles published 1 or 2 years prior to the
JCR year is one. The citation count includes articles citing other
articles in the same journal.
Table 1's first two rows summarize the W&L Currency Factor
for 2010 for: (1) all refereed law journals in W&L and (2) for the
54 refereed law journals that also appear in JCR. The third row
summarizes the JCR Impact Factor for the refereed journals that also
appear in W&L. For the 54 refereed law journals included in JCR, the
average Impact Factor is 0.84, with a range of 0 to 2.27, so the average
article is cited less than once per year in JCR journals in the period
shortly after publication.
JCR limits citations credited to journals to citations appearing in
journals in the JCR database. For law journals as of the time of this
analysis, those data are limited to 133 journals, of which 54 are
refereed (not edited by students). Because the scope of subject matter
category coverage will be shown to have an important influence on JCR
factors, Table 2 shows the top 25 subject matter categories (of 56 total
categories), by total cites, in JCR's social science database for
2010. Economics is the dominant field in terms of the numbers of
citations, journals, and articles, but psychology's many subfields,
if aggregated, would exceed the economics numbers.
III. CONSISTENCY OF RANKINGS
We use W&L and JCR to address the consistency of journal
rankings across the two systems. One expects to observe consistency, but
a major difference is the groups of journals they count in computing
impact measures. W&L specializes in law journals; JCR's journal
pool spans many fields. Bao, Lo, and Mixon (2010, 352) provide evidence
that combining articles in all research fields to generate rankings can
introduce bias into rankings. They construct a new journal ranking using
econometrics articles as a group of specialty articles. They find that
the intellectual influence of an article as measured by citations to it
using the new ranking is much higher than if it were published in
higher-ranked general interest economics journals such as American
Economic Review. "[U]sing the existing economics journal rankings
to evaluate econometricians' research productivity is an
error-ridden system because it imposes a substantial downward bias
against them." They observe that the prevailing practice by
academic institutions of judging article quality by where articles are
published, in contrast to their impact as measured by citations, is
problematic.
JCR's inclusion of many disciplines but an incomplete sample
of law journals may introduce an analogous problem with the reliability
of its rankings. If citations to law-related articles can be reasonably
expected to occur in law journals, then a database that emphasizes
non-law journals may understate the impact of law journals in their core
field of interest, law. It may also provide a noisy signal of quality
based on the makeup of JCR's non-law sample of journals. JCR's
consistency with W&L can provide some insight into this issue.
Figure 1 shows the relation between the W&L Currency Factor and
JCR's Impact Factor for 34 refereed law journals appearing in both
systems. These 34 journals include those with an Impact Factor or
Currency Factor of at least 0.5. Figure 1 does not show a strong
association between the two measures.
The two measures have a correlation coefficient of -0.18 (using log
transformations) that is significant at p- .33. Since both ranking
systems use multiple measures, perhaps a more composite measure of the
two ranking systems would improve the correlation. We use factor
analysis results to construct a single measure for each ranking system
and show that relation in Figure 2. The correlation coefficient remains
low, 0.20, with p =. 15. It is as if the two systems, albeit both
nominally based on citations, are ranking different universes of
journals.
IV. EXPLAINING THE DIFFERENCES IN RANKINGS
We now explore the absence of correlation between W&L and JCR
rankings. W&L seeks to evaluate law journals along several
dimensions and focuses on that goal by emphasizing law-related journals.
JCR's goal is much broader; it seeks to evaluate a vast body of
scholarship that includes many disciplines. In the course of doing so,
it may not perform well with respect to disciplines that it
underrepresents or does not fully apprehend (Neuhaus, Marx, and Daniel
2009).
A. Source of Ranking Difference
Insight into the sources of ranking difference emerges by examining
the JCR database in more detail. Table 2 shows that economics has by far
the most articles and journals in the database, though the multiple
psychology categories if combined would outrank economics. Since the
basic citation unit in JCR is an article, Table 2 shows that economics
articles had 14,402 opportunities to be cited in JCR. Law articles had
about one-fourth that amount yet law has many journals not included in
JCR. A concern about using a more limited set of journals is that
citations based on that set provide no credit for articles citing a work
that do not appear in the journal list included in the study (Bao, Lo,
and Mixon 2010, 348, n.5). So the more law-oriented the content of a
journal, the lower its baseline JCR expected citations compared to
economics.
[FIGURE 1 OMITTED]
[FIGURE 2 OMITTED]
JCR's underrepresentation of law journals is substantial. The
Australian Research Council (ARC) journal list contains 1,060 journals
with the word "law" or the word "legal" in the title
and lists 1.167 journals as law journals (and five more as law and legal
studies). (7) It contains 563 journals with the letters
"econom" in the title. If ARC reasonably proxies the number of
journals in a category, then JCR contains 12.5% of law journals and
54.2% of economics journals. As a check on the total number of economics
journals, the EconLit database on EBSCO lists 657 journals. (8) As a
check on the total number of law journals, recall that Westlaw includes
985 journals. So while ARC undoubtedly omits many journals, it is not
dramatically far off as a proxy for a subject category's journals
and has the attractive feature of presumed consistent methodology in
locating and classifying journals. JCR's selection of journals for
inclusion is proprietary. Whatever the precise measure, JCR vastly
overrepresents economics journals compared to law journals. The more the
law emphasis in a journal the greater the bias against law-focused
journals in JCR's database. (9)
[FIGURE 3 OMITTED]
Although economics articles had the most opportunities to be cited
in JCR, economics as a category performed poorly by other measures. The
larger number of articles in a field provides more opportunities to be
cited. But it also increases the denominator used to compute impact
factors for a field as a whole. So the ex ante relation between the
number of articles in a category and impact is not obvious.
Figure 3 shows the relation between a category's number of
articles and its median impact factor. The relation is clearly a
positive one. Articles in larger fields, as measured by total articles
in JCR, tend to have more impact. But economics is somewhat of an
outlier. The solid line in the figure shows the predicted values from a
regression ([R.sup.2] = 0.07) of a category's (log) median journal
impact as the dependent variable and (log) number of articles as the
explanatory variable. Economics is distant from the line. The dashed
line in the figure shows the predicted values if one excludes economics
([R.sup.2] = 0.09). That single category noticeably pulls down the
relation.
[FIGURE 4 OMITTED]
What might lead to a category's outlier performance in Figure
3? Recall, the various impact factors are computed at the journal level
not across the field. A field that has more articles per journal has
more opportunities to be cited. For instance, law journals typically
publish longer articles than economics journals, therefore the average
number of articles in law journals will be fewer than in economics
journals. Hence a particular law journal will likely have less
opportunities to be cited than an economics journal. Figure 3 says
nothing about the average number of articles per journal in a category.
Figure 4 shows the relation between the (log) average number of articles
per journal (number of articles/number of journals) in a category and
its (log) median impact factor. The relation is positive and fields with
a larger average number of articles per journal tend to have higher
impact factors. The line in Figure 4 shows the predicted values from a
regression of a category's (log) median journal impact as the
dependent variable and (log) average number of articles per journal as
the explanatory variable ([R.sup.2] = 0.60, slope coefficient 0.96, and
95% CI: 0.74-1.17). Now when controlling for the average number of
articles, rather than the total number, in a field category economics is
no longer exceptional. As a matter of record, in this regression model
the absolute value of all the studentized residuals (10) are all less
than 2.5, hence there are no large outliers.
How does the overrepresentation of economics journals or the
selection of journals for inclusion in JCR and Westlaw influence law
journal rankings? Table 3 shows the refereed law journals that are
included in both W&L and JCR's rankings. The journals are
ordered by W&L's Currency Factor for 2010, as shown in the
first numerical column. The table also shows each journal's JCR
Impact Factor and the journal's rank by those measures. One need
not read far down the list to observe startling effects. By the
W&L's Currency Factor measure, SCR is the second-ranked
refereed law journal, just after the Journal of Empirical Legal Studies.
Yet SCR ranks 22nd according to JCR's Impact Factor. If one ordered
the list by JCR measures, similar effects emerge. The JLE ranks seventh
of all refereed law journals based on JCR's Impact Factor. It ranks
18th based on W&L's Currency Factor.
Further evidence about the source of W&L-JCR inconsistency
comes from considering the student law journals that our analysis thus
far excludes. Student law journals are rarely interdisciplinary because
the student editors tend to lack expertise outside of law and because
the supply of pure law articles is sufficient to populate the many
existing journals. Indeed, the reason for many refereed journals'
existence is the lack of student expertise in other fields, such as
economics, psychology, and statistics. So rankings of student law
journals are likely to be less influenced by the selection of journals
by JCR with respect to its nonlaw databases.
The rankings of the student-dominated non-refereed journals are
much more consistent across the two ranking systems. The correlation
coefficient between W&L's Impact Factor and JCR's Currency
Factor for the 90 law journals (now adding 56 student journals) with
impact factors greater than 0.5 is 0.27, significant at p = .01. Remove
the 34 refereed journals (shown above to have low correlation) in that
sample and the correlation coefficient for the remaining 56 student
journals increases to 0.43, significant at p = .001.
B. Difference in Rankings: Two Case Studies
A case study of why SCR and JLE perform so differently in the two
ranking systems, as shown in Figure 2 and Table 3, helps illuminate the
JCR's limitations for law journals. The SCR-JLE comparison is not
randomly chosen. Both journals are highly respected faculty-edited
journals controlled by the same elite institution, the University of
Chicago, and founded within a few years of each other. Chicago's
law school and economics department are unquestionably elite.
The journals' more detailed pedigree also suggests that they
are both excellent journals. SCR is edited by University of Chicago law
professors, is one of the first faculty-edited law journals (founded in
1960), and its articles over the years have been written by a Who's
Who of constitutional law professors. Constitutional law, its central
topic, is of course a central topic in U.S. law schools, especially
elite ones, and some schools likely owe much of their contemporary
prestige in large part to the visibility of their constitutional law
scholars (Eisenberg and Wells 1998, 407-9). Constitutional scholars tend
to be cited more than other legal scholars (Eisenberg and Wells 1998,
408, 410). JLE also has unquestioned pedigree and historical importance.
JLE's creation was at the core of activity that attracted Ronald
Coase and Judge Richard Posner to the University of Chicago. (11)
So some level of subjective quality control exists; we are
comparing two elite law-related journals. Yet the two citation systems
produce startlingly different and, in the case of JCR's ranking of
SCR, ludicrous, results. The objective measure of SCR's impact by
W&L is consistent with its reputation and pedigree. JCR's
measure is not. How can it perform so much more poorly in JCR than in
W&L?
We probe further by examining the journals that JCR reports cite to
SCR and JLE. Table 4 shows, for JCR year 2010, the journals citing to
SCR and JLE. These citing journals are the ones counted in JCR's
measures. For JCR year 2010, the citing journals must have been
published in 2010 and must have cited articles in SCR and JLE published
in 2008 or 2009. We combine those two years.
The table establishes that JLE's prominence in JCR is due to
its being cited in many economics, finance, and accounting journals, the
vast majority of which are not in the Westlaw journals database used by
W&L. We estimate that only 18 of the 70 citations to JLE in Table 4
are in journals included in Westlaw. JCR reports that its Impact Factor
ranking of JLE in 2010 is based on 60 articles (12) and that its ranking
of SCR in 2010 is based on 17 articles. (13) Since JCR divides the
number of citations by the number of articles, the 14 citations in Table
4 to SCR means that SCR would far outperform JLE if JCR were limited to
journals in Westlaw. So the dominance of economics journals in
JCR's data contributes to the relative performance of JLE compared
to SCR.
JCR's inclusion criteria are part of the explanation but so
are Westlaw's exclusion criteria since it does not include most of
the nonlaw journals in Table 4 that cite to JLE articles. Similarly,
Westlaw's inclusion of many more law journals than JCR must
contribute to SCR's better performance in the W&L system. As
noted above, Westlaw includes approximately 1,000 law journals and JCR
includes only 133 law journals, a surprisingly low number since another
citation system, Scopus, includes nearly 400 law journals. JLE should of
course receive credit for the many economics journals in which it is
cited. But it is questionable to include JLE and SCR in a category in
which they are purportedly ranked against one another on a
methodologically rigorous basis yet in a system that vastly
overrepresents economics journals and vastly underrepresents law
journals.
SCR's poorer performance in JCR is thus not likely
attributable to actual lower impact. It rests instead in the databases
used to compute measures. How else can Regulation and Governance or the
Journal of the American Academy of Psychiatry and Law outperform SCR, as
they do in JCR? These are fine journals but that they have more legal
scholarly impact than SCR is not credible. Regulation and Governance
likely benefits from the many economics journals in JCR and Journal of
the American Academy of Psychiatry and Law likely benefits from
JCR's strong social science content. A similar effect may favor
psychology journals in JCR. As Table 2 shows, psychology is highly
represented in the JCR data. A similar effect helps explain the
prominence of SCR in W&L. W&L uses Westlaw, which is a
law-centered database and U.S law-centered to a large degree. Journals
with more purely law-related content ought to fare better in it than in
the JCR database.
JLE and SCR focus on such different areas of law that direct
comparison of them may not fully resolve an issue. Our second case study
compares the rankings for ALER and JLE, two excellent journals in the
same joint discipline, law and economics. No journal is more
historically important than JLE in promoting social science in legal
academia (Eisenberg 2011). ALER, as of this article's writing, was
edited by two prominent law school professors, Steven Shavell at Harvard
and John Donohue at Stanford, and is the official journal of the
American Law and Economics Association (ALEA), which is administered out
of Yale Law School. (14) JLE is a joint publication of Chicago's
business and law schools and not directly associated with ALEA. JLE and
ALER are thus two journals that both emphasize law economics and that
are connected to elite institutions. They differ, however, in their
connections to legal academia.
Table 5, based in part on a table in Eisenberg (2011), compares
ALER and JLE performance in recent years in the W&L database. Table
5 shows the W&L rankings of the two journals for four recent
available years. Both are sufficiently highly ranked in W&L to be in
the top 10% of refereed journals. Despite JLE's excellence and
prominence in the history of law and economics, however, JLE has been
passed by ALER in quantitatively measurable impact among legal
academics. ALER's stronger centering in law schools likely has
contributed to its growing relative ascendancy in a law journal
dominated database relative to JLE. Centering in law schools likely
leads to less technical articles on average and to topics likely of
greater interest to legal academics and attorneys. JLE, however,
continues to outperform ALER in JCR, as shown in Table 3, although both
journals do substantially better there than in W&L.
C. Context
JCR's inclusion of so many disciplines precludes it from
accounting for important within-and across-discipline context. Legal
scholarship has distinctive features inapplicable to most disciplines.
It is characterized by the presence of many student-edited law journals.
As noted above, within academically-oriented law journals, it is
questionable whether student journals should be included in the same
category as refereed journals. Refereed law journals tend to publish
different kinds of articles than student journals and the articles tend
to be much shorter. These characteristics have unknown effects on
citation patterns. At a minimum, one should do what W&L does, which
allows the user to isolate refereed journals if they so desire.
Legal scholarship can also have a wider than usual array of target
audiences. The target audience can vary from the practitioners of law,
to judges, to academia, to policymakers. W&L's system reflects
this diversity in part by tracking citations to legal scholarship by
cases as well as by journals. A journal with a strong presence in case
law through citations to it by cases receives no credit for that in
JCR's multiple measures. This helps explain the poorer performance
of the American Bankruptcy Law Journal (ABU) in JCR than in W&L, as
shown in Figures 1 and 2. Yet ABU is the most-cited law journal in
cases. A ranking system purporting to assess law journals should allow
users to understand this. The Business Lawyer, the journal of the
Business Law Section of the American Bar Association, is also frequently
cited by courts yet fares poorly in JCR. A journal's rank perhaps
ought not suffer because it succeeds in reaching nonacademic audiences
in a discipline in which those audiences are essential. This seems
particularly misguided with respect to law or journals of other
professions.
V. CONCLUSION
We use data from W&L and JCR to evaluate the ranking of
refereed law journals. Both systems base rankings on citation measures
yet the rankings are uncorrelated. This raises a question about the
sources of their lack of correlation. We identify a source as JCR's
underrepresentation of law journals compared to other subject areas.
Although all ranking methods have limitations, JCR's treatment
of law journals is troublesome. JCR originated as part of a
citation-tracking system intended to enable scholars to track the
history and promulgation of scientific ideas. But it is now also used to
rank journals and thus scholars in many fields. (15) The original
conception was largely immune to lack of institutional knowledge about
the ranked disciplines and had little downside because the goal was to
trace the development of ideas. Gaps in development due to journal
database limitations were simply gaps and were unlikely to be
affirmatively harmful or misleading or even to persist.
JCR's newer journal ranking function is more vulnerable to the
makeup of its journal database. Biases in JCR's database that
distort rankings can have obvious harmful effects on scholars who
publish in inappropriately ranked journals and on the journals
themselves. The concern is not purely abstract and our results echo
Neuhaus, Marx, and Daniel's (2009) findings with respect to
multidisciplinary journals. As interdisciplinary scholarship grows,
JCR's varying coverage of subject categories becomes of greater
concern. The original Garfield concept, still of value in many contexts
through JCR's continuing efforts, has become a process that cannot
account for the nuances of the many disciplines it ranks. Most
fundamentally, it can produce unreasonable ranking results.
ABBREVIATIONS
ALEA: American Law and Economics Association
ALER: American Law and Economics Review
JCR: Journal Citation Reports
JLE: Journal of Law and Economics
JLR: Journals and Law Reviews
SCR: Supreme Court Review
W&L: Washington and Lee Law Library
doi: 10.1111/ecin. 12133
Online Early publication July 29, 2014
REFERENCES
Australian Research Council (ARC). "Excellence in Research for
Australia (ERA), 2012 Journal List." 2012. Accessed May 30, 2012.
http://www.arc.gov.au/era/era_2012/era Joumal_list.htm.
Bao, Y., M. Lo, and F. G. Mixon Jr. "General-Interest Versus
Specialty Journals: Using Intellectual Influence of Econometrics
Research to Rank Economics Journals and Articles." Journal of
Applied Econometrics., 25, 2010, 345-53.
Bergstrom, C. T. and T. C. Bergstrom. "The Economics of
Scholarly Journal Publishing." 2006. Accessed May 30, 2012.
http://octavia.zoology.washington.edu/publishing/.
Braun, T., W. Glanzel, and A. Schubert "How Balanced Is the
Science Citation Index's Journal Coverage? A Preliminary Overview
of Macrolevel Statistical Data," in The Web of Knowledge: A
Festschrift in Honor of Eugene Garfield, edited by B. Cronin and H. B.
Atkins. Medford, NJ: Information Today, Inc., 2000, 251-77.
Cook, R. D., and S. Weisberg Residuals and Influence in Regression.
New York: Chapman and Hall, 1982.
Cronin, B., and H. B. Atkins "Introduction: The Scholar's
Spoor." in The Web of Knowledge: A Festschrift in Honor of Eugene
Garfield, edited by B. Cronin and H. B. Atkins. Medford, NJ: Information
Today, Inc., 2000, 1-10.
Currie. R. R., and G. S. Pandher. "Finance Journal Rankings
and Tiers: An Active Scholar Assessment Methodology." Journal of
Banking & Finance, 35, 2011, 7-20.
Davis, P. "Reward or Persuasion? The Battle to Define the
Meaning of a Citation. " Learned Publishing, 22(1), 2009, 5-11.
Eisenberg, T. "Assessing SSRN-Based Law School Rankings."
Indiana Law Journal, 81, 2006, 285-91.
--. "The Origins, Nature, and Promise of Empirical Legal
Studies and a Response to Concerns." University of Illinois Law
Review, 2011(5), 2011, 1713-38.
Eisenberg, T., and M. T. Wells. "Ranking and Explaining the
Scholarly Impact of Law Schools." Journal of Legal Studies, 27,
1998, 373-413.
Garfield, E. "Citation Indexes for Science." Science,
3159, 1955, 108-11.
--. "Citation Analysis as a Tool in Journal Evaluation."
Science, 178(4060), 1972, 471-79.
JCR. "About Journal Citation Reports. " 2011a. Accessed
May 30, 2012. http://admin-apps. webofknowledge. com/JCR/help/h
Jcrabout. htm.
--. "Journal Source Data. " 2011b. Accessed May 30,
"2012. http://admin-apps. webofknowledge.
com/JCR/help/h_sourcedata.htm.
Meadows, J. "The Growth of Journal Literature: A Historical
Perspective," in The Web of Knowledge: A Festschrift in
Honor of Eugene Garfield, edited by B. Cronin and H. B. Atkins.
Medford, NJ: Information Today, Inc., 2000, 87-108.
Moosa, I. "The Demise of the ARC Journal Ranking Scheme: An Ex
Post Analysis of the Accounting and Finance Journals." Accounting
and Finance, 51. 2011, 809-36.
Neuhaus, C., W. Marx, and H.-D. Daniel. "The Publication and
Citation Impact Profiles of Angewandte Chemie and the Journal of the
American Chemical Society Based on the Sections of Chemical Abstracts: A
Case Study on the Limitations of the Journal Impact Factor."
Journal of the American Society for Information and Science Technology,
60, 2009, 176-83.
Teles, S. M. The Rise of the Conservative Legal Movement: The
Battle for Control of the Law. Princeton, NJ: Princeton University
Press, 2008.
Thomson Reuters. "Journal Citation Reports." 2012.
Accessed May 30, 2012. http://thomsonreuters.
com/products_services/science/science_products/a-z/journalcitationreports/.
Washington and Lee Law School. "Law Journal: Submissions and
Ranking." 2012. Accessed May 30, 2012.
http://lawlib.wlu.edU/LJ/method.asp#methodology.
Westlaw Summary. "Journals & Law Reviews Database,
Westlaw." 2012. Accessed March 28,2012. http://web2.
westlaw.com/scope/default.aspx?db=JLR&RP=/scope/default.
wl&RS=WLW 12.01 &VR=2.0&S V=Split&
FN=_top&MT=Westlaw&MST
Wouters, P. "Garfield as Alchemist," in The Web of
Knowledge: A Festschrift in Honor of Eugene Garfield, edited by B.
Cronin and H. B. Atkins. Medford, NJ: Information Today, Inc., 2000,
65-72.
(1.) W&L also includes a cost-performance measure, Cites per
Cost, which is not a pure citation measure.
(2.) To avoid double counting of journals, the 985 figure excludes
journals in JLR that changed names and in which the former name occupies
a line in the journal listing. The W&L Website lists 1,683 journals,
many of which are listed for informational purposes other than ranking.
As of March 27, 2012, an online query limited to the Currency Factor for
ranked journals listed nonzero values for 1,210 journals. Since the JLR
database contains only 985 journals, it is not clear why the W&L
search, which uses the JLR database, ranks more than 985 journals. It
may be that the JLR database list is incomplete in what it includes
(e.g., Westlaw erroneously states that it is limited to U.S. and
Canadian sources) or that the W&L list of ranked journals treats
some as ranked that are not in the JLR database.
(3.) The following statement accompanies the Currency Factor
description:
It would have been desirable to create this index from
the final three years of the survey period, but the data
on which it's based, being automatically created from
annual data collected to calculate impact-factor, is in
a form requiring the use of the first three years of each
survey period. For any journal that began publication
after the beginning of the survey period the three years
will be the first three years of the journal's existence.
(4.) Wilson Index to Legal Periodicals and Books,
http://www.ebscohost.com/academic/index-to-legalperiodicals-books.
(5.) W&L reports that if other sources had to be used, the next
preference was Westlaw (if Westlaw comprehensively added articles for
the years needed), followed by Lexis, then Legal Resource Index, then
Legal Journals Index (United Kingdom), and then any other index in which
the journal was indexed. W&L also states, "Often a manual count
was made by physically examining the tables of contents for the journal
years needed. In cases where indexing was not available and a manual
count was not feasible, then an extrapolation was made from what was
known. As these variant sources undoubtedly have differing definitions
as to what is a countable entity this introduces variability into the
counts."
(6.) Note that W&L's Currency Factor is not analogous to
JCR's similarly named Immediacy Index.
(7.) Even ARC'S impressive database of about 22,000 journals
is likely a substantial undercount. One estimate is that there were
71.000 scientific journal titles as of 1987 (Meadows 2000, 92), a figure
that does not include social science journals.
(8.) One source refers to there being 300 economics journals in
2000 (Bergstrom and Bergstrom 2006).
(9.) A similar concern about database content exists when using the
SSRN system to rank law schools (Eisenberg 2006, 289-90).
(10.) Studentized residuals equal the usual regression residual
divided by an estimate of its standard deviation, their use is an
important technique in the detection of outliers (Cook and Weisberg
1982). If the regression model is true, about 5% of observations will
have studentized residuals outside of the ranges [-2, 2],
(11.) Judge Posner, the most visible law and economics scholar, was
the first recipient of the American Law and Economics Association Coase
medal. See http://aler. oxfordjournals.org/cgi/content/full/ahq009v 1
(Accessed August 29, 2010). He was preceded at Chicago by Nobel
prize-winning economist Ronald Coase, who was preceded by Aaron
Director. Director played a central role in developing law and economics
at Chicago and his influence increased with JLE's creation in 1958
(Teles 2008, 95), just 2 years before SCR first published. Coase doubted
that he would have gone to Chicago absent JLE (Teles 2008, 96). Coase
and Director, in turn, were instrumental in attracting Posner to Chicago
from Stanford (Teles 2008,97-98).
(12.) http://admin-apps.webofknowledge.com.proxy.
library.cornell.edu/JCR/JCR?RQ=RECORD&rank=71&
journal=J+LAW+ECON.
(13.) http://admin-apps.webofknowledge.com.proxy.
library.cornell.edu/JCR/JCR?RQ=RECORD&rank=71& journal = SUPREME
+ COURT + REV.
(14.) The editors have since changed.
(15.) The shift in intended use has been noted in, for example
Davis (2009).
Eisenberg: Deceased.
Wells: Charles A. Alexander Professor of Statistical Science and
Professor of Social Statistics, Department of Statistical Science,
Cornell University, 1190 Comstock Hall, Ithaca, NY 14853. Phone 607 255
8801, Fax 607 255 4698, E-mail mtwl@comell.edu
TABLE 1
Descriptive Statistics of Measures of Refereed
Law Journal Impact
N M SD Min Max
W&L: All refereed 562 0.139 0.321 0 3.42
journals in W&L
W&L: Refereed law 54 0.520 0.671 0 3.42
journals also in JCR
JCR: All refereed law 54 0.840 0.629 0 2.27
journals in JCR
TABLE 2
Summary Characteristics of Academic Categories, JCR Data 2010
Rank Category
1 Economics
2 Psychiatry
3 Management
4 Psychology, Experimental
5 Business
6 Psychology, Multidisciplinary
7 Psychology, Clinical
8 Public, Environmental, and Occupational Health
9 Psychology, Developmental
10 Psychology, Social
11 Sociology
12 Psychology, Applied
13 Education and Educational Research
14 Business Finance
15 Health Policy and Services
16 Environmental Studies
17 Social Sciences, Mathematical Methods
18 Law
19 Political Science
20 Nursing
21 Psychology, Educational
22 Linguistics
23 Gerontology
24 Anthropology
25 Social Sciences Interdisciplinary
Median Aggregate
Total Impact Impact No. of
Rank Cites Factor Factor Journals Articles
1 380,146 0.75 1.188 305 14,403
2 352,344 1.565 2.918 110 7,887
3 279,688 1.22 1.768 144 5,898
4 220,555 1.813 2.465 81 5,629
5 217,885 1.365 1.597 103 4,629
6 216,785 1.065 1.868 120 5,755
7 209,743 1.524 2.27 104 5,517
8 202,288 1.286 1.721 116 9,245
9 154,240 1.525 2.352 66 3,675
10 144,205 1.18 1.686 58 3,141
11 112,171 0.767 0.923 132 4,159
12 104,570 1.326 1.658 69 2,484
13 99,229 0.649 0.906 184 6,862
14 93,384 0.758 1.29 76 3,122
15 84,593 1.397 1.942 58 3,689
16 84,454 1.11 1.749 78 4,479
17 84,016 0.9 1.272 43 1,930
18 76,290 0.786 1.178 133 3,761
19 76,087 0.655 0.806 141 5,078
20 70,584 0.957 1.033 87 5,141
21 67,250 1.054 1.416 50 1,669
22 66,868 0.565 0.96 144 3,355
23 65,901 1.082 2.124 30 2,093
24 60,091 0.68 1.228 76 2,756
25 59,906 0.643 1.023 84 3,611
TABLE 3
Refereed Law Journal Ranks by W&L and JCR Measures, 2010
Currency Impact
Factor Factor
Journals (W&L) (JCR)
Journal of Empirical Legal Studies 3.42 1.565
Supreme Court Review 2.44 0.824
Journal of Legal Studies 1.91 2.239
American Journal of International Law 1.61 0.865
American Business Law Journal 1.55 1.576
Business Lawyer 1.41 1
Antitrust Law Journal 1.34 0.49
Law and Social Inquiry 0.91 0.965
American Bankruptcy Law Journal 0.91 0.513
Journal of Law, Economics, and Organization 0.89 1.595
American Law and Economics Review 0.82 0.696
American Journal of Comparative Law 0.81 0.965
Journal of Legal Education 0.8 0.396
Law and Society Review 0.74 1.558
Journal of Legal Medicine 0.69 0.6
International Journal of Constitutional Law 0.66 0.754
Journal of the Copyright Society of USA 0.61 0.239
Journal of Law and Economics 0.53 1.617
Food and Drug Law Journal 0.5 0.514
Psychology Public Policy and Law 0.49 2.16
Family Law Quarterly 0.46 0.42
Journal of International Economic Law 0.43 0.95
Law and Human Behavior 0.39 2.268
Journal of Law, Medicine and Ethics 0.35 1.294
Issues in Law and Medicine 0.31 0.312
Regulation and Governance 0.3 1.488
Law and Philosophy 0.29 0.314
Melbourne University Law Review 0.24 0.276
Judicature 0.24 0.429
European Law Journal 0.21 0.789
International Review of Law and Economics 0.2 0.3
International Review of the Red Cross 0.19 0.354
Behavioral Sciences and Law 0.19 1.505
Modern Law Review 0.18 0.326
Journal of Law and Society 0.13 0.772
International Journal of Transitional 0.11 1.756
Justice
Justice System Journal 0.1 0.295
Journal of African Law 0.09 0.16
Chinese Journal of International Law 0.08 0.206
Common Market Law Review 0.08 2.194
World Trade Review 0.08 1.231
Juvenile and Family Court Journal 0.07 0.067
European Business Organization Law Review 0.07 0.36
Social and Legal Studies 0.06 0.673
Netherlands Quarterly Human Rights 0.05 0
European Journal of Migration and Law 0.04 0.303
Journal of Environmental Law 0.04 0.359
Journal of American Academy of Psychiatry 0.02 1.785
and Law
International Environmental Agreements: 0.01 1.128
Politics, Law and Economics
Psychiatry, Psychology and Law 0.01 0.494
Asian Journal of WTO and International 0 0.333
Health Law and Policy
Asia Pacific Law Review 0 0.059
European Constitutional Law Review 0 0.74
Review of Central and East European Law 0 0.286
W&L
Currency
Factor JCR Impact
Journals Rank Factor Rank
Journal of Empirical Legal Studies 1 10
Supreme Court Review 2 22
Journal of Legal Studies 3 2
American Journal of International Law 4 21
American Business Law Journal 5 9
Business Lawyer 6 17
Antitrust Law Journal 7 33
Law and Social Inquiry 8 18
American Bankruptcy Law Journal 8 31
Journal of Law, Economics, and Organization 10 8
American Law and Economics Review 11 27
American Journal of Comparative Law 12 18
Journal of Legal Education 13 36
Law and Society Review 14 11
Journal of Legal Medicine 15 29
International Journal of Constitutional Law 16 25
Journal of the Copyright Society of USA 17 49
Journal of Law and Economics 18 7
Food and Drug Law Journal 19 30
Psychology Public Policy and Law 20 4
Family Law Quarterly 21 35
Journal of International Economic Law 22 20
Law and Human Behavior 23 1
Journal of Law, Medicine and Ethics 24 14
Issues in Law and Medicine 25 43
Regulation and Governance 26 13
Law and Philosophy 27 42
Melbourne University Law Review 28 48
Judicature 28 34
European Law Journal 30 23
International Review of Law and Economics 31 45
International Review of the Red Cross 32 39
Behavioral Sciences and Law 32 12
Modern Law Review 34 41
Journal of Law and Society 35 24
International Journal of Transitional 36 6
Justice
Justice System Journal 37 46
Journal of African Law 38 51
Chinese Journal of International Law 39 50
Common Market Law Review 39 3
World Trade Review 39 15
Juvenile and Family Court Journal 42 52
European Business Organization Law Review 42 37
Social and Legal Studies 44 28
Netherlands Quarterly Human Rights 45 54
European Journal of Migration and Law 46 44
Journal of Environmental Law 46 38
Journal of American Academy of Psychiatry 48 5
and Law
International Environmental Agreements: 49 16
Politics, Law and Economics
Psychiatry, Psychology and Law 49 32
Asian Journal of WTO and International 51 40
Health Law and Policy
Asia Pacific Law Review 51 53
European Constitutional Law Review 51 26
Review of Central and East European Law 51 47
TABLE 4
Journals that Cite Articles in the Journal of Law and
Economics and in Supreme Court Review, JCR 2010
No. of Cites No. of Cites
to Journal to Supreme
of Imw and Court
Citing Journals Economics Review
Accounting Review 1
Administrative Law Review 1
American Business Law 2
Journal
American Economic Journal 1
Economic Policy
American Economic Review 1
American Law and 2
Economics Review
BE Journal of Economic 2
Analysis and Policy
British Journal of Industrial 1
Relations
Columbia Law Review 1
Cornell Law Review 1
Duke Law Journal 2
Economic Letters 1
Energy Economics 1
Environmental Management 1
Explorations in Economic 3
History
Financial Management 2
Information Economics and 1
Policy
Innovation: Management, 1
Policy
International Journal of 1
Industrial Organization
Journal of Accounting and 4
Economics
Journal of Accounting and 2
Public Policy
Journal of Accounting 1
Research
Journal of Banking and 1
Finance
Journal of Consumer Affairs 1
Journal of Corporate Finance 2
Journal of Economic 1
Literature
Journal of Economics and 1
Management Strategy
Journal of Economic 1
Perspectives
Journal of Empirical Finance 2
Journal of Empirical Legal 2
Studies
Journal of Environmental 2
Economics and Management
Journal of Finance 2
Journal of Financial Economics 2
Journal of Health Economics 3
Journal of Public Economics 2
Journal of Regulatory 1
Economics
Land Economics 2
Michigan Law Review 2
New York University Law Review 4
Northwest University Law 2 3
Review
Public Choice 1
Regulation and Governance 1
Research Policy 1
Review of Economics and 1
Statistics
Review of Industrial 1
Organization
Stanford Law Review 2
University of Chicago Law 2
Review
University of Illinois Law 2
Review
Virginia Law Review 2
Wisconsin Law Review 1
Yale Law Journal 1
Total 70 14
TABLE 5
Journal of Law and Economics and American Law and Economics
Review Rank Among Refereed Law Journals, W&L 2010
2011 2010 2009 2008
JLE ALER JLE ALER JLE ALER JLE ALER
Currency factor 53 25 40 23 46 16 21 16
Impact factor 51 25 40 19 40 19 28 16
Combined 45 32 35 26 34 24 25 22
