Which journal rankings best explain academic salaries? evidence from the University of California.
Gibson, John ; Anderson, David L. ; Tressler, John 等
I. INTRODUCTION
The ranking of an academic journal is important to authors,
universities, journal publishers, and research funders. Rankings are
gaining prominence as countries adopt regular research assessment
exercises that especially reward publication in high-impact journals.
(1) Moreover, journal rankings are used as a proxy for the value of
published research in many other assessments of departments and
individuals (e.g., Combes and Linnemer 2003; Coupe 2003; Kalaitzidakis,
Mamuneas, and Stengos 2003; Macri and Sinha 2006). (2) Yet despite this
role, even in a rankings-oriented subject like economics there is no
consensus on which set of journal rankings is best or most appropriate
for a given setting. In order to provide some guidance, in this article,
we use academic labor market data to sift between different schemes for
ranking economics journals.
The basis of our approach is that for research-intensive
universities, hiring, promotion, and salary advancement are likely
directly related to perceived research impacts. When faculty debate
merits of publication in one journal or another during labor market
deliberations, they are choosing over actions with costly consequences.
In contrast, some other ways of ranking journals, such as citation-based
metrics, may be prone to strategic manipulation, as we discuss below. In
our specific example, we relate salaries of economists in the University
of California system to their lifetime publications in 700 different
academic journals. These data enable us to test amongst various sets of
competing journal rankings, looking in particular at how aggressively
each down-weights lower-ranked journals. We also examine how the
academic labor market discounts older articles, co-authored articles,
and the structure of research portfolios.
As far as we are aware, this is the first article to use labor
market data to examine journal rankings. Most studies in this area
instead consider variants of citation-based metrics. (3) The most common
citation-based metric is the Two Year Impact Factor: the total citations
by all journals in a database in a particular year to papers published
in the journal considered over the previous 2 years, divided by the
2-year total of papers in that journal. Since the path-breaking work by
Liebowitz and Palmer (1984), journal rankings in economics generally
weight citations by an assessment of the quality of the source journal
to provide adjusted impact factors. However, this growing reliance on
impact factors may encourage strategic behavior. For example, editors
may coerce authors to add citations to their journals in order to
inflate impact factors. Wilhite and Fong (2012) find 175 coercing
journals in their survey of researchers in economics, sociology,
psychology, and business. (4) Collusion between journals is also
possible, where authors of articles in one or more journals deliberately
cite papers from another journal. (5) Similarly, individual authors
could form cross-citation clubs to raise their personal citation counts,
which also inflate the impact factors of the journals they publish in.
Hence, what is required for ranking schemes is a robust measure of
journal quality which is not easily manipulated.
There is a substantial empirical literature on the academic labor
market for economists, but this research has not focused on uncovering
measures of journal quality. (6) Instead, academic earning equations are
used to consider the negative impact of seniority (Bratsberg, Ragan, and
Warren 2003; Moore, Newman, and Turnbull 1998; 2010; Ransom 1993), the
returns to co-authorship (Sauer 1998; Moore, Newman, and Turnbull 2001;
Hilmerand Hilmer 2005; Hilmer, Hilmer, and Ransom 2012), and the returns
to the quantity versus the quality of research, with citations typically
a proxy for quality and article counts a proxy for quantity (Hamermesh,
Johnson, and Weisbrod 1982; Hilmer, Hilmer, and Ransom 2012). (7)
Quantity and quality are revisited by Hamermesh and Pfann (2012), who
find that both matter to salary whereas citations (quality) are an
important determinant of reputation (using Nobel prizes, Clark medals,
Econometric Society fellowships and departmental reputation as proxies)
but the quantum of publications is not.
The closest study in the literature to this article is Ellison
(2013), who examines a particular academic labor market outcome--which
young economists (a PhD since 1988) gained tenure at top 25 U.S.
economics departments by 2006/2007. There is a hierarchy of departments,
so where each economist gains tenure is a proxy for the labor
market's assessment of quality. Tenure decisions are actions with
costly consequences, so should be highly informative. Ellison therefore
uses this information to assess the widely used h-index of Hirsch
(2005), which is the largest number h such that the author has written h
papers that have each been cited at least h times. Ellison compares the
original h-index with a more generalized form, finding the academic
labor market in economics to support the case for a version that
emphasizes a smaller number of more highly cited papers. Of relevance to
the current study is the explicit emphasis placed by Ellison on labor
market information:
... I propose that a useful criterion for assessing competing
indexes (and for assessing whether the indexes have any value at all) is
to examine whether they are consistent with labor market outcomes.
(Ellison 2013, 63)
Although Ellison (2013) does not examine journal rankings, the same
logic of seeking congruence with labor market data can be applied to
sifting between the various journal ranking schemes, using information
from salary decisions that have costly consequences. We also focus on
journal rankings rather than citations because the time lag from
publication to receiving citations is impractical for research
assessment exercises that often examine just the past 6 years (Tressler
and Anderson 2012). For this reason, journal quality measures remain
widely used as a proxy for the value of published research, not just in
tea room conversations but also when universities make forward-looking
decisions on hiring, tenure, promotion, and salary increases.
II. JOURNAL ASSESSMENT MEASURES
This section outlines key characteristics of some representative
journal assessment measures that have been used in the economics
literature. A wide range of these journal ranking schemes have been
proposed by economists, with no consensus on which is the best or which
should be used in particular situations. Indeed, according to Engemann
and Wall (2009, 128)
There is no such thing as the correct ranking of economics
journals. Instead, there is a universe of rankings,
each the result of a set of subjective decisions by
its constructor ...
These subjective decisions include which journals to cover, over
which years, and whether to rely on citations or expert opinion. If
rankings are citation-based further decisions are needed about whether
to adjust for the citing sources and for self-citations by authors and
journals. Approaches that rely on expert opinion are also subject to
potential bias, not only from nonresponse by those surveyed but also at
any earlier stage by choosing whose opinion is sought, over what
universe of journals. For example, perceptions of older and younger
academics may differ on the ranking for new journals, such as the
American Economic Journals.
Laband and Piette (1994) ("LP94") apply the pioneering
methodology of Liebowitz and Palmer (1984), where adjusted impact
factors are determined by using the sum of citations to each journal in
an iterative process. The adjusted impact factors are used to weight the
citation sources and to provide journal weights. This approach is
sometimes referred to as an Eigenfactor approach. The ISI Journal
Citation Reports provide the database of articles published in 1990 and
their citations to papers published from 1985 to 1989. Only 130
economics journals are given weights in the Laband and Piette scheme,
which is the least permissive of any scheme considered here.
Kodrzycki and Yu (2006) also apply the Liebowitz and Palmer
methodology, but adjust for the citing intensity of subdisciplines.
While also using the ISI Journal Citation Reports, they develop a list
of journals commonly used by economists, rather than relying primarily
on the ISI list of economics journals. Unlike Liebowitz and Palmer, and
Laband and Piette, they consider citations from all social science
journals, to provide one set of journal assessment measures
("K&Y_all"), and citations from those journals they
classified as economics journals for a second set of measures
("K&Y_econ"). (8) Citations are from 2003 to articles
published over the 8 years from 1996 to 2003. The rankings are provided
for 181 economics journals. A more recent application of the Liebowitz
and Palmer methodology by Kalaitzidakis, Mamuneas, and Stengos (2011)
("KMS") uses the average of the citations in each of the years
2003 to 2008 to articles published in the preceding 10 years. (9) The
ISI Journal Citation Reports database is used and only journals
classified as economics journals are considered, which gives nonzero
rankings to 209 journals.
While iterative (or Eigenfactor) approaches are widely used, some
popular journal rankings are based on the simpler direct count of
citations. For example, Coupe (2003) uses an average of 2-year impact
factors from 1994 to 2000 based on the ISI Journal Citation Reports for
273 economics and related journals. (10) Coupe uses this average, along
with three other journal weighting schemes, to provide rankings of
departments and individual economists. (11) The Research Papers in
Economics (RePEc) Website provides a number of different journal impact
assessments using the RePEc citation database. The assessment measure
used in this article is the basic impact factor based on direct
citations, which is corrected for self-cites to journals, but not
self-cites to an author's own papers. This is one of the most
permissive schemes, in that impact factors are provided for 984
journals. (12)
[FIGURE 1 OMITTED]
The other permissive ranking, in the sense of trying to not exclude
any economics journal, is from Combes and Linnemer (2010). These authors
use a number of approaches in order to cover all EconLit journals. They
start with 304 journals drawn from EconLit and the ISI Journal Citation
Reports, then combine several citation indices, adjusted for the field
of specialization, and use a Google Scholar h-index to regress their
index on Google Scholar citation data to extrapolate it to all EconLit
journals. Assessment measures are proposed based on assumptions about
the desired degree of down-weighting of lower-ranked journals, which
Combes and Linnemer call "convexity." They propose a
"medium" convexity measure, "CLm," and a
"high" convexity measure, "CLh," both of which are
available for the same set of 1,168 journals. This convexity can be
thought of as the elasticity of the weight given to a particular journal
with respect to its ranking, as we show below.
The difference between the "medium" and the
"high" convexity measures is illustrated in Figure 1, for the
top 102 journals in the Combes-Linnemer scheme. These comprise their
"AAA," "AA," and "A" groups, with the
bottom-ranked journal in the "A" group being Applied
Economics. The "medium" and "high" convexity indexes
are similar for the top four journals but then a large gap opens up,
with the fifth ranked journal (Review of Economic Studies) being
equivalent to either 0.81 of the top journal (Quarterly Journal of
Economics [QJE]) under medium convexity or else just 0.66 under high
convexity. The relative penalty for lower-ranked journals then grows,
with the 20th journal being either 0.54 or 0.29 of the QJE and the 50th
ranked being just 0.30 or 0.09 of the QJE. To provide a sense of the
type of journal at various ranks, Figure 1 highlights the position of
four general interest journals of varying quality and their CLm and CLh
indexes: the Economic Journal (64.5, 41.6), Economics Letters (30.4,
9.2), Economic Inquiry (24.2, 5.9), and the Southern Economic Journal
(19.0, 3.6). These are large gaps in the assessment of relative impact
and so it should be possible to detect which degree of convexity is most
congruent with labor market data.
The final approach to ranking journals we consider gives an
alternative to using citations. Instead, Mason, Steagall, and Fabritius
(1997) ("MSF") derive reputational weights from a survey of
chairs of economics department in the United States. All 965 departments
listed in the December 1989 American Economic Review (AER) were surveyed
in 1992 and 1993 with a response rate of 22.4% yielding 216 usable
replies. Respondents ranked journals on a 0 to 4 scale (allowing
noninteger scores). This ranking has a relatively low degree of
convexity, in the sense of not heavily penalizing lower-ranked journals.
(13) But these reputational weights are only available for 142 journals
so MSF is the second least permissive of the schemes that we consider.
It is clear from this discussion that journal assessment schemes
differ in three main ways: the ranking of journals, the degree of
coverage or nonzero weights, and the rate at which the weights decline
for lower-ranked journals. We illustrate these aspects by applying each
of the nine schemes considered to the career publications by the
University of California economists in our sample (described below).
Applying these schemes to the output of actual economists helps show the
important impact of coverage assumptions. In fact, the two least
permissive schemes (MSF and LP94) would exclude over one-third of the
academic articles published by economists working in economics
departments in the largest research-intensive public university system
in the United States (Table 1). It is unclear whether the academic labor
market also places zero value on publication in these excluded journals.
Even the most permissive schemes that attempt to cover the universe of
economics journals (RePEc and Combes-Linnemer) would miss over one-tenth
of the articles published by these economists. (14)
To show the rate at which the weights decline for lower-ranked
journals under each of the nine journal ranking schemes we estimate:
log(relative weight) = [alpha] + (Slog(rank). Relative weight of each
journal under each ranking scheme is calculated by dividing the
journal's actual weight by the weight given to the highest-ranked
journal in the scheme (typically the QJE). The regressions are estimated
on the nonzero weighted journals published in by the academics in our
sample and the coefficient of interest is the elasticity of the relative
weight with respect to the rank of the journal. The rank elasticities in
Table 1 range from -0.22, for the least aggressive scheme (MSF), to
-1.92 for the scheme (LP94) that reduces weights the fastest when moving
down the ranks. Although the CLh weights are proposed by Combes and
Linnemer as having high convexity (i.e., a rapid decline in the weight
given to lower-ranked journals, as shown in Figure 1), it in fact has
only a moderate elasticity of the weight with respect to rank. Four
other journal ranking schemes have a more elastic response of journal
weights to rank (K&Y_all, K&Y_econ, KMS, and LP94), and these
same four schemes also exclude at least one-quarter of the articles
published to date by the economists in the sample used below. Thus,
these four schemes can be considered to be especially aggressive in
their focus on perceived journal quality.
The final column of Table 1 reports the average lifetime weighted
journal output of the economists in our sample, in terms of AER-sized
pages (using 1/n for co-authored papers). This is the key variable which
will be used to explain academic salaries, and it varies from 144 pages
under the MSF journal rankings to just 36 pages with the LP94 weights.
This fourfold difference should be large enough to allow the salary data
to discriminate between the various schemes, which are ranked in Table 1
from least aggressive to most aggressive, in terms of the combined
impact of the assumptions about noncoverage and convexity. III.
III. DATA
Our approach of using labor market data to uncover the implied
quality of academic journals can be applied to any group of academics,
in any discipline, in any country. We decided to focus on economists
within economics departments in the nine campuses of the University of
California system for three reasons. First, the public disclosure
database of California state worker salaries
(http://www.sacbee.com/statepay/) is unusually detailed, as we describe
below. Second, this gives us a well-defined target sample frame that is
likely to be of inherent interest--the largest research-intensive public
university system in the United States. While the salary returns to
various journals may differ in private universities or in other public
systems, for the first study of journal rankings using salary data it
makes sense to start with the largest system (i.e., there should be good
external validity in studying the University of California). Third,
while all University of California campuses are research-intensive, they
span the perceived quality range from excellent (e.g., Berkeley), to
very good (e.g., Davis) to those that are less highly ranked (e.g.,
Riverside) and emerging (e.g., Merced). This range of quality allows us
to test if some journal ranking schemes do a better job of matching
labor market outcomes in the best departments. (15)
On the other hand, some features of compensation in the University
of California system may differ from other universities, potentially
limiting the generality of our results so we briefly discuss these here.
The University of California takes a system-wide approach to academic
personnel policy and practice. Thus in setting academic salaries,
individual campuses operate within the constraints of a
multi-institution system using a similar approach to appointments,
re-appointments, merit reviews, and promotion. This approach centers on
review procedures in which academics are judged by their peers, with
evaluations based on research and other factors. After department
discussions and voting, recommendations are made to deans or equivalent
and, with a few exceptions, final decisions are made by campus
chancellors. Thus, overall, these procedures are not significantly more
centralized than those that would apply in most public and private
institutions, with the key inputs being from the department chairperson
and departmental assessments of research contributions.
There are two particular aspects of salary setting in the
University of California system that should be noted. The first is the
process of merit reviews or regular consideration for advancement on the
salary scale within ranks, normally undertaken every 2 years. While
merit reviews follow the same general process as appointments,
promotions, and tenure decisions, the regular nature of these reviews
may mean that quality evaluations are less rigorous with salary
increases given for publishing in any of a wide range of outlets. Hence,
salary variations resulting from merit increases may provide a less
reliable signal of research quality than they would in universities
without this type of pro-forma review. But potentially offsetting this
factor, reviews undertaken by the University of California have
suggested that remuneration has lagged behind benchmark institutions.
(16) Thus it is likely that external factors, such as competing offers
from other institutions and the need to be competitive in making new
appointments, may have greater influence on salary decisions than
signals from the merit review process.
In order for labor market data to provide a valid signal of
perceived journal quality, the sample has to be relatively homogeneous
in terms of the weight placed on research performance in salary
determination. So we excluded anyone with significant nondepartment
administrative responsibilities (e.g., Deans) and those with primarily
teaching roles (including affiliated faculty, adjuncts and those
obviously on part-time contracts). While economists infiltrate many
other departments we did not consider those for our sample since the
returns to publishing in particular economics journals appear to differ
between economics-related departments within the same universities
(Hilmer, Hilmer, and Lusk 2012). The heterogeneity introduced by
expanding the sample would increase the risk of omitting from the salary
equation factors that are correlated with some of the journal rankings.
Instead, a homogeneous sample and a well-specified model are likely to
provide a better test-bed for comparing the various journal rankings.
Moreover, while our sample of 223 is smaller than in many other studies
using academic earnings equations, it has the advantage of being for a
well-defined population of interest rather than simply a hodge-podge of
universities with publicly available salary data.
The salary data are unusually detailed, with the base salary
reported for 2007, 2009, and 2010 and the total salary reported for
those 3 years and also for 2008. Total salary is more temporally
volatile than base salary, with squared deviations of annual salary
around the multiyear mean for an individual being 32% higher, on
average, when using total salary rather than base salary. (17) When we
calculate the ratio of total salary to base salary for each individual,
it ranged from 0.8 to 2.7, suggesting that the total salary received in
any year may not be a good guide to the long-run "permanent"
salary. Moreover, while the total-to-base ratio averages 1.10 across all
3 years, it fell from 1.14 in 2007 to 1.07 in 2010, presumably because
the worsening financial position of the State of California meant that
cuts were being made in extraordinary salary payments. For these reasons
we use the base salary rather than the total salary.
Another helpful feature of the salary data provided by the
Sacramento Bee Website is that details are provided on the nature of the
employment contract, in terms of the pay period. Almost all academics in
economics departments at the University of California are on academic
year rather than financial year contracts (in contrast to, say, those in
agricultural economics departments). In a few cases, especially at UC
Berkeley, some economists are on law school scales, so we include a
dummy variable in our regressions for individuals whose reported salary
is not for a standard scale and 9-month academic year. (18) In contrast,
some previous studies of faculty salaries have had to drop individuals
for whom it was unclear if their reported salaries were on a 9-month
academic year basis (Hamermesh and Pfann 2012).
In addition to salaries we gathered data on gender, the year of
first appointment and of hire at the current department, the year of
obtaining a PhD (and the department and university), and whether the
current appointment was named (such as an endowed or distinguished
chair). These details were available from departmental web pages and
online curriculum vitae (CVs) for most academics and otherwise we
obtained them from dissertation databases and from changes in
affiliations on journal articles to date movements. The online CVs also
provided the initial information on publications, which were
supplemented with searches of EconLit, RePEc, and the Web of Science.
Measuring the research outputs of academics with common names can be
difficult, but with so many of the sample having their CVs online it
helped cross-validate the database search results. We restrict attention
to articles that were actually published (with pagination) by the end of
2010. Since our focus is on journal articles, we did not include book
reviews, book chapters, editorial introductions, or conference
proceedings. (19) The one exception is AER Papers and Proceedings (even
though many CVs listed this in the "nonrefereed" section)
because seven of
the journal assessment schemes make no distinction between the May
issue of the AER and other issues, while LP94 weight the May issue at
one-quarter of ordinary issues. Only KMS give Papers and Proceedings a
weight of zero. In total, our procedures recorded 5,721 articles in 700
different journals that the 223 economists in our sample had published
in over their careers.
Table A1 presents definitions and summary statistics for the
variables used in the academic earnings equations. The dependent
variable is (log) base salary in 2010, with a mean for the underlying
salary data of $156,700 and a range from $78,000 to $310,000 (the
maximum total salary is $458,900). (20) The average economist in the
sample had spent 12 years at the current university and 18 years at all
appointments. One-sixth of the sample is female. Three indicator
variables for atypical salary levels (and potentially influential data
points) are included: whether the academic is on a nonstandard contract,
whether they have a named position (which may fund additional salary),
and whether they are a Nobel Prize winner (only one individual).
Finally, three indicators of PhD quality are also included: the rank of
the PhD-granting department in either the 1995 National Research Council
rankings or the Amir and Knauff (2008) rankings, and an indicator for
those economists whose PhD was not from an economics department.
IV. RESULTS
The first step in our analysis is to obtain well-specified academic
earnings equations to then use as the testing ground for comparing each
journal ranking scheme. We then compare nine salary equations, where
each uses a different journal ranking scheme to aggregate lifetime
publications into a scalar measure for each academic. We use model
comparisons statistics and formal non-nested tests to establish which of
these models is closer to the truth, in order to see which of the
journal ranking schemes is most congruent with the salary data. We then
consider the robustness of the results to assumptions about discounting
for the age of articles and for co-authorship.
In Table 2, we report the results of various specifications which
suggest the following:
indicators of quality for the PhD-granting department are not
relevant to salary for this sample (columns 1 to 3); all three of the
indicators that we use for atypical salary levels (and potentially
influential data points) are statistically significant (columns 4 to 6);
the effects of seniority and experience on salary are best modeled as
quadratics; there is weak evidence of a premium for males; and, location
fixed effects are highly significant. These location effects not only
capture cost of living differences but also departmental (and campus)
reputational effects, differences in average teaching loads and in the
quality of PhD students, and other amenities. (21) Based on these
observations, the equation in column 7 of Table 2 is used as the base
specification, to which we will then add an output variable measuring
lifetime publications in journals, as weighted under each of the nine
assessment schemes. Even without the output variable, the base
specification explains 72% of the variation in log salary, which is
higher than the predictive power of academic earnings equations in other
studies. Hence we expect that this sample and regression specification
will provide a good test bed for comparing the various journal ranking
schemes.
To create the output variable, the number of pages of each of the
5,721 journal articles published by our sample members are multiplied by
the assessment weight of the journal (which varies between the ranking
schemes). We also adjust for the number of authors of each article
(using the "1/n rule") and standardize pages to the size of a
typical page in the AER. (22) Thus for each article published by each
individual academic the measured output is:
Article Pages X Size Correction
x (1 /number of authors)
X Journal Assessment Weight
and to calculate the lifetime output measure we sum over articles
published from the year of the first article until the end of 2010. The
full results for the nine different earnings equations, where each in
turn uses a different set of journal assessment weights to summarize
lifetime output, are reported in Table SI. For these models, the
[R.sup.2] ranges from 0.76 to 0.78, so the incremental [R.sup.2] from
including the lifetime output measure is 0.04 to 0.06. However, since
lifetime output is correlated with experience and seniority, another way
to measure the explanatory power of this variable would be to include it
first. If we run a simple regression of log salary on each of the
lifetime output variables, the [R.sup.2] values would range from 0.46
(using MSF or Coupe weights) to 0.52 (using CLm or CLh weights).
The coefficients on the output measures and a series of model
comparison statistics are reported in Table 3. The nine different
earnings equations are non-nested, in the sense that it is not possible
to impose a set of linear restrictions to derive one model from the
other. A standard procedure for model comparison in this case is to use
information criteria, with the Akaike's Information Criteria (AIC)
and Schwarz's Bayesian Information Criteria (BIC) typically used.
We can also compare [R.sup.2] since all the equations have the same
number of explanatory variables. The maximized [R.sup.2] and
log-likelihood, and the minimized loss of information, is for the
earnings equation that uses CLm--the medium convexity weights of Combes
and Linnemer (2010). Even though the MSF weights are the second least
permissive, in excluding one-third of the articles published to date by
this sample, they provide the second best congruence with the labor
market data. The greatest loss of information comes from using the KMS
weights, which are the third least permissive and have the second
largest elasticity of journal weights with respect to the journal
ranking (see Table 1).
Our focus is on which journal ranking scheme best fits the salary
data, but it is also worth interpreting the magnitudes of some of the
regression coefficients. The best fitting model uses CLm weights to
aggregate over all journal articles for each academic (effectively
calculating the number of standardized pages of (T/E-equivalent
quality). For example, Economic Inquiry has a CLm weight of 0.242 and a
page size correction factor of 0.9, so a 20-page sole-authored article
in Economic Inquiry converts into 4.4 standardized pages of
(L/C-equivalent quality (20 x 0.9 x 0.242). The semi-elasticity in the
first column of Table 3 indicates that such an article would be
associated with an annual salary that was 0.6% higher, which is an
increase of $940 at the mean. (23) With a 40-year career and a 5%
discount rate, for the average economist in the sample (who is 18 years
into their career) such an article would have a net present value of $
12,400.
Amongst the other variables previously identified from the
specification search in Table 2, all are statistically significant at
conventional levels except for Male (which has /-statistics between 1.0
and 1.5). The location fixed effects are smaller than in Table 2,
suggesting that some of the apparent salary premium at UC Berkeley was
productivity-related, but all remain statistically significant. (24) The
quadratics suggest that for the average economist, salary is maximized
after 30 years of labor market experience and minimized after 27 years
of seniority at the current university.
A. Formal Non-Nested Tests
Non-nested tests can help formally discriminate between the
competing models in Table 3. These test the validity of one linear
model, [H.sub.0] as opposed to its non-nested alternative [H.sub.1]:
[H.sub.0] : y = x[beta] + [[epsilon].sub.0]
[H.sub.1] : y = Z[gamma] + [[epsilon].sub.1]
where X and Z are matrices of explanatory variables, and neither is
a linear combination of the other, [beta] and [gamma] are corresponding
parameter vectors, and [[epsilon].sub.0] and [[epsilon].sub.1] are
random errors. Forming a "compound" model with each competing
measure of lifetime output included at the same time is not advisable
because of possible multicollinearity. Moreover, this artificial nesting
approach does not distinguish between [H.sub.0] and [H.sub.1]; instead,
it distinguishes between each competing model and a hybrid (Greene
2012). This can be seen by writing the compound model as:
y = [bar.X][bar.[beta]] + [bar.Z][bar.[gamma]] + W[delta] +
[epsilon]
where [bar.X] holds the set of variables in X not in Z, [bar.Z]
holds the set of variables in Z not in X, and W has the variables common
to the two models. While the test of [bar.[gamma]] = 0 might seem to
reject [H.sub.1] and [bar.[beta]] = 0 might reject [H.sub.0], since
[delta] remains a mixture of parts of [beta] and [gamma] it is not
established by the F-test on the compound model that either of these
parts is zero (Greene 2012).
Instead, we use Vuong's (1989) likelihood ratio test that does
not presume that either competing model is "true," and instead
determines which competitor has verisimilitude (i.e., is closer to the
truth). This approach relies on the Kullback-Leibler Information
Criterion (KLIC), which, intuitively, is the log-likelihood function
under the hypothesis of the true model minus the log-likelihood function
for the (potentially mis-specified) model under the assumption of the
true model. One model is "better" than another if it is closer
to the "truth" under the KLIC (Greene 2012, 535). Vuong's
test is directional, with large positive values favoring the null model
while large negative values favor the alternative (and values between
-1.96 and +1.96 are inconclusive, for 95% significance). We corroborate
results for a subset of the bilateral comparisons using Pesaran's
(1974) version of a Cox likelihood ratio test, where the null model is
rejected against the alternative if there are large negative values of
the test statistic. The test is then reversed to see if the alternative
is rejected against the null.
The pairwise comparisons of each model, using Vuong's test to
see which is closer to the truth, are reported in Table 4. For ease of
interpretation, the models are ordered with those using the most
inclusive ranking schemes listed first. The format of the table is that
each cell contains a bilateral z-statistic test result, with significant
positive values counting as evidence in favor of the model in the column
against the model in the row and negative values counting as evidence
for the row model against the column model. The model that uses CLm, the
medium convexity weights of Combes and Linnemer, is favored against all
of the competing models except for the one using MSF weights, for which
the comparison is inconclusive (z = -1.40). The comparison of models
that use the CLm and CLh weights to calculate lifetime output yields a
significant rejection of the high convexity weights (z = 2.27), which is
notable since there is no difference in the set of journals or the
rankings and just a difference in the weights. The only other
significant results in the table are that the model using KMS weights is
rejected against the model using CLh weights, and also weakly rejected
against the model using MSF weights (at p = 0.052). (25)
The results in Table 4 suggest that more inclusive journal
weighting schemes are most congruent with salaries of University of
California economists. To see how robust this finding is, Cox-Pesaran
tests were carried out to compare the models using CLm and CLh weights,
and to compare the models using MSF, KMS, and LP94 weights, since these
capture the extremes in terms of least and most aggressive
down-weighting for lower-ranked journals. The model using CLh weights is
rejected against the one using CLm weights (p = 0.00) while there is no
reverse rejection (p = 0.30). Similarly, the model using LP94 weights is
rejected against the model using CLm weights (p = 0.00) but not the
reverse (p = 0.26). When the least aggressive MSF weights are used, the
model using KMS weights (the second most aggressive) is rejected against
it (p = 0.00) while there is no reverse rejection (p = 0.23), and the
models with MSF weights and LP94 weights reject against each other. Thus
the congruence of the less-aggressive journal weighting schemes with the
salary data appears to be a robust finding that does not depend on using
just one type of non-nested test.
B. Should Older Articles Be Discounted?
The results reported thus far treat an article published in, say,
1978 the same as one from 2008; adjustments are made for length,
coauthors, page sizes, and journal quality, but not for vintage. To test
if this assumption of no age discounting is appropriate, we calculated
for each article published by each academic in year t:
Article Pages x (1 /[(Age).sup.[delta]])
X SizeCorrection
X (1/number of authors)
x Journal Assessment Weight
where Age = 2011-1, and the age discount factor, 8 varied from 0 to
2, in increments of 0.1. In other words, we allowed for no age
discounting ([delta] = 0), for inverse age discounting ([delta]=1) where
a 20-year-old article has 1/20th the impact on current salary of a
1-year-old article, and for a variety of more extreme and intermediate
cases. The best-fitting model, using CLm weights, was estimated for each
of these 21 values of [delta] and the maximized log-likelihoods are
compared in Figure 2. There appears to be weak age discounting, with the
log-likelihood maximized at [delta] = 0.4, which is four-points above
the value at [delta] = 0. The maximized likelihood declines steeply at
higher discount rates. (26)
[FIGURE 2 OMITTED]
To check if our finding of greater congruence between salary and
the less aggressive journal weighting schemes is robust to different
assumptions about the age discounting of articles, the academic earnings
equations were re-estimated. Since there is no rule-of-thumb for
(1/[(Age).sup.0.4]) we discounted according to the inverse of the square
root of age, (1/[(Age).sup.0.5]) noting that there was only a half-point
difference in the maximized log-likelihoods at 8 = 0.4 and 0.5. The full
set of estimation results are reported in Table S2 and the results of
the Vuong non-nested tests are in Table 5.
The salary data continue to favor the less aggressive journal
weighting schemes when age-discounting of articles is allowed for. The
model using the least aggressive MSF weights previously weakly rejected
against two models but it now rejects against four (KMS, Coupe, and both
Kodrzycki and Yu schemes). Similarly the model with CLm weights now
rejects against six models (and against the model using Coupe weights at
p = 0.118). Moreover, the models using six of the assessment schemes now
reject against the model using the KMS weights (which are the second
most aggressive), whereas previously only three models rejected against
this scheme. The final change caused by allowing age discounting is that
the model using the simple RePEc impact factors now rejects against
three others (both Kodrzycki and Yu schemes and KMS) whereas previously
that model rejected against no others.
C. Do Results Depend on Co-Authored Articles Being Fully Pro-Rated?
The results reported thus far rely on applying the "1/n
rule" to fully pro-rate co-authored articles. Liebowitz (2013)
argues that full proration is required for researchers to choose
efficient sized teams, and early empirical results from regressing
salaries on co-authored articles were consistent with this (Sauer 1998).
(27) But more recent salary studies reject proration and support the
hypothesis that each author of a co-authored paper receives the same
value as for a sole-authored paper (Hilmer, Hilmer, and Ransom 2012).
Similarly, in a small survey of department chairs, Liebowitz (2013)
reports that each author of a paper with two authors receives an average
of 89% of the value of a sole-authored paper. In a similar survey 30
years earlier the average was 70%, so the trend within economics appears
to be away from full proration.
To see what the data indicate about proration, and to check
robustness of our key finding that salary data are most supportive of
the more inclusive journal weighting schemes, we calculated for each
article published by each academic in year t:
Article Pages x (1/[(Age).sup.[delta]])
X Size Correction
x (1/number of authors).sup.[alpha]]
X Journal Assessment Weight
where the co-authorship discount factor, a varied from 0 (no
proration) to 1 (full proration), and age discount factors of 0 and
[delta] = 0.5 are used. The best-fitting model, using CLm weights, was
re-estimated with a incrementing by 0.05 each time, with Figure 3
displaying the maximized log-likelihoods. (28) If age discounting is
ignored, the salary data are consistent with [alpha] = 0.25 (although
the log likelihood is fairly flat over 0 [less than or equal to] [alpha]
[less than or equal to] 0.4) and significantly reject full proration.
With [alpha] = 0.25 a three-authored paper gives each author
three-quarters of the value of a sole-authored paper, while the
"1/n rule" would give each author only one-third of the value.
However, the significant negative correlation between the age of
articles and the number of coauthors (r = -0.23) makes results on
co-author discounting sensitive to how age discounting is treated.
Specifically, if articles are discounted by the square root of their age
([delta] = 0.5) the salary data are more consistent with full proration,
with the log-likelihood maximizing at [alpha] = 0.7 and no significant
difference from [alpha] = 1. A comparison of Figures 2 and 3 suggests
that age discounting makes a bigger difference to the fit of an academic
salary model than co-authorship discounting; also, using age-discounted
articles raises log-likelihood by an average of 1.4 points.
Although the results do not provide clear evidence for or against
use of the "1/n rule," in terms of the main finding of this
article this ambiguity does not matter. Irrespective of whether we use
age discounted articles and [alpha] = 0.7 for the degree of proration,
or no age discount and [alpha] = 0.25, the results of the non-nested
testing are very similar to what is reported in Tables 4 and 5. (29) The
salary model with CLm weights is favored over all competing models (at p
< 0.05), except for the model using MSF weights where the comparison
is inconclusive (z = -1.13), and the model using KMS weights is also
rejected against models using CLh and MSF weights. This is the same
pattern as shown in Table 4, so we view our finding that the labor
market data are most consistent with the more inclusive journal
weighting schemes that do not aggressively down-weight lower-ranked
journals as a robust one.
[FIGURE 3 OMITTED]
V. EXTENSIONS
Our main research question of what the academic labor market says
about various journal ranking schemes has been answered. Using salaries
from economics departments in the University of California as our labor
market indicator, we find support for those journal ranking schemes that
are more inclusive and which do not aggressively penalize publication in
lower-ranked journals. However, our salary regressions can also answer
several related questions, which we consider in this section.
A. Are Results Different for the Best Departments?
Combes and Linnemer (2010) suggest their high convexity index is
useful to compare the best departments and their medium convexity index
is suitable for middle-ranked departments. We therefore examine whether
the finding that salaries are best explained by journal ranking schemes
that have the lowest rate of decline in the journal weights when moving
down the ranks is also found if we restrict attention to the top four
economics departments in the University of California system: Berkeley,
San Diego, Davis, and Los Angeles. In keeping with the results on age
discounting, the published output to date of each academic is calculated
using the inverse of the square root of the age of each article. The
full estimation results are reported in Table S3 and the results of the
Vuong non-nested tests for the subsample of academics in the top four
departments are in Table 6.
There is no evidence that salaries of academics in the top four
departments are more congruent with the journal ranking schemes that
more heavily penalize the lower-ranked journals. In fact, the Vuong
tests suggest that the salary regression that uses the MSF rankings,
which have the smallest elasticity of the journal weight to the journal
rank, now rejects against six of the other regression models (and is
inconclusive against the models with CLm and LP94 weights). Moreover,
the Cox-Pesaran test shows the model using the LP94 weights, which most
heavily penalize lower-ranked journals, is rejected against the model
using MSF weights (p = .00) but the reverse rejection is only weakly
significant (p = .07). In terms of the specific claim of Combes and
Linnemer that their high convexity weights are most suitable for
comparing the best departments, the salary evidence suggests the
opposite. Under both the Vuong test (z = 3.00) and the Cox-Pesaran test
(p = .00) the model using CLh weights is significantly rejected against
the model using CLm weights. In other words, the pattern of salaries in
the top four economics departments in the UC system are no more
supportive of journal ranking schemes that heavily penalize the
lower-ranked journals than is the pattern of economist salaries in the
UC system as a whole. A similar result could be found by interacting a
dummy variable for being in the top four departments with the journal
quality measures, and regardless of whether age discounted or not, none
of these interaction variables are statistically significant. (30)
B. What Weight Should Be Placed on AER Papers and Proceedings?
The journal assessment schemes that we use differ in their
treatment of the Papers and Proceedings May issue of the AER. While
seven of the schemes do not discriminate, KMS places zero weight on
articles in the May issue and LP94 gives them about one-quarter of the
weight of ordinary issues. The Papers and Proceedings issue is a common
outlet for the University of California economists in our sample, with
160 articles published there (and 242 in the other issues of the AER).
We therefore use our academic earnings equations to see what the data
indicate about the appropriate weight to place on Papers and Proceedings
compared with ordinary issues of the AER. Our best-fitting model, using
CLm weights with articles discounted according to the inverse of the
square-root of their age, was re-estimated 101 times, incrementally
decreasing the weight for articles in the Papers and Proceedings issue
from 100% of an ordinary AER down to zero. The maximized log-likelihoods
from this search procedure are illustrated in Figure 4.
The salary data suggest that treating one page in the Papers and
Proceedings issue as equivalent to 58% of a page in the other issues of
the AER is most appropriate. Thus journal assessment schemes that do not
discriminate between the May issue and other issues (likely because they
rely on ISI data that do not distinguish journal issue numbers when
counting citations) would seem to overstate the impact of Papers and
Proceedings. On the other hand, the two assessment schemes that do
discriminate may have down-weighted Papers and Proceedings too heavily.
Nevertheless we caution that the difference in the maximized
log-likelihoods is very small, across any of the values of the relative
weight term for Papers and Proceedings. These small differences suggest
that even for journals in which our sample have published a large number
of articles, deriving journal-specific impact factors from the salary
data--what might be dubbed "Market Impact Factors"--may be
difficult, despite our statistical power to discriminate between
different schemes for weighting the entire spectrum of journals.
C. Does the Portfolio Matter?
Our focus thus far has been on the total published output (weighted
by journal quality, age of the article and number of co-authors) without
considering the portfolio of articles. This assumes that it is possible
to substitute more articles in lower-ranked journals for fewer,
higher-ranked articles in order to achieve the same salary. But instead
it may be that failure to publish at least one article in the top
journals influences tenure, hiring, or promotion decisions at good
departments (Card and DellaVigna 2013). In this case, failure to publish
in such journals would be expected to impose a penalty even if overall
research output would suggest the same salary. To consider this issue we
study the influence of not having published in the top three journals;
the AER (not including the Papers and Proceedings issue), JPE, and QJE,
and also in the top five by adding Econometrica and the Review of
Economic Studies. (31)
As shown in Table 7 a significant faction of University of
California economists do not (yet) have publications in these journals;
44.4% of all faculty and 28.8% of full professors do not have at least
one article in the top three journals. Setting the bar as ever
publishing in the top five journals, there are still 32.7% of all
faculty and 17.3% of full professors who do not meet this standard. (32)
To estimate the effects on salary we re-estimated the salary regressions
reported in Table S2, adding a dummy variable for the economists with no
articles in the top three journals or for those with no articles in the
top five. As shown in the table there is no statistically significant
penalty for not having published in any of the top three journals
conditional on overall research output. (33) But when we consider the
economists not having any articles in any of the top five journals,
there does appear to be a salary penalty, of approximately 5%-7%.
To consider this result further we look at the impact of not having
an article in each of the top five journals separately. Conditional on
overall research output, not having an Econometrica article in a
portfolio imposes a penalty of 11% -13% across the nine salary models
considered. (34) In contrast, none of the top three journals show any
statistically significant salary penalty from being absent from a
portfolio while the Review of Economic Studies shows a small penalty,
which is statistically significant in just one of the nine salary
models. We interpret this as a premium for demonstrating the technical
competence and theoretical rigor that is typically associated with
Econometrica articles. In contrast, AER, JPE, and QJE are closer
substitutes for each other, and a dominant trend for all three of these
journals over recent decades has been the rising share of empirical
articles (Hamermesh 2013). Consequently, an article in any of the top
three journals may not stand out in influencing the perception of a
research record in the way that an Econometrica article stands out. Once
this perception is gained, there is no further gain from additional
Econometrica articles that is over and above the payoff to published
research which is already captured by the number of quality-, age- and
co-author-adjusted journal pages.
VI. CONCLUSIONS
In this article, we have compared nine different sets of economics
journal assessment measures to find which is most consistent with labor
market outcomes. These journal assessment measures differ according to
the journals covered, the ranking, and the rate at which the weights
decline for lower-ranked journals. The most aggressive schemes, in terms
of either ignoring or down-weighting lower-ranked journals, exclude more
than one-quarter of the lifetime output of our sample of University of
California economists and imply a very substantial penalty for
publishing in lower-ranked journals. For example, an article in a
journal like Economic Inquiry that is just outside the top 50 journals
is equivalent to less than 10% of a similarly sized article in the
American Economic Review or the Quarterly Journal of Economics.
The clear picture that emerges from the empirical results is that
the labor market does not reward publication in ways consistent with the
weights implied by the most aggressive journal assessment measures.
Instead, it is more inclusive schemes, where the weights for each
journal do not decline sharply with the rank of the journal, such as
from Mason, Steagall, and Fabritius (1997) and the CLm index of Combes
and Linnemer (2010), that have the greatest congruence with academic
salaries. This finding is robust to different assumptions about the age
discounting and co-authorship discounting of articles and also holds if
we restrict attention just to the best departments. Indeed, this last
result, that a model using high convexity weights is rejected against a
less convex alternative if tested on the top four University of
California economics departments is contrary to the claim of Combes and
Linnemer (2010) that a high convexity index is more suited for comparing
the best departments.
We view this congruence with labor market information as an
important criterion for the reasonableness of a journal ranking scheme.
While journal ranking schemes that are very aggressive in down-weighting
lower-ranked journals can be derived from citation data, such data may
be manipulated by editors who coerce authors to add superfluous
citations and by authors and editors who collude in cross-citation clubs
to raise the citation counts for particular journals or particular
individuals. In contrast, labor market decisions have costly
consequences, so the evidence for more inclusive journal ranking schemes
that is revealed by labor market outcomes is a pattern that comes from
information that should be less prone to strategic manipulation.
ABBREVIATIONS
AER: American Economic Review
AIC: Akaike's Information Criteria
BIC: Bayesian Information Criteria
CVs: Curriculum Vitae
KLIC: Kullback-Leibler Information Criterion
QJE: Quarterly Journal of Economics
RePEc: Research Papers in Economics
doi: 10.1111/ecin.12107
Online Early publication May 29, 2014
APPENDIX
TABLE A1
Variable Definitions, Means (M), and Standard Deviations (SD)
Variable M SD Description
Salary 156.66 57.07 Base salary in 2010 ($000)
Log (annual salary) 11.90 0.35 Logarithm of 2010 base salary
Experience (years) 18.00 12.43 Years since first appointment
(or receipt of PhD if
earlier)
Seniority (years) 12.15 9.97 Years of employment at
current university
Male 0.83 0.37 Person is male (=1) or
female (=0)
Holder of a named 0.19 0.40 Person holds an endowed or
chair named position or a
distinguished chair
Not standard pay 0.03 0.16 Person is not on a standard,
scale 9-month, academic year
pay scale
Nobel Prize winner 0.00 0.07 Winner of the Nobel Prize
PhD field not 0.08 0.27 Person holds a PhD granted
economics from a department that is
not economics
PhD rank (Amir and 31.61 34.55 Score for PhD-granting
Knauff 2008) department (100 = best)
using the placement-based
ranking of Amir and Knauff
(2008)
PhD rank 3.78 1.63 Score for PhD-granting
(Goldberger. department (5 = best) in
Maher, and the 1995 National Research
Flattau 1995) Council rankings
Note: N = 223.
REFERENCES
Abelson, P. "The Ranking of Economics Journals by the Economic
Society of Australia." Economic Papers, 28(2), 2009, 176-80.
Amir, R., and M. Knauff. "Ranking Economics Departments
Worldwide on the Basis of PhD Placement." Review of Economics and
Statistics, 90(1), 2008, 185-90.
Bratsberg, B., J. Ragan, and J. Warren. "Negative Returns to
Seniority: New Evidence in Academic Markets." Industrial and Labor
Relations Review, 56(2), 2003, 306-23.
--. "Does Raiding Explain the Negative Return to
Seniority." Economic Inquiry, 48(3), 2010, 704-21.
Card, D., and S. DellaVigna. "Nine Facts about Top Journals in
Economics." Journal of Economic Literature, 51(1), 2013, 144-61.
Chang, C.-L., M. McAleer, and L. Oxley. "What Makes a Great
Journal Great in Economics? The Singer Not the Song." Journal of
Economic Surveys, 25(2), 2011, 326-61.
Combes, P.-P., and L. Linnemer. "Where Are the Economists Who
Publish? Publication Concentration and Rankings in Europe Based on
Cumulative Publications." Journal of the European Economic
Association, 1(6), 2003, 1250-308.
--. "Inferring Missing Citations: A Quantitative Multi
Criteria Ranking of All Journals in Economics." Groupement de
Recherche en Economic Quantitative d'Aix Marseille (GREQAM),
Document de Travail, no 2010-28, 2010.
Coupe. T. "Revealed Performances: Worldwide Rankings of
Economists and Economics Departments, 1990-2000." Journal of the
European Economic Association, 1(6), 2003, 1309-45.
--. "What Do We Know about Ourselves? On the Economics of
Economics." Kyklos, 57(2), 2004, 197-216.
Ellison, G. "How Does the Market Use Citation Data? The Hirsch
Index in Economics." American Economic Journal: Applied Economics,
5(3), 2013, 63-90.
Engemann, K., and H. Wall. "A Journal Ranking for the
Ambitious Economist." Federal Reserve Bank of St. Louis Review,
91(3), 2009, 127-39.
Gibson, J. "Research Productivity in New Zealand University
Economics Departments: Comment and Update." New Zealand Economic
Papers, 34(1), 2000, 73-88.
Goldberger, M. L., B. A. Maher, and P. E. Flattau, eds.
Research-Doctorate Programs in the United States: Continuity and Change.
National Research Council. Washington, DC: National Academy Press, 1995.
Greene, W. Econometric Analysis. 7th ed. New York: Prentice Hall,
2012.
Hamermesh, D. "Six Decades of Top Economics Publishing: Who
and How?" Journal of Economic Literature, 51(1), 2013, 162-72.
Hamermesh, D., and G. Pfann. "Reputation and Earnings: The
Roles of Quality and Quantity in Academe." Economic Inquiry, 50(1),
2012, 1-16.
Hamermesh, D., G. Johnson, and B. Weisbrod. "Scholarship,
Citations and Salaries: Economic Rewards in Economics." Southern
Economic Journal, 49(2), 1982, 472-81.
Hilmer, C., and M. Hilmer. "How Do Journal Quality,
Co-authorship, and Author Order Affect Agricultural Economists'
Salaries?" American Journal of Agricultural Economics, 87(2), 2005,
509-23.
Hilmer, C., M. Hilmer, and J. Lusk. "A Comparison of Salary
Structures between Economics and Agricultural Economics
Departments." Applied Economic Perspectives and Policy, 34(3),
2012a, 489-514.
Hilmer, C., M. Hilmer, and M. Ransom. "Fame and the Fortune of
Academic Economists: How the Market Rewards Influential Research in
Economics." Discussion Paper No. 6960, IZA, Bonn, 2012b.
Hirsch, J. "An Index to Quantify an Individual's
Scientific Research Output." Proceedings of the National Academy of
Sciences, 102(46), 2005, 16569-72.
Kalaitzidakis, P., T. Mamuneas, and T. Stengos. "Rankings of
Academic Journals and Institutions in Economics." Journal of the
European Economic Association, 1(6), 2003, 1346-66.
--. "An Updated Ranking of Academic Journals in
Economics." Working Paper 9/2010, Economics Department, University
of Guelph, Guelph, Canada, 2010.
--. "An Updated Ranking of Academic Journals in
Economics." Canadian Journal of Economics, 44(4), 2011, 1525-38.
Kodrzycki, Y., and P. Yu. "New Approaches to Ranking Economics
Journals." B.E. Journal of Economic Analysis and Policy:
Contributions to Economic Analysis and Policy, 5(1), 2006, Article 24.
Laband, D., and M. Piette. "The Relative Impact of Economics
Journals." Journal of Economic Literature, 32(2), 1994, 640-66.
Liebowitz, S. J. "Willful Blindness: The Inefficient Reward
Structure in Academic Research." Economic Inquiry, 2013. DOI:
10.1111/ecin. 12039.
Liebowitz, S. J., and J. P. Palmer. "Assessing the Relative
Impact of Economics Journals." Journal of Economic Literature,
22(1), 1984, 77-88.
Macri, J., and D. Sinha. "Rankings Methodology for
International Comparisons of Institutions and Individuals: An
Application to Economics in Australia and New Zealand." Journal of
Economic Surveys, 20(1), 2006, 111-56.
Mason, P., J. Steagall, and M. Fabritius. "Economics Journal
Rankings by Type of School: Perceptions versus Citations."
Quarterly Journal of Business and Economics, 36(1), 1997, 69-79.
Moore. W., R. Newman, and G. Turnbull. "Do Faculty Salaries
Decline with Seniority?" Journal of Labor Economics, 16(2), 1998,
352-66.
--. "Reputational Capital and Academic Pay." Economic
Inquiry, 39(4), 2001, 663-71.
--. "Academic Pay in the United Kingdom and the United States:
The Differential Returns to Productivity and the Lifetime Earnings
Gap." Southern Economic Journal, 73(3), 2007, 717-32.
OECD. "Performance-based Funding for Public Research in
Tertiary Education Institutions," Workshop Proceedings, OECD
Publishing, 2010. Accessed July 6. 2012.
http://dx.doi.org/10.1787/9789264094611-en
Oswald, A. "An Examination of the Reliability of Prestigious
Scholarly Journals: Evidence and Implications for Decision-Makers."
Economica, 74(296), 2007, 21-31.
Pesaran, M. "On the General Problem of Model Selection."
Review of Economic Studies, 41(2), 1974, 153-71.
Ransom, M. "Seniority and Monopsony in the Academic Labor
Market." American Economic Review, 83(1), 1993, 221-33.
Sauer. R. "Estimates of the Returns to Quality and
Coauthorship in Economic Academia." Journal of Political Economy,
96(4), 1998, 856-66.
Tressler, J., and D. Anderson. "The Merits of Using Citations
to Measure Research Output in Economics Departments: The New Zealand
Case." Agenda, 19(1), 2012, 17-37.
University of California. "Facts About the University of
California, Total Compensation for UC Employees," October 2009.
Accessed June 6, 2013. http://
compensation.universityofcalifornia.edu_total_comp_ facts_nov2009.pdf.
Vuong, Q. "Likelihood Ratio Tests for Model Selection and
Non-nested Hypotheses." Econometrica, 57(2), 1989, 307-33.
Wilhite, A. W., and E. A. Fong. "Coercive Citation in Academic
Publishing." Science, 335, 2012, 542-43.
SUPPORTING INFORMATION
Additional Supporting Information may be found in the online
version of this article:
Table S1. Salary Regressions for UC Economists with Lifetime Output
of Journal Articles Not Age Discounted
Table S2. Salary Regressions for UC Economists with Lifetime Output
of Journal Articles Square-Root-Age Discounted
Table S3. Salary Regressions for UC Economists at the Top Four
Departments (Articles Square Root-Age Discounted)
(1.) For an international review of performance-based research
funding in tertiary education see OECD (2010).
(2.) Government research assessment exercises tend to rely more on
peer review assessments. Critiques of the use of journal rankings in
evaluating the research they publish are Oswald (2007) and Chang,
McAleer, and Oxley (2011).
(3.) Journal rankings based on expert opinion have also been
important. Examples in economics include assessments provided by Mason,
Steagall, and Fabritius (1997) and the Economic Society of Australia
(Abelson 2009).
(4.) See also the supporting analysis available from
www.sciencemag.org/cgi/content/full/335/6068/542/DCl.
(5.) The journal Technological and Economic Development of Economy
rose to third in the economics category of the ISI Journal Citation
Reports in 2010 by having 60% of citations coming from five journals
from the same university and another 23% of citations from journals
published by a neighboring university.
(6.) Coupe (2004) provides a somewhat dated review of research on
the market for academic economists.
(7.) Academic earnings equations have also been used to compare
returns to research productivity between countries (Moore, Newman, and
Terrell 2007) and to compare rankings of departments (Gibson 2000).
(8.) Kodczychi and Yu also provide assessment measures based on
citations from a set of journals they refer to as "policy"
oriented, which we do not use in this article.
(9.) This is an update of the widely used Kalaitzidakis, Mamuneas,
and Stengos (2003) measures. In our analysis the journal impact factors
used are taken from their 2010 Working Paper.
(10.) The list we use was obtained from http//homepages.
ulb.ac.be/~tCoupe/update/journals.html (on August 20, 2007).
(11.) Coupe (2003) also uses direct citation measures to provide
rankings of departments and individual economists.
(12.) The list we use was obtained on May 6, 2012 and as of July
2012 RePEc had grown to cover 1,004 journals.
(13.) For example, the relative weight for the Southern Economic
Journal compared with the AER is from 0.02 to 0.20, with an average of
0.12, for eight of the nine schemes that we use. But the MSF scheme
gives it a weight of 0.73.
(14.) These are all academic articles, as captured in the union of
the EconLit, RePEc, and Web of Science databases, so the under-coverage
of articles is not because they are not part of the peer-reviewed
scientific literature.
(15.) Combes and Linnemer (2010, 2) suggest that their high
convexity index "is useful to compare the best departments",
while their medium convexity index is better suited to study
middle-ranked departments.
(16.) See for example, University of California (2009).
http://compensation.universityofcalifornia.edu/total_comp_
facts_nov2009.pdf.
(17.) This calculation is limited to individuals with 3 years of
data for each type of salary and with no decline in base salary over
time (which may signal only a partial year's employment as could
occur from someone moving to another position). For these individuals,
the squared deviation of annual salary from the 3-year average has a
mean (median) of $1,443 ($312) for total salary and $1,093 ($213) for
base salary (all in millions).
(18.) At UC Berkeley the salaries for the nine steps of the
Faculty-Ladder ranks for the Law School Professor Series are about
$40,000 higher per academic year than the same nine steps on the series
for Business/Economics/Engineering.
(19.) We did gather the number of authored books (but not edited
volumes) from EconLit but this proved to have no explanatory power in
the earnings equations.
(20.) In a few cases the 2010 base salary was lower than in 2009 or
2007, which may signal partial year employment as could occur from
someone moving to another position, so for these individuals we used
their maximum base salary from 2007 or 2009.
(21.) Of course some of these factors may also differ within
departments. For example, faculty with good PhD students who publish in
the best journals may have reduced salary relative to others who get
paid more as compensating differential for working with lower quality
students who publish in lower-ranked journals. The RePEc Genealogy
database may eventually allow linking all of these academics with their
students, as a topic for future research.
(22.) Page correction factors were supplied by Joseph Macri, based
on his work with the late Dependra Sinha. A value of 0.72 was used for
journals with no factor available. This is the average page size for the
lowest-ranked journals in Gibson (2000), and these are typically the
ones without their own size correction factors available.
(23.) This result is broadly consistent with other estimates in the
literature of the value of research output in the academic labor market.
See Coupe (2004,208) for a brief review of such results.
(24.) The UCLA fixed effects become larger (and more significant)
positive values once output measures are included.
(25.) The model using Coupe weights is also weakly rejected against
the model using MSF weights (at p = .099).
(26.) If the models with different values of 6 were instead
estimated after using the CLh weights to summarize lifetime output, the
log-likelihood would maximize at [delta] = 0.3 (although at just 0.35
points above the log-likelihood at [delta] = 0.4). So this evidence of
weak age discounting is not specific to the journal weighting scheme
used.
(27.) See Coupe (2004, 202) for an overview of the earlier
literature on co-author discounting.
(28.) We also used CLh weights and included 8 = 0.3 (the age
discount factor that maximized the likelihood for those weights) and
find very similar patterns to what is shown in Figure 3.
(29.) The results of these sensitivity analyses are available from
the authors.
(30.) The result closest to statistical significance comes from the
salary regressions using the Kodrzycki and Yu (2006) scheme that is
restricted to economics journals, where the interaction variables have
f-statistics of 1.56 (age discounted) and 1.79 (no age discounting).
(31.) These five journals are the top journals whose trends have
recently been described by Card and DellaVigna (2013).
(32.) Professors with no top five journal articles are
predominantly at UC Riverside, but a total of eight UC departments had
at least one of these professors without an article in a top five
journal.
(33.) This noneffect is the same no matter which journal ranking
scheme is used for calculating research output.
(34.) We also considered whether the salary data favor modelling a
penalty for not having an article in Econometrica or a premium for each
Econometrica article. For the best-fitting salary regression with CLm
weights, Cox-Pesaran non-nested tests show a salary model using the
number of Econometrica articles is rejected against a model using the
dummy for no Econometrica articles (p = .00) while there is no reverse
rejection (p = .30).
JOHN GIBSON, DAVID L. ANDERSON and JOHN TRESSLER, We are grateful
to the editor, two anonymous referees, Peter Timmer, Dean Scrimgeour,
Izi Sin, and audiences at the Melbourne Institute, Motu and the
University of Canterbury for helpful comments.
Gibson: Professor, Department of Economics, Waikato Management
School, University of Waikato, Hamilton 3240, New Zealand. Phone
64-7-838-4289, Fax 64-7-829-5931, E-mail jkgibson@waikato.ac.nz
Anderson: Emeritus Professor, School of Business, Queen's
University, Kingston, Ontario K7L 3N6, Canada. Phone 1-790-470-0408, Fax
1-613-533-2301, E-mail dla@queensu.ca
Tressler: Associate Professor, Department of Economics, Waikato
Management School, University of Waikato, Hamilton 3240, New Zealand.
Phone 64-7-856-2889, Fax 64-7-838-4331, E-mail tressler@waikato.ac.nz
TABLE 1
Indicators of the Inclusiveness of Various Journal Ranking Schemes,
as Applied to Publications of UC Economists
Percentage of
Lifetime Articles
Source of Impact Factor with Zero Weight
MSF: Mason, Steagall, and Fabritius 33.3
CLm: Coomes-Linnemer (medium) 10.9
CLh: Coomes-Linnemer (high) 10.9
RePEc Simple Impact Factor 12.6
Coupe (2003) 22.0
K&Y_all: Kodrzycki and Yu (2006) 26.4
K&Y_econ: Kodrzycki and Yu (2006) 26.0
KMS: Kalaitzidakis. Mamuneas, 29.6
and Stengos (2011)
LP94: Laband and Piette (1994) 36.4
Rank-Elasticity
Regression (a)
Source of Impact Factor Elasticity SE [R.sup.2]
MSF: Mason, Steagall, and Fabritius -0.22 0.01 0.74
CLm: Coomes-Linnemer (medium) -0.67 0.01 0.97
CLh: Coomes-Linnemer (high) -1.35 0.01 0.97
RePEc Simple Impact Factor -1.17 0.04 0.76
Coupe (2003) -0.78 0.03 0.81
K&Y_all: Kodrzycki and Yu (2006) -1.48 0.05 0.85
K&Y_econ: Kodrzycki and Yu (2006) -1.53 0.05 0.84
KMS: Kalaitzidakis. Mamuneas, -1.74 0.07 0.80
and Stengos (2011)
LP94: Laband and Piette (1994) -1.92 0.09 0.82
Average Lifetime
Output for UC
Source of Impact Factor Economists (b)
MSF: Mason, Steagall, and Fabritius 143.6
CLm: Coomes-Linnemer (medium) 106.3
CLh: Coomes-Linnemer (high) 69.7
RePEc Simple Impact Factor 65.2
Coupe (2003) 45.5
K&Y_all: Kodrzycki and Yu (2006) 40.0
K&Y_econ: Kodrzycki and Yu (2006) 37.0
KMS: Kalaitzidakis. Mamuneas, 38.2
and Stengos (2011)
LP94: Laband and Piette (1994) 35.9
Note: Author's calculations based on 5,721 journal articles
produced by 213 University of California economists and journal
weights from the sources noted.
(a) Estimated over the journals with non-zero weights for each
scheme using log(relative weight) = [alpha] + [beta]log(rank).
(b) Total number of standardized pages (with co-authors given 1
In) published in career through 2010, where journals are weighted
such that the highest-ranked journal for each scheme has weight
1.0 and there is no age-discounting for older articles.
TABLE 2
Salary Regressions for UC Economists: Individual Characteristics,
Salary Attributes, and LocationFixed Effects
(1) (2) (3)
Seniority (years) -0.020 -0.020 -0.020
(3.05) ** (3.02) ** (3.04) **
Seniority squared (/100) 0.023 0.023 0.024
(1.21) (1.22) (1.24)
Experience (years) 0.052 0.052 0.052
(10.94) ** (10.99) ** (10.97) **
Experience squared (/100) -0.072 -0.072 -0.072
(6.06) ** (6.14) ** (6.17) **
Male 0.060 0.058 0.058
(1.47) (1.41) (1.43)
PhD field not economics -0.037
(0.90)
PhD rank (Amir and Knauff 2008) 0.000
(0.33)
PhD rank (Goldberger, Maher, 0.001
and Flattau 1995)
(0.08)
Nobel prize winner
Holder of a named chair
Not standard pay scale
Davis -0.449 -0.445 -0.448
(8.71) ** (8.38) ** (8.65) **
Irvine -0.392 -0.386 -0.391
(8.74) ** (8.27) ** (8.81) **
Merced -0.454 -0.458 -0.465
(9.08) ** (8.10) ** (8.62) **
Riverside -0.472 -0.466 -0.472
(8.83) ** (8.17) ** (8.63) **
San Diego -0.200 -0.199 -0.200
(3.85) ** (3.78) ** (3.84) **
Santa Barbara -0.406 -0.400 -0.405
(7.86) ** (7.50) ** (7.84) **
Santa Cruz -0.439 -0.435 -0.440
(7.87) ** (7.69) ** (7.93) **
Los Angeles 0.033 0.036 0.033
(0.60) (0.65) (0.60)
Constant 11.696 11.689 11.692
[R.sup.2] (210.07) ** (202.75) ** (189.47) **
0.70 0.69 0.69
(4) (5)
Seniority (years) -0.020 -0.022
(2.95) ** (3.38) **
Seniority squared (/100) 0.024 0.029
(1.25) (1.61)
Experience (years) 0.052 0.051
(10.72) ** (10.21) **
Experience squared (/100) -0.073 -0.074
(6.06) ** (6.16) **
Male 0.058 0.066
(1.44) (1.65)+
PhD field not economics
PhD rank (Amir and Knauff 2008)
PhD rank (Goldberger, Maher,
and Flattau 1995)
Nobel prize winner 0.431
(7.37) **
Holder of a named chair 0.146
(3.84) **
Not standard pay scale
Davis -0.449 -0.410
(8.67) ** (8.16) **
Irvine -0.389 -0.362
(8.71) ** (7.89) **
Merced -0.462 -0.445
(9.07) ** (6.61) **
Riverside -0.470 -0.412
(8.61) ** (7.42) **
San Diego -0.200 -0.142
(3.85) ** (2.72) **
Santa Barbara -0.422 -0.358
(8.46) ** (7.07) **
Santa Cruz -0.440 -0.384
(7.99) ** (7.30) **
Los Angeles 0.034 0.069
(0.62) (1.27)
Constant 11.695 11.662
[R.sup.2] (209.67) ** (207.02) **
0.70 0.71
(6) (7)
Seniority (years) -0.020 -0.021
(3.06) ** (3.28) **
Seniority squared (/100) 0.024 0.030
(1.26) (1.60)
Experience (years) 0.052 0.050
(10.79) ** (9.89) **
Experience squared (/100) -0.072 -0.075
(6.08) ** (6.02) **
Male 0.065 0.071
(1.61) (I.81)+
PhD field not economics
PhD rank (Amir and Knauff 2008)
PhD rank (Goldberger, Maher,
and Flattau 1995)
Nobel prize winner 0.346
(6.36) **
Holder of a named chair 0.137
(3.70) **
Not standard pay scale 0.113 0.115
(2.17) * (2.07) *
Davis -0.432 -0.397
(7.97) ** (7.62) **
Irvine -0.376 -0.347
(7.99) ** (7.18) **
Merced -0.451 -0.428
(8.35) ** (6.38) **
Riverside -0.463 -0.404
(8.39) ** (7.19) **
San Diego -0.185 -0.131
(3.43) ** (2.45) *
Santa Barbara -0.389 -0.359
(7.18) ** (7.00) **
Santa Cruz -0.425 -0.373
(7.44) ** (6.92) **
Los Angeles 0.048 0.084
(0.84) (1.49)
Constant 11.678 11.647
[R.sup.2] (202.31) ** (202.41) **
0.70 0.72
Notes: Dependent variable is log of base salary for the 2010
academic year, as reported at: http://www.sacbee.com/statepay/
with economists at UC Berkeley as the excluded group for the
fixed effects. N = 223, robust t statistics in parentheses.
(+) Significant at 10%; *significant at 5%; **significant at 1%.
TABLE 3
Comparisons of Academic Earnings Equations Using Different
Journal Assessment Weights to Compute Lifetime Output
Academic Earnings
Equation Regression
Semi-Elasticity (a) Robust SE [R.sup.2]
MSF: Mason, Steagall 0.009 0.001 0.77
and Fabritius
CLm: 0.013 0.002 0.78
Coomes-Linnemer
(medium)
CLh: 0.017 0.003 0.77
Coomes-Linnemer
(high)
RePEc Simple Impact 0.016 0.002 0.77
Factor
Coupe(2003) 0.023 0.004 0.76
K&Y all: Kodrzycki 0.023 0.004 0.76
and Yu(2006)
K&Y_econ: 0.025 0.004 0.76
Kodrzycki and Yu
(2006)
KMS: Kalaitzidakis, 0.024 0.005 0.76
Mamuneas, and
Stengos (2011)
LP: Laband and Piette 0.027 0.004 0.77
(1994)
Bayesian
Maximized Akaike's Info Info
Log-Likelihood Criteria Criteria
MSF: Mason, Steagall 83.66 -133.33 -75.40
and Fabritius
CLm: 87.49 -140.99 -83.07
Coomes-Linnemer
(medium)
CLh: 83.06 -132.12 -74.20
Coomes-Linnemer
(high)
RePEc Simple Impact 80.00 -126.00 -68.07
Factor
Coupe(2003) 79.28 -124.57 -66.65
K&Y all: Kodrzycki 79.07 -124.13 -66.21
and Yu(2006)
K&Y_econ: 78.92 -123.84 -65.92
Kodrzycki and Yu
(2006)
KMS: Kalaitzidakis, 75.63 -117.27 -59.34
Mamuneas, and
Stengos (2011)
LP: Laband and Piette 80.27 -126.53 -68.61
(1994)
Notes: The results are from nine separate regressions, where each
includes all of the variables in column 7 of Table 2 plus the
total number of standardized pages (with co-authors given 1 In)
published in each economist's career through 2010, where journals
are weighted such that the highest-ranked journal for each scheme
has weight 1.0 and there is no age-discounting for older
articles. Full results of the regressions are reported in Table
SI. N = 223.
(a) The semi-elasticity shows the percentage increase in annual
(academic year) salary for a 10-page increase in total career
output of weighted journal articles, where the weight for each
journal in each ranking scheme is used to convert into pages of
QJE-equivalent quality.
TABLE 4
Vuong Test Results Comparing Academic
Earnings Functions with Different Journal
Assessment Weights Used to Calculate Lifetime
Output
MSF (a) (a) (b) (c) (d)
CLm (b) -1.40
CLh (c) 0.17 2.27#
RePEc (d) 1.28 3.00# 1.19
Coupe (e) 1.65@ 2.41# 0.95 0.29
K&Y_all (f) 1.31 2.80# 1.41 0.56
K&Y_econ (g) 1.34 2.81# 1.46 0.68
KMS (h) 1.94@ 3.49# 3.11# 1.45
LP94 (i) 0.80 2.24# 1.11 -0.08
MSF (e) (f) (g) (h)
CLm
CLh
RePEc
Coupe
K&Y_all 0.08
K&Y_econ 0.13 0.49
KMS 0.89 1.13 1.11
LP94 -0.23 -0.37 -0.42 -1.37
Notes: Cell values are z-statistics, calculated from the models
reported in Table S1. Significant positive values favor the model
in the column against the model in the row and negative values
favor the row model over the column model. Test values in bold
are statistically significant at 5% level, those in italics are
significant at 10% level.
Note: Test values in bold are indicated with #.
Note: Those in italics are indicated with @.
TABLE 5
Vuong Test Results When Lifetime Output Is
Calculated with Journal Articles
Square-Root-Age Discounted
MSF (a) (a) (b) (c) (d)
CLm (b) -0.18
CLh (c) 1.45 3.29#
RePEc (d) 1.20 1.76@ -0.70
Coupe (e) 1.71@ 1.56 -0.14 0.47
K&Y_all (f) 2.22# 2.69# 1.01 2.33#
K&Y_econ (g) 2.14# 2.60# 0.92 2.21#
KMS (h) 2.80# 3.94# 3.26# 2.62#
LP94 (i) 1.47 1.93@ 0.18 0.68
MSF (e) (f) (g) (h)
CLm
CLh
RePEc
Coupe
K&Y_all 1.42
K&Y_econ 1.30 -0.72
KMS 1.88@ 1.36 1.44
LP94 0.26 -0.81 -0.72 -1.89@
Notes: Cell values are z-statistics, calculated from the models
reported in Table S2. Significant positive values favor the model in
the column against the model in the row and negative values favor the
row model over the column model. Test values in bold are statistically
significant at 5% level, those in italics are significant at 10% level.
Note: Test values in bold are indicated with #.
Note: those in italics are indicated with @.
TABLE 6
Vuong Test Results for the Subsample at the Top
Four University of California Departments
(Lifetime Output Calculated with Journal
Articles Square-Root-Age Discounted)
MSF (a) (a) (b) (c) (d)
CLm (b) 0.95
CLh (c) 2.20# 3.00#
RePEc (d) 1.67@ 1.00 -1.21
Coupe (e) 2.04# 0.98 -0.51 0.41
K&Y_all (f) 2.64# 2.09# 0.71 2.20#
K&Y_econ (g) 2.57# 2.02# 0.63 2.09#
KMS (h) 3.03# 3.35# 2.86# 2.69#
LP94 (i) 1.35 0.85 -0.94 0.13
MSF (e) (f) (g) (h)
CLm
CLh
RePEc
Coupe
K&Y_all 1.75@
K&Y_econ 1.62 -0.63
KMS 2.13# 1.49 1.55
LP94 -0.15 -1.38 -1.30 -2.42#
Notes: Cell values are z-statistics, calculated from the models
reported in Table S3. Significant positive values favor the model
in the column against the model in the row and negative values
favor the row model over the column model. Test values in bold
are statistically significant at 5% level, those in italics are
significant at 10% level.
Note: Test values in bold are indicated with #.
Note: those in italics are indicated with @.
TABLE 7
Portfolio Effects from Absence of Top Journals in
Publication Records of UC Economists
Salary Penalty
Percent without from Not Having
Articles in an Article in Top
Top Journals Journals (a)
Full Full Using CLm
Sample Professors Weights
Top three journals 44.4 28.4 -0.002 (0.08)
Top five journals 32.7 17.3 -0.052(1.71) (+)
Quarterly Journal of 70.0 59.1 0.012 (0.42)
Economics
American Economic Review 56.1 40.2 -0.012 (0.40)
Journal of Political 71.3 57.5 -0.022 (0.67)
Economy
Econometrica 72.2 62.2 -0.112(3.48) **
Review of Economic Studies 74.9 66.9 -0.030 (0.93)
Salary Penalty
from Not Having
an Article in Top
Journals (a)
Most Significant Average of
Model 9 Models
Top three journals -0.025 (0.83) -0.008 (0.26)
Top five journals -0.068 (2.21) * -0.061 (1.95) (+)
Quarterly Journal of -0.022 (0.75) 0.005 (0.39)
Economics
American Economic Review -0.036(1.16) -0.018(0.55)
Journal of Political -0.034(1.05) -0.024 (0.71)
Economy
Econometrica -0.134 (4.15) ** -0.119(3.54) **
Review of Economic Studies -0.062 (2.01) * -0.043 (1.33)
Notes: Author's calculations based on 223 University of
California economists. Robust t-statistics in parentheses.
(a) Estimates of the salary penalty are based on the
specifications reported in Table S2, which have been augmented
with a dummy variable for whether the individual had not
published in any of the journals or groups of journals listed in
the first column. These regressions condition on overall research
output (weighted by journal quality, age of the article and
number of co- authors). The results summarize 63 regressions
(seven journals or groups of journals crossed with nine different
journal ranking schemes), with column 4 reporting the results for
the overall best-fitting model (using CLm weights), column 5
reporting the most significant result for that particular
journal(s), and column 6 reporting the average over all nine
regressions for each of the journal(s).
(+) Significant at 10%; * significant at 5%; ** significant at 1%.
