Is It Worth while to Pay Referees?

Lai, Ching-chong

Juin-jen Chang [*]

Ching-chong Lai [+]

There are puzzles in refereeing scholarly articles: Why are referees willing to review a paper without payment, and is it worthwhile to pay referees in order to raise the review rate? Two interesting results are found in this article. First, when reviewing services are driven by reciprocity, the equilibrium participation of referees may exhibit the so-called self-fulfilling feature. Second, the optimal payment may not be zero if the referee receives the benefit of reputation gained by refereeing an article. In particular, this article will show that those journals whose status quo review rate is lower tend to pay reviewers more while journals whose status quo review rate is higher do not find it worthwhile to pay referees enough. This result implies that, in order to raise its quality, a journal with a low review rate is more likely to adopt a strategy to increase pay and attract a critical mass of referees.

1. Introduction

Economics tells us that people make decisions by comparing costs and benefits; their behavior changes when the costs or benefits change; that is, people respond to incentives. Yet referees for journals are often paid nothing or paid little for their work. Engers and Gans (1998) provide a reasonable starting point for the mystery of why referees are willing to review a paper without payment. The basic idea is that referees are willing to review an article since they are concerned about the quality of journals and the editor can take advantage of this concern by reducing the monetary payment. However, there still exist some interesting and unsolved questions. Is it worthwhile to pay referees? Which journals would be more likely to pay referees more?

To shed light on these questions, this article sets up a model in which the referee can receive the benefit of academic reputation gained by refereeing an article. Laband (1990), Hamermesh (1994), and Engers and Gans (1998) point Out that, if the participation rate of referees is high, then the review process could not only serve as a good screening mechanism but also provide added value to the paper and hence improve the quality of the journal. The higher the journal's quality, the more reputation benefits the referee can obtain. As a consequence, the referee's reputation benefits from reviewing a paper will increase with the total participation of referees. In other words, participating in refereeing papers not only improves the journal's quality but also the referee reaps a better academic reputation. When more potential referees are willing to review articles for the journal, reviewers reap a higher academic reputation stemming from reviewing an article for a high-quality journal. Thus, a referee providin g a reviewing service not only benefits himself but also other reviewers. Reviewing articles for a journal is thereby viewed as a reciprocal activity among all referees.

Two interesting new results are found in our analysis. First, when reviewing a paper is driven by reciprocity, the possible snowballing effect emerges and so the equilibrium participation of referees may exhibit the so-called self- fulfilling feature and generates multiple equilibria. This explains why academic journals of different caliber have different review rates. Second, due to the fact that an academic reputation has an intrinsic mechanism to enhance the positive benefit of the referee's payment, the optimal payment may not be zero. This article will show that journals whose status quo review rate is lower tend to pay reviewers more, while journals whose status quo review rate is higher do not find it worthwhile to pay referees enough. This result implies that, in order to raise its quality, a journal with a low review rate is more likely to adopt a strategy to increase pay and attract a critical mass of referees.

2. The Model

We start by formulating the model to illustrate the interaction between the editor of a journal and potential referees. In general, referees have the option to decline to review a paper. When a referee is not willing to review the paper, it will be returned to the editor; delay costs thus are imposed on the journal by a fixed amount [delta]. [1] In the meantime, the editor must send the paper out again. Accordingly, the journal's expected total delay cost, D, can be written as D = (1 - [micro]) ([delta] + D), where [micro] is the proportion of referees who opt to review the article. From this equation, we derive

D = 1 - [micro]/[micro] [delta] = (1/[micro] - 1)[delta]. (1)

On the other hand, from the standpoint of a referee, s/he receives a reward w but incurs costs c if s/he is willing to review the article. Without loss of generality, c is assumed to display a uniform distribution, f(c), on the interval [0, 1], and its cumulative density function is F(c) = c.

The referee's academic reputation can possibly benefit when the referee reviews a paper for a high-quality journal. This beneficial effect is evidenced by two facts. First, every year, most journals publish a list of all referees to express its appreciation for their services. Second, most scholars will select some established journals in which they have served as referees and list them on their resume. If the participation rate of referees is high, the review process could act not only as a good screening mechanism but also provide added value to the paper and hence improve the journal's quality (Laband 1990; Hamermesh 1994). The higher the journal's quality, the more reputation benefits the referee can obtain. Thus, it is plausible to specify that the referee's reputation benefiting from reviewing paper B is positively related to the level of the total referees' participation rate [micro]: B = B([micro]); B' [greater than] 0, B" [less than] 0. The payoff of a reviewing referee thus can be expressed as

W + B([micro]) - c. (2)

Following Engers and Gans (1998), if the referee declines to review the paper, then s/he incurs the expected delay cost since s/he is concerned about the quality of the journal. The referee's payoff in declining to review is -([delta] + D). Substituting Equation 1 into this payoff yields

-([delta] + D) = -[delta]/[micro]. (3)

For a referee, reviewing an article is worthwhile if the payoff from the review is larger than that of declining to review it, or equivalently, w + B([micro]) - c + [delta]/[micro] [greater than or equal to] 0. We can now find the critical level c that makes the marginal potential referee just indifferent between reviewing the paper and not. That is

w + B([micro]) - c + [delta]/[micro]. = 0. (4)

As is evident, referees with a lower cost c are more likely to review the article. Because c is distributed uniformly between zero and one, the proportion of referees who opt to review the article, [micro], is then

[micro] = [[[integral].sup.c].sub.o] f(c) dc = c. (5)

A graphical presentation is helpful in understanding the character of referees' equilibrium participation. In Figure 1, the pairs [micro] and c that satisfy Equation 4 are depicted by the decision rule (DR) locus. It follows from Equation 4 that the slope of the DR locus is ambiguous; that is,

[partial][micro]/[partial]c[\.sub.DR] = 1/B' - [delta]/[[micro].sup.2] [greater than/equal to/less than] 0, if B' [greater than/equal to/less than] [delta]/[[micro].sup.2]. (6)

Any combination of ([micro], c) in DR makes the referee indifferent to reviewing the paper. Given the level of [micro], for pairs ([micro], c) to the right of DR, reviewing the paper is not worthwhile due to c [greater than] c. The participation rate of referees, p., will decline as shown by the arrow in Figure 1. Analogously, for pairs ([micro], c) to the left of DR, the reviewing rate will grow.

In Figure 1, the distribution schedule (DS) traces the associations of [micro] and c that fulfill Equation 5. Clearly, the DS locus is the 45-degree line. Since Equation 5 is a definitional relation, the economy is not allowed to deviate from the DS locus.

Figure 1 shows that there could exist multiple equilibria of referees' participation in our model. As indicated by the arrows in Figure 1, the stability condition requires that the slope of DR should be larger than that of DS, that is, (1 - B')[[micro].sup.2] + [delta] [greater than] 0. [2] The value of [[micro].sup.E] can be viewed as a threshold level for referees' participation. If the benefit B is high enough, due to a higher status quo participation of referees ([micro] [greater than] [[micro].sub.E]), then it will motivate more potential referees to review the paper. The critical-mass effect thus pushes the review rate to a higher equilibrium, [[micro].sub.H]. On the contrary, if the participation of referees is low ([micro] [less than] [[micro].sub.E]) initially, then the additional reviewing benefit B is too small to attract referees' participation. The review rate will fall to a lower equilibrium [[micro].sub.L]. In other words, since the reputation effect exhibits a feedback, a journal could attract higher participation by referees or be stuck in a low-level equilibrium. In essence, the more successful journals are more attractive to work for and hence find it easier to recruit referees and maintain their advantage, while the less successful ones have a harder time recruiting reviewers and hence remain less successful. This feature of evolution is similar to the so-called self-fulfilling diagram provided by Schelling (1978). Such a result explains why academic journals of different quality have different review rates.

We next discuss whether journals with lower review rates can increase their pay to attract more referees and hence move to a high-review equilibrium. Figure 2 will be utilized to explore this interesting issue. [3] Figure 2 assumes that DS stands for the distribution schedule and [DR.sub.0] is the decision rule locus that is associated with a lower pay, [w.sub.0]. Can a small payment be sufficient enough to lead the review rate from a lower level, denoting [micro] [less than] [[micro].sub.E] to a higher one? The answer is not unambiguous and it depends on the status quo participation of referees. If the status quo referee participation of the journal is the value of [micro] at point A, then it is easy to infer from Equation 4 that a slight increase in pay from [w.sub.0] to [w.sub.1] will push the [DR.sub.0] locus to shift rightward to [DR.sub.1]. Since the referees' status quo participation at point A now exceeds the level of threshold at point E', the self-fulfilling feature stemming from the reputation benefits of reviewing papers will create a critical-mass effect and push the review rate to a higher equilibrium point, H'. However, if the initial review rate is considerably low, where [micro]. corresponds with point L, then such a policy only slightly improves the referees' participation but does not escape the dilemma of low equilibrium. Under such a situation, perhaps a big push by means of increasing referees' payment to a great extent is absolutely necessary. If the financial fund is allowed and if referees' pay can dramatically rise from [w.sub.0] to [w.sub.2], then Figure 2 reveals that the lower review rate may have a chance to escape this dilemma and is thereby pushed to a high-review equilibrium, H". [4]

Utilizing Equations 4 and 5, the equilibrium of referees' participation can be derived as

[micro] = [micro](w, [delta]), [[micro].sub.w] = [[micro].sup.2]/[(1 - B')[[micro].sup.2] + [delta]] [greater than] 0, [[micro].sub.[delta]] = [micro]/[(1 - B')[[micro].sup.2] + [delta]] [greater than] 0. (7)

Equation 7 indicates that either a positive payment or an increase in the delay cost can motivate more referees to put forth effort to review an article.

We now address the editor's optimization problem. The total costs to editor Z include the total expected delay costs of journal D and payment cost w. Given the referees' reaction in Equation 7, the editor's optimization problem is to choose an optimal w to minimize the total costs,

[min.sub.w] Z = D + w = (1/[micro] - 1) [delta] + w, s.t. [micro] = [micro](w, [delta]) (8)

The first-order condition with respect to w is

[Z.sub.w] = 1 -[delta]/(1 - B')[[micro].sup.2] + [delta] = 0, (9)

and the second-order condition requires [[micro].sub.w][2[micro](1 - B') - B"[[micro].sup.2]] [greater than] 0 in order to ensure that a minimum is satisfied. In Equation 9, term 1 presents the marginal cost of raising the referees' pay, while the term [delta]/[(1 - B')[[micro].sup.2] + [delta]] is its marginal benefit stemming from a reduction in the expected delay cost.

If the referee's benefit is absent (B = 0 and hence B' = 0), then the first-order condition in Equation 9 reduces to [Z.sub.w] = [[micro].sup.2]/([[micro].sup.2] + [delta]) [greater than] 0. The reason for this result is quite obvious. A rise in w will lower the expected delay cost and raise the payment cost, but the former marginal benefit definitely falls short of the latter marginal cost. This implies that a positive pay will always increase the editor's total costs when the referee's private benefit is absent. The result is the conclusion proposed by Engers and Gans (1998, pp. 1342-3): "... the benefit an editor obtains from a higher review rate is more than offset by the costs of higher pay. ... it is never optimal for the editor to set w [greater than] 0." [5] However, when the referee's benefit of building up a reputation is present (B [greater than] 0) and is positively related to [micro] (B' [greater than] 0), Equation 9 indicates that the optimal w could be positive. It is obvious that, due to the s elf-fulfilling feature, the effect of lowering the delay cost will be enhanced as B is taken into the picture. Once the decreased delay cost of raising w outweighs the increased payment cost, the optimal payment is not zero.

Many journals in reality do not actually pay referees cash, but they implicitly give reviewers some kind of rewards. Some journals (e.g., Journal of Economic Dynamics and Control) give their referees a 1-year subscription. Other journals offer timely referees a waiver (e.g., Southern Economic Journal) or a discount (e.g., Contemporary Economic Policy) of the submission fee for a submission within a year of the review. To some extent, these observations provide good evidence to back up our result: Editors still attempt to pay referees some implicit payments to encourage reviewing articles for their journals and hope that it will decrease the journals' delay costs and further raise their quality. [6]

Based on the fact that the review rate could present multiple equilibria, it is interesting to further investigate whether journals with different review rates (e.g., a higher review rate, [[micro].sub.H], and a lower one, [[micro].sub.L]) may adopt different pay strategies to improve their journal quality. Figure 3 sketches the equilibrium payment and helps us answer this question. Let MC stand for the marginal cost of adjusting w, while MB is the marginal benefit of adjusting w. It is clear from Equation 9 that MC = 1 and MB [delta]/[(1 - B')[[micro].sup.2] + [delta]]. In Figure 3, given that MC = 1 is independent of the referee's payment, w, the MC curve hence is horizontal. In addition, it is easy to infer that MB is negatively related to the referee's payment, w, and hence the MB locus is drawn downward sloping. [7] Furthermore, differentiating MB with respect to [micro] gives

[partial]MB/[partial][micro] = [partial]/[partial][micro] [[delta]/(1 - B')[[micro].sup.2] + [delta]] = - 2[micro](1 - B') - [[micro].sup.2]B"/[[(1 - B')[[micro].sup.2] + [delta]].sup.2] [less than] 0. (10)

Equation 10 indicates that the extent of MB decreases with the status quo review rate [micro]. As a result, Figure 3 reveals that a higher review rate, say [[micro].sub.H], is associated with a lower marginal benefit, MB([[micro].sub.H]), and a lower review rate, say [[micro].sub.L], is associated with a higher MB([[micro].sub.L]). Accordingly, we can conclude that journals with a lower initial review rate tend to pay reviewers more (say, [w.sub.1]), while journals whose status quo review rate is higher do not pay referees enough (say, [w.sub.2]). This implies that a journal with a low review rate would be more likely to adopt a strategy to increase pay and attract a critical mass of referees, thereby improving journal quality and the overall review rate by a large amount.

3. Concluding Remarks

Based on the fact that referees' academic reputations can benefit from refereeing papers, this article sets up a simple model to examine the proposition of Engers and Gans (1998). Two interesting results are found in our analysis. First, when reviewing services are driven by reciprocity, the possible snowballing effect emerges and so the equilibrium participation of referees may exhibit the so-called self-fulfilling feature and generate multiple equilibria. In essence, the more successful journals are more attractive to work for and hence find it easier to recruit referees and maintain their advantage, while the less successful ones have a harder time recruiting reviewers and hence remain less successful. Second, from the standpoint of the editor, the optimal payment to the referee may not be zero. In reality, editors indeed attempt to pay referees some implicit payments in order to encourage article reviews for their journals and hope this will decrease the journals' delay costs and further raise their quali ty.

(*.) Department of Economics, Fu-Jen Catholic University, Hsingchuang, Taipei 24205, Taiwan; E-mail ecosOOO4@mails.fju.edu.tw.

(+.) Sun Yat-Sen Institute for Social Sciences and Philosophy, Academia Sinica, Nankang, Taipei 11529, Taiwan; Email cclai@ssp.sinica.cdu.tw corrcsponding author.

We gratefully acknowledge insightful comments and suggestions received on a previous version of this article from the editor and an anonymous referee. Any errors or shortcomings are our responsibility.

(1.) The term [delta] can be regarded as the journal's costs of publishing the good paper with a lag time and hence losing its pioneering contribution.

(2.) There could be a single stable equilibrium or more than two stable equilibria.

(3.) We are grateful to the editor for bringing this interesting question to our attention.

(4.) For a journal, the financial constraint, of course, should be a factor in determining referees' pay.

(5.) Obviously, Engers and Gans' (1998) result crucially depends on the difference of the editor's subjective preference between the journal's expected delay cost D and the payment cost w. For example, if the editor's subjective total costs are specified as Z = [gamma]D + w, where [gamma] ([greater than or equal to] 0) stands for the editor's subjective weight on the journal's expected delay cost, the first-order condition will change to [Z.sub.w], = [[[micro].sup.2] - ([gamma] - 1)[delta]]/([[micro].sup.2] + [delta]) = 0; that is, as long as the weight that the editor puts on the expected delay cost is large enough, the editor's optimal payment may be positive.

(6.) Of course, these implicit payments may create different effects in improving the journal's quality. If a journal offers its referees a waiver of the submission fee, then this encourages scholars not only to referee a paper for the journal but also to submit their paper to this journal. Given that most referees have an excellent performance in academic research, in addition to raising referees' incentive for participation (and hence decreasing expected delay costs), such a policy can also attract more high-quality articles from the referees for the journal. If so, the journal's quality will thereby further improve.

(7.) From Equation 9, we have -[[micro].sub.w][2[micro](1 - B') - B"[[micro].sup.2]]/[[(1 - B')[[micro].sup.2] + [delta]].sup.2] [less than] 0.

References

Engers, Maxim, and Joshua S. Gans. 1998. Why referees are not paid (enough). American Economic Review 88:1341-9.

Hamermesh, Daniel S. 1994. Facts and myths about refereeing. Journal of Economic Perspectives 8:153-63.

Laband, David N. 1990. Is there value-added from the review process in economies? Preliminary evidence from authors.

Quarterly Journal of Economics 105:341-52.

Schelling, Thomas C. 1978. The micromotives of macrobehavior. New York: W. W. Norton.

[Graph omitted]

[Graph omitted]

[Graph omitted]