Refereeing For Journals

An introduction to the wonderful world of journal refereeing

1. How do people get chosen as referees for papers?

The basic rule is: editors are desperate to find referees for submissions, and especially desperate to avoid using the same small pool of referees over and over again. (In part because they want to avoid having a small collection of gatekeepers having too much control over what's published, but in larger part because they want to avoid having their small pool of referees rise up in rebellion against being overworked.) So they're looking for anything they can find that will lead them to qualified referees. I'd say the four major tools are:

1. The editor's own knowledge of the field. Journals vary in how much expertise editors typically have in the area of submissions they are handling. With an area-specific journal, or with a journal that has sub-editors handling specific areas, the managing editor probably knows the area reasonably well, and can put together a preliminary list of qualified people. With other journals in which a single editor handles all submissions, individual expertise may be much less useful.
2. The bibliography of the paper. (More ambitiously, the ancestral of the bibliography relation.) This is a quick way to get some names of other people working on the same topics. (Professional tip, though: editors typically aren't stupid, so don't try to control who will be refereeing your paper by judiciously including and excluding names in your bibliography. It's not hard to spot (and it works poorly if the ancestral is examined).)
3. Some judicious googling/consulting of PhilPapers. This is a close cousin to consulting the ancestral of the bibliography.
4. Suggestions made by potential referees who decline. (See below on declining invitations to referee.)

2. Will I get invited as a graduate student to referee papers?

It definitely could happen. If you've published in a given area, the odds are increased considerably. (My experience is that early-career, you tend to get used as a referee on one specific topic, and then as your career continues, you get refereeing requests on a wider and wider range of topics. Early on, almost everything I was asked to referee was on compositionality. These days, I feel like I get refereeing requests across the field.) Also, someone else (maybe one of your committee members, or some more senior member of the profession who knows your work) might suggest you.

3. I've gotten a referee request. What's in it for me to accept?

Good question. Honestly, given a certain axiology, the answer is: nothing. No one is going to give you a job because you're a diligent referee of papers, and it's just barely the case that anyone will ever even thank you or think better of you for doing it. If you were to go through your entire professional career declining every single referee request you ever got, it would have more or less no real impact on your professional success. (The requesting is distributed enough among different journals and editors that it might never even be noticed that you're systematically declining.) That said:

1. It really is necessary for sustaining the current structure of the field that people referee papers. (Whether the current structure is worth sustaining is a big question I won't try to tackle here.) If you don't do some of it, others will have to do more.
2. You really are, in at least a small and incremental way, helping people by refereeing. By exercising some quality control, you're helping readers of journals more reliably get useful philosophical insight from articles they read. And by providing helpful feedback and commentary, you're helping authors create better work.
3. Refereeing is not a bad way to get some extra insight into what's going on in areas that you're interested in. You get to take an early look at, and have some extra incentive to think carefully about, some of that work. Sometimes this even leads to some ideas that contribute to something you're writing. (That said, refereeing isn't a terribly efficient way of achieving these ends. If that's all you're trying to do, you're almost certainly better off just making your own selections of relevant papers from journals and people's websites. Maybe, depending on your personality, the obligation to write comments helps encourage better uptake than you'd get if you were just reading of your own accord.)
4. It's a chance to put a small stamp of your own on the field, by having some limited ability to have your own philosophical standards respected. (See below on the danger of abusing that tiny bit of authority, though.) 
5. It can be fun. Or maybe that's just me. In the end, it's some doing of philosophy, and doing philosophy is fun. It's enjoyable and rewarding to work through a paper and think carefully about what's good and bad in the paper.

Very roughly, I think that people's obligation to referee papers increases as their career advances. A potted statistical analysis: suppose the field-average acceptance rate for journals is 10%. Then on average, for every paper that someone published, ten papers need to get evaluated by journals. Maybe 30% of those get desk rejected. That leads 7 needing referees. So overall for your career, you should be aiming to referee seven papers for every one that you submit. But typically that should come in the form of doing very little reviewing in the early years (maybe one or two papers a year), and a lot more reviewing in later years (maybe 10-15 papers a year).

4. I've gotten a referee request. What should I do now?

The most important rule: respond quickly. Ideally, respond, either accepting or declining the referee request, within 24 hours. One of the main things that slows down the referee process is waiting to hear back from reviewers. It's not uncommon to need to ask a dozen potential referees before getting a couple of acceptances. If each invitation leads to three or four days wait for a response, it can get to be almost a month before referees are even assigned.  Much better promptly to decline to referee a paper than to wait a week and then accept.

Take a quick look at the paper. (If you're ambitious, this can be an actual read through the paper, but it needn't be that thorough.) Decide whether you are competent to evaluate the paper, and then think about whether you've got time to do the refereeing. (Within the time frame that the journal requests. But honestly, the time frame shouldn't matter that much. The whole refereeing process shouldn't take more than a few hours, and, while it's perfectly reasonable to conclude that you don't want to carve out a few hours from your research schedule, if you've decided that you can do so, you should then be able to get it done within any reasonable time period.)

If you decide you are competent, and that you're willing to take on the assignment, accept the review request. For most journals, this is just a matter of clicking a link in the request email. For some less technically sophisticated ones, you may just have to email the editor directly.

If you decide not to accept the assignment, decline promptly. It's extremely helpful if you can then suggest some other possible names for referees. (However, speaking from personal experience: don't let difficulty in coming up with alternative names slow down sending the decline. Again, better a decline today with no suggestions than a decline three days from now with some suggestions.) Less obvious suggestions are especially helpful. The editor is probably already aware that Kit Fine would be a good fit for this submission on grounding, but might not know about less senior figures. (The less senior figures are also typically more likely to accept refereeing assignments.) Journals usually ask you to include contact information for potential referees. Unless you've got that at your fingertips, I wouldn't bother. It's easy enough for editors to look up someone's email (their journal editing software might even have it available), and I think it's again too easy to delay responding because you don't want to take the time to look up the emails yourself.

5. What does a good referee report look like?

When you're writing a referee report, you're writing for two different audiences, with different goals for the two audiences:

• You're writing for the editor. Here your goal is to provide the editor with evaluative remarks sufficient to let the editor make an informed decision about whether the paper is worth publishing. 
• You're writing for the author. Here your goal is to provide the author with evaluative remarks that are helpful in thinking about how to improve the paper.The official structure of things is that the first of the audiences is your primary one: you're primarily writing for the editor, so your obligation is to be helpful to the editor. If you can be helpful to the author in the process, that's great, but it's supererogatory. But whether to respect the official structure is really up to you.

I recommend the following overall structure:

• A short paragraph giving your overall assessment of the paper. ("A potentially important contribution to the literature on question X, which raises a novel and intriguing argument against the positions of Y and Z. However, there is a powerful response W available to argument X that surprisingly is not discussed in the paper. Without a reasonable reply to W, the paper doesn't seem to me publishable, but if the author can incorporate a helpful discussion of W, I think the resulting paper would significantly advance discussion of topic X.")  This opening paragraph lets the editor quickly see roughly what you thought about the paper, and lets the author see your overall opinion before getting into the details.
• A paragraph giving your own summary of what the paper is up to -- what claim is at issue, how the main arguments run. Producing a summary like this will probably be helpful to you in evaluating the paper. It's also useful in letting editor and author know how to react to the comments you're making. If your summary of the paper matches the author's own conception of the paper, it will make it easier for the author to see that your critiques should be taken seriously.  If your understanding of the paper diverges from the author's conception, that will be helpful in knowing how to situate and interpret your comments. Your summary can also help the author see where they need to clarify their presentation to avoid misinterpretation.
• Your core critical remarks about the paper. For this part of the report, don't get bogged down in all the minutiae. Just focus on laying out a few central points that you think are most urgent for improving the paper (or for showing why the paper is beyond hope). Explain a central equivocation in the argument, or an unjustified and unreasonable background assumption, or an important section of the literature that's relevant but undiscussed, or an important potential response to the paper's argument that ought to be mentioned.
• Any smaller critical points. How much detail to go into is largely up to you. Your job in providing the editor with the information they need should probably be done at this point, so now you're mostly being helpful to the author. If you're so inclined, you can go through smaller worries about particular parts of the paper, or even detail typographical errors in the paper. If you're recommending that the paper be accepted or given a "revise and resubmit" verdict, there's some extra reason to make these smaller points, because you're envisioning the paper appearing in print, and this might be the last chance to get things fixed. If you think the paper is a clear reject, there's less reason to go into these details. Even if the author later produces a viable revision of the paper, it's likely to be different enough from the current version that small criticisms will become irrelevant by the time it's revised.                                                              

Beyond this, the main goal in writing your report should be to do good philosophy. Your approach should be much the same as in reading and commenting on a student's (or colleague's) paper: you want to avoid making superficial "gotcha" objections, and instead think through as carefully as you can to what the important core issues are, and give what guidance you can about how to approach those issues most profitably. A couple of things to keep in mind:

• It's not your job to fix the paper, or to write/sketch out the perfect alternative paper. You should think things through enough to be able to say helpful things, but it's fine if you see problems without having a picture of how to fix those problems. And don't, in the process of thinking through the paper and the issues, get so fixated on what seems to you the right thing to say that you end up wanting to reject the paper just because it isn't the paper you would (now) have written. (Finish the referee report, and go write that paper, if you feel strongly about it.) A paper can be important and worth publishing even if it isn't in the end actually right.
• Don't be a jerk about your critical comments. It is undeniably fun to fine-tune the rhetorical zip of your objections, but save the zingers to share with your friends later, and be calm and measured in your actual remarks. Of course, some (many!) papers shouldn't be published (or yet be published), and some papers that should be published should still have some stuff fixed up first. So you shouldn't hesitate to point out problems when you see them. But being nasty about pointing them out isn't going to achieve anything. In fact, it's mostly counterproductive. Nastiness is just going to make the editor worry that you're not impartial and that your concerns shouldn't be taken too seriously, and put up the author's back so that they'll be less likely to be receptive to your concerns. No one is keeping score here, and there's no one to impress with your verbal fireworks.
• Do say enough to give some justification for your evaluative remarks. A report that simply says, "the paper's arguments are unpersuasive" isn't much use to either editor or author. As with everything else in refereeing, this is a balancing act. Better to give justification, but also better to be timely with your report. If writing detailed justifications is going to make your report late, make do with a sketchier justification. Sometimes problems with a paper are systematic and hard to pin down. In such cases, sometimes the best you can do is to give a few representative examples. If the problem is that the use of terminology in the paper is vague and sloppy throughout, pick a few sentences where the problem is particularly bad, and walk through what's wrong with those sentences. then say that the rest of the paper is filled with similar problems that you're not going to detail.

6. How long should my referee report be?

As with seminar (and journal) papers, long enough to say something useful, and no longer than that. I know how helpful that bit of guidance is in paper-writing.

I'd say I average maybe 1500-2000 words for a referee report. But there's high variance in length. Some papers are hopeless enough that all that's needed is a brief report setting out a few central problems, and then it's not worth spending time and energy with suggestions for improvement, because the project is too far away from any plausible success. Some (much fewer) papers and just ready to go and don't need much done with them. Some papers have as their central problem that they just aren' very interesting -- what's said is reasonable enough, but it doesn't go anywhere novel or productive. Again, there may be little to say in such cases. But some papers will show a lot of promise, but still need enough work that you're motivated to put in more time working through the problems in detail and tracing out various options.

7. How do I decide what verdict to reach?

My own preference is to avoid giving a verdict at all. I'd rather just give my substantive thoughts on the paper, and then leave it to the editor to decide what the practical upshot of those thoughts should be. (Among other things, I don't always know what kind of quality standards a given journal is striving for, and how their publishing backlog is affecting the competition level.) Some journals are friendly to this, but a lot of journals have online referee report submission systems that require a verdict to be entered. (In these cases, I often enter a default verdict of "decline", and then include a note to the editor that I don't give verdict recommendations and that the "decline" verdict should be ignored.)


Your preferences may be different, of course. A couple of points to keep in mind:

• Negative verdicts are likely to be more causally efficacious than positive ones. An editor is much less likely to reject a paper despite a positive report than to accept despite a negative report. (I'm not convinced that that's a good thing, but it's the way it is.) So be sure you think the paper should be rejected before giving a negative verdict. (On the other hand, the default position should probably be that papers should be declined unless they've made a sufficient positive case for acceptance, rather than the other way around.)
• I'd recommend staying away from "revise and resubmit" verdicts as much as possible. I think in most cases they're a way to avoid taking a stand even when it's in fact clear what needs to happen to the paper. Keep in mind that "revise and resubmit' verdicts contribute to clogging up the journal system -- any such verdict is very likely going to lead to a resubmission of the paper in the near future, and the need to evaluate it again. (It's reasonably likely, too, that you'll be asked to referee the next drat of the paper. So "revise and resubmit" often ends up just being a way of putting off the inevitable.)

8. What happens after I send in my report?

In most cases, your report simply vanishes into the void, and you never hear anything about it ever again. A few journals have a policy of keeping referees in the loop on the decision process -- with those journals, you'll be copied on the editorial decision letter, and be able to see what verdict the editor reached and why. You'll then also be able to see what the other referees (if there were any) said about the paper -- this can be a nice opportunity to compare your evaluations to other people's. I think more (all) journals should follow this policy -- I'd encourage you to ask editors what happened with papers you reviewed, and push them to adopt the inclusion policy.

If you don't hear anything from the journal, you might eventually see the paper appear in print. If you're very lucky, the published version of the paper might even include a footnote or two thanking anonymous referees for raising various points. (Keep in mind, though, that what such footnotes often mean is "I just want everyone to know that it wasn't my idea to talk about this silly point -- I was just trying to placate an unreasonable referee.")

You will, of course, have resisted the temptation to google the paper's title while you've been writing your referee report. (In all likelihood, knowing who the author is wouldn't have much effect on your evaluation -- it's not that hard to figure out which papers are good and which aren't. But there's still value in maintaining the appearance of integrity in the system.) You should continue to resist after you send in your report. If you made a "revise and resubmi" recommendation, you may well be called on to referee a later version of the paper, and it would be best if anonymity were preserved for that round of review. If you recommend rejection, it's still likely that the author will try again, with a later version or a different journal. If you were picked as a good fit as referee once, you're not unlikely to be picked again.