Doing Economics

You have convinced me to review as much as I can. How can I be a good
reviewer? Beyond responding to invitations in time, the key to being a good
reviewer is to (i) write constructive reviews that are (ii) at the right level for
the journal you are reviewing for and (iii) make comments at the right
margin for the paper you have been asked to read.
Unpacking the foregoing, by writing constructive reviews, what I mean is
this: when you point out a problem, do your best to offer the way in which
the authors can address it, even if you do not think the authors can do so.
For example, if the authors would require more data to truly establish the
result they are after, the authors would much rather be told “I do not think
the authors can establish their desired result with the data they currently

---

have, but they should be able to do so with more data” than “I do not
believe the empirical results.”
By suggesting that you write reviews at the right level for the journal you
are reviewing for, what I mean is that not every journal is at the level of the
American Economic Review, and pointing out every possible flaw you can
find about a paper you are reviewing for a field journal or a second-tier
general journal to make yourself look smart in the eyes of the editor in
charge is likely to have the opposite effect; the quality of a review is only
weakly correlated with its length.
Finally, and on a related note, by suggesting that you make comments at
the correct margin for the paper, what I mean is that if you know that a
paper has a fundamental flaw that the authors will be unable to address, it is
perfectly fine to write a short (i.e., between a half-page and one page)
review pointing out that flaw, and explaining why you do not think the
authors can realistically fix it. But if you see a path to publication for a
paper and recommend that the authors be given a chance at revising and
resubmitting, then by all means give the authors a full list of what you
expect them to have done for the next version, as this will shorten the
review process considerably, and it will maximize the chances you will only
have to read the paper one more time instead of two, three, or more times.
And for the love of God, when you do recommend a revise and resubmit,
do not decline the editor’s request to review the paper when a revision is
submitted.
What follows are some guidelines about how to structure and write a
review.
1. When you receive a request to review a manuscript: Read the title
and the abstract of the paper, determine whether you are a good fit
for the paper, and respond immediately. You would be surprised
how frequently some reviewers simply do not respond to review
requests. As I have alluded to above, you might sometimes think
you are not a good fit for the paper. Ultimately, however, the
editor has asked you to review the paper for a reason. Often, she
simply wants your advice as a non-expert, to see if the paper
makes sense to (and will thus be cited) by people who are not
experts on the topic. If you have refereed the paper before, or if

---

there is a conflict of interest (i.e., the paper was written by one of
your colleagues, one of your coauthors, one of your students, or
one of your advisors), you should let the editor know about it and
give her a chance to reconsider her request.
2. Once you have agreed to referee a paper: Do not wait until it is
too late to do the review. You can certainly wait until you get a
first reminder from the journal that your review is due, but do not
wait until the second such reminder. Really, the earlier you submit
your review, the better for everyone involved—you do not know
your (future) self as well as you think you do.
6
3. Once you decide to get started on your review, read the title, the
abstract, the introduction, and the conclusion of the paper. Do you
have a good idea of what the authors are doing? Perhaps more
importantly, are you convinced that it is a worthwhile topic? If it
is an empirical paper, can you understand from the tables what
relationships the authors are after? If you answer “No” to any of
those questions, you should encourage the editor to reject the
paper. This may sound harsh, but before submitting, authors
should work hard on the “sell” of their paper, i.e., on convincing
the reader that the paper is worth their time. And generally
speaking, the profession could certainly benefit from better
writing. It is true that a groundbreaking idea that suffers from a
bad sell deserves a second chance. The two, however, are rarely
orthogonal to one another.
4. Corollary to the previous rule, as a referee, you don’t “accept,”
“give a revise and resubmit,” or “reject” a paper, you merely give
your advice as to what is to be done in a cover letter which the
editor will not share with the authors. So as a reviewer, to claim
that you have “rejected” this paper and “given a revise and
resubmit” to that paper seriously overstates your role in the peer-
review process.
5. If you believe the paper should be rejected after reading the
abstract, the introduction, the conclusion and, if the paper
contains any, the tables, write up your review. Make sure you
explain clearly why the paper is a nonstarter. More importantly,
offer some constructive ways in which the authors can make their

---

paper ready for prime time. Do so in less than two single-spaced
pages, and skip the tedium (e.g., typos, missing references, etc.)
6. If you believe the paper should be given a chance after reading
the abstract, the introduction, the conclusion and, if the paper
contains any, the tables, read the paper thoroughly, then write up
your review. Do so in as much space as you need, and do not skip
anything, even the tedium.
7. A good review should start by summarizing the paper in one or
two short paragraphs. Do not paraphrase the abstract—this is your
chance to give your point of view to the authors as to what the
paper really does, and sometimes your interpretation of what
matters in the paper differs from theirs.
8. Then, offer a numbered list of major comments—those are the
potential deal breakers, i.e., the things the authors have to do to
get you to recommend that the paper be published—and a
numbered list of minor comments—the small things you would
like the authors to do in order to improve upon the paper. If you
are going to recommend rejection, you can skip the specific
comments. These two lists are a clear indication of what you view
as negotiable or not, and numbered comments increase efficiency
as the authors can refer to specific comments more easily when
responding to your review.
9. Again, make sure you give the authors suggestions as to how to
address your comments. At the very least do so for your general
(i.e., nonnegotiable) comments. If you do not think the authors
can reasonably address your comments, you should recommend
rejection so as to not waste anyone’s time.
10. What is a constructive comment? There is a general
misunderstanding of what constructive means that interprets the
word as “A comment that I like.” That is not what constructive
means; a constructive comment is one that can actually be
addressed, and one that is not demeaning to the authors. When in
doubt as to whether a comment will be seen as demeaning, err on
the side of being nicer.
11. That being said, you are not a coauthor on the paper, so you
should not push the authors to write the paper you would have

---

written on the topic. Moreover, it is extremely bad form to push
the authors to cite all of your papers on the topic. Encourage them
to cite the ones that are truly germane to their work—no more, no
less. And encourage the authors to polish their writing. More
specifically, if you can think of more (or better) motivations for
their work, do not hold back, as this will enhance the paper’s
citation potential, and the editor will be grateful for that since for
her, citations are the coin of the realm.
12. Once your review is written up, it is time to write your cover
letter to the editor. That letter will generally not be shared with
the authors (though sometimes excerpts from cover letters are
shared with authors if it useful to do so), so you can be as candid
as you want. Specifically, you make your recommendation
—“reject,” “weak revise and resubmit” (same as “major
revision”), “strong revise and resubmit” (same as “minor
revision”), or “accept”—to the editor, and you give your
arguments as to why you make this specific recommendation. Do
not cut and paste from your review. Again, this is an occasion for
you to be as candid as you want about the reasons behind your
recommendation.
13. The cover letter is also your chance to flag potential ethical
problems. For example, if you believe the authors are “double
dipping” (i.e., publishing only slightly different versions of the
same paper), if you believe the paper is simultaneously being
reviewed somewhere else, or if you believe the authors have
plagiarized part of their paper, the cover letter is the place to
mention it.
14. Once your review and your cover letter are written up, submit
them to the editor. There is no good reason to sit on them; as long
as you write clearly and cogently, no one will care about your
prose in a referee report or cover letter.
15. Once you have submitted your referee report: The more
considerate editors will let you know the decision they have made
on the basis of your review. Often, they will agree with you. This
is especially likely when you recommend rejection, as several
editors will typically require the majority or all of the referees to

---

recommend a “revise and resubmit” before they ask the authors
for a revised version. Sometimes, however, the editors will go
against your review. Do not take those cases personally, but if this
happens often, you should take some time to reflect upon why it
does.
16. You will eventually meet many of the authors of papers you
have refereed. Resist the temptation to out yourself as a reviewer,
even if you think you were especially kind as a referee. If you
have ever spent any time on social media, you know how easy it
is to misconstrue the intention behind a written comment; you
never know whether your well-intended constructive remark may
have been interpreted as an insult by the authors.
17. Think carefully before breaking any of these rules. Many seem
to believe that they can get away with not responding to editors,
with taking an unduly long amount of time before submitting
their reviews, with being insulting to the authors of the papers
they referee, and so forth. You get the idea. Those people forget
that editors will remember such bad behavior and keep track of
who is a good citizen of the profession—and who is not.
18. The converse of the previous rule is that good refereeing is a
very good way to build a solid reputation for good citizenship in
the profession. This is something that is overlooked all too often
in a profession supposedly full of smart people who have a deep
understanding of incentives.