Pre- and Post-Course Assignments

Notes

The pre- and post-course assignments have been designed to simulate a "real" peer-review process. The emails that are sent to the participants about the pre- and post-course assignments include an introduction to the assignment and are then followed by a peer review request that has been adapted from requests used by the journal partner. The assignments can use real manuscripts that have passed peer review but are not yet published, pre-prints, or papers that have already been published.

Pre-course Assignment

Rationale

To provide the instructor with an idea of the peer review ability of incoming course participants prior to starting the course and to provide participants with a simulated real-world experience of being invited to review a paper at the partnering journal.

Assignment description

We request four manuscripts from the partner journal that have passed editorial triage to the peer review stage but have not yet been published from our journal partner. If you are not working with a journal partner, any pre-print or published paper would also be sufficient. If any peer reviews have been published together with the paper, instructors should be aware that students may access/use this content while preparing their assignments.

We assign participants to review one of the four manuscripts at random. They receive an invitation from us via email to review their assigned manuscript as if they were getting the request directly from the journal, with the primary difference being that their review should be submitted to us via return email rather than uploaded via the journal partner's submission system. Beyond what is provided in the journal partner's usual reviewer invitation email, no guidance on how to write a peer-review report is provided. However, participants are told they may use any resources available to them (as do actual invited peer reviewers).

Participants are given a a two-week deadline to complete the review assignment, and they are able to request a one-week extension if needed. Beginning at the two-week deadline, we send chase emails every 3 days until the review is received.

Assessment of the pre-course assignment

Other than for the purposes of our program evaluation study, we have not performed a formal assessment of the pre-course assignment. Since the purpose of the assignment is to give participants a first experience with receiving a peer review assingment "cold" (before receiving any training), we do not believe that giving them specific feedback on the review itself is valuable at this stage. We recommend that the instructors read through the submitted assignments prior to starting the course to get an idea of the partcipants' levels and degree of familiarity with the reviewing task. We have also incorporated a detailed self-reflection exercise as one of the homework assignments. Here, participants review their own pre-course assignment after having attended the lectures, reflecting on the report's strengths and weaknesses, which is then discussed as a group.

We have also used the pre-course assignment to help us assign the workshop groups. If you are running a course with multiple workshop groups, you could try to distribute participants who generate stronger baseline review reports evenly between the groups.

Related email templates

Email: Pre-course assignment

Post-course Assignment

Rationale

To provide participants with specific feedback on their own review reports after they have completed the course.

Assignment description

We use the same four manuscripts that were distributed as the pre-course assignment. For the post-course assignment, we recommend that each participant be assigned one of the four pre-course assignments that they have not previously reviewed.

Ideally, based on the timeframe of the course, the manuscripts that the participants have reviewed for the pre- and post- course assignments will be published shortly after the completion of the course. The participants will then be able to compare their own reviews to the published reviews for the papers (if partnering with an open review journal).

Related email templates

Email: Post-course assignment

Assessment of the post-course assignment

The teaching team provides feedback on the post-course assignment following submission. We developed a Review Report Quality Score, with items adapted from previous studies and tailored towards the specific criteria of The BMJ [maybe also cite?]. The following table outlines the 13 items, divided into general and specific criteria, each worth 2 points with the exception of the last language item which is only worth 1 point.

	Peerspectives Review Report Quality Score (Criteria specific for The BMJ)	Points possible
Specific content	Comments on study objectives, rationale	2
	Comments on methods-related aspects (e.g. study design, statistical analyses, sampling, suitability of reported effect sizes/confidence intervals/p-values, reporting quality - relevant for replicability/reproducibility)	2
	Comments on tables, figures, visual aspects of the manuscript	2
	Comments on the interpretation of results and whether the conclusions are supported by the data	2
	Comments on specific manuscript strengths (study, theory, methods, argument) and limitations (including biases)	2
	Comments on manuscript structure, flow, writing style and/or language	2
General items	Structure: includes a short summary as well as some specific comments	2
	Specific reflection on importance, relevance and suitability, also given the journal's specific scope/readership	2
	Focus on “global concerns” (larger structural, logic/reasoning issues) rather than detailed “local concerns” (spelling, grammar, formatting)	2
	Thorough, respectful, constructive critique, including positive and negative comments, also offering suggestions rather than simply labeling problems	2
	Appropriate comment density, overall review length demonstrates the reviewer’s investment in peer review while not overwhelming the reader	2
	Filled out reviewer form completely, including declaring any competing interests or no competing interests	2
	Review is written in understandable English (language, meaning of statements clear, few, if any, distracting typos/grammatical mistakes) - 1 point only	1

Scoring

Each item is scored based on the following criteria, awarding 0, 1 or 2 points depending on the completeness of reporting. No partial points are awarded.

Points	Suggested criteria
0	Item not addressed or insufficiently addressed in your review or not directly evident in your the review report
1	Item may be partially or indirectly addressed within your review report, but lacks supporting details, clarity and/or otherwise not fully up to the standards expected of peer reviewers
2	Item clearly captured within your review report and reflects sufficient standards for peer reviewing

Feedback

To provide an interpretation of the overall review report quality, the points of all 13 items are summed up to a total score, which falls into one of the four categories outlined in the table below. This should give students an immediate impression on how they performed and if their work was up to the standard that BMJ editors expect to see in high quality reviews. The course instructors can add additional comments after these standard evaluation if there are notable positive or negative aspects, e.g., praising thorough checks of supplement, study protocol and trial registration or advising against the use of inappropriate language, etc.

Total Points	Suggested comments
23-25	Nice job! Editors and authors alike would appreciate your high-quality report. Keep up the great work.
20-22	Your work represents a good starting point. Your report had many strengths but was missing a few important elements.
15-19	Your review had some strengths but was missing several important elements.
<15	Your review was missing several important elements. Editors would likely not consider your review to be complete.

Students receive an email with the individual scoring of all items, as well as the overall feedback and any additional comments (if applicable). In case the reviewed papers are still currently under consideration at the journal partner, reports that scored >20 points can be forwarded to the journal as an actual review report. In cases where manuscripts have been published in the meantime, a link to the paper will be shared with the student.

Related email templates

Email: add template for sending out feedback, including individual scores and what happened to the reviewed manuscripts (if already known)