Reviews

Overview

EDAS supports a range of review styles for conferences and journals, from conferences where each paper receives three reviews of the same type, to multiple levels, types or iterations of review. For example, there can be separate review forms for abstract and full paper reviews, or in-depth reviews and meta reviews. (Meta reviews are summaries of in-depth reviews.)

Conferences seem to fall into several common review models:

Single review cycle (1 review round):

Authors submit a full-length (e.g., 6-page) manuscript by the deadline. The paper is reviewed by, say, three reviewers. Authors of accepted papers are invited to submit a final (proceedings) paper by a specified deadline, without another review. The final manuscript is published in the conference proceedings.

Abstract, then full paper (2 review rounds):

Authors first submit an extended abstract, e.g., of one page, either as plain text or as a PDF file. This abstract is reviewed either by the technical program chairs or one or more reviewers. Abstracts that pass this stage are labeled as "abstract accepted", the others as "rejected". Authors then submit a full manuscript which is reviewed again, typically with a more extensive review. After that review cycle, papers are labeled as either "rejected" or "accepted". As before, authors submit the final proceeding paper.

Full paper, then revision (2 review rounds):

Authors submit a full paper for review, which is reviewed and either accepted, rejected or requires a minor or major revision. Papers with revisions are submitted again, possibly accompanied by a file describing the changes made in response to the first-round reviews, and reviewed by a subset of the original reviewers, possibly with additional reviewers. After the additional reviews, the paper may be rejected or accepted. If accepted, authors submit the final publication version.

This model can be implemented in two ways: (1) The chair creates a revision for the minor/major revision papers, with a new paper number. (2) The chair creates a new review type, copying the reviewers from the first review round to the new review type. The authors of minor/major revision papers submit the final version, but the paper is subject to rejection or may be changed to accept. Auxiliary files can be used to capture the author changes. If accepted, the author may replace the final version of the manuscript.

Abstract, then full paper, then revision (3 review rounds):

This model builds on the previous one, but adds a third review cycle for papers that are labeled as requiring a minor or major revision.

There are the same two options as before, with the addition of the abstract submission phase.

Full paper, then revision, repeated (N review rounds):

Journals tend to have multi-round review model, where each round yields one of three decisions: reject, minor or major revision or accept. If the author is asked to provide a revision, it is reviewed according to the simple review cycle above.

This model is best implemented using paper revisions.

Reviews can be assigned either manually or automatically, by interests and paper claims ("bids"). Chairs and, if allowed by the conference, TPC members can add any number or reviewers to each paper by going to the main paper page or by using the various means of assignments in Reviews:Assign.

Configuration

The basic review configuration is found at Reviews:Configure, e.g., who can serve on the technical program committee or how review reminders are sent. The actual review forms are then defined in Reviews:Review forms, where you can define as many forms as needed. (Use the "You can add more review types or inherit review types (not the reviews themselves)." link.) Review deadlines and other parameters specific to a review form are also configured there. Once you create the review form, click on "Configure review forms:" links to add questions.

Review forms can have (almost) any number of questions, either with numeric or text answers. Each question can be restricted to be visible only to some combination of chairs, track chairs, group leaders and authors, among others. (A common configuration problem is that a track chair is granted general read access to reviews, but none of the questions are visible to track chairs.)

Review process

The review process typically goes through the following steps:

1. The conference chairs create a list of TPC candidates via one of the options in TPC:Add. If the conference inherited the settings from a previous year, the list may only need to be modified.
2. The conference chairs notify the prospective TPC members via TPC:Notify:Notify TPC invitees.
3. TPC members pick topics (areas) of interest, designating them as "of interest", "neutral", "no interest". These topics are used to limit the number of papers TPC members can "claim" (or, as some conferences call it, "bid") for possible review, as only papers having at least one topic "of interest" and no topics "not of interest" are included.
4. If the conference plans to assign papers automatically, TPC members can claim (bid for) papers, where claiming is short-hand for "I want to review this paper". There are three levels of interest: "review if needed" (lowest), "can review", and "want to review". In addition, TPC members can mark papers as "cannot review". Papers in that category will not be assigned to that TPC member. For automated assignment, all papers that match the areas of interest are initially placed in the "can review" category. If a TPC member declares no interests, he or she is assumed to be interested in all conference topics and will thus be assigned papers randomly within the tracks he or she is assigned to.
5. Reviews are assigned to the reviewers, either automatically or manually, or both, taking into account topic preferences and conflicts of interest (Reviews:Assign).
6. Reviewers are notified by email (Reviews:Notify reviewers), either receiving one message for each review or one for all reviews. That way, notifications can be delayed until all reviewers have been assigned, by selecting the initial review state option in the review form configuration.
7. Reviewers confirm the review. Reviewers are reminded periodically to confirm their reviews, if configured in Reviews:Configure.
8. Reviewers complete the review or, if permitted, delegate the review to others.
9. Tardy reviewers are reminded (configured at Reviews:Configure).

Reviewers and TPC members

EDAS supports either a two-tier (single-level) or three-tier (two-level) review model, consisting of one or more conference chairs, TPC members and reviewers. In a two-tier model, the chair assigns reviewers to papers, who then complete or delegate their reviews. In a three-tier model, conference or track chairs assign TPC members to papers manually or automatically, while either the chairs or the TPC members can assign reviewers to papers. For journals, TPC chairs are renamed to editors instead, but are otherwise equivalent.

Many conferences have just one type of review, but particularly larger conferences may also use multiple different review types (Reviews:Review forms). Each review type has separate due dates, templates, restrictions on who can assign the reviews, and the number of reviews for each paper and TPC member.

Conferences may use multiple review types to implement meta reviews. Meta reviews are summaries of the other in-depth paper reviews. In that approach, each paper is assigned one meta review and, say, three regular reviews. The meta reviewer can be one of the three reviewers, or be a fourth reviewer.

If allowed by the conference configuration, TPC members and reviewers can delegate the review to somebody else, e.g., a graduate student working for a faculty member. Delegated reviews can still be viewed and edited by the original assigned reviewer; they can also be reclaimed if the person the review was delegated to fails to complete the review.

To facilitate TPC meetings for large conferences, TPC members and papers can be assigned to TPC groups, with one or more groupleaders. Each paper can belong to at most one TPC group, but each TPC member can belong to any number of TPC groups.

Track Chairs

Track chairs are members of the TPC that have chair-like privileges for papers in one or more designated tracks for the conference. They can assign reviewers manually or automatically, notify reviewers, delete reviewers and view all reviews. They cannot modify conference parameters, track parameters and email templates.

Group Leaders

TPC members can be designated as groupleaders. They can review papers, but they can also be granted additional privileges (in Reviews:Configure) to assign papers, view reviews and see reviewer identities. In most cases, group leaders only have special privileges for those papers explicitly assigned to the same TPC group as the groupleader.

Assigning Papers to Reviewers

There are several ways to assign papers to TPC members and reviewers. These approaches are complementary, e.g., you can assign papers to reviewers automatically and then later add additional reviewers manually. Reviews can be assigned in bulk via Reviews:Assign, e.g., to all session chairs or to specific TPC members. Alternatively, you can assign reviews for each paper by going to the paper page and using the button next to the "Reviews" heading.

Note that you need to first configure the review questions for your conference via Reviews:Review forms. Each review type can designate who can assign those reviews.

Conflicts of Interest

TPC members can declare conflicts of interest by adding and deleting names in a list. When creating a new user account, the list of conflicts is initially populated by all other EDAS users that share the same non-generic email domain, share the same name, have been co-authors or have the same affiliation. When reviewers claim ("bid") papers for review, EDAS will omit papers where one or more of the authors is on the reviewer's conflict-of-interest list. For manual assignment, it will not assign such papers.

Assigning Reviewers and TPC Members Automatically

TPC members are sent an email (People:Email) containing instructions and the {CLAIM} placeholder. (To use a placeholder, include it in the message text.) From the link contained in the email, TPC members can then claim a list of papers. There is no inherent deadline, but obviously claims submitted after the assignment is run are not taken into account. The list shown to TPC members is limited to papers that match at least one topic of interest; papers that contain a topic in the "not of interest" list are omitted. It also excludes papers where there is a known conflict of interest. If TPC members are assigned to specific tracks, only papers in their track are shown. TPC members can assign each paper into four categories, namely papers that they would like to review, can review, are willing to review if needed and cannot review. Reviewers are only assigned papers from the first three categories.

TPC members can be assigned to a particular track and will only be able to claim and review papers from their track.

Chairs and track chairs can see the number of claims for each TPC member via TPC:Claims & assignments. Clicking on the name in that listing leads to the person's overview page, which also shows the claims.

The automated review assignment has four goals: (1) every paper receives the required number of reviews; (2) no TPC member reviews more than the maximum review load; (3) the TPC members most interested in the paper are assigned as reviewers; (4) the review load of TPC members is roughly equal. The third and fourth objective are not always achievable at the same time, so the automated assignment mechanism allows the chair to choose whether to first consider TPC interest or favor equal assignment. As a third option, the algorithm can also use a weighted combination of the two. When load or interest are equal between two TPC candidate reviewers, the TPC reviewer with the higher interest or lower load is always chosen.

The algorithm assigns papers up to the specified maximum number of reviews to TPC members. Each paper is assigned reviewers in turn, starting with the paper with the smallest number of claims. The assignment algorithms sorts papers with the same number of claims randomly, and thus the assignment can differ if re-run.

If a TPC member does not claim any papers, they are assigned based on their declared interests. If a TPC member does not indicate any interests, they are assumed to be interested in all topics and thus can be assigned any random paper in the conference.

To assign reviews based on claims, go to Reviews:Assign 'automatically based on claims'.

Notifying reviewers

The TPC member or reviewer will receive an email asking him to review the paper. The TPC member or reviewer can accept or decline this invitation. Once he has accepted, he will be sent another email containing detailed review instructions.

Reviewers and TPC members can always check their EDAS home page or My reviews tab to find the papers that they have been assigned.

Conferences can be configured for two kinds of review notifications. Reviewers can receive an email notification for each review they have been asssigned, or a single email if one or more new reviews have been assigned. For the latter, they need to consult their EDAS review listing to determine which reviews they are responsible for.

Delegating Reviews

If allowed by configuration, both a TPC member and a reviewer can delegate ("hand off") their review to another person. They remain listed as the original reviewer for the paper and can check the status of the review and view the review itself once it is done. In the new interface, somebody who has been delegated the review cannot further delegate it; if he or she refuses the review, it reverts back to the original reviewer.

Reminding Reviewers

Reviewers are reminded automatically or manually via Reviews:Reminders. Reminders are sent to reviewers that have not yet confirmed their review or where the review is almost due or beyond due. The reminder ("nag") interval and the warn-ahead period are configured in Reviews:Configure. Reminders for overdue reviews can be sent either in one message for each paper or in one message for each tardy reviewer, depending on which email template is defined. You can also send one message per reviewer to reviewers that have not yet confirmed all their reviews, via People:Email.

Review Visibility

Review visibility is controlled in two places: the conference configuration at Conference:Configure, Permissions ("read reviews (beyond the reviews person has written") determines what roles can read reviews, in general. Each review question can then determine which roles have access, via Reviews:Review forms, "Visibility".

Last updated by Henning Schulzrinne