Reviews

Overview

The basic steps for assigning reviews are:

Details are below.

Reviewers and TPC members

EDAS supports a three-tier review model, consisting of one or more conference chairs, TPC members and reviewers. Conference chairs assign TPC members to papers manually or automatically, while either the chairs or the TPC members can assign reviewers to papers. For journals, TPC chairs are renamed to editors instead, but are otherwise equivalent. There are separate review forms (Reviews/Configure), notification letters (Conference/Configure) and deadlines for regular and TPC reviews.

TPC reviews can only be assigned to TPC members, while both TPC members and anybody else can serve as a reviewer.

If desired, TPC members can be assigned to a particular track and will only be able to claim and review papers from their track.

Smaller conferences typically have only the TPC members review the papers, with maybe a few delegations to outsiders. On the other hand, a very large conference, such as IEEE Infocom, may use a two or three-tier review model. The TPC members assign reviewers and provide a summary of the reviews, using the separate TPC review form defined in Reviews/Configure. They may also be a reviewer for the same paper and may initiate discussions among the reviewers for the paper. They are often also called lead reviewers.

To facilitate TPC meetings for large conferences, TPC members and papers can be assigned to TPC groups, with groupleaders.

Track Chairs

Track chairs are members of the TPC that have chair-like privileges for papers in one or more designated tracks for the conference. They can assign reviewers manually, delete reviewers and view all reviews. They cannot run the automated review assignment mechanism for the conference and cannot notify all authors. They also cannot modify conference parameters, track parameters and email templates.

Group Leaders

TPC members can be designated as groupleaders. They can review papers, but they can also be granted additional privileges (in Reviews/Configure) to assign papers, view reviews and see reviewer identities. In most cases, group leaders only have special privileges for those papers explicitly assigned to the same TPC group as the groupleader.

Assigning Papers to Reviewers

There are several ways to assign papers to TPC members and reviewers. These approaches are complementary, e.g., you can assign papers to reviewers automatically and then later add additional reviewers manually. Note that you need to first configure the review questions for your conference via Reviews/Configure (bottom of page).

Conferences can be configured, via Reviews/Configure, so that TPC members cannot assign reviews.

Conferences with a two-level review system can operate in two ways, depending on whether the reviewers are assigned from a fixed list, a technical program committee, or not. If there is a fixed list, reviewers can be entered as TPC members and selected from a list.

Conflicts of Interest

TPC members can declare conflicts of interest by adding and deleting names in a list. When creating a new user account, the list of conflicts is initially populated by all other EDAS users that share the same non-generic email domain, share the same name, have been co-authors or have the same affiliation. When reviewers claim papers for review, EDAS will omit papers where one or more of the authors is on the reviewer's conflict-of-interest list. For manual assignment, it will not assign such papers.

Assigning Reviewers and TPC Members Automatically

  1. TPC members are sent an email (People/Send email) containing instructions and the {CLAIM} placeholder. From the link contained in the email, TPC members can then claim a list of papers. There is no inherent deadline, but obviously claims submitted after the assignment is run are not taken into account. The list shown to TPC members is limited to papers that match at least one topic of interest. It excludes papers where there is a known conflict of interest. If TPC members are assigned to specific tracks, only papers in their track are shown. TPC members can assign each paper into four categories, namely papers that they would like to review, can review, are willing to review if absolutely needed and cannot review. Reviewers are only assigned papers from the first three categories.

    The goal of the automatic review assignment is to ensure that every paper has the required number of TPC and regular reviews. A secondary objective is to ensure that papers are reviewed by TPC members that are most interested in the paper. After those criteria, the algorithm tries to ensure that the review load is balanced between reviewers. Clearly, it's impossible to optimize for all three criteria at once, so some amount of unequal distribution is unavoidable.

    The algorithm assigns papers up to the specified maximum number of reviews to TPC members. Each paper is assigned reviewers in turn, starting with the paper with the smallest number of claims. Reviews are assigned to be the most willing reviewer, i.e., if there's a reviewer that wants to review the paper, they will get precedence over reviewers who are merely willing to review the paper. If there's a tie between TPC members, the one with the smallest number of papers assigned gets it.

    If a TPC member does not claim any papers, they are assigned based on their declared interests. If a TPC member does not indicate any interests, they are assumed to be interested in all topics.

  2. Configure the number of papers each TPC member should review and the number of TPC reviews and regular reviews each paper should have. Both are configured via 'Chair/Configure/Reviews'.
  3. You can see the number of claims for each TPC member via Chair/People/TPC. Clicking on the name in that listing leads to the person's overview page, which also shows the claims.
  4. To assign reviews based on claims, go to 'Reviews/Assign automatically based on claims'.

Assigning Reviewers and TPC Members Manually

After you have defined a review form (via Reviews/Configure), you can add reviewers. To add TPC members or reviewers manually, pull up the basic paper information, e.g., via Papers/List, and then click on 'Add reviewer' or 'Add TPC member' below the basic paper information. The TPC member or reviewer will receive an email asking him to review the paper. The TPC member or reviewer can accept or decline this invitation. Once he has accepted, he will be sent another email containing detailed review instructions.

Reviewers and TPC members can always check their EDAS home page to find the papers that they have been assigned.

Note that the notifications can be delayed until all reviewers have been assigned, by checking the appropriate box in the "Reviews" configuration menu. Pending notifications are sent via "Reviews/Notify reviewers".

Conferences can be configured for two kinds of review notifications. Reviewers can receive an email notification for each review they have been asssigned, or a single email if one or more new reviews have been assigned. For the latter, they need to consult their EDAS review listing to determine new reviews.

A TPC reviewer, if allowed by the review configuration, can assign additional non-TPC reviews for a paper, but no additional TPC reviews. Thus, there's one additional review outstanding after the TPC member assigns another review.

If allowed by configuration, both a TPC member and a reviewer can delegate ("hand off") their review to another person. They remain listed as the original reviewer for the paper and can check the status of the review and view the review itself once it is done. In the new interface, somebody who has been delegated the review cannot further delegate it; if he or she refuses the review, it reverts back to the original reviewer. (The old interface allows delegating delegated reviews, but that turned out to be confusing.)

Reminding Reviewers

Reviewers can be reminded via "Review/Remind reviewers". Sometimes browsers time out and the notification does not complete. To ensure that this doesn't happen, click on the "Remind reviewers" link again, until "0 Reviewers Reminded" appears.


Last updated by Henning Schulzrinne