Netdev Program Review Guidelines

These guidelines have been been established to provide a fair and unbiased review process of program proposals submitted to Netdev Society.

Background

Netdev Society welcomes submissions for proposals for technical presentations at Netdev Society conferences. Proposals are to be judged for acceptance on their technical merit without bias or subjectivity. To this end, Netdev Society has instituted a blind peer review process. Proposals are reviewed by a program committee for inclusion in Netdev Society programs. The program committee is a set of volunteers that have demonstrated technical expertise and have made significant contributions to the Netdev community. It is important to note that the program committee performs its technical reviews independently of conference organizers that deal with logistics or work with sponsors. In particular, corporate sponsorship of a Netdev conference in no way guarantees that any proposals or presentations are accepted.

The procedures described here were motivated by the SIGCOMM peer review process as well as the consensus decision model of IETF. The procedures have been adapted and scaled appropriately to work for Netdev Society.

Roles

This section describes the roles in Netdev conference submissions and review process.

Submitters

Submitters are individuals that make proposals for inclusion in the Netdev conference program. A submission is done via the submission page. The content describes the proposal and identifies the submitters.

Submissions include the following information:

  • Name(s) of the submitter(s).
  • Title of the submission.
  • Submission type (one of talk, workshop, tutorial).
  • Label (one of moonshot, nuts'n bolts, hands-on).
  • Estimate of length of time for presentation.
  • Affiliations of submitters (needed for conflict of interest check).
  • PC members who have conflicts of interest with this submission. This includes mentors, people with the same affiliation, and any recent (~2 years) coauthors and collaborators.
  • Description of proposal.

Please note that submitters must not communicate directly with any program committee members concerning their proposal. The program shepherd serves as the sole liaison between submitters and the program committee.

Program committee

The role of the Program Committee (PC) is to evaluate submitted proposals. For a each proposal, a subset of committee members are selected to review the proposal. Proposals are evaluated on their technical merits, suitability for the Netdev conference, appropriateness for the potential audience, and overall relevance of the work to the community. Ultimately, the committee makes a consensus decision whether to accept a proposal for a Netdev conference. The PC cannot see the submitters name or affiliation.

Program committee chair

The program committee chair is responsible for the overall flow and content of the conference. The chair is a member of the program committee and participates in proposal discussions. Generally, the chair will participate in discussions for all proposals (except in cases of potential conflict of interest) and will ensure that progress is made and that consensus is reached concerning acceptance for a Netdev conference. The chair may at their discretion reference other proposals they are aware of for the purposes of comparing similar proposals and promoting a good balance of presentations. Part of the chair’s responsibility is determining when consensus has been reached, however in normal technical discussions the chair should ‘take their hat off’ so that their opinion carries no more weight than other committee members.

Program shepherd

The role of the program shepherd(s) is the communications conduit between submitters and the program committee. The shepherd is not a member of the program committee and does not participate in the decision process or technical discussions. If the program committee requires clarification from a submitter concerning a proposal, they will refer questions to the program shepherd who will then forward to the submitter. There may be more than one program shepherd for any conference to distribute load.

Program scheduler

The program scheduler handles the scheduling of presentations. They will work to appropriately schedule time slots for presentations to ensure a good ordering and to satisfy the requested presentation times. The scheduler may request adjustments to presentations to fit in allotted time.

Review panel

For each proposal a subset of program committee members is selected by the program shepherd to review the proposal and make a consensus decision about whether proposal is accepted. Minimally, a review panel will consist of four program committee members, however it may have more.

The shepherd will select the review panel for a proposal based on the following factors:

  • Potential conflicts of interest either declared or perceived
  • Areas of expertise of program committee members
  • Load balancing work amongst program committee members
  • Random selection

Submissions and review procedures

The Netdev submission and program committee procedures are as follows:

  1. Create a new account on: https://0x15.netdevconf.info

  2. Check inbox for email verification link.

  3. Go to the link from step 1, update your profile with your name and any conflicts of interest you know of. Save your profile using the "Save changes" button at the bottom, and you'll be taken to the submission site's front page.

  4. On the front page, click "New submission" and submit your proposal using the submissions form.

  5. An acknowledgment is sent to the submitter once the system verifies all the relevant fields are entered, and the proposal is registered in the proposal system. The shepherd receives the submission notification and starts the review process.

  6. The submissions system automatically calculates potential conflicts of interest based on submitters input on conflict of interest, then program shepherd sends a notification and COIQ to the selected subset of the program committee. If any member of the subset indicates a conflict of interest, the shepherd replaces those members. The query includes the label, submission type, title, and description-- with the last two items being anonymized. Each member of the committee considers whether they have a potential conflict of interest if they were to review the proposal. For instance, if they recognize the work as being done by a colleague they are working closely with then that would be considered a conflict.

  7. Committee members reply directly to the shepherd as to whether they have a conflict of interest or would otherwise prefer to recuse themselves from reviewing a proposal. If a committee member has questions about potential conflict of interest then they should send those directly to the program shepherd. Note that there is no discussion about the proposal during the COIQ period.

  8. Once all members have replied to the conflict of interest query (or seven days have elapsed) the program shepherd will select a group of members for the review panel of a proposal. The criteria for selection are described above in 'program shepherd role' section.

  9. The shepherd initiates a review of the proposal. This is done by sending a Request for Review (RFR). The email includes the anonymized title, label, submission type and anonymized description. It is sent to the members of the review panel for the proposal.

  10. The review panel discusses the proposal. Only the members of the panel participate in the discussion. No other PC members can see the discussion. In this stage, there are no side discussions concerning the proposal with other members of the committee, the submitters, or anyone else outside of the review panel.

  11. The review panel may have questions or require clarifications from a submitter. The panel creates a Shepherd Request (SR) email. The shepherd will follow up with the submitter and provide their anonymized responses to the panel.

  12. The panel discusses the proposal until consensus is reached and an acceptance decision is made.

    There are three possible decisions:

    • Accepted - proposal accepted for inclusion in the conference.
    • Accepted with contingency - proposal is accepted assuming that some contingencies are met. Examples of contingencies are: a request to the submitter to frame the subject in a certain way, eliminate overt references to commercial products, work with another submitter to ensure that potentially overlapping proposals are resolved.
    • Not accepted - proposal is not accepted for inclusion


  13. Once a decision is made, one member of the panel is chosen by the panel to write a Decision and Explanation (DAE) (this is not necessarily the program committee chair). This includes an explanation by the panel for its decision. If the proposal was accepted with contingencies then the panel will detail the contingencies. In the case a proposal is not accepted it is important to give useful feedback to the submitter for future proposals.

  14. The panel's decision is sent to the shepherd. Unless there are procedural issues with the decision, the panel decision is binding. The decision is logged in the program system.

  15. The shepherd updates the system to notify the submitter of the decision. If a proposal is accepted it may be announced.

  16. If the decision is accepted with contingencies, the program shepherd will follow up with the submitter to make sure the contingencies are met. The program shepherd may also consult with the review panel concerning the contingencies.

  17. Once a proposal has been accepted and any contingencies have been met, the program scheduler will schedule the presentation in the agenda. If any adjustments must be made to the length of presentation or other scheduling constraints considered then those will be negotiated between the program scheduler and the submitter. The program scheduler may also consult with the program committee chair or members of the program committee for optimizing the program agenda.