GENWiki

Premier IT Outsourcing and Support Services within the UK

User Tools

Site Tools


rfc:rfc6385

Internet Engineering Task Force (IETF) M. Barnes Request for Comments: 6385 Polycom Category: Informational A. Doria ISSN: 2070-1721 Research Consultant

                                                         H. Alvestrand
                                                                Google
                                                          B. Carpenter
                                                University of Auckland
                                                          October 2011
           General Area Review Team (Gen-ART) Experiences

Abstract

 The General Area Review Team (Gen-ART) has been doing reviews of
 Internet-Drafts (I-Ds) since 2004.  This document discusses the
 experience and the lessons learned over the past 7 years of this
 process.  The review team initially reviewed the I-Ds before each of
 the IESG telechats.  Since late 2005, review team members have been
 assigned to review I-Ds during IETF Last Call, unless no IETF Last
 Call is necessary for the I-D.  The same reviewer then reviews any
 updates when the I-D is placed on an IESG telechat agenda.

Status of This Memo

 This document is not an Internet Standards Track specification; it is
 published for informational purposes.
 This document is a product of the Internet Engineering Task Force
 (IETF).  It represents the consensus of the IETF community.  It has
 received public review and has been approved for publication by the
 Internet Engineering Steering Group (IESG).  Not all documents
 approved by the IESG are a candidate for any level of Internet
 Standard; see Section 2 of RFC 5741.
 Information about the current status of this document, any errata,
 and how to provide feedback on it may be obtained at
 http://www.rfc-editor.org/info/rfc6385.

Barnes, et al. Informational [Page 1] RFC 6385 Gen-ART October 2011

Copyright Notice

 Copyright (c) 2011 IETF Trust and the persons identified as the
 document authors.  All rights reserved.
 This document is subject to BCP 78 and the IETF Trust's Legal
 Provisions Relating to IETF Documents
 (http://trustee.ietf.org/license-info) in effect on the date of
 publication of this document.  Please review these documents
 carefully, as they describe your rights and restrictions with respect
 to this document.  Code Components extracted from this document must
 include Simplified BSD License text as described in Section 4.e of
 the Trust Legal Provisions and are provided without warranty as
 described in the Simplified BSD License.

Table of Contents

 1. Introduction ....................................................2
 2. Who Are the Gen-ART Members? ....................................3
 3. Goals of Gen-ART ................................................3
 4. Gen-ART Reviews .................................................4
    4.1. IETF Last Call Review Process ..............................4
    4.2. IESG Telechat Review Process ...............................5
    4.3. Form of Review .............................................5
    4.4. Gen-ART Process Overview ...................................8
 5. Secretarial Process ............................................10
    5.1. Maintaining Review Spreadsheet ............................10
    5.2. Last Call Assignment Procedure ............................12
    5.3. Telechat Assignment Procedure .............................13
    5.4. Capturing Reviews .........................................14
 6. Results ........................................................14
 7. Impressions ....................................................15
    7.1. Reviewers' Impressions ....................................15
    7.2. General Area Directors' Impressions .......................17
    7.3. Gen-ART Secretaries' Impressions ..........................18
 8. Needed Improvements ............................................18
 9. Applicability ..................................................20
 10. Security Considerations .......................................20
 11. Acknowledgments ...............................................20
 12. Informative References ........................................21

1. Introduction

 The General Area Review Team (Gen-ART) was created personally by the
 General Area Director in 2004.  This document discusses the
 experiences and the lessons learned as the process has evolved over
 the past 7 years.  The process described in this document reflects
 that which was in place at the time this document was published.

Barnes, et al. Informational [Page 2] RFC 6385 Gen-ART October 2011

 This process is likely to continue to change over time.  The review
 team has been retained by subsequent General Area Directors.  It has
 no official role in the IETF standards process, except as a set of
 individuals entitled, like everyone, to comment on Internet-Drafts
 (I-Ds).  Its volunteers, including a secretary and the team of
 reviewers, serve at the invitation of the General AD.  Both the
 reviews and the reviewer names are public.

2. Who Are the Gen-ART Members?

 The reviewers are typically individuals that have a fair amount of
 experience within various IETF Working Groups (WGs), have authored WG
 I-Ds and RFCs, and are often considered to be subject matter experts
 (SMEs) in their particular areas of work.  The current review team is
 comprised of such technical experts, including several WG chairs as
 well as past and current Internet Architecture Board (IAB) members.
 Several past and current ADs have served as reviewers.  Two past
 General ADs have also served as reviewers, with one currently
 serving.
 Members of the review team sometimes excuse themselves from the team
 for various reasons, typically due to "day job" demands.  However,
 they often rejoin (for periods of time) as their schedules allow.
 Also, some reviewers remain on the team, while their review workload
 is decreased by assigning them just one I-D (at Last Call time) to
 review each month.  Section 11 provides a list of currently active
 reviewers, along with those who have served on the review team in the
 past.

3. Goals of Gen-ART

 The original and continuing goal of the Gen-ART was, and is, to
 offload from the General AD some of the burden of IESG reviews.  The
 load for the bi-weekly IESG reviews is often quite large;
 occasionally, there are more than 20 I-Ds scheduled for discussion in
 a single telechat.  Thus, ADs also have less than a week's notice for
 many of the I-Ds on the telechat agenda.
 Gen-ART was based on a model that had proved productive in the
 Operations (OPS) Directorate: quick review close to telechat time, to
 advise the AD on issues that remain serious.  By having a trusted
 group of reviewers read and evaluate the I-Ds, the General AD would
 be able to concentrate on those I-Ds where there was a concern
 expressed by the reviewer.  The reviewers are expected to provide
 feedback based on a whole set of criteria, including the criteria

Barnes, et al. Informational [Page 3] RFC 6385 Gen-ART October 2011

 summarized in Section 4.3.  The overall objective is to ensure that
 the I-Ds are well structured; can be easily understood, at least at a
 high level; and provide a reasonable basis for implementation (for
 I-Ds intended for the Standards Track).
 While other area (and WG) directorates/review teams existed prior to
 Gen-ART and more have been established since Gen-ART, the roles of
 each are fairly distinct.  Thus, there is little overlap between the
 goals and review criteria for the various review teams.  It is also
 very valuable for these other review teams to operate independently.
 For example, when both Gen-ART reviews and Security Directorate
 (SecDir) reviews raise the same sorts of concerns, it's a clear red
 flag that the I-D needs more work before progressing.  In addition,
 due to the typical thoroughness (and objectiveness) of the various
 review teams' reviews, the sponsoring AD and document shepherd are
 often able to work with the editors/WG (and vice versa, depending
 upon area and WG structure) to improve the overall quality of the
 final I-D.  It should also be noted that some ADs take the Gen-ART
 reviews into consideration when preparing their own evaluations.
 Statistics from the Gen-ART reviews over the past 6+ years show a
 trend of increased quality and readiness for progression of I-Ds by
 the time they are placed on the telechat agenda.  Additional
 statistics are discussed in Section 6.

4. Gen-ART Reviews

4.1. IETF Last Call Review Process

 While the original process was meant only for reviews just before the
 IESG telechat, the decision was made to include IETF Last Call (LC)
 reviews in early 2005.  Over time the latter has proven to be quite
 effective.  Assigning the I-Ds at IETF LC time typically gives a
 reviewer more time to review an I-D.  And, in some cases, the IETF LC
 version is the one to appear on the telechat.  Thus, by the time I-Ds
 are added to the telechat agenda, a majority (typically at least 70%)
 have already been reviewed.  For those I-Ds that have been
 up-versioned, the amount of time dedicated to re-review depends upon
 the review summary for the IETF LC review.
 The assignments at IETF LC time evolved to minimize the gap between
 LC announcements and assignment time, with the secretary doing LC
 assignments every Thursday night.  This typically allows the reviewer
 at least one week and sometimes two to three weeks to complete the
 review.  The reviews are obviously most helpful when done on or
 before the end of the IETF LC.

Barnes, et al. Informational [Page 4] RFC 6385 Gen-ART October 2011

 The Last Call assignments are done on a fairly strict round-robin
 basis to ensure a fair workload amongst all the reviewers.  Reviewers
 that are unavailable (vacations, etc.) during the review period
 timeframe obviously are excluded from that round of assignments, but
 remain in the same queue position for the next round.  The order is
 occasionally modified to avoid assigning an editor/author or WG chair
 their own I-Ds.  A reviewer may also NACK an assignment if they feel
 they may have some bias (although corporate affiliations are not
 considered to be sources of bias) or they don't feel they can review
 the I-D in a timely manner.
 The assignment process is completely manual, although a spreadsheet
 tremendously facilitates the process.  The details are described in
 Section 5.  Ideally, this process could be automated.  However,
 manual intervention would still be required to maintain the
 appropriate available reviewer list (unless reviewers took on the
 task of maintaining their data in some sort of database).  Further
 details on the tools necessary to automate the entire process are
 provided in Section 8.

4.2. IESG Telechat Review Process

 The process for reviewing I-Ds when they appear on the IESG agenda is
 as follows:
 o  The "nearly final" IESG meeting agenda generally appears on
    Thursday night, less than one week before the IESG telechat.  The
    Gen-ART secretary uses this as the input for the assignment
    process.
 o  For I-Ds reviewed at IETF Last Call, a new review is only asked
    for if the I-D is revised.  In this case the reviewer, typically
    the person who did the Last Call review, only needs to check that
    any open issues were resolved.  Often the draft will not have
    changed between IETF LC and the IESG telechat review.  Section 4.4
    provides the step-by-step telechat review assignment process, with
    specific details on the maintenance of the review assignment data,
    which is in turn maintained in review spreadsheets (Section 5).

4.3. Form of Review

 Rather than invent new guidelines, the Gen-ART requirements for the
 form of a review stole liberally from "Careful Additional Review of
 Documents (CARD) by Senior IETF Reviewers (SIRS)" [SIRS], making
 adaptations for the special "late, quick review" case and the nature
 of the General Area's concerns.

Barnes, et al. Informational [Page 5] RFC 6385 Gen-ART October 2011

 Each review must start with a summary statement chosen from or
 adapted from the following list:
 o  This draft is ready for publication as a [type] RFC, where [type]
    is Informational, Experimental, etc.  (In some cases, the review
    might recommend publication as a different [type] than requested
    by the author.)
 o  This draft is basically ready for publication, but has nits that
    should be fixed before publication.
 o  This draft is on the right track but has open issues, described in
    the review.
 o  This draft has serious issues, described in the review, and needs
    to be rethought.
 o  This draft has very fundamental issues, described in the review,
    and further work is not recommended.
 o  Unfortunately, I don't have the expertise to review this draft.
 The length of a review can vary greatly according to circumstances,
 and it is considered acceptable for purely editorial comments to be
 sent privately if it's obvious that the I-D needs substantial
 revision.  All substantive comments, however, must be included in the
 public review.  Wherever possible, comments should be written as
 suggestions for improvement rather than as simple criticism.
 Explicit references to prior work and prior IETF discussion should be
 given whenever possible.
 Reviewers are asked to review for all kinds of problems, such as
 basic architectural or security issues, Internet-wide impact,
 technical nits, problems of form and format (such as IANA
 Considerations or incorrect references), and editorial issues.  Since
 these reviews are on I-Ds that are supposed to be finished, the
 review should consider "no issue too small" -- but should cover the
 whole range, from the general architectural level to the editorial
 level.
 All reviews should apply generally agreed-upon IETF criteria,
 such as:
 o  [RFC1958]: "Architectural Principles of the Internet"
 o  [RFC3426]: "General Architectural and Policy Considerations"
 o  [RFC3439]: "Some Internet Architectural Guidelines and Philosophy"

Barnes, et al. Informational [Page 6] RFC 6385 Gen-ART October 2011

 o  ID-Checklist: The "ID checklist" document maintained by the IESG
 o  [RFC2223bis]: "Instructions to Request for Comments (RFC) Authors"
    as updated by [RFC-STYLE]: "RFC Document Style"
 o  [RFC5226]: BCP 26 - "Guidelines for Writing an IANA Considerations
    Section in RFCs"
 o  [RFC3552]: BCP 72 - "Guidelines for Writing RFC Text on Security
    Considerations"
 o  Any other applicable architectural or procedural documents.  It is
    considered important that reviews give precise references to such
    criteria when relevant to a comment.
 Of special interest to the General area, because they do not fall
 under any other area, are:
 o  A clear description of why the I-D or protocol is useful to the
    Internet.
 o  Adherence to IETF formalities, such as capitalized "must",
    "should", etc. in normative statements, per the ID-Checklist.
 o  Useful and reasonable IANA considerations.  Ensure that all
    necessary registries are defined/referenced, and ensure definition
    and compliance with IANA assignment criteria.
 o  Correct dependencies for normative references.
 o  That the I-D is written in reasonably clear English.
 o  Checking the updates/obsoletes information.
 o  Running idnits and checking the output.
 o  Checking that things imported by reference, especially from other
    RFCs, make sense (notably definitions of terms, security
    considerations, and lists of criteria) and ensuring they are used
    as intended by the referenced document.
 o  That examples (e.g., Fully Qualified Domain Names (FQDNs),
    telephone numbers, IP addresses) are taken from the right spaces.

Barnes, et al. Informational [Page 7] RFC 6385 Gen-ART October 2011

4.4. Gen-ART Process Overview

 The following provides a general overview of the Gen-ART process
 along with some basic rules associated with assignments.  The very
 precise details of the secretary's process are provided in Section 5.
 o  The availability of reviewers and the order of assignments for the
    next round of Last Call document assignments are updated weekly
    and are available on the directory where all the assignments and
    reviews are cached.
 o  At telechat assignment time, all previously reviewed I-Ds are
    assigned to the reviewer who reviewed them previously, assuming
    that reviewer is available.  Otherwise, these I-Ds are assigned to
    a new person in the process described below.
 o  The secretary attempts to avoid assigning I-Ds that might conflict
    with other IETF roles such as WG chairs, other directorates, etc.
    However, in the cases where the secretary doesn't note the
    conflict, the reviewer should notify the secretary and Gen-ART
    mailing list so another reviewer may be assigned.
 o  It should be emphasized that assignment is never made according to
    a reviewer's technical specialty.  Even though it happens, when,
    for example, routing I-Ds fall on routing experts or MIB documents
    fall on MIB doctors, it is coincidental.  To the reviewer, the
    choice looks random.
 o  There is an attempt to evenly distribute I-Ds amongst reviewers at
    LC time by using a round-robin process, starting from where the
    previous week's assignments stopped.
 o  Typically, there is no attempt made to actually equalize the load,
    as the length and complexity of the I-Ds are not taken into
    account in this process.  (Thus, a reviewer could end up with a
    couple of hundred-page I-Ds, but this is statistically rare.)
    However, in the case of a reviewer that might receive more than
    one new LC I-D at one time, the secretary does try to ensure that
    both are not large I-Ds.
 o  Once the assignments are made, the web pages that list the reviews
    and the assignments are posted.  Since the telechat agenda is not
    published until the end of the day on the Thursdays prior to the
    telechats (i.e., one week prior), the secretary needs to complete
    the assignments on that Thursday evening.  This often requires
    working later in the evening and also requires an Internet
    connection even when traveling.

Barnes, et al. Informational [Page 8] RFC 6385 Gen-ART October 2011

 o  If the reviewers notice any problems or conflict of interest, a
    bargaining process, shifting I-Ds from one reviewer to another,
    takes place.  The secretary updates the assignment files with any
    new assignments.
 o  Once the review has been completed, the reviewer sends the review
    to the Gen-ART list, ideally using the template provided in the
    review assignment emails.  Typically, reviews are also sent to
    authors, the responsible AD, and the WG chairs/document shepherd.
    The only case where this might not be done is when there are no
    issues found for a re-review and none had been found on an initial
    review.  Sending the review to the authors, ADs, and/or WG chairs/
    Proto Shepherds was originally voluntary but is now considered
    standard practice.  Reviewers may also send the reviews to the
    IETF discussion list, but that is entirely at the discretion of
    the reviewer, in which case the author must be copied on the
    review to ensure they see any follow-up discussion.  Reviewers may
    also send the comments to the WG; however, this typically causes
    the review to end up in the moderation queue, as most reviewers do
    not want to subscribe to the WG lists for the I-Ds they review.
    Thus, it is expected that any of the original recipients (i.e.,
    authors, WG chairs/document shepherd, or responsible AD) may
    forward the review to the WG mailing list if they believe it is
    necessary.  In the past, sending these reviews resulted in
    confusion among the authors, who may not have been expecting a
    Gen-ART review and may not be familiar with Gen-ART.  Thus,
    reviewers are reminded to prepend to the email the description of
    Gen-ART and the purpose of the review.  This information is part
    of the standard template provided in the review assignment emails.
 o  The secretary gathers the reviews, sometimes edits them for
    format, and records the review in the spreadsheet on the web
    pages, including the synopsis.  This is typically done on
    Thursday.  This is one aspect of the process that can be easily
    delegated such that one volunteer uploads all the reviews and then
    the secretary need only update the fields in the spreadsheet.  If
    the reviewer has not provided a synopsis ("Summary" field in the
    template), the secretary makes a best guess based on the review
    details.  Note that in most cases the reviewers do include a
    synopsis.
 o  Ideally, the reviews should be posted to the Gen-ART mailing list
    by close of business on the East coast on Tuesday.  This is
    necessary to allow the General AD time to consider the reviews
    prior to the telechat.  If the reviews are received after Tuesday,
    the review may not be read by the AD before the IESG telechat.
    Due to time constraints, the spreadsheets containing review
    summaries/assignments are only updated on Thursday evenings when

Barnes, et al. Informational [Page 9] RFC 6385 Gen-ART October 2011

    the new LC assignments and upcoming telechat assignments are done.
    Ideally, the reviews would get uploaded on the Tuesdays prior to
    the telechat along with the updated spreadsheets.
 o  If the AD concludes that the concerns raised by the reviewer
    warrant placing a DISCUSS comment on the I-D, the AD will do so,
    and the DISCUSS must be resolved before the I-D advances.
    Usually, the reviewer will be involved in the resolution process,
    but the responsibility for the DISCUSS rests with the AD.

5. Secretarial Process

 This section summarizes the details of managing the review materials,
 including the spreadsheet used to track all reviews and the HTML
 files containing the review assignments.  Please note that these
 details represent a snapshot of a process that has been implemented
 -- the details are very likely to change over time, in particular as
 the needed improvements highlighted in Section 8 are carried out.

5.1. Maintaining Review Spreadsheet

 A spreadsheet is used to enter all the I-Ds at the time of assignment
 and to capture all the reviews.  For IETF LC assignments, the
 assignments are completed before adding the I-Ds to the spreadsheet
 as described in Section 5.2.  For telechat assignments, I-Ds are
 obviously only added in the cases where there is no previous LC
 assignment.  For the other I-Ds, the appropriate fields are updated
 as described in Section 5.3.
 All the reviews can be accessed from the spreadsheet via hyperlinks
 from specific fields, as summarized below.  The following information
 is maintained in the spreadsheet (in the order listed):
 1.  "Chat/LC Date": Indicates either the date on which the LC review
     is due or the date of the telechat.
 2.  "Document": Filename for the I-D, which includes a hyperlink to
     the IETF I-D tracker.
 3.  "Assigned": Name of the reviewer assigned to that I-D.
 4.  "Category": This field contains one of the following self-
     explanatory values: "Doc - WG", "Doc - Ind/AD", or "IETF LC".
     Note that Gen-ART does not review I-Ds submitted directly to the
     RFC Editor.  The "IETF LC" value is of course entered for all
     I-Ds at LC time.  It is changed to one of the other appropriate
     values, based on the information in the telechat agenda.

Barnes, et al. Informational [Page 10] RFC 6385 Gen-ART October 2011

 5.  "Previous Review": This includes a link to any previous reviews.
     For example, when a doc appears on a telechat agenda, if an IETF
     LC review was done, this field is updated with the review summary
     for the LC review (i.e., the information from the "Current Review
     Summary" as described below is copied to this column).  The field
     is set to "New" when an I-D is first assigned/added to the
     spreadsheet.  In the case of returns, this field has a value of
     "Return" or "Return/IETF-LC" for I-Ds for which there is an LC
     review.  It should be noted that since Gen-ART started doing
     reviews at LC time, there seem to be far fewer returns on the
     agenda.
 6.  "Current Review Summary": This field includes the review type and
     version number of the document that is to be reviewed or has been
     reviewed (e.g., LC: -02).  When the field also contains a review
     summary after the review type/version number (e.g., Telechat: -06
     Ready), the active hyperlink points to the review.  Occasionally,
     a reviewer will re-review an I-D prior to its telechat
     assignment, in which case it is added to the spreadsheet, but the
     date does not change to maintain consistency in the date field,
     since the reviews themselves contain the review date.
 The following summarizes the steps to add a new I-D to the
 spreadsheet:
 1.  In order to optimize steps, blank rows are first inserted for the
     number of new I-Ds to be added.
 2.  To minimize data entry, a row with default fields (including the
     hyperlinks) is kept at the end of the file.  There is a separate
     default row for IETF LC versus telechat assignments.  This row is
     copied into each of the new blank rows.  The dates are then
     entered (this allows double-checking that all I-Ds from the
     review assignments are accounted for, especially LC).
 3.  The I-D name is then copied to the name field as well as being
     appended to the hyperlink for the "Review Summary" field.  The
     hyperlink is included as part of the default row.  This minimizes
     the steps to enter the reviews in the spreadsheet.
 4.  The data is also sorted by "Chat/LC Date", "Assigned", and
     "Document".  The file is then saved and closed.
 5.  The file is then reopened and saved as HTML.
 6.  The file is opened a second time and sorted by "Assigned",
     "Chat/LC Date", and "Document" to provide the I-D reviewers an
     easy way to find any outstanding assignments.

Barnes, et al. Informational [Page 11] RFC 6385 Gen-ART October 2011

5.2. Last Call Assignment Procedure

 The secretary can cache the Last Call assignments as they are
 announced and/or check the IETF announcement mailing list archives.
 The assignments are done on Thursday evening, along with any telechat
 assignments.  This optimizes the process in terms of batch changes to
 files.
 The assignments are listed in an HTML file.  The following are the
 steps in creating that file:
 1.  The order of assignment is actually created the week before, with
     the details below.  Thus, before starting the new assignments,
     the current file is saved for editing for the following week.
     The current file-naming convention is "reviewersyymmdd-lc.html"
     (e.g., for July 8th, 2010, the file reviewers100708-lc.html was
     created, and the file for the following week is named
     reviewers100715-lc.html).
 2.  Since the file is already prepared with the appropriate ordering
     of reviewers, the assignments are done in the order of due dates.
     The link to the I-D in the Datatracker is copied into the
     assignment file along with the intended RFC status for each of
     the new LC I-Ds.
 3.  The "Due Date" paragraph from the Last Call announcement is
     shortened as follows: "IETF LC ends on:", keeping the date.
 4.  Once the assignment file is complete, the new I-Ds are added to
     the spreadsheet as described above.
 5.  The assignment file for the next week is then updated to reflect
     the next reviewer in the round-robin process, by simply cutting
     and pasting the names in the list in a block and removing any
     "one doc per month" reviewers (annotated with an "*") that have
     already received their monthly assignment.  If the next round of
     assignments occurs at the beginning of a new month, the "one doc
     per month" reviewers are added back into the list (in the normal
     order -- alphabetically by first name).
 6.  The assignment files and updated spreadsheets are then cached on
     the Gen-ART server.
 7.  An email providing a link to the assignment file along with the
     updated spreadsheets is sent to the Gen-ART mailing list.  This
     email has a standard form, such that the reviewers can simply cut
     and paste the template to include the Gen-ART context statement
     and link to the FAQ.

Barnes, et al. Informational [Page 12] RFC 6385 Gen-ART October 2011

5.3. Telechat Assignment Procedure

 Since LC assignments are now the starting point for Gen-ART I-D
 reviews, the telechat assignments are generally straightforward, as
 the majority of the I-Ds are already in the spreadsheet.  The
 following details the steps:
 1.  The telechat agenda is typically available around 6PM PDT.  In
     order to create the assignment HTML file, the agenda is created
     from the email announcing the upcoming telechat agenda.  The
     filename has the following format, with the date corresponding to
     the telechat date (versus the date of assignment, as is the case
     for Last Call assignments): "reviewersyymmdd.html".
 2.  Rows are added to the agenda for the reviewers' names.
 3.  The reviewers' names are then added to the weekly assignment
     file.
 4.  As each reviewer is added to the assignment file, the review
     spreadsheet is updated as follows:
  • "Chat/LC Date" is changed to the telechat date.
  • The link to the LC review, if available, is copied as the link

for the "Previous Review" column.

  • The "text" for the "Current Review" is updated to reflect the

new review type (i.e., Telechat) and version.

 5.  In the case of an I-D that did not go through IETF LC, a reviewer
     is assigned using the order in the file to be used for Last Call
     assignments for the next week.
 6.  Once the reviewer(s) have been determined, the LC assignment file
     for the next week is updated.
 7.  Any new I-Ds are then added to the spreadsheet (and the updates
     saved) per the steps described in Section 5.1.
 8.  The assignment files and updated spreadsheets are then cached on
     the Gen-ART server.
 9.  An email providing a link to the assignment file along with the
     updated spreadsheets is sent to the Gen-ART mailing list.  This
     email has a standard form, such that the reviewers can simply cut
     and paste the template to include the Gen-ART context statement
     and link to the FAQ.

Barnes, et al. Informational [Page 13] RFC 6385 Gen-ART October 2011

5.4. Capturing Reviews

 As noted in Section 4.4, the spreadsheet is typically updated with
 the review summaries on Thursday evenings just prior to entering the
 data for that week's LC and any telechat assignments.  The following
 summarizes the steps to capture the reviews:
 1.  Currently, an additional volunteer is assisting the secretary in
     caching the email reviews as they arrive.
 2.  In the cases where the review is included inline in the body of
     the email, the review is cut and pasted into a text file and
     saved with the reviewer's last name appended to the filename --
     e.g., draft-ietf-xyz-00-smith.txt.
 3.  In the case where the review is included as an attachment to the
     email, the file can be directly saved and uploaded.
 4.  The volunteer uploads the reviews by around 5PM CST on Thursdays;
     thus, they are available to the secretary at the time that week's
     assignments are done.  This sequence is necessary to ensure the
     information for I-Ds on the upcoming telechat is up to date.
 5.  The review summary is entered into the "Current Summary" field.
     Note that the hyperlink to the review (added at assignment time)
     will automatically work when the file is uploaded.
 6.  Once all the reviews have been entered and the spreadsheets
     formatted, the review spreadsheet is saved and files uploaded per
     the last three steps in Section 5.1.

6. Results

 Over the past 7 years, the Gen-ART has provided reviewing services to
 3 ADs and has done around two thousand publicly available reviews.
 The reviews have been executed with a team of around a dozen full-
 time reviewers and other reviewers receiving one I-D assignment each
 month.  There are currently 9 reviewers in the latter category.  The
 full-time reviewers receive 2-3 assignments each month.  In terms of
 improving quality, the number of I-Ds that are now "Ready" at the
 time of the telechat (since the reviews are now initiated at LC time)
 has increased.  The review term "Ready" means the reviewer believes
 that the document has no outstanding editorial or technical issues.
 Based on the data from 2007, there were over 250 I-Ds assigned at LC
 time that went through IESG review.  Of those 250 I-Ds, 82% of the LC
 reviews (205 I-Ds) were completed.  Of the completed reviews, 70%
 (144 I-Ds) were "Ready" at the time of the telechat.  Of those 144
 I-Ds, roughly 1/4 had been deemed "Ready" (with no nits) at LC time

Barnes, et al. Informational [Page 14] RFC 6385 Gen-ART October 2011

 (based on a sample of 50 reviews).  For the I-Ds that were not
 reviewed at LC time, only about 1/4 of those were deemed "Ready" when
 they were reviewed for the telechat.  So, doing the Gen-ART reviews
 at Last Call time does seem to improve the quality of the I-Ds for
 the telechat.

7. Impressions

 This section is divided into 3 subsections: the impressions as
 gathered from the Gen-ART, the impressions of the ADs for whom they
 worked, and the impressions of the secretaries of Gen-ART.

7.1. Reviewers' Impressions

 The following list of comments are excerpted and edited from comments
 sent in by the reviewers of Gen-ART in response to the request:
 "We'd like to ask you each to write a few lines about your personal
 experience and lessons learned as a Gen-ART reviewer".
 o  We really do find problems, but we don't find problems with
    most I-Ds.
 o  Comments seem to be in three areas: editorial/grammar, editorial/
    what-the-heck-does-this-mean, and actual problems.  I'm seeing
    fewer reviews in the first category, which is a good thing.
 o  It is becoming rarer that we hear back "these guys have suffered
    enough; I'm voting no objection" (I'm remembering an LDAP I-D that
    had been around so long it had 2119 referenced AS A DRAFT -- some
    people suffered a lot).
 o  The direct assignment of reviews is necessary and effective.  It
    does not matter much as far as I can tell what scheme is used to
    actually do the assignment.
 o  Folks are very open to the reviews that come out of Gen-ART.  This
    somewhat surprised me, because I have seen resistance to outside
    reviews in other cases.
 o  The improvements that have come about (for example, one of my
    latest, an I-D about the SIPPING conference) have made a big
    difference to the comprehensibility and usability of the I-Ds --
    and provide a useful incentive to keep going.
 o  Some form of review like this is desperately needed.  While most
    of the stuff we see is good, every once in a while really bad
    errors have made their way all the way to IESG vote.

Barnes, et al. Informational [Page 15] RFC 6385 Gen-ART October 2011

 o  Reading this stuff is interesting.  I like having a reason to read
    a wide range of materials.
 o  I am more than convinced that this can be, and is, a valuable
    process.  It is, in my opinion, a pity that Senior IETF Reviewers
    (SIRS) and so on did not take off, because this late-stage
    reviewing is a poor substitute for doing the same thing at a much
    earlier stage.  Very few of the drafts that have come past my
    screen are truly fully ready for IESG review.  It is actually a
    joy to find the occasional nugget that is both well written and is
    a proper technical job, such that the reviewer really can say
    "This is ready".
 o  I have certainly found the process intellectually stimulating!  It
    encourages me to take a wider interest in what is going on in the
    IETF, but consumes a fair bit of time to do a proper job, and
    requires a very wide knowledge to be able to properly catch the
    cross-area implications: I hope (believe!) that this is something
    that one gets better at with experience and doing a few of these
    reviews.
 o  There is probably a very limited pool of people who have both the
    time and the inclination to keep on doing these reviews.  It does
    require a fair bit of dedication.
 o  It is difficult to avoid correcting the English, even if that is
    not really the point: Often, really bad English (whether as a
    result of non-mother-tongue authors with limited grasp or mother-
    tongue authors using informal language) obscures/corrupts what is
    being said or just makes it impossible to read.
 o  Mostly authors welcome the comments: I think most of them
    understand the concept of "ego-free reviewing", and we have
    generally been constructive rather than destructive.
 o  Part of the job of Gen-ART is to think the unthinkable from
    another point of view, to challenge (apparently undocumented)
    assumptions, and apply experience from other fields.

Barnes, et al. Informational [Page 16] RFC 6385 Gen-ART October 2011

7.2. General Area Directors' Impressions

 It should be noted that these impressions are from multiple General
 Area Directors; thus, the "I"s are not necessarily associated with a
 specific AD.
 o  It's essential.  The reviewing load for the IESG <shout>DOES NOT
    SCALE</shout>.
 o  Without Gen-ART, I would be a much less effective General AD.
 o  On a single fortnight example, the IESG had 21 drafts on the
    agenda.  It is just impossible (to conscientiously review all the
    documents), and no wonder we sometimes miss serious issues.
 o  So I think a distributed review team with about 30 trusted
    reviewers needs to be institutionalized.  I suspect that will need
    to be formalized in a BCP sooner or later -- with their reviews
    having a formal position in the standards process, and the
    expectation that the whole IESG truly reviews all I-Ds being
    relaxed.
 o  We've learned that polite, well reasoned, constructive reviews are
    very positively received by authors and WGs.  Dismissive reviews
    are counter-productive.  And reviews sent in private eventually
    show up in public, so it's better to go public at the start.
 o  Normally, LC reviews are available in good time for the draft to
    be revised before reaching the IESG agenda.  It is important that
    this happens, except for an emergency situation where the
    responsible AD has good reason to place the draft on the agenda
    immediately.  In that case, it would be preferable for the AD to
    inform the Gen-ART, so that the review can be expedited.
 o  The other problem is a big detail -- between late Thursday or
    early Friday when the secretary sends out the assignments, and
    Wednesday when the General AD likes to start filling in ballots
    based on the reviews received by close of business on Tuesday,
    there are only three work days (plus possible volunteer time over
    the weekend).  Now even with only one I-D to review, that may be a
    real challenge.  Sometimes, a lucky reviewer will get 130 pages
    (e.g., draft-ietf-nntpext-base-27).  That doesn't compute.

Barnes, et al. Informational [Page 17] RFC 6385 Gen-ART October 2011

 o  There are some mechanical issues.  The process followed is far too
    manual.  Everything needs to be robotic except for the judgment
    calls about which reviewer gets which draft.  Similarly, the
    reviewer should be able to just paste the review into a web form,
    click, and it's sent off to everyone appropriate and posted to the
    review site.

7.3. Gen-ART Secretaries' Impressions

 Serving as the secretary of Gen-ART is a worthwhile experience.  From
 a personal point of view, it gives the secretary an easy way to track
 all of the work going through the IESG review process and see how the
 work flowed through that process.  Also, by reviewing and sometimes
 creating the one-line abstracts that go on the review web page, the
 secretary has an opportunity to really get a survey of the work being
 approved by the IETF.
 The nature of these reviews is informal, and originally the reviews
 were only intended for the General AD, though they were made public.
 During 2004, there was little if any interaction between authors and
 reviewers.  There was some discussion during 2004 about trying to
 expand the role of Gen-ART to a more formal, early-review model,
 i.e., to evolve it into a form of SIRS.  The original Gen-ART
 secretary was against such a transformation, because she felt it
 would put at risk something that worked.  She believed that there
 were risks inherent in formalizing the reviews and adding mechanisms
 for standardizing the resultant review process.  Another concern
 involves the interaction between reviewers and authors.  As discussed
 above, it has become the practice to send reviews to the authors with
 an explanation about the nature of Gen-ART reviews.  While it is
 clear that this has resulted in improved RFCs, it has also resulted
 in increased workload for the reviewers.
 The secretary thinks that Gen-ART is an experiment that works well,
 but she also believes it is fragile.  The secretary is often
 concerned about overburdening reviewers, and feels it is her
 responsibility to keep them from burning out.  Adding additional
 reviewers to the review team would help to alleviate this concern.
 In terms of the process, adding additional reviewers has minimal
 impact.

8. Needed Improvements

 The current size of the review team introduces a fairly heavy
 workload for the individual reviewers that are not on the "one doc
 per month" assignment cycle.  Additional reviewers would be really
 helpful to alleviate this workload.  It is also important to note
 that having additional reviewers adds minimal workload to the

Barnes, et al. Informational [Page 18] RFC 6385 Gen-ART October 2011

 secretary's process; thus, the only blocking point is finding the
 right folks that are interested in this type of volunteer role.  As
 noted in Section 7.2, 30 would be a good size for the review team.
 This would cut the workload for an individual reviewer in half (given
 the current model of 9 reviewers on the "one doc per month"
 assignment cycle).
 Obviously, automation of the process would be a good thing.  However,
 Gen-ART secretaries are not necessarily highly motivated to
 transition to a more automated approach until a significant part of
 the process is automated.  In more recent consideration of this
 situation, it likely would be best to first automate the process of
 entering the reviews, as that benefits the review team as a whole.
 This automation should allow the reviewers to enter the reviews via a
 web interface that would automatically generate the appropriate
 emails -- quite similar to how the draft "Upload" tool currently
 works.  Also, given consistent naming conventions for the review
 forms, this step would automate some of the process for the
 secretary, as the reviews would automatically appear via the
 spreadsheet hyperlinks, although there would still be a need to
 manually enter the summary.  But this would eliminate the need to
 edit/normalize and upload files and, hopefully, eliminate the problem
 encountered with unflowed text in emails and getting the review
 properly formatted using some text editors.
 Section 5 was written to facilitate the process of determining tools
 requirements, by providing the very detailed steps currently applied
 to the process.  As noted above, automating the upload of the reviews
 could be a good first step.  This is somewhat starting at the end of
 the process.  However, it seems that by automating in this direction,
 we may have optimal results; since one of the earliest steps in the
 process is the task of assigning reviewers, it likely needs the most
 manual intervention, even with tools available.
 The current SecDir secretary does use some tools for assignments and
 generating assignment emails.  These tools could be considered for
 use by the Gen-ART secretary.  Since the SecDir reviews are not
 cached and the information maintained for those reviews is less
 detailed, there would be no reusability of that aspect.  However, if
 the Gen-ART spreadsheet can be automatically populated (with
 assignments and completed reviews), the SecDir may be able to make
 use of that same tool.

Barnes, et al. Informational [Page 19] RFC 6385 Gen-ART October 2011

9. Applicability

 As implemented today, the process has no formal role in the IETF
 standards process.  But as trust in the review team has built, and as
 the team itself has learned to deliver reviews that are generally
 well received, they have had a significant impact on I-D quality and
 on timeliness.  Rather than becoming a roadblock, they have (in
 general) allowed the General AD to feel more confident in reaching
 decisions and be more precise in resolving issues.  Since reviews now
 typically appear during IETF Last Call, the reviews, like the SecDir
 reviews, are now generally expected.  So, the role of the team has
 evolved to be more formal than in the past (i.e., when this document
 was first drafted in 2005).  However, the handling of the reviews
 remains entirely within the scope of the ADs, document shepherds, WG
 chairs, and authors as they deem appropriate.

10. Security Considerations

 Since this is an informational I-D about an open process, the
 security considerations are specific to the process and users
 involved in the process.  The primary concern would be to limit the
 people that have the ability to create and update the Gen-ART data/
 files to ensure that the integrity of the data is maintained.  For
 example, each Gen-ART reviewer should have a unique user name/
 password, just as folks do to access any other IETF-maintained data,
 as appropriate.

11. Acknowledgments

 Initial comments were received from the members of the Gen-ART, and
 the experiences discussed in this document were derived from their
 hard work over the last 7+ years.  We thank the past reviewers of the
 Gen-ART:
    Mark Allman
    Harald Alvestrand (creator of Gen-ART)
    Ron Bonica
    Scott Brim
    Gonzalo Camarillo
    Sharon Chisholm
    Spencer Dawkins
    Lakshminath Dondeti
    Avri Doria (past secretary)
    Pasi Eronen
    Dorothy Gellert
    Eric Gray
    Avashalom Houri
    Glenn Kowack

Barnes, et al. Informational [Page 20] RFC 6385 Gen-ART October 2011

    John Loughney
    Lucy Lynch
    Enrico Marocco
    Michael Patton
    Stefan Santesson
    Robert Sparks
    Tom Taylor
    Sean Turner
    Christian Vogt
    Suzanne Woolf
 We also thank the current team of reviewers/secretary:
    Mary Barnes (past secretary, 2005-2010)
    Richard Barnes
    David Black
    Ben Campbell
    Brian Carpenter (past General AD)
    Elwyn Davies
    Francis Dupont
    Roni Even
    Miguel-Angel Garcia
    Vijay Gurbani (assisting secretary to upload reviews)
    Wassim Haddad
    Joel Halpern
    Suresh Krishnan
    Peter McCann
    Jean Mahoney (secretary as of Jan. 2011)
    Alexey Melnikov
    Kathleen Moriarty

12. Informative References

 [RFC1958]     Carpenter, B., Ed., "Architectural Principles of the
               Internet", RFC 1958, June 1996.
 [RFC2223bis]  Reynolds, J., Ed., and R. Braden, Ed., "Instructions to
               Request for Comments (RFC) Authors", Work in Progress,
               August 2004.
 [RFC-STYLE]   Braden, R., Ginoza, S., and A. Hagens, "RFC Document
               Style", September 2009,
               <http://www.rfc-editor.org/rfc-style-guide/rfc-style>.
 [RFC3426]     Floyd, S., "General Architectural and Policy
               Considerations", RFC 3426, November 2002.

Barnes, et al. Informational [Page 21] RFC 6385 Gen-ART October 2011

 [RFC3439]     Bush, R. and D. Meyer, "Some Internet Architectural
               Guidelines and Philosophy", RFC 3439, December 2002.
 [RFC3552]     Rescorla, E. and B. Korver, "Guidelines for Writing RFC
               Text on Security Considerations", BCP 72, RFC 3552,
               July 2003.
 [RFC5226]     Narten, T. and H. Alvestrand, "Guidelines for Writing
               an IANA Considerations Section in RFCs", BCP 26,
               RFC 5226, May 2008.
 [SIRS]        Carpenter, B. and D. Crocker, "Careful Additional
               Review of Documents (CARD) by Senior IETF Reviewers
               (SIRS)", Work in Progress, June 2003.

Barnes, et al. Informational [Page 22] RFC 6385 Gen-ART October 2011

Authors' Addresses

 Mary Barnes
 Polycom
 TX
 USA
 EMail: mary.ietf.barnes@gmail.com
 Avri Doria
 Research Consultant
 Providence, RI
 USA
 EMail: avri@acm.org
 Harald Alvestrand
 Google
 Kungsbron 2
 11122 Stockholm
 SE
 EMail: harald@alvestrand.no
 Brian E. Carpenter
 University of Auckland
 PB 92019
 Auckland, 1142
 New Zealand
 Phone:
 EMail: brian.e.carpenter@gmail.com

Barnes, et al. Informational [Page 23]

/data/webs/external/dokuwiki/data/pages/rfc/rfc6385.txt · Last modified: 2011/10/18 18:23 by 127.0.0.1

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki