GENWiki

Premier IT Outsourcing and Support Services within the UK

User Tools

Site Tools


rfc:rfc5047

Network Working Group M. Chadalapaka Request for Comments: 5047 HP Category: Informational J. Hufferd

                                                          Brocade Inc.
                                                             J. Satran
                                                                   IBM
                                                               H. Shah
                                                  Broadcom Corporation
                                                          October 2007
                  DA: Datamover Architecture for
       the Internet Small Computer System Interface (iSCSI)

Status of This Memo

 This memo provides information for the Internet community.  It does
 not specify an Internet standard of any kind.  Distribution of this
 memo is unlimited.

Abstract

 The Internet Small Computer System Interface (iSCSI) is a SCSI
 transport protocol that maps the SCSI family of application protocols
 onto TCP/IP.  Datamover Architecture for iSCSI (DA) defines an
 abstract model in which the movement of data between iSCSI end nodes
 is logically separated from the rest of the iSCSI protocol in order
 to allow iSCSI to adapt to innovations available in new IP
 transports.  While DA defines the architectural functions required of
 the class of Datamover protocols, it does not define any specific
 Datamover protocols.  Each such Datamover protocol, defined in a
 separate document, provides a reliable transport for all iSCSI PDUs,
 but actually moves the data required for certain iSCSI PDUs without
 involving the remote iSCSI layer itself.  This document begins with
 an introduction of a few new abstractions, defines a layered
 architecture for iSCSI and Datamover protocols, and then models the
 interactions within an iSCSI end node between the iSCSI layer and the
 Datamover layer that happen in order to transparently perform remote
 data movement within an IP fabric.  It is intended that this
 definition will help map iSCSI to generic Remote Direct Memory Access
 (RDMA)-capable IP fabrics in the future comprising TCP, the Stream
 Control Transmission Protocol (SCTP), and possibly other underlying
 network transport layers, such as InfiniBand.

Chadalapaka, et al. Informational [Page 1] RFC 5047 DA October 2007

Table of Contents

 1. Motivation ......................................................4
    1.1. Intent .....................................................4
    1.2. Interpretation of Requirements .............................5
 2. Definitions and Acronyms ........................................5
    2.1. Definitions ................................................5
    2.2. Acronyms ...................................................6
 3. Architectural Layering of iSCSI and Datamover Layers ............7
 4. Design Overview .................................................9
 5. Architectural Concepts .........................................10
    5.1. iSCSI PDU Types ...........................................10
         5.1.1. iSCSI Data-Type PDUs ...............................10
         5.1.2. iSCSI Control-Type PDUs ............................11
    5.2. Data_Descriptor ...........................................11
    5.3. Connection_Handle .........................................11
    5.4. Operational Primitive .....................................12
    5.5. Transport Connection ......................................13
 6. Datamover Layer and Datamover Protocol .........................13
 7. Functional Overview ............................................14
    7.1. Startup ...................................................14
    7.2. Full Feature Phase ........................................15
    7.3. Wrap-up ...................................................15
 8. Operational Primitives Provided by the Datamover Layer .........16
    8.1. Send_Control ..............................................16
    8.2. Put_Data ..................................................17
    8.3. Get_Data ..................................................17
    8.4. Allocate_Connection_Resources .............................18
    8.5. Deallocate_Connection_Resources ...........................19
    8.6. Enable_Datamover ..........................................19
    8.7. Connection_Terminate ......................................20
    8.8. Notice_Key_Values .........................................20
    8.9. Deallocate_Task_Resources .................................20
 9. Operational Primitives Provided by the iSCSI Layer .............21
    9.1. Control_Notify ............................................21
    9.2. Connection_Terminate_Notify ...............................22
    9.3. Data_Completion_Notify ....................................22
    9.4. Data_ACK_Notify ...........................................23
 10. Datamover Interface (DI) ......................................23
    10.1. Overview .................................................23
    10.2. Interactions for Handling Asynchronous Notifications .....24
         10.2.1. Connection Termination ............................24
         10.2.2. Data Transfer Completion ..........................24
         10.2.3. Data Acknowledgement ..............................25
    10.3. Interactions for Sending an iSCSI PDU ....................25
         10.3.1. SCSI Command ......................................26
         10.3.2. SCSI Response .....................................26
         10.3.3. Task Management Function Request ..................26

Chadalapaka, et al. Informational [Page 2] RFC 5047 DA October 2007

         10.3.4. Task Management Function Response .................27
         10.3.5. SCSI Data-Out and SCSI Data-In ....................27
         10.3.6. Ready To Transfer (R2T) ...........................28
         10.3.7. Asynchronous Message ..............................28
         10.3.8. Text Request ......................................28
         10.3.9. Text Response .....................................28
         10.3.10. Login Request ....................................29
         10.3.11. Login Response ...................................29
         10.3.12. Logout Command ...................................29
         10.3.13. Logout Response ..................................30
         10.3.14. SNACK Request ....................................30
         10.3.15. Reject ...........................................30
         10.3.16. NOP-Out ..........................................30
         10.3.17. NOP-In ...........................................30
    10.4. Interactions for Receiving an iSCSI PDU ..................31
         10.4.1. General Control-Type PDU Notification .............31
         10.4.2. SCSI Data Transfer PDUs ...........................31
         10.4.3. Login Request .....................................32
         10.4.4. Login Response ....................................32
 11. Security Considerations .......................................33
    11.1. Architectural Considerations .............................33
    11.2. Wire Protocol Considerations .............................33
 12. References ....................................................34
    12.1. Normative References .....................................34
    12.2. Informative References ...................................34
 Appendix A. Design Considerations and Examples ....................35
    A.1. Design Considerations for a Datamover Protocol ............35
    A.2. Examples of Datamover Interactions ........................35
 Acknowledgements ..................................................44

Table of Figures

 Figure 1. Datamover Architecture Diagram, with the RDMAP Example ...8
 Figure 2. A Successful iSCSI Login on Initiator ...................37
 Figure 3. A Successful iSCSI Login on Target ......................37
 Figure 4. A Failed iSCSI Login on Initiator .......................38
 Figure 5. A Failed iSCSI Login on Target ..........................38
 Figure 6. iSCSI Does Not Enable the Datamover .....................39
 Figure 7. A Normal iSCSI Connection Termination ...................40
 Figure 8. An Abnormal iSCSI Connection Termination ................40
 Figure 9. A SCSI Write Data Transfer ..............................41
 Figure 10. A SCSI Read Data Transfer ..............................42
 Figure 11. A SCSI Read Data Acknowledgement .......................43
 Figure 12. Task Resource Cleanup on Abort .........................44

Chadalapaka, et al. Informational [Page 3] RFC 5047 DA October 2007

1. Motivation

1.1. Intent

 There are relatively new standard protocols that enable Remote Direct
 Memory Access (RDMA) and Remote Direct Data Placement (RDDP)
 technologies to work over IP fabrics.  The principal value
 proposition of these technologies is that they enable one end node to
 place data in the final intended buffer on the remote end node, thus
 eliminating the need for a receive path data copy that moves the data
 to its final location.  The data copy avoidance in turn eliminates
 unnecessary memory bandwidth consumption, substantially decreases the
 reassembly buffer size requirements, and preserves CPU cycles that
 would otherwise be spent in copying.
 The iSCSI specification [RFC3720] defines a very detailed data
 transfer model that employs SCSI Data-In PDUs, SCSI Data-Out PDUs,
 and R2T PDUs, in addition to the SCSI Command and SCSI Response PDUs
 that respectively create and conclude the task context for the data
 transfer.  In the traditional iSCSI model, the iSCSI protocol layer
 plays the central role in pacing the data transfer and carrying out
 the ensuing data transfer itself.  An alternative architecture would
 be for iSCSI to delegate a large part of this data transfer role to a
 separate protocol layer exclusively designed to move data, which in
 turn is possibly aided by a data movement and placement technology
 such as RDMA.
 If iSCSI were operating in such RDMA environments, iSCSI would be
 shielded from the low-level data transfer mechanics but would only be
 privy to the conclusion of the requested data transfer.  Thus, there
 would be an effective "off-loading" of the work that an iSCSI
 protocol layer is expected to perform, compared to today's iSCSI end
 nodes.  For such RDMA environments, it is highly desirable that there
 be a standard architecture to separate the data movement part of the
 iSCSI protocol definition from the rest of the iSCSI functionality.
 This architecture precisely defines what a Datamover layer is and
 also describes the model of interactions between the iSCSI layer and
 the Datamover layer (Section 6).  In order to satisfy this need, this
 document presents a Datamover Architecture for iSCSI (DA) and
 summarizes a reasonable model for interactions between the iSCSI
 layer and the Datamover layer for each of the iSCSI PDUs that are
 defined in [RFC3720].  Note that while DA is motivated by the advent
 of RDMA over TCP/IP technology, the architecture is not dependent on
 RDMA in its design.  DA is intended to be a generic architectural
 framework for allowing different types of Datamovers based on
 different types of RDMA and transport protocols.  Adoption of this
 model will help iSCSI proliferate into more environments.

Chadalapaka, et al. Informational [Page 4] RFC 5047 DA October 2007

1.2. Interpretation of Requirements

 This document introduces certain architectural abstractions and
 builds an abstract functional interface model between iSCSI and
 Datamover protocol layers based on those abstractions.  This
 architectural style is motivated by the following desires:
    a) Provide guidance to Datamover protocol designers with respect
       to the functional boundary between iSCSI and the Datamover
       protocols.  This guidance is critical since a significant part
       of the [RFC3720] protocol definition is left unchanged by DA
       architecture and the iSCSI notions from [RFC3720] (e.g., tasks,
       ITTs) are leveraged by the Datamover protocol.
    b) Aid existing iSCSI implementations to rapidly adapt to DA
       architecture, largely by leveraging the architectural
       abstractions into implementation constructs -- e.g., functions,
       APIs, modules.
 However, note that DA architecture does not intend to impose any
 implementation specifics per se.  When a DA architectural concept
 (e.g., Operational Primitive) is described as mandatory ("MUST") or
 recommended ("SHOULD") of a layer (iSCSI or Datamover) in this
 document, the intent is that an implementation respectively MUST or
 SHOULD produce the same protocol action as what the model describes.
 Specifically, no implementation compliance in terms of names, modules
 or API arguments etc. is implied by this Architecture by such use of
 [RFC2119] terms, only a functional compliance is sought.

2. Definitions and Acronyms

2.1. Definitions

 I/O Buffer - A buffer that is used in a SCSI Read or Write operation
    so that SCSI data may be sent from or received by the buffer.
 Datamover protocol  - A Datamover protocol is a data transfer wire
    protocol for iSCSI that meets the requirements stated in Section
    6.
 Datamover layer - A Datamover layer is a protocol layer within an end
    node that implements the Datamover protocol.
 Datamover-assisted - An iSCSI connection is said to be "Datamover-
    assisted" when a Datamover layer is enabled for moving control and
    data information on that iSCSI connection.

Chadalapaka, et al. Informational [Page 5] RFC 5047 DA October 2007

 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
 document are to be interpreted as described in [RFC2119].

2.2. Acronyms

 Acronym        Definition
 -------------------------------------------------------------
 DA             Datamover Architecture for iSCSI
 DDP            Direct Data Placement Protocol
 DI             Datamover Interface
 IANA           Internet Assigned Numbers Authority
 IETF           Internet Engineering Task Force
 I/O            Input - Output
 IP             Internet Protocol
 iSCSI          Internet SCSI
 iSER           iSCSI Extensions for RDMA
 ITT            Initiator Task Tag
 LO             Leading Only
 MPA            Marker PDU Aligned Framing for TCP
 PDU            Protocol Data Unit
 RDDP           Remote Direct Data Placement
 RDMA           Remote Direct Memory Access
 R2T            Ready To Transfer
 R2TSN          Ready To Transfer Sequence Number
 RDMA           Remote Direct Memory Access
 RDMAP          Remote Direct Memory Access Protocol
 RFC            Request For Comments

Chadalapaka, et al. Informational [Page 6] RFC 5047 DA October 2007

 SAM            SCSI Architecture Model
 SCSI           Small Computer Systems Interface
 SN             Sequence Number
 SNACK          Selective Negative Acknowledgment - also
                Sequence Number Acknowledgement for Data
 TCP            Transmission Control Protocol
 TTT            Target Transfer Tag

3. Architectural Layering of iSCSI and Datamover Layers

 Figure 1 illustrates an example of the architectural layering of
 iSCSI and Datamover layers, in conjunction with a TCP/IP
 implementation of RDMAP/DDP ([DDP]) layers in an iSCSI end node.
 Note that RDMAP/DDP/MPA and TCP protocol layers are shown here only
 as an example, and in reality, DA is completely oblivious to protocol
 layers below the Datamover layer.  The RDMAP/DDP/MPA protocol stack
 provides a generic transport service with direct data placement.
 There is no need to tailor the implementation of this protocol stack
 to the specific ULP to benefit from these services.

Chadalapaka, et al. Informational [Page 7] RFC 5047 DA October 2007

        Initiator stack                            Target stack
    +----------------+     SCSI application   +----------------+
    | SCSI Layer     |     protocols          | SCSI Layer     |
    +----------------+                        +----------------+
           ^                                          ^
           |                                          |
           v                                          v
    +----------------+     iSCSI protocol     +----------------+
    | iSCSI Layer    |    (excluding data     | iSCSI Layer    |
    +----------------+       movement)        +----------------+
           ^                                          ^
    --  ---+--  ---- DI (Datamover Interface)---  ----+---  ----
           v                                          v
    +----------------+      a Datamover       +----------------+
    | Datamover Layer|       protocol         | Datamover Layer|
    +----------------+                        +----------------+
           ^                                          ^
   +-------+----------+                     +---------+-----------+
   |       v          |                     |         v           |
   |+---------------+ |                     | +-----------------+ |
   || RDMAP/DDP/MPA | |    RDMAP/DDP/MPA    | | RDMAP/DDP/MPA   | |
   || Layers        | |    protocols        | | Layers          | |
   |+---------------+ |                     | +-----------------+ |
   |       ^          |                     |         ^           |
   |       | network  |                     |         | network   |
   |       | transport|                     |         | transport |
   |       v          |                     |         v           |
   |+---------------+ |                     | +----------------+  |
   || TCP Layer     | |    TCP protocol     | | TCP Layer      |  |
   |+---------------+ |                     | +----------------+  |
   |       ^          |                     |         ^           |
   +-------+----------+                     +---------+-----------+
           +------------------------------------------+
            Figure 1.  Datamover Architecture Diagram,
                      with the RDMAP Example
 The scope of this document is limited to:
    1. Defining the notion of a Datamover layer and a Datamover
       protocol (Section 6).
    2. Defining the functionality distribution between the iSCSI layer
       and the Datamover layer, along with the communication model
       between the two (Operational Primitives).

Chadalapaka, et al. Informational [Page 8] RFC 5047 DA October 2007

    3. Modeling the interactions between the blocks labeled as "iSCSI
       Layer" and "Datamover Layer" in Figure 1 -- i.e., defining the
       interface labeled "DI" in the figure -- for each defined iSCSI
       PDU, based on the Operational Primitives.

4. Design Overview

 This document discusses and defines a model for interactions between
 the iSCSI layer and a "Datamover layer" (see Section 6) operating
 within an iSCSI end node, presumably communicating with one or more
 iSCSI end nodes with similar layering.  The model for interactions
 for handling different iSCSI operations is called the "Datamover
 Interface" (DI, Section 10), while the architecture itself is called
 the "Datamover Architecture for iSCSI" (DA).  It is likely that the
 architecture will have implications on the Datamover wire protocols
 as DA places certain requirements and functionality expectations on
 the Datamover layer.  However, this document itself neither defines
 any new wire protocol for the Datamover layer, nor any potential
 modifications to the iSCSI wire protocol to employ the Datamover
 layer.  The scope of this document is strictly limited to specifying
 the architectural framework and the minimally required interactions
 that happen within an iSCSI end node to leverage the Datamover layer.
 The design ideas behind DA can be summarized as follows:
    1) DA defines an abstract functional interface model of the iSCSI
       layer's interactions with a Datamover layer below -- i.e., DA
       models the interactions between the logical "bottom" interface
       of iSCSI and the logical "top" interface of a Datamover.
    2) DA guides the wire protocol for a Datamover layer by defining
       the iSCSI knowledge that the Datamover layer may utilize in its
       protocol definition (as an example, this document completely
       limits the notion of "iSCSI session" to the iSCSI layer).
    3) DA is designed to allow implementation of the Datamover layer
       either in hardware or in software.
    4) DA is not a wire protocol spec, but an architecture that also
       models the interactions between iSCSI and Datamover layers
       operating within an iSCSI end node.
    5) DA by design seeks to model the iSCSI-Datamover interactions in
       a way that the modeling is independent of the specifics of
       either a particular iSCSI revision or an instantiation of a
       Datamover layer.

Chadalapaka, et al. Informational [Page 9] RFC 5047 DA October 2007

    6) DA introduces and relies on the notion of a defined set of
       Operational Primitives (could be seen as entry point
       definitions in implementation terms) provided by each layer to
       the other to carry out the request-response interactions.
    7) DA is intended to allow Datamover protocol definitions with
       minimal changes to existing iSCSI implementations.
    8) DA is designed to allow the iSCSI layer to completely rely on
       the Datamover layer for all data transport needs.
    9) DA models the architecturally required minimal interactions
       between an operational iSCSI layer and a Datamover layer to
       realize the iSCSI-transparent data movement.  There may be
       several other interactions in a typical implementation in order
       to bootstrap a Datamover layer (or an iSCSI layer) into
       operation, but they are outside the scope of this document.
 Note that in summary, DA is architected to support many different
 Datamover protocols operating under the iSCSI layer.  One such
 example of a Datamover protocol is iSER [iSER].

5. Architectural Concepts

5.1. iSCSI PDU Types

 This section defines the iSCSI PDU classification terminology, as
 defined and used in this document.  Out of the set of legal iSCSI
 PDUs defined in [RFC3720], as we will see in Section 5.1.1, the iSCSI
 layer does not request a SCSI Data-Out PDU carrying solicited data
 for transmission across the Datamover Interface per this
 architecture.  For this reason, the SCSI Data-Out PDU carrying
 solicited data is excluded in the iSCSI PDU classification we
 introduce in this section (for SCSI Data-Out PDUs for unsolicited
 Data, see Section 5.1.2).  The rest of the legal iSCSI PDUs that may
 be exchanged across the Datamover Interface are defined to consist of
 two classes:
    1) iSCSI data-type PDUs
    2) iSCSI control-type PDUs

5.1.1. iSCSI Data-Type PDUs

 An iSCSI data-type PDU is defined as an iSCSI PDU that causes data
 transfer, transparent to the remote iSCSI layer, to take place
 between the peer iSCSI nodes on a Full Feature Phase iSCSI
 connection.  A data-type PDU, when requested for transmission by the

Chadalapaka, et al. Informational [Page 10] RFC 5047 DA October 2007

 sender iSCSI layer, results in the associated data transfer without
 the participation of the remote iSCSI layer, i.e., the PDU itself is
 not delivered as-is to the remote iSCSI layer.  The following iSCSI
 PDUs constitute the set of iSCSI data-type PDUs:
    1) SCSI Data-In PDU
    2) R2T PDU
 In an iSCSI end node structured as an iSCSI layer and a Datamover
 layer as defined in this document, the solicitation for Data-Out
 (i.e., R2T PDU) is not delivered to the initiator iSCSI layer, per
 the definition of an iSCSI data-type PDU.  The data transfer is
 instead performed via the mechanisms known to the Datamover layer
 (e.g., RDMA Read).  This in turn implies that a SCSI Data-Out PDU for
 solicited data is never requested for transmission across the
 Datamover Interface at the initiator.

5.1.2. iSCSI Control-Type PDUs

 Any iSCSI PDU that is not an iSCSI data-type PDU and also not a
 solicited SCSI Data-Out PDU is defined as an iSCSI control-type PDU.
 Specifically, note that SCSI Data-Out PDUs for unsolicited Data are
 defined as iSCSI control-type PDUs.

5.2. Data_Descriptor

 A Data_Descriptor is an information element that describes an
 iSCSI/SCSI data buffer, provided by the iSCSI layer to its local
 Datamover layer or provided by the Datamover layer to its local iSCSI
 layer for identifying the data associated respectively with the
 requested or completed operation.
 In implementation terms, a Data_Descriptor may be a scatter-gather
 list describing a local buffer, the exact structure of which is
 subject to the constraints imposed by the operating environment on
 the local iSCSI node.

5.3. Connection_Handle

 A Connection_Handle is an information element that identifies the
 particular iSCSI connection for which an inbound or outbound iSCSI
 PDU is intended.  A connection handle is unique for a given pair of
 an iSCSI layer instance and a Datamover layer instance.  The
 Connection_Handle qualifier is used in all invocations of any
 Operational Primitive for connection identification.

Chadalapaka, et al. Informational [Page 11] RFC 5047 DA October 2007

 Note that the Connection_Handle is conceptually different from the
 Connection Identifier (CID) defined by the iSCSI specification.
 While the CID is a unique identifier of an iSCSI connection within an
 iSCSI session, the uniqueness of the Connection_Handle extends to the
 entire iSCSI layer instance coupled with the Datamover layer
 instance, across possibly multiple iSCSI sessions.
 In implementation terms, a Connection_Handle could be an opaque
 identifier exchanged between the iSCSI layer and the Datamover layer
 at the connection login time.  One may also consider it to be similar
 in scope of uniqueness to a socket identifier.  The exact structure
 and modalities of exchange of a Connection_Handle between the two
 layers is implementation-specific.

5.4. Operational Primitive

 An Operational Primitive, in this document, is an abstract functional
 interface procedure that requests another layer perform a specific
 action on the requestor's behalf or notifies the other layer of some
 event.  The Datamover Interface between an iSCSI layer instance and a
 Datamover layer instance within an iSCSI end node uses a set of
 Operational Primitives to define the functional interface between the
 two layers.  Note that not every invocation of an Operational
 Primitive may elicit a response from the requested layer.  This
 document describes the types of Operational Primitives that are
 implicitly required and provided by the iSCSI protocol layer as
 defined in [RFC3720], and the semantics of these Primitives.
 Note that ownership of buffers and data structures is likely to be
 exchanged between the iSCSI layer and its local Datamover layer in
 invoking the Operational Primitives defined in this architecture.
 The buffer management details, including how buffers are allocated
 and released, are implementation-specific and thus are outside the
 scope of this document.
 Each Operational Primitive invocation needs a certain "information
 context" (e.g., Connection_Handle) for performing the specific action
 being requested.  The required information context is described in
 this document by a listing of "qualifiers" on each invocation, in the
 style of function call arguments.  There is no specific
 implementation implied in this notation.  The "qualifiers" of any
 Operational Primitive invocation specified in this document thus
 represent the mandatory information context that the Operational
 Primitive invocation MUST consider in performing the action.  While
 the qualifiers are required, the method of realizing the qualifiers
 (passed synchronously with invocation, or retrieved from task
 context, or retrieved from shared memory etc.) is really up to the
 implementations.

Chadalapaka, et al. Informational [Page 12] RFC 5047 DA October 2007

 When an Operational Primitive implementation is described as
 mandatory ("MUST") or recommended ("SHOULD") of a layer (iSCSI or
 Datamover) in this document, the intent is that an implementation
 respectively MUST or SHOULD produce the same protocol action as what
 the model describes.

5.5. Transport Connection

 The term "Transport Connection" is used in this document as a generic
 term to represent the end-to-end logical connection as defined by the
 underlying reliable transport protocol.  For this document, all
 instances of Transport Connection refer to a TCP connection.

6. Datamover Layer and Datamover Protocol

 This section introduces the notion of a "Datamover layer" and
 "Datamover protocol" as meant in this document, and defines the
 requirements on a Datamover protocol.
 A Datamover layer is the implementation component that realizes a
 Datamover protocol functionality in an iSCSI-capable end node in
 communicating with other iSCSI end nodes with similar capabilities.
 More specifically, a "Datamover layer" MUST provide the following
 functionality and the "Datamover protocol" MUST consist of the wire
 protocol required to realize the following functionality:
    1) guarantee that all the necessary data transfers take place when
       the local iSCSI layer requests transmitting a command (in order
       to complete a SCSI command, for an initiator), or
       sending/receiving an iSCSI data sequence (in order to complete
       part of a SCSI command for a target).
    2) transport an iSCSI control-type PDU as-is to the peer Datamover
       layer when requested to do so by the local iSCSI layer.
    3) provide notification and delivery to the iSCSI layer upon
       arrival of an iSCSI control-type PDU.
    4) provide an initiator-to-target data acknowledgement of SCSI
       read data back to the target iSCSI layer, when requested.
    5) provide an asynchronous notification upon completion of a
       requested data transfer operation that moved data without
       involving the iSCSI layer.
    6) place the SCSI data into the I/O buffers or pick up the SCSI
       data for transmission out of the data buffers that the iSCSI
       layer had requested to be used for a SCSI I/O.

Chadalapaka, et al. Informational [Page 13] RFC 5047 DA October 2007

    7) provide an error-free (i.e., must have at least the same level
       of assurance of data integrity as the CRC32C iSCSI data
       digest), reliable, in-order delivery transport mechanism over
       IP networks in performing the data transfer, and asynchronously
       notify the iSCSI layer upon iSCSI connection termination.
 Note that this architecture expects that each compliant Datamover
 protocol will define the precise means of satisfying the requirements
 specified in this section.
 In order to meet the functional requirements listed in this section,
 certain Datamover protocols may require pre-posted buffers from the
 local iSCSI protocol layer via mechanisms outside the scope of this
 document.  In some implementations, the absence of such buffers may
 result in a connection failure.  Datamover protocols may also realize
 these functional requirements via methods not explicitly listed in
 this document.

7. Functional Overview

 This section presents an overview of the functional interactions
 between the iSCSI layer and the Datamover layer as intended by this
 Architecture.

7.1. Startup

 The iSCSI Login Phase on an iSCSI connection occurs as defined in
 [RFC3720].  The Architecture assumes that at the end of the Login
 Phase, both the initiator and target, if they had so decided,
 transition the connection to being Datamover-assisted.  The precise
 means of how an iSCSI initiator and an iSCSI target agree on having
 the connection Datamover-assisted is defined by the Datamover
 protocol.  The only architectural requirement is that all iSCSI
 interactions in the iSCSI Full Feature Phase MUST be Datamover-
 assisted subject to the prior agreement, meaning that the Datamover
 protocol is in the iSCSI-to-iSCSI communication path below the iSCSI
 layer on either side as shown in Figure 1.  DA defines the
 Enable_Datamover Operational Primitive (Section 8.6) to bring about
 this transition to a Datamover-assisted connection.
 The Architecture also assumes that the Datamover layer may require a
 certain number of opaque local resources for making a connection
 Datamover-assisted.  DA thus defines the
 Allocate_Connection_Resources Operational Primitive (Section 8.4) to
 model this interaction.  This Primitive is intended to be invoked on
 each side once the two sides decide (as previously noted) to have the
 connection be Datamover-assisted.  The expected sequence of Primitive
 invocations is depicted in Figures 2 and 3 in Section 13.2.  Figures

Chadalapaka, et al. Informational [Page 14] RFC 5047 DA October 2007

 4, 5, and 6 illustrate how the Primitives may be employed to deal
 with various legal login outcomes.

7.2. Full Feature Phase

 All iSCSI peer communication in the Full Feature Phase happens
 through the Datamover layers if the iSCSI connection is Datamover-
 assisted.  The Architecture assumes that a Datamover layer may
 require a certain number of opaque local resources for each new iSCSI
 task.  In the normal course of execution, these task-level resources
 in the Datamover layer are assumed to be transparently allocated on
 each task initiation and deallocated on the conclusion of each task
 as appropriate.  In exception scenarios however -- scenarios that do
 not yield a SCSI Response for each task such as ABORT TASK operation
 -- the Architecture assumes that the Datamover layer needs to be
 notified of the individual task terminations to aid its task-level
 resource management.  DA thus defines the Deallocate_Task_Resources
 Operational Primitive (Section 8.9) to model this task-resource
 management.  In specifying the ITT qualifier for the
 Deallocate_Task_Resources Primitive, the Architecture further assumes
 that the Datamover layer tracks its opaque task-level local resources
 by the iSCSI ITT.  DA also defines Send_Control (Section 8.1),
 Put_Data (Section 8.2), Get_Data (Section 8.3),
 Data_Completion_Notify (Section 9.3), Data_ACK_Notify (Section 9.4),
 and Control_Notify (Section 9.1) Operational Primitives to model the
 various Full Feature Phase interactions.
 Figures 9, 10, and 11 in Section 13.2 show some Full Feature Phase
 interactions -- SCSI Write task, SCSI Read task, and a SCSI Read Data
 acknowledgement, respectively.  Figure 12 in Section 13.2 illustrates
 how an ABORT TASK operation can be modeled leading to deterministic
 resource cleanup on the Datamover layer.

7.3. Wrap-up

 Once an iSCSI connection becomes Datamover-assisted, the connection
 continues in that state until the end of the Full Feature Phase,
 i.e., the termination of the connection.  The Architecture assumes
 that when a connection is normally logged out, the Datamover layer
 needs to be notified so that its connection-level opaque resources
 (see Section 7.1) may be freed up.  DA thus defines a
 Connection_Terminate Operational Primitive (Section 8.7) to model
 this interaction.  The Architecture further assumes that when a
 connection termination happens without iSCSI layer's involvement
 (e.g., TCP RST), the Datamover layer is capable of locally cleaning
 up its task-level and connection-level resources before notifying the
 iSCSI layer of the fact.  DA thus defines the

Chadalapaka, et al. Informational [Page 15] RFC 5047 DA October 2007

 Connection_Terminate_Notify Operational Primitive (Section 9.2) to
 model this interaction.
 Figures 7 and 8 in Section 13.2 illustrate the interactions between
 the iSCSI and Datamover layers in normal and unexpected connection
 termination scenarios.

8. Operational Primitives Provided by the Datamover Layer

 While the iSCSI specification itself does not have a notion of
 Operational Primitives, any iSCSI layer implementing the iSCSI
 specification functionally requires the following Operational
 Primitives from its Datamover layer.  Thus, any Datamover protocol
 compliant with this architecture MUST implement the Operational
 Primitives described in this section.  These Operational Primitives
 are invoked by the iSCSI layer as appropriate.  Unless otherwise
 stated, all the following Operational Primitives may be used both on
 the initiator side and the target side.  In general programming
 terminology, this set of Operational Primitives may be construed as
 "down calls".
    1) Send_Control
    2) Put_Data
    3) Get_Data
    4) Allocate_Connection_Resources
    5) Deallocate_Connection_Resources
    6) Enable_Datamover
    7) Connection_Terminate
    8) Notice_Key_Values
    9) Deallocate_Task_Resources

8.1. Send_Control

 Input qualifiers: Connection_Handle, iSCSI PDU-specific qualifiers
 Return Results: Not specified.
 An iSCSI layer requests that its local Datamover layer transmit an
 iSCSI control-type PDU to the peer iSCSI layer operating in the
 remote iSCSI node by this Operational Primitive.  The Datamover layer

Chadalapaka, et al. Informational [Page 16] RFC 5047 DA October 2007

 performs the requested operation, and may add its own protocol
 headers in doing so.  The iSCSI layer MUST NOT invoke the
 Send_Control Operational Primitive on an iSCSI connection that is not
 yet Datamover-assisted.
 An initiator iSCSI layer requesting the transfer of a SCSI Command
 PDU or a target iSCSI layer requesting the transfer of a SCSI
 response PDU are examples of invoking the Send_Control Operational
 Primitive.  As Section 10.3.1 illustrates later on, the iSCSI PDU-
 specific qualifiers in this example are: BHS and AHS,
 DataDescriptorOut, DataDescriptorIn, ImmediateDataSize, and
 UnsolicitedDataSize.

8.2. Put_Data

 Input qualifiers: Connection_Handle, contents of a SCSI Data-In PDU
 header, Data_Descriptor, Notify_Enable
 Return Results: Not specified.
 An iSCSI layer requests that its local Datamover layer transmit the
 data identified by the Data_Descriptor for the SCSI Data-In PDU to
 the peer iSCSI layer on the remote iSCSI node by this Operational
 Primitive.  The Datamover layer performs the operation by using its
 own protocol means, completely transparent to the remote iSCSI layer.
 The iSCSI layer MUST NOT invoke the Put_Data Operational Primitive on
 an iSCSI connection that is not yet Datamover-assisted.
 The Notify_Enable qualifier is used to request the local Datamover
 layer to generate or not generate the eventual local completion
 notification to the iSCSI layer for this Put_Data invocation.  For
 detailed semantics of this qualifier, see Section 9.3.
 A Put_Data Primitive may only be invoked by an iSCSI layer on the
 target to its local Datamover layer.
 A target iSCSI layer requesting the transfer of an iSCSI read data
 sequence (also known as a read burst) is an example of invoking the
 Put_Data Operational Primitive.

8.3. Get_Data

 Input qualifiers: Connection_Handle, contents of an R2T PDU,
 Data_Descriptor, Notify_Enable
 Return Results: Not specified.

Chadalapaka, et al. Informational [Page 17] RFC 5047 DA October 2007

 An iSCSI layer requests that its local Datamover layer retrieve
 certain data identified by the R2T PDU from the peer iSCSI layer on
 the remote iSCSI node and place it into the buffer identified by the
 Data_Descriptor by invoking this Operational Primitive.  The
 Datamover layer performs the operation by using its own protocol
 means, completely transparent to the remote iSCSI layer.  The iSCSI
 layer MUST NOT invoke the Get_Data Operational Primitive on an iSCSI
 connection that is not yet Datamover-assisted.
 The Notify_Enable qualifier is used to request that the local
 Datamover layer generate or not generate the eventual local
 completion notification to the iSCSI layer for this Get_Data
 invocation.  For detailed semantics of this qualifier, see Section
 9.3.
 A Get_Data Primitive may only be invoked by an iSCSI layer on the
 target to its local Datamover layer.
 A target iSCSI layer requesting the transfer of an iSCSI write data
 sequence (also known as a write burst) is an example of invoking the
 Get_Data Operational Primitive.

8.4. Allocate_Connection_Resources

 Input qualifiers: Connection_Handle[, Resource_Descriptor ]
 Return Results: Status.
 By invoking this Operational Primitive, an iSCSI layer requests that
 its local Datamover layer perform all the Datamover-specific resource
 allocations required for the Full Feature Phase of an iSCSI
 connection.  The Connection_Handle identifies the connection for
 which the iSCSI layer is requesting resources to be allocated.
 Allocation of these resources is a step towards eventually
 transitioning the connection to become a Datamover-assisted iSCSI
 connection.  Note that the Datamover layer however does not allocate
 any Datamover-specific task-level resources upon invocation of this
 Primitive.
 An iSCSI layer, in addition, optionally specifies the
 implementation-specific resource requirements for the iSCSI
 connection to the Datamover layer by passing an input qualifier
 called Resource_Descriptor.  The exact structure of a
 Resource_Descriptor is implementation-dependent, and hence
 structurally opaque to DA.
 A return result of Status=success means that the
 Allocate_Connection_Resources invocation corresponding to that

Chadalapaka, et al. Informational [Page 18] RFC 5047 DA October 2007

 Connection_Handle succeeded.  If an Allocate_Connection_Resources
 invocation is made for a Connection_Handle for which an earlier
 invocation succeeded, the return Status must be success and the
 request will be ignored by the Datamover layer.  A return result of
 Status=failure means that the Allocate_Connection_Resources
 invocation corresponding to that Connection_Handle failed.  There
 MUST NOT be more than one Allocate_Connection_Resources Primitive
 invocation outstanding for a given Connection_Handle at any time.
 The iSCSI layer must invoke the Allocate_Connection_Resources
 Primitive before the invocation of the Enable_Datamover Primitive.

8.5. Deallocate_Connection_Resources

 Input qualifiers: Connection_Handle
 Return Results: Not specified.
 By invoking this Operational Primitive, an iSCSI layer requests that
 its local Datamover layer deallocate all the Datamover-specific
 resources that may have been allocated earlier for the Transport
 Connection identified by the Connection_Handle.  The iSCSI layer may
 invoke this Operational Primitive when the Datamover-specific
 resources associated with the Connection_Handle are no longer
 necessary (such as the Login failure of the corresponding iSCSI
 connection).

8.6. Enable_Datamover

 Input qualifiers: Connection_Handle, Transport_Connection_Descriptor
 [, Final_Login_Response_PDU]
 Return Results: Not specified.
 By invoking this Operational Primitive, an iSCSI layer requests that
 its local Datamover layer assist all further iSCSI exchanges on the
 iSCSI connection (i.e., to make the connection Datamover-assisted)
 identified by the Connection_Handle, for which the Datamover-specific
 resource allocation was earlier made.  The iSCSI layer MUST NOT
 invoke the Enable_Datamover Operational Primitive for an iSCSI
 connection unless there is a corresponding prior resource allocation.
 The Final_Login_Response_PDU input qualifier is applicable only for a
 target, and contains the final Login Response that concludes the
 iSCSI Login Phase and which must be sent as a byte stream as expected
 by the initiator iSCSI layer.  When this qualifier is used, the
 target-Datamover layer MUST transmit this final Login Response before
 Datamover assistance is enabled for the Transport Connection.

Chadalapaka, et al. Informational [Page 19] RFC 5047 DA October 2007

 The iSCSI layer identifies the specific Transport Connection
 associated with the Connection_Handle to the Datamover layer by
 specifying the Transport_Connection_Descriptor.  The exact structure
 of this Descriptor is implementation-dependent.

8.7. Connection_Terminate

 Input qualifiers: Connection_Handle
 Return Results: Not specified.
 By invoking this Operational Primitive, an iSCSI layer requests that
 its local Datamover layer terminate the Transport Connection and
 deallocate all the connection and task resources associated with the
 Connection_Handle.  When this Operational Primitive invocation
 returns to the iSCSI layer, the iSCSI layer may assume the full
 ownership of all the iSCSI-level resources, e.g., I/O Buffers,
 associated with the connection.  This Operational Primitive may be
 invoked only with a valid Connection_Handle, and the Transport
 Connection associated with the Connection_Handle must already be
 Datamover-assisted.

8.8. Notice_Key_Values

 Input qualifiers: Connection_Handle, Number of keys, a list of Key-
 Value pairs.
 Return Results: Not specified.
 By invoking this Operational Primitive, an iSCSI layer requests that
 its local Datamover layer take note of the negotiated values of the
 listed keys for the Transport Connection.  This Operational Primitive
 may be invoked only with a valid Connection_Handle, and the Key-Value
 pairs MUST be the current values that were successfully agreed upon
 by the iSCSI peers for the connection.  The Datamover layer may use
 the values of the keys to aid the Datamover operation as it deems
 appropriate.  The specific keys to be passed as input qualifiers and
 the point(s) in time this Operational Primitive is invoked are
 implementation-dependent.

8.9. Deallocate_Task_Resources

 Input qualifiers: Connection_Handle, ITT
 Return Results: Not specified.
 By invoking this Operational Primitive, an iSCSI layer requests that
 its local Datamover layer deallocate all Datamover-specific resources

Chadalapaka, et al. Informational [Page 20] RFC 5047 DA October 2007

 that earlier may have been allocated for the task identified by the
 ITT qualifier.  The iSCSI layer uses this Operational Primitive
 during exception processing when one or more active tasks are to be
 terminated without corresponding SCSI Response PDUs.  This Primitive
 MUST be invoked for each active task terminated without a SCSI
 Response PDU.  This Primitive MUST NOT be invoked by the iSCSI layer
 when a SCSI Response PDU normally concludes a task.  When a SCSI
 Response PDU normally concludes a task (even if the SCSI Status was
 not a success), the Datamover layer is assumed to have automatically
 deallocated all Datamover-specific task resources for that task.
 Refer to Section 7.2 for a related discussion on the Architectural
 assumptions on the task-level Datamover resource management,
 especially with respect to when the resources are assumed to be
 allocated.

9. Operational Primitives Provided by the iSCSI Layer

 While the iSCSI specification itself does not have a notion of
 Operational Primitives, any iSCSI layer implementing the iSCSI
 specification would have to provide the following Operational
 Primitives to its local Datamover layer.  Thus, any iSCSI protocol
 implementation compliant with this architecture MUST implement the
 Operational Primitives described in this section.  These Operational
 Primitives are invoked by the Datamover layer as appropriate and when
 the iSCSI connection is Datamover-assisted.  Unless otherwise stated,
 all the following Operational Primitives may be used both on the
 initiator side and the target side.  In general programming
 terminology, this set of Operational Primitives may be construed as
 "up calls".
    1) Control_Notify
    2) Connection_Terminate_Notify
    3) Data_Completion_Notify
    4) Data_ACK_Notify

9.1. Control_Notify

 Input qualifiers: Connection_Handle, an iSCSI control-type PDU.
 Return Results: Not specified.
 A Datamover layer notifies its local iSCSI layer, via this
 Operational Primitive, of the arrival of an iSCSI control-type PDU
 from the peer Datamover layer on the remote iSCSI node.  The iSCSI
 layer processes the control-type PDU as defined in [RFC3720].

Chadalapaka, et al. Informational [Page 21] RFC 5047 DA October 2007

 A target iSCSI layer being notified of the arrival of a SCSI command
 is an example of invoking the Control_Notify Operational Primitive.
 Note that implementations may choose to describe the "iSCSI control-
 type PDU" qualifier in this notification using a Data_Descriptor
 (Section 5.2) and not necessarily one contiguous buffer.

9.2. Connection_Terminate_Notify

 Input qualifiers: Connection_Handle
 Return Results: Not specified.
 A Datamover layer notifies its local iSCSI layer on an unsolicited
 termination or failure of an iSCSI connection providing the
 Connection_Handle associated with the iSCSI Connection.  The iSCSI
 layer MUST consider the Connection_Handle to be invalid upon being so
 notified.  The iSCSI layer processes the connection termination as
 defined in [RFC3720].  The Datamover layer MUST deallocate the
 connection and task resources associated with the terminated
 connection before notifying the iSCSI layer of the termination via
 this Operational Primitive.
 A target iSCSI layer is notified of an ungraceful connection
 termination by the Datamover layer when the underlying Transport
 Connection is torn down.  Such a Connection_Terminate_Notify
 Operational Primitive may be triggered, for example, by a TCP RESET
 in cases where the underlying Transport Connection uses TCP.

9.3. Data_Completion_Notify

 Input qualifiers: Connection_Handle, ITT, SN
 Return Results: Not specified.
 A Datamover layer notifies its local iSCSI layer on completing the
 retrieval of the data or upon sending the data, as requested in a
 prior iSCSI data-type PDU, from/to the peer Datamover layer on the
 remote iSCSI node via this Operational Primitive.  The iSCSI layer
 processes the operation as defined in [RFC3720].
 SN may be either the DataSN associated with the SCSI Data-In PDU or
 R2TSN associated with the R2T PDU depending on the SCSI operation.
 Note that, for targets, a TTT (see [RFC3720]) could have been
 specified instead of an SN.  However, the considered choice was to
 leave the SN to be the qualifier for two reasons -- a) it is generic
 and applicable to initiators and targets as well as Data-In and
 Data-Out, and b) having both SN and TTT qualifiers for the

Chadalapaka, et al. Informational [Page 22] RFC 5047 DA October 2007

 notification is considered onerous on the Datamover layer, in terms
 of state maintenance for each completion notification.  The
 implication of this choice is that iSCSI target implementations will
 have to adapt to using the ITT-SN tuple in associating the solicited
 data to the appropriate task, rather than the ITT-TTT tuple for doing
 the same.
 If Notify_Enable is set in either a Put_Data or a Get_Data
 invocation, the Datamover layer MUST invoke the
 Data_Completion_Notify Operational Primitive upon completing that
 requested data transfer.  If the Notify_Enable was cleared in either
 a Put_Data or a Get_Data invocation, the Datamover layer MUST NOT
 invoke the Data_Completion_Notify Operational Primitive upon
 completing that requested data transfer.
 A Data_Completion_Notify invocation serves to notify the iSCSI layer
 of the Put_Data or Get_Data completion, respectively.  As earlier
 noted in Sections 8.2 and 8.3, specific Datamover protocol
 definitions may restrict the usage scope of Put_Data and Get_Data,
 and thus implicitly the usage scope of Data_Completion_Notify.
 A target iSCSI layer being notified of the retrieval of a write data
 sequence is an example of invoking the Data_Completion_Notify
 Operational Primitive.

9.4. Data_ACK_Notify

 Input qualifiers: Connection_Handle, ITT, DataSN
 Return Results: Not specified.
 A target Datamover layer notifies its local iSCSI layer of the
 arrival of a previously requested data acknowledgement from the peer
 Datamover layer on the remote (initiator) iSCSI node via this
 Operational Primitive.  The iSCSI layer processes the data
 acknowledgement notification as defined in [RFC3720].
 A target iSCSI layer being notified of the arrival of a data
 acknowledgement for a certain SCSI Read data PDU is the only example
 of invoking the Data_ACK_Notify Operational Primitive.

10. Datamover Interface (DI)

10.1. Overview

 This section describes the model of interactions between iSCSI and
 Datamover layers when the iSCSI connection is Datamover-assisted so
 the iSCSI layer may carry out the following:

Chadalapaka, et al. Informational [Page 23] RFC 5047 DA October 2007

  1. send iSCSI data-type PDUs and exchange iSCSI control-type PDUs,

and

  1. handle asynchronous notifications such as completion of data

sequence transfer and connection failure.

 This chapter relies on the notion of Operational Primitives (Section
 5.4) to define DI.

10.2. Interactions for Handling Asynchronous Notifications

10.2.1. Connection Termination

 As stated in Section 9.2, the Datamover layer notifies the iSCSI
 layer of a failed or terminated connection via the
 Connection_Terminate_Notify Operational Primitive.  The iSCSI layer
 MUST consider the connection unusable upon the invocation of this
 Primitive and handle the connection termination as specified in
 [RFC3720].

10.2.2. Data Transfer Completion

 As stated in Section 9.3, the Datamover layer notifies the iSCSI
 layer of a completed data transfer operation via the
 Data_Completion_Notify Operational Primitive.  The iSCSI layer
 processes the transfer completion as specified in [RFC3720].

10.2.2.1. Completion of a Requested SCSI Data Transfer

 To notify the iSCSI layer of the completion of a requested iSCSI
 data-type PDU transfer, the Datamover layer uses the
 Data_Completion_Notify Operational Primitive with the following input
 qualifiers.
    a) Connection_Handle.
    b) ITT: Initiator Task Tag semantics as defined in [RFC3720].
    c) SN: DataSN for a SCSI Data-in/Data-out PDU, and R2TSN for an
       iSCSI R2T PDU.  The semantics for both types of sequence
       numbers are as defined in [RFC3720].
 The rationale for choosing SN is explained in Section 9.3.
 Every invocation of the Data_Completion_Notify Operational Primitive
 MUST be preceded by an invocation of the Put_Data or Get_Data
 Operational Primitive with the Notify_Enable qualifier set by the
 iSCSI layer at an earlier point in time.

Chadalapaka, et al. Informational [Page 24] RFC 5047 DA October 2007

10.2.3. Data Acknowledgement

 [RFC3720] allows the iSCSI targets to optionally solicit data
 acknowledgement from the initiator for one or more Data-In PDUs, via
 setting of the A-bit on a Data-In PDU.  The Data_ACK_Notify
 Operational Primitive with the following input qualifiers is used by
 the target Datamover layer to notify the local iSCSI layer of the
 arrival of data acknowledgement of a previously solicited iSCSI read
 data acknowledgement.  This Operational Primitive thus is applicable
 only to iSCSI targets.
    a) Connection_Handle.
    b) ITT: Initiator Task Tag semantics as defined in [RFC3720].
    c) DataSN: of the next SCSI Data-In PDU, which immediately follows
       the SCSI Data-In PDU with the A-bit set to which this
       notification corresponds, with semantics as defined in
       [RFC3720].
 Every invocation of the Data_ACK_Notify Operational Primitive MUST be
 preceded by an invocation of the Put_Data Operational Primitive by
 the iSCSI target layer with the A-bit set to 1 at an earlier point in
 time.

10.3. Interactions for Sending an iSCSI PDU

 This section discusses the model of interactions for sending each of
 the iSCSI PDUs defined in [RFC3720].  A Connection_Handle (see
 Section 5.3) is assumed to qualify each of these interactions so that
 the Datamover layer can route it to the appropriate Transport
 Connection.  The qualifying Connection_Handle is not explicitly
 listed in the subsequent sections.
 Note that the defined list of input qualifiers represents the
 semantically required set for the Datamover layer to consider in
 implementing the Primitive in each interaction described in this
 section (see Section 5.4 for an elaboration).  Implementations may
 choose to deduce the qualifiers in ways that are optimized for the
 implementation specifics.  Two examples of this are:
    1. For SCSI command (Section 10.3.1), deducing the
       ImmediateDataSize input qualifier from the DataSegmentLength
       field of the SCSI Command PDU.
    2. For SCSI Data-Out (Section 10.3.5.1), deducing the
       DataDescriptorOut input qualifier from the associated SCSI
       command invocation qualifiers (assuming such state is

Chadalapaka, et al. Informational [Page 25] RFC 5047 DA October 2007

       maintained) in conjunction with BHS fields of the SCSI Data-Out
       PDU.

10.3.1. SCSI Command

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of a SCSI Command
 PDU.
    a) BHS and AHS, if any, of the SCSI Command PDU as defined in
       [RFC3720].
    b) DataDescriptorOut: that defines the I/O Buffer meant for Data-
       Out for the entire command, in the case of a write or
       bidirectional command.
    c) DataDescriptorIn: that defines the I/O Buffer meant for Data-In
       for the entire command, in the case of a read or bidirectional
       command.
    d) ImmediateDataSize: that defines the number of octets of
       immediate unsolicited data for a write/bidirectional command.
    e) UnsolicitedDataSize: that defines the number of octets of
       immediate and non-immediate unsolicited data for a
       write/bidirectional command.

10.3.2. SCSI Response

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of a SCSI Response
 PDU.
    a) BHS of the SCSI Response PDU as defined in [RFC3720].
    b) DataDescriptorStatus: that defines the iSCSI buffer that
       contains the sense and response information for the command.

10.3.3. Task Management Function Request

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of a Task
 Management Function Request PDU.
    a) BHS of the Task Management Function Request PDU as defined in
       [RFC3720].

Chadalapaka, et al. Informational [Page 26] RFC 5047 DA October 2007

    b) DataDescriptorOut: that defines the I/O Buffer meant for Data-
       Out for the entire command, in the case of a write or
       bidirectional command.  (Only valid if Function="TASK REASSIGN"
       - [RFC3720].)
    c) DataDescriptorIn: that defines the I/O Buffer meant for Data-In
       for the entire command, in the case of a read or bidirectional
       command.  (Only valid if Function="TASK REASSIGN" - [RFC3720].)

10.3.4. Task Management Function Response

 The Send_Control Operational Primitive with the following input
 qualifier is used for requesting the transmission of a Task
 Management Function Response PDU.
    a) BHS of the Task Management Function Response PDU as defined in
       [RFC3720].

10.3.5. SCSI Data-Out and SCSI Data-In

10.3.5.1. SCSI Data-Out

 The Send_Control Operational Primitive with the following input
 qualifiers is used by the initiator iSCSI layer for requesting the
 transmission of a SCSI Data-Out PDU carrying the non-immediate
 unsolicited data.
    a) BHS of the SCSI Data-Out PDU as defined in [RFC3720].
    b) DataDescriptorOut: that defines the I/O Buffer with the Data-
       Out to be carried in the iSCSI data segment of the PDU.

10.3.5.2. SCSI Data-In

 The Put_Data Operational Primitive with the following input
 qualifiers is used by the target iSCSI layer for requesting the
 transmission of the data carried by a SCSI Data-In PDU.
    a) BHS of the SCSI Data-In PDU as defined in [RFC3720].
    b) DataDescriptorIn: that defines the I/O Buffer with the Data-In
       being requested for transmission.

Chadalapaka, et al. Informational [Page 27] RFC 5047 DA October 2007

10.3.6. Ready To Transfer (R2T)

 The Get_Data Operational Primitive with the following input
 qualifiers is used by the target iSCSI layer for requesting the
 retrieval of the data as specified by the semantic content of an R2T
 PDU.
    a) BHS of the Ready To Transfer PDU as defined in [RFC3720].
    b) DataDescriptorOut: that defines the I/O Buffer for the Data-Out
       being requested for retrieval.

10.3.7. Asynchronous Message

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of an Asynchronous
 Message PDU.
    a) BHS of the Asynchronous Message PDU as defined in [RFC3720].
    b) DataDescriptorSense: that defines an iSCSI buffer that contains
       the sense and iSCSI Event information.

10.3.8. Text Request

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of a Text Request
 PDU.
    a) BHS of the Text Request PDU as defined in [RFC3720].
    b) DataDescriptorTextOut: that defines the iSCSI Text Request
       buffer.

10.3.9. Text Response

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of a Text Response
 PDU.
    a) BHS of the Text Response PDU as defined in [RFC3720].
    b) DataDescriptorTextIn: that defines the iSCSI Text Response
       buffer.

Chadalapaka, et al. Informational [Page 28] RFC 5047 DA October 2007

10.3.10. Login Request

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of a Login Request
 PDU.
    a) BHS of the Login Request PDU as defined in [RFC3720].
    b) DataDescriptorLoginRequest: that defines the iSCSI Login
       Request buffer.
 Note that specific Datamover protocols may choose to disallow the
 standard DA Primitives from being used for the iSCSI Login Phase.
 When used in conjunction with such Datamover protocols, an attempt to
 send a Login Request via the Send_Control Operational Primitive
 invocation is clearly an error scenario, as the Login Request PDU is
 being sent while the connection is in the iSCSI Full Feature Phase.
 It is outside the scope of this document to specify the resulting
 implementation behavior in this case -- [RFC3720] already defines the
 error handling for this error scenario.

10.3.11. Login Response

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of a Login
 Response PDU.
    a) BHS of the Login Response PDU as defined in [RFC3720].
    b) DataDescriptorLoginResponse: that defines the iSCSI Login
       Response buffer.
 Note that specific Datamover protocols may choose to disallow the
 standard DA Primitives from being used for the iSCSI Login Phase.
 When used in conjunction with such Datamover protocols, an attempt to
 send a Login Response via the Send_Control Operational Primitive
 invocation is clearly an error scenario, as the Login Response PDU is
 being sent while in the iSCSI Full Feature Phase.  It is outside the
 scope of this document to specify the resulting implementation
 behavior in this case -- [RFC3720] already defines the error handling
 for this error scenario.

10.3.12. Logout Command

 The Send_Control Operational Primitive with the following input
 qualifier is used for requesting the transmission of a Logout Command
 PDU.

Chadalapaka, et al. Informational [Page 29] RFC 5047 DA October 2007

    a) BHS of the Logout Command PDU as defined in [RFC3720].

10.3.13. Logout Response

 The Send_Control Operational Primitive with the following input
 qualifier is used for requesting the transmission of a Logout
 Response PDU.
    a) BHS of the Logout Response PDU as defined in [RFC3720].

10.3.14. SNACK Request

 The Send_Control Operational Primitive with the following input
 qualifier is used for requesting the transmission of a SNACK Request
 PDU.
    a) BHS of the SNACK Request PDU as defined in [RFC3720].

10.3.15. Reject

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of a Reject PDU.
    a) BHS of the Reject PDU as defined in [RFC3720].
    b) DataDescriptorReject: that defines the iSCSI Reject buffer.

10.3.16. NOP-Out

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of a NOP-Out PDU.
    a) BHS of the NOP-Out PDU as defined in [RFC3720].
    b) DataDescriptorNOPOut: that defines the iSCSI Ping data buffer.

10.3.17. NOP-In

 The Send_Control Operational Primitive with the following input
 qualifiers is used for requesting the transmission of a NOP-In PDU.
    a) BHS of the NOP-In PDU as defined in [RFC3720].
    b) DataDescriptorNOPIn: that defines the iSCSI Return Ping data
       buffer.

Chadalapaka, et al. Informational [Page 30] RFC 5047 DA October 2007

10.4. Interactions for Receiving an iSCSI PDU

 The only PDUs that are received by an iSCSI layer operating on a
 Datamover layer are the iSCSI control-type PDUs.  The Datamover layer
 delivers the iSCSI control-type PDUs as they arrive, qualifying each
 with the Connection_Handle (see Section 5.3) that identifies the
 iSCSI connection for which the PDU is meant.  The subsequent
 processing of the iSCSI control-type PDUs proceeds as defined in
 [RFC3720].

10.4.1. General Control-Type PDU Notification

 This sub-section describes the general mechanics applicable to
 several control-type PDUs.  The following sub-sections note
 additional considerations for control-type PDUs that are not covered
 in this sub-section.
 The Control_Notify Operational Primitive is used to notify the iSCSI
 layer of the arrival of the following iSCSI control-type PDUs: SCSI
 Command, SCSI Response, Task Management Function Request, Task
 Management Function Response, Asynchronous Message, Text Request,
 Text Response, Logout Command, Logout Response, SNACK, Reject, NOP-
 Out, NOP-In.

10.4.2. SCSI Data Transfer PDUs

10.4.2.1. SCSI Data-Out

 The Control_Notify Operational Primitive is used to notify the iSCSI
 layer of the arrival of a SCSI Data-Out PDU carrying the non-
 immediate unsolicited data.  Note however that the solicited SCSI
 Data-Out arriving on the target does not cause a notification to the
 iSCSI layer using the Control_Notify Primitive because the solicited
 SCSI Data-Out was not sent by the initiator iSCSI layer as control-
 type PDUs.

10.4.2.2. SCSI Data-In

 The Datamover layer does not notify the iSCSI layer of the arrival of
 the SCSI Data-in at the initiator, because SCSI Data-in is an iSCSI
 data-type PDU (see section 5.1).  The iSCSI layer at the initiator
 however may infer the arrival of the SCSI Data-In when it receives a
 subsequent notification of the SCSI Response PDU via a Control_Notify
 invocation.
 While this document does not contemplate the possibility of a Data-In
 PDU being received at the initiator iSCSI layer, specific Datamover
 protocols may define how to deal with an unexpected inbound SCSI

Chadalapaka, et al. Informational [Page 31] RFC 5047 DA October 2007

 Data-In PDU that may result in the initiator iSCSI layer receiving
 the Data-In PDU.  This document leaves the details of handling this
 error scenario to the specific Datamover protocols, so each may
 define the appropriate error handling specific to the Datamover
 environment.

10.4.2.3. Ready To Transfer (R2T)

 Because an R2T PDU is an iSCSI data-type PDU (see Section 5.1) that
 is not delivered as-is to the initiator iSCSI layer, the Datamover
 layer does not notify the iSCSI layer of the arrival of an R2T PDU.
 When an iSCSI node sends an R2T PDU to its local Datamover layer, the
 local and remote Datamover layers transparently bring about the data
 transfer requested by the R2T PDU.
 While this document does not contemplate the possibility of an R2T
 PDU being received at the initiator iSCSI layer, specific Datamover
 protocols may define how to deal with an unexpected inbound R2T PDU
 that may result in the initiator iSCSI layer receiving the R2T PDU.
 This document leaves the details of handling this error scenario to
 the specific Datamover protocols, so each may define the appropriate
 error handling specific to the Datamover environment.

10.4.3. Login Request

 The Control_Notify Operational Primitive is used for notifying the
 target iSCSI layer of the arrival of a Login Request PDU.  Note that
 specific Datamover protocols may choose to disallow the standard DA
 Primitives from being used for the iSCSI Login Phase.  When used in
 conjunction with such Datamover protocols, the arrival of a Login
 Request necessitating the Control_Notify Operational Primitive
 invocation is clearly an error scenario, as the Login Request PDU is
 arriving in the iSCSI Full Feature Phase.  It is outside the scope of
 this document to specify the resulting implementation behavior in
 this case -- [RFC3720] already defines the error handling in this
 error scenario.

10.4.4. Login Response

 The Control_Notify Operational Primitive is used to notify the
 initiator iSCSI layer of the arrival of a Login Response PDU.  Note
 that specific Datamover protocols may choose to disallow the standard
 DA Primitives from being used for the iSCSI Login Phase.  When used
 in conjunction with such Datamover protocols, the arrival of a Login
 Response necessitating the Control_Notify Operational Primitive
 invocation is clearly an error scenario, as the Login Response PDU is
 arriving in the iSCSI Full Feature Phase.  It is outside the scope of
 this document to specify the resulting implementation behavior in

Chadalapaka, et al. Informational [Page 32] RFC 5047 DA October 2007

 this case -- [RFC3720] already defines the error handling in this
 error scenario.

11. Security Considerations

11.1. Architectural Considerations

 DA enables compliant iSCSI implementations to realize a control and
 data separation in the way they interact with their Datamover
 protocols.  Note however that this separation does not imply a
 separation in transport mediums between control traffic and data
 traffic -- the basic iSCSI architecture with respect to tasks and PDU
 relationships to tasks remains unchanged.  [RFC3720] defines several
 MUST requirements on ordering relationships across control and data
 for a given task besides a mandatory deterministic task allegiance
 model -- DA does not change this basic architecture (DA has a
 normative reference to [RFC3720]) for allow any additional
 flexibility in compliance in this area.  To summarize, sending bulk
 data transfers (prompted by Put_Data and Get_Data Primitive
 invocations) on a different transport medium would be as ill-advised
 as sending just the Data-Out/Data-In PDUs on a different TCP
 connection in RFC 3720-based iSCSI implementations.  Consequently,
 all the iSCSI-related security text in [RFC3723] is directly
 applicable to a DA-enabled iSCSI implementation.
 Another area with security implications is the Datamover connection
 resource management model, which DA defines -- particularly the
 Allocate_Connection_Resources Primitive.  An inadvertent realization
 of this model could leave an iSCSI implementation exposed to denial-
 of-service attacks.  As Figures 2 and 3 in Section 13.2 illustrate,
 the most effective countermeasure to this potential attack consists
 of performing the Datamover resource allocation when the iSCSI layer
 is sufficiently far along in the iSCSI Login Phase that it is
 reasonably certain that the peer side is not an attacker.  In
 particular, if the Login Phase includes a SecurityNegotiation stage,
 an iSCSI end node MUST defer the Datamover connection resource
 allocation (i.e., invoking the Allocate_Connection_Resources
 Primitive) to the LoginOperationalNegotiation stage [RFC3720] so that
 the resource allocation happens post-authentication.  This
 considerably minimizes the potential for a denial-of service attack.

11.2. Wire Protocol Considerations

 In view of the fact that the DA architecture itself does not define
 any new wire protocol or propose modifications to the existing
 protocols, there are no additional wire protocol security
 considerations in employing DA itself.  However, a DA-compliant iSCSI
 implementation MUST comply with all the iSCSI-related requirements

Chadalapaka, et al. Informational [Page 33] RFC 5047 DA October 2007

 stipulated in [RFC3723] and [RFC3720].  Note further that in
 realizing DA, each Datamover protocol must define and elaborate as
 appropriate on any additional security considerations resulting from
 the use of that Datamover protocol.
 All Datamover protocol designers are strongly recommended to refer to
 [RDDPSEC] for the types of security issues to consider.  While
 [RDDPSEC] elaborates on the security considerations applicable to an
 RDDP-based Datamover [iSER], the document is representative of the
 type of analysis of resource exhaustion and the application of
 countermeasures that need to be done for any Datamover protocol.

12. References

12.1. Normative References

 [RFC3720] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and
           E. Zeidner, "Internet Small Computer Systems Interface
           (iSCSI)", RFC 3720, April 2004.
 [RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
           Travostino, "Securing Block Storage Protocols over IP", RFC
           3723, April 2004.
 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119, March 1997.

12.2. Informative References

 [DDP]     Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
           Data Placement over Reliable Transports", RFC 5041, October
           2007.
 [iSER]    Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah, H.,
           and P. Thaler, "Internet Small Computer System Interface
           (iSCSI) Extensions for Remote Direct Memory Access (RDMA)",
           RFC 5046, October 2007.
 [RDDPSEC] Pinkerton, J. and E. Deleganes, "Direct Data Placement
           Protocol (DDP) / Remote Direct Memory Access Protocol
           (RDMAP) Security", RFC 5042, October 2007.

Chadalapaka, et al. Informational [Page 34] RFC 5047 DA October 2007

Appendix A. Design Considerations and Examples

A.1. Design Considerations for a Datamover Protocol

 This section discusses the specific considerations for RDMA-based and
 RDDP-based Datamover protocols.
    a) Note that the modeling of interactions for SCSI Data-Out
       (Section 10.3.5.1) is only used for unsolicited data transfer.
    b) The modeling of interactions for SNACK (Sections 10.3.14 and
       10.4.1) is not expected to be used given that one of the design
       requirements on the Datamover is that it "guarantees an error-
       free, reliable, in-order transport mechanism" (Section 6).  The
       interactions for sending and receiving a SNACK are nevertheless
       modeled in this document because the receiving iSCSI layer can
       deterministically deal with an inadvertent SNACK.  This also
       shows the DA designers' intent that DI is not meant to filter
       certain types of PDUs.
    c) The onus is on a reliable Datamover (per requirements stated in
       Section 6) to realize end-to-end data acknowledgements via
       Datamover-specific means.  In view of this, even use of data-
       ACK-type SNACKs are unnecessary.  Consequently, an initiator
       may never request sending a SNACK Request in this model
       assuming that the proactive (timeout-driven) SNACK
       functionality is turned off in the legacy iSCSI code.
    d) Note that the current DA model for bootstrapping a
       Connection_Handle into service -- i.e., associating a new iSCSI
       connection with a Connection_Handle -- clearly implies that the
       iSCSI connection must already be in Full Feature Phase when the
       Datamover layer comes into the stack.  This further implies
       that the iSCSI Login Phase must be carried out in the
       traditional "Byte streaming mode" with no assistance or
       involvement from the Datamover layer.

A.2. Examples of Datamover Interactions

 The figures described in this section provide some examples of the
 usage of Operational Primitives in interactions between the iSCSI
 layer and the Datamover layer.  The following abbreviations are used
 in this section.
 Avail - Available
 Abted - Aborted

Chadalapaka, et al. Informational [Page 35] RFC 5047 DA October 2007

 Buf - I/O Buffer
 Cmd - Command
 Compl - Complete
 Conn - Connection
 Ctrl_Ntfy - Control_Notify
 Dal_Tk_Res - Deallocate_Task_Resources
 Data_Cmp_Nfy - Data_Completion_Notify
 Data_ACK_Nfy - Data_ACK_Notify
 DM - Datamover
 Imm - Immediate
 Snd_Ctrl - Send_Control
 Msg - Message
 Resp - Response
 Sol - Solicited
 TMF Req - Task Management Function Request
 TMF Res - Task Management Function Response
 Trans - Transfer
 Unsol - Unsolicited

Chadalapaka, et al. Informational [Page 36] RFC 5047 DA October 2007

       |   | Allocate_Connection_Resources  | D |    ^
       |   |------------------------------->| a |    |
       |   |    Connection resources are    | t |    |
       | i |     successfully allocated     | a |    |   iSCSI
       | S |                                | m |    |   Login
       | C |                                | o |    |   Phase
       | S |                                | v |    |
       | I |                                | e |    |
       |   |                                | r |    | Login Phase
       | L | Final Login Response (success)          v succeeds
       | a |<----------------------------------------^
       | y |                                | L |    |   iSCSI
       | e |       Enable_Datamover         | a |    |   Full
       | r |------------------------------->| y |    |   Feature
       |   |     Datamover is enabled       | e |    |   Phase
       |   |                                | r |    |
       |   |   Full Feature Phase           |   |    |
       |   |   control and data Transfer    |   |    v
         Figure 2.  A Successful iSCSI Login on Initiator
       |   | Notice_Key_Values              |   |      |
       |   |------------------------------->|   |      |
       |   |  Datamover layer is notified   |   |      |
       |   |  of the negotiated key values  |   |      |
       |   |                                |   |      |
       |   | Allocate_Connection_Resources  |   |      |
       |   |------------------------------->| D |      |
       |   |    Connection resources are    | a |      |
       | i |     successfully allocated     | t |      |   iSCSI
       | S |                                | a |      |   Login
       | C |                                | m |Final |   Phase
       | S |                                | o |Login |
       | I |Enable_Datamover(Login Response)| v |Resp  |
       |   |------------------------------->| e |---->vLogin Phase
       | L |     Datamover is enabled       | r |      ^ succeeds
       | a |                                |   |      |
       | y |                                | L |      |   iSCSI
       | e |                                | a |      |   Full
       | r |                                | y |      |   Feature
       |   |                                | e |      |   Phase
       |   |      Full Feature Phase        | r |      |
       |   |   control and data Transfer    |   |      |
       |   |                                |   |      v
         Figure 3.  A Successful iSCSI Login on Target

Chadalapaka, et al. Informational [Page 37] RFC 5047 DA October 2007

       |   | Allocate_Connection_Resources  | D |      ^
       |   |------------------------------->| a |      |
       |   |    Connection resources are    | t |      |
       | i |     successfully allocated     | a |      |   iSCSI
       | S |                                | m |      |   Login
       | C |                                | o |      |   Phase
       | S |                                | v |      |
       | I |                                | e |      |
       |   |                                | r |      | Login
       |   |                                |   |      | Phase
       | L | Final Login Response (failure)            v fails
       | a |<------------------------------------------
       | y |                                | L |
       | e | Deallocate_Connection_Resources| a |
       | r |------------------------------->| y |
       |   |     Datamover-specific         | e |
       |   |     connection resources freed | r |
       |   |                                |   |
       |   |
       |   | Connection terminated by standard means
       |   |--------------------------------------------->
          Figure 4.  A Failed iSCSI Login on Initiator
       |   | Allocate_Connection_Resources  | D |      ^
       |   |------------------------------->| a |      |
       |   |    Connection resources are    | t |      |
       | i |     successfully allocated     | a |      |   iSCSI
       | S |                                | m |      |   Login
       | C |                                | o |      |   Phase
       | S |                                | v |      |
       | I |                                | e |      |
       |   |                                | r |      | Login
       |   |                                |   |      | Phase
       | L | Final Login Response (failure)            v fails
       | a |---------------------------------------------->
       | y |                                | L |
       | e | Deallocate_Connection_Resources| a |
       | r |------------------------------->| y |
       |   |     Datamover-specific         | e |
       |   |     connection resources freed | r |
       |   |                                |   |
       |   |
       |   | Connection terminated by standard means
       |   |-------------------------------------------->
           Figure 5.  A Failed iSCSI Login on Target

Chadalapaka, et al. Informational [Page 38] RFC 5047 DA October 2007

       |   | Allocate_Connection_Resources  | D |     ^
       |   |------------------------------->| a |     |
       |   |    Connection resources are    | t |     |
       | i |     successfully allocated     | a |     |   iSCSI
       | S |                                | m |     |   Login
       | C |                                | o |     |   Phase
       | S |                                | v |     |
       | I |                                | e |     |
       |   |                                | r |     |
       | L | Login non-Final Request/Response         |
       | a |<-----------------------------------------|
       | y |    iSCSI layer decides not to  | L |     |
       | e |    enable Datamover for this   | a |     |
       | r |    connection                  | y |     |
       |   |                                | e |     |
       |   | Deallocate_Connection_Resources| r |     |
       |   |------------------------------->|   |     |
       |   |     All Datamover-specific     |   |     |
       |   |     resources deallocated      |   |     |
       |   |                                |   |     | Login
       |   |                                |   |     | Phase
       |   |                                          | continues
       |   | Regular Login negotiation continues      |
       |   |<---------------------------------------->|
       |   |                                          .
       |   |                                          .
       |   |                                          .
        Figure 6.  iSCSI Does Not Enable the Datamover

Chadalapaka, et al. Informational [Page 39] RFC 5047 DA October 2007

       |   |                                |   |   ^
       |   |  Full Feature Phase Control &  |   |   |
       |   |    Data Transfer Using DM      | D |   | iSCSI
       |   |                                | a |   | Full Feature
       | i |                                | t |   | Phase
       | S |                                | a |   | (DM Enabled)
       | C |                                | m |   |
       | S |    Successful iSCSI Logout     | o |   |
       | I |                                | v |   v
       |   |     Connection_Terminate       | e |
       | L |------------------------------->| r |
       | a |   Connection is terminated     |   |
       | y |   Datamover-specific resources | L | Transport
       | e |   deallocated, both connection | a | Connection
       | r |   level & task level           | y | is terminated
       |   |                                | e |
       |   |                                | r |
       |   |                                |   |
       |   |                                |   |
         Figure 7.  A Normal iSCSI Connection Termination
       |   |                                |   |   ^
       |   |  Full Feature Phase Control &  | D |   | iSCSI
       |   |    Data Transfer Using DM      | a |   | Full Feature
       | i |                                | t |   | Phase
       | S |                                | a |   | (DM Enabled)
       | C |                                | m |   v
       | S |                                | o |<--Transport
       | I |   Datamover-specific resources | v | Connection
       |   |   deallocated, both connection | e | Terminated (e.g.
       | L |   level & task level           | r | unexpected
       | a |                                |   | FIN/RESET)
       | y |                                | L |
       | e |   Connection_Terminate_Notify  | a |
       | r |<-------------------------------| y |
       |   |                                | e |
       |   |                                | r |
       |   |                                |   |
         Figure 8.  An Abnormal iSCSI Connection Termination

Chadalapaka, et al. Informational [Page 40] RFC 5047 DA October 2007

      <-----Initiator----->                <-------Target------->
      |  |          |  | DM Msg holding |  |            |  |
 SCSI |  |          |  | SCSI Cmd PDU & |  |            |  |SCSI
 Cmd  |  | Snd_Ctrl |  |Unsol Imm Data  |  |Ctrl_Notify |  |Cmd
 ---->|  |--------->|  |--------------->|  |----------->|  |--->
      |  |          |  |                |  |            |  |
      |  |          |  | DM Msg holding |  |            |  |
      |  | Snd_Ctrl |  |SCSI Dataout PDU|  |Ctrl_Notify |  |
      |  |--------->|  |--------------->|  |----------->|  |
      |  |    .     |  |        .       |  |     .      |  |Unsol
      |  |    .     | D|        .       | D|     .      |  |Data
      |  |    .     | a| DM Msg holding | a|     .      |  |Trans
      | i| Snd_Ctrl | t|SCSI Dataout PDU| t|Ctrl_Notify | i|
      | S|--------->| a|--------------->| a|----------->| S|
      | C|          | m|                | m|            | C|Buf
      | S|          | o|                | o|            | S|Avail
      | I|          | v|                | v|  Get_Data  | I|(R2T)
      |  |          | e|----------------| e|<-----------|  |<----
      | L|          | r||Solicited Data | r|            | L|  .
      | a|          |  ||  Transfer     |  |            | a|  .
      | y|          | L|--------------->| L|      .     | y|Buf
      | e|          | a|        .       | a|      .     | e|Avail
      | r|          | y|        .       | y|  Get_Data  | r|(R2T)
      |  |          | e|----------------| e|<-----------|  |<----
      |  |          | r||Solicited Data | r|            |  |
      |  |          |  ||   Transfer    |  |            |  |
      |  |          |  |--------------->|  |Data_Cmp_Nfy|  |Data
      |  |          |  |                |  |----------->|  |Trans
      |  |          |  |                |  |            |  |Compl
      |  |          |  | DM Msg holding |  |            |  |
 SCSI |  |          |  |SCSI Resp PDU & |  |            |  |SCSI
 Resp |  |Ctrl_Ntfy |  |  Sense Data    |  |  Snd_Ctrl  |  |Resp
 <----|  |<---------|  |<---------------|  |<-----------|  |<----
      |  |          |  |                |  |            |  |
               Figure 9.  A SCSI Write Data Transfer

Chadalapaka, et al. Informational [Page 41] RFC 5047 DA October 2007

      <-----Initiator----->                <-------Target------->
      |  |          |  |                |  |            |  |
 SCSI |  |          |  | DM Msg holding |  |            |  |SCSI
 Cmd  |  | Snd_Ctrl |  |  SCSI Cmd PDU  |  |Ctrl_Notify |  |Cmd
 ---->|  |--------->|  |--------------->|  |----------->|  |--->
      |  |          |  |                |  |            |  |
      |  |          | D|    SCSI Read   | D|            |  |Buf
      |  |          | a|  Data Transfer | a|  Put_Data  |  |Avail
      | i|          | t|<---------------| t|<-----------| i|<----
      | S|          | a|        .       | a|     .      | S|  .
      | C|          | m|        .       | m|     .      | C|  .
      | S|          | o|        .       | o|     .      | S|  .
      | I|          | v|    SCSI Read   | v|     .      | I|Buf
      |  |          | e|  Data Transfer | e|  Put_Data  |  |Avail
      | L|          | r|<---------------| r|<-----------| L|<----
      | a|          |  |                |  |            | a|
      | y|          | L|                | L|            | y|
      | e|          | a|                | a|Data_Cmp_Nfy| e|Data
      | r|          | y|                | y|----------->| r|Trans
      |  |          | e|                | e|            |  |Compl
      |  |          | r| DM Msg holding | r|            |  |
 SCSI |  |          |  |SCSI Resp PDU & |  |            |  |SCSI
 Resp |  |Ctrl_Ntfy |  |  Sense Data    |  |  Snd_Ctrl  |  |Resp
 <----|  |<---------|  |<---------------|  |<-----------|  |<----
      |  |          |  |                |  |            |  |
               Figure 10.  A SCSI Read Data Transfer

Chadalapaka, et al. Informational [Page 42] RFC 5047 DA October 2007

      <-----Initiator----->                <-------Target------->
      |  |          |  |                |  |            |  |
 SCSI |  |          |  | DM Msg holding |  |            |  |SCSI
 Cmd  |  | Snd_Ctrl |  |  SCSI Cmd PDU  |  |Ctrl_Notify |  |Cmd
 ---->|  |--------->|  |--------------->|  |----------->|  |---->
      |  |          |  |                |  |            |  |
      |  |          | D|    SCSI Read   | D|  Put_Data  |  |Buf
      |  |          | a|  Data Transfer | a|Data_in.A=1 |  |Avail
      | i|          | t|<---------------| t|<-----------| i|<----
      | S|          | a|        .       | a|     .      | S|  .
      | C|          | m|        .       | m|Data_ACK_Nfy| C|  .
      | S|          | o|                | o|----------->| S|  .
      | I|          | v|                | v|     .      | I|
      |  |          | e|                | e|     .      |  |
      | L|          | r|                | r|            | L|
      | a|          |  |                |  |            | a|
      | y|          | L|                | L|            | y|
      | e|          | a|                | a|            | e|Data
      | r|          | y|                | y|            | r|Trans
      |  |          | e|                | e|            |  |Compl
      |  |          | r| DM Msg holding | r|            |  |
 SCSI |  |          |  |SCSI Resp PDU & |  |            |  |SCSI
 Resp |  |Ctrl_Ntfy |  |  Sense Data    |  |  Snd_Ctrl  |  |Resp
 <----|  |<---------|  |<---------------|  |<-----------|  |<----
      |  |          |  |                |  |            |  |
         Figure 11.  A SCSI Read Data Acknowledgement

Chadalapaka, et al. Informational [Page 43] RFC 5047 DA October 2007

      <-----Initiator----->                <-------Target------->
      |  |          |  |                |  |            |  |
 SCSI |  |          |  | DM Msg holding |  |            |  |SCSI
 Cmd  |  | Snd_Ctrl |  |  SCSI Cmd PDU  |  |Ctrl_Notify |  |Cmd
 ---->|  |--------->|  |--------------->|  |----------->|  |---->
      |  |          |  |                |  |            |  |
      |  |          | D|    SCSI Read   | D|            |  |Buf
      |  |          | a|  Data Transfer | a|  Put_Data  |  |Avail
      | i|          | t|<---------------| t|<-----------| i|<----
      | S|          | a|        .       | a|     .      | S|  .
 Abort| C|          | m| DM Msg holding | m|     .      | C|Abort
 Task | S| Snd_Ctrl | o|  Abort TMF Req | o|Ctrl_Notify | S|Task
 ---->| I|--------->| v|--------------->| v|----------->| I|---->
      |  |          | e|       .        | e|     .      |  |
 Abort| L|          | r|  DM Msg holding| r|            | L| .
 Done | a|Ctrl_Ntfy |  |   Abort TMF Res|  | Snd_Ctrl   |  |Abted
 <----| y|<---------| L|<---------------| L|<-----------| y|<----
      | e|          | a|                | a|            | e|
      | r|          | y|                | y|            | r|
      |  |          | e|                | e|            |  |
      |  |          | r|                | r|            |  |
      |  |          |  |                |  |            |  |
      |  |Dal_Tk_Res|  |                |  |Dal_Tk_Res  |  |
      |  |--------->|  |                |  |<-----------|  |
      |  |          |  |                |  |            |  |
          Figure 12.  Task Resource Cleanup on Abort

Acknowledgements

 The IP Storage (IPS) Working Group in the Transport Area of
 IETF has been responsible for defining the iSCSI protocol
 (apart from a host of other relevant IP Storage protocols).
 The authors are grateful to the entire working group, whose
 work allowed this document to build on the concepts and
 details of the iSCSI protocol.
 In addition, the following individuals reviewed and
 contributed to the improvement of this document.  The authors
 are grateful for their contribution.
 John Carrier
 Adaptec, Inc.
 691 S. Milpitas Blvd., Milpitas, CA 95035, USA
 Phone: +1 (360) 378-8526
 EMail: john_carrier@adaptec.com

Chadalapaka, et al. Informational [Page 44] RFC 5047 DA October 2007

 Hari Ghadia
 Adaptec, Inc.
 691 S. Milpitas Blvd., Milpitas, CA 95035, USA
 Phone: +1 (408) 957-5608
 EMail: hari_ghadia@adaptec.com
 Hari Mudaliar
 Adaptec, Inc.
 691 S. Milpitas Blvd., Milpitas, CA 95035, USA
 Phone: +1 (408) 957-6012
 EMail: hari_mudaliar@adaptec.com
 Patricia Thaler
 Agilent Technologies, Inc.
 1101 Creekside Ridge Drive, #100, M/S-RG10,
 Roseville, CA 95678, USA
 Phone: +1-916-788-5662
 EMail: pat_thaler@agilent.com
 Uri Elzur
 Broadcom Corporation
 16215 Alton Parkway, Irvine, CA 92619-7013, USA
 Phone: +1 (949) 585-6432
 EMail: Uri@Broadcom.com
 Mike Penna
 Broadcom Corporation
 16215 Alton Parkway,Irvine, CA 92619-7013, USA
 Phone: +1 (949) 926-7149
 EMail: MPenna@Broadcom.com
 David Black
 EMC Corporation
 176 South St., Hopkinton, MA 01748, USA
 Phone: +1 (508) 293-7953
 EMail: black_david@emc.com
 Ted Compton
 EMC Corporation
 Research Triangle Park, NC 27709, USA
 Phone: +1-919-248-6075
 EMail: compton_ted@emc.com
 Dwight Barron
 Hewlett-Packard Company
 20555 SH 249, Houston, TX 77070-2698, USA
 Phone: +1 (281) 514-2769
 EMail: Dwight.Barron@Hp.com

Chadalapaka, et al. Informational [Page 45] RFC 5047 DA October 2007

 Paul R. Culley
 Hewlett-Packard Company
 20555 SH 249, Houston, TX 77070-2698, USA
 Phone: +1 (281) 514-5543
 EMail: paul.culley@hp.com
 Dave Garcia
 Hewlett-Packard Company
 19333 Vallco Parkway, Cupertino, CA 95014, USA
 Phone: +1 (408) 285-6116
 EMail: dave.garcia@hp.com
 Randy Haagens
 Hewlett-Packard Company
 8000 Foothills Blvd, MS 5668, Roseville CA, USA
 Phone: +1-916-785-4578
 EMail: randy_haagens@hp.com
 Jeff Hilland
 Hewlett-Packard Company
 20555 SH 249, Houston, TX 77070-2698, USA
 Phone: +1 (281) 514-9489
 EMail: jeff.hilland@hp.com
 Mike Krause
 Hewlett-Packard Company, 43LN
 19410 Homestead Road, Cupertino, CA 95014, USA
 Phone: +1 (408) 447-3191
 EMail: krause@cup.hp.com
 Jim Wendt
 Hewlett-Packard Company
 8000 Foothills Blvd, MS 5668, Roseville CA, USA
 Phone: +1-916-785-5198
 EMail: jim_wendt@hp.com
 Mike Ko
 IBM
 650 Harry Rd, San Jose, CA 95120, USA
 Phone: +1 (408) 927-2085
 EMail: mako@us.ibm.com
 Renato Recio
 IBM Corporation
 11501 Burnett Road, Austin, TX 78758, USA
 Phone: +1 (512) 838-1365
 EMail: recio@us.ibm.com

Chadalapaka, et al. Informational [Page 46] RFC 5047 DA October 2007

 Howard C. Herbert
 Intel Corporation
 MS CH7-404,5000 West Chandler Blvd., Chandler, AZ 85226, USA
 Phone: +1 (480) 554-3116
 EMail: howard.c.herbert@intel.com
 Dave Minturn
 Intel Corporation
 MS JF1-210, 5200 North East Elam Young Parkway
 Hillsboro, OR 97124, USA
 Phone: +1 (503) 712-4106
 EMail: dave.b.minturn@intel.com
 James Pinkerton
 Microsoft Corporation
 One Microsoft Way, Redmond, WA 98052, USA
 Phone: +1 (425) 705-5442
 EMail: jpink@microsoft.com
 Tom Talpey
 Network Appliance
 375 Totten Pond Road, Waltham, MA 02451, USA
 Phone: +1 (781) 768-5329
 EMail: thomas.talpey@netapp.com

Chadalapaka, et al. Informational [Page 47] RFC 5047 DA October 2007

Authors' Addresses

 Mallikarjun Chadalapaka
 Hewlett-Packard Company
 8000 Foothills Blvd.
 Roseville, CA 95747-5668, USA
 Phone: +1-916-785-5621
 EMail: cbm@rose.hp.com
 John L. Hufferd
 Brocade, Inc.
 1745 Technology Drive
 San Jose, CA 95110, USA
 Phone: +1-408-333-5244
 EMail: jhufferd@brocade.com
 Julian Satran
 IBM, Haifa Research Lab
 Haifa University Campus - Mount Carmel
 Haifa 31905, Israel
 Phone +972-4-829-6264
 EMail: Julian_Satran@il.ibm.com
 Hemal Shah
 Broadcom Corporation
 5300 California Avenue
 Irvine, California 92617, USA
 Phone: +1-949-926-6941
 EMail: hemal@broadcom.com
 Comments may be sent to Mallikarjun Chadalapaka.

Chadalapaka, et al. Informational [Page 48] RFC 5047 DA October 2007

Full Copyright Statement

 Copyright (C) The IETF Trust (2007).
 This document is subject to the rights, licenses and restrictions
 contained in BCP 78, and except as set forth therein, the authors
 retain all their rights.
 This document and the information contained herein are provided on an
 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
 OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

 The IETF takes no position regarding the validity or scope of any
 Intellectual Property Rights or other rights that might be claimed to
 pertain to the implementation or use of the technology described in
 this document or the extent to which any license under such rights
 might or might not be available; nor does it represent that it has
 made any independent effort to identify any such rights.  Information
 on the procedures with respect to rights in RFC documents can be
 found in BCP 78 and BCP 79.
 Copies of IPR disclosures made to the IETF Secretariat and any
 assurances of licenses to be made available, or the result of an
 attempt made to obtain a general license or permission for the use of
 such proprietary rights by implementers or users of this
 specification can be obtained from the IETF on-line IPR repository at
 http://www.ietf.org/ipr.
 The IETF invites any interested party to bring to its attention any
 copyrights, patents or patent applications, or other proprietary
 rights that may cover technology that may be required to implement
 this standard.  Please address the information to the IETF at
 ietf-ipr@ietf.org.

Chadalapaka, et al. Informational [Page 49]

/data/webs/external/dokuwiki/data/pages/rfc/rfc5047.txt · Last modified: 2007/10/27 00:16 by 127.0.0.1

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki