GENWiki

Premier IT Outsourcing and Support Services within the UK

User Tools

Site Tools


rfc:rfc3640

Network Working Group J. van der Meer Request for Comments: 3640 Philips Electronics Category: Standards Track D. Mackie

                                                        Apple Computer
                                                        V. Swaminathan
                                                 Sun Microsystems Inc.
                                                             D. Singer
                                                        Apple Computer
                                                            P. Gentric
                                                   Philips Electronics
                                                         November 2003
   RTP Payload Format for Transport of MPEG-4 Elementary Streams

Status of this Memo

 This document specifies an Internet standards track protocol for the
 Internet community, and requests discussion and suggestions for
 improvements.  Please refer to the current edition of the "Internet
 Official Protocol Standards" (STD 1) for the standardization state
 and status of this protocol.  Distribution of this memo is unlimited.

Copyright Notice

 Copyright (C) The Internet Society (2003).  All Rights Reserved.

Abstract

 The Motion Picture Experts Group (MPEG) Committee (ISO/IEC JTC1/SC29
 WG11) is a working group in ISO that produced the MPEG-4 standard.
 MPEG defines tools to compress content such as audio-visual
 information into elementary streams.  This specification defines a
 simple, but generic RTP payload format for transport of any non-
 multiplexed MPEG-4 elementary stream.

Table of Contents

 1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
 2.  Carriage of MPEG-4 Elementary Streams Over RTP . . . . . . . .  4
     2.1.  Signaling by MIME Format Parameters  . . . . . . . . . .  4
     2.2.  MPEG Access Units  . . . . . . . . . . . . . . . . . . .  5
     2.3.  Concatenation of Access Units  . . . . . . . . . . . . .  5
     2.4.  Fragmentation of Access Units  . . . . . . . . . . . . .  6
     2.5.  Interleaving . . . . . . . . . . . . . . . . . . . . . .  6
     2.6.  Time Stamp Information . . . . . . . . . . . . . . . . .  7
     2.7.  State Indication of MPEG-4 System Streams  . . . . . . .  8
     2.8.  Random Access Indication . . . . . . . . . . . . . . . .  8

van der Meer, et al. Standards Track [Page 1] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

     2.9.  Carriage of Auxiliary Information  . . . . . . . . . . .  8
     2.10. MIME Format Parameters and Configuring Conditional Field  8
     2.11. Global Structure of Payload Format . . . . . . . . . . .  9
     2.12. Modes to Transport MPEG-4 Streams  . . . . . . . . . . .  9
     2.13. Alignment with RFC 3016  . . . . . . . . . . . . . . . . 10
 3.  Payload Format . . . . . . . . . . . . . . . . . . . . . . . . 10
     3.1.  Usage of RTP Header Fields and RTCP  . . . . . . . . . . 10
     3.2.  RTP Payload Structure  . . . . . . . . . . . . . . . . . 11
           3.2.1.  The AU Header Section  . . . . . . . . . . . . . 11
                   3.2.1.1.  The AU-header  . . . . . . . . . . . . 12
           3.2.2.  The Auxiliary Section . . . . . . . . . . . . .  14
           3.2.3.  The Access Unit Data Section . . . . . . . . . . 15
                   3.2.3.1.  Fragmentation. . . . . . . . . . . . . 16
                   3.2.3.2.  Interleaving . . . . . . . . . . . . . 16
                   3.2.3.3.  Constraints for Interleaving . . . . . 17
                   3.2.3.4.  Crucial and Non-Crucial AUs with
                             MPEG-4 System Data . . . . . . . . . . 20
     3.3.  Usage of this Specification. . . . . . . . . . . . . . . 21
           3.3.1.  General. . . . . . . . . . . . . . . . . . . . . 21
           3.3.2.  The Generic Mode . . . . . . . . . . . . . . . . 22
           3.3.3.  Constant Bit Rate CELP . . . . . . . . . . . . . 22
           3.3.4.  Variable Bit Rate CELP . . . . . . . . . . . . . 23
           3.3.5.  Low Bit Rate AAC . . . . . . . . . . . . . . . . 24
           3.3.6.  High Bit Rate AAC. . . . . . . . . . . . . . . . 25
           3.3.7.  Additional Modes . . . . . . . . . . . . . . . . 26
 4.  IANA Considerations. . . . . . . . . . . . . . . . . . . . . . 27
     4.1.  MIME Type Registration . . . . . . . . . . . . . . . . . 27
     4.2.  Registration of Mode Definitions with IANA . . . . . . . 33
     4.3.  Concatenation of Parameters. . . . . . . . . . . . . . . 33
     4.4.  Usage of SDP . . . . . . . . . . . . . . . . . . . . . . 34
           4.4.1.  The a=fmtp Keyword . . . . . . . . . . . . . . . 34
 5.  Security Considerations. . . . . . . . . . . . . . . . . . . . 34
 6.  Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 35
 APPENDIX: Usage of this Payload Format. . .  . . . . . . . . . . . 36
 Appendix A.  Interleave Analysis . . . . . . . . . . . . . . . . . 36
 A.  Examples of Delay Analysis with Interleave. . .  . . . . . . . 36
     A.1.  Introduction . . . . . . . . . . . . . . . . . . . . . . 36
     A.2.  De-interleaving and Error Concealment  . . . . . . . . . 36
     A.3.  Simple Group Interleave  . . . . . . . . . . . . . . . . 36
           A.3.1.  Introduction . . . . . . . . . . . . . . . . . . 36
           A.3.2.  Determining the De-interleave Buffer Size  . . . 37
           A.3.3.  Determining the Maximum Displacement . . . . . . 37
     A.4.  More Subtle Group Interleave . . . . . . . . . . . . . . 38
           A.4.1.  Introduction . . . . . . . . . . . . . . . . . . 38
           A.4.2.  Determining the De-interleave Buffer Size. . . . 38
           A.4.3.  Determining the Maximum Displacement . . . . . . 39
     A.5.  Continuous Interleave  . . . . . . . . . . . . . . . . . 39
           A.5.1.  Introduction . . . . . . . . . . . . . . . . . . 39

van der Meer, et al. Standards Track [Page 2] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

           A.5.2.  Determining the De-interleave Buffer Size  . . . 40
           A.5.3.  Determining the Maximum Displacement . . . . . . 40
 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
 Normative References . . . . . . . . . . . . . . . . . . . . . . . 41
 Informative References . . . . . . . . . . . . . . . . . . . . . . 41
 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 42
 Full Copyright Statement . . . . . . . . . . . . . . . . . . . . . 43

1. Introduction

 The MPEG Committee is Working Group 11 (WG11) in ISO/IEC JTC1 SC29
 that specified the MPEG-1, MPEG-2 and, more recently, the MPEG-4
 standards [1].  The MPEG-4 standard specifies compression of audio-
 visual data into, for example an audio or video elementary stream.
 In the MPEG-4 standard, these streams take the form of audio-visual
 objects that may be arranged into an audio-visual scene by means of a
 scene description.  Each MPEG-4 elementary stream consists of a
 sequence of Access Units; examples of an Access Unit (AU) are an
 audio frame and a video picture.
 This specification defines a general and configurable payload
 structure to transport MPEG-4 elementary streams, in particular
 MPEG-4 audio (including speech) streams, MPEG-4 video streams and
 also MPEG-4 systems streams, such as BIFS (BInary Format for Scenes),
 OCI (Object Content Information), OD (Object Descriptor) and IPMP
 (Intellectual Property Management and Protection) streams.  The RTP
 payload defined in this document is simple to implement and
 reasonably efficient.  It allows for optional interleaving of Access
 Units (such as audio frames) to increase error resiliency in packet
 loss.
 Some types of MPEG-4 elementary streams include "crucial" information
 whose loss cannot be tolerated.  However, RTP does not provide
 reliable transmission, so receipt of that crucial information is not
 assured.  Section 3.2.3.4 specifies how stream state is conveyed so
 that the receiver can detect the loss of crucial information and
 cease decoding until the next random access point has been received.
 Applications transmitting streams that include crucial information,
 such as OD commands, BIFS commands, or programmatic content such as
 MPEG-J (Java) and ECMAScript, should include random access points, at
 a suitable periodicity depending upon the probability of loss, in
 order to reduce stream corruption to an acceptable level.  An example
 is the carousel mechanism as defined by MPEG in ISO/IEC 14496-1 [1].
 Such applications may also employ additional protocols or services to
 reduce the probability of loss.  At the RTP layer, these measures
 include payload formats and profiles for retransmission or forward
 error correction (such as in RFC 2733 [10]), that must be employed

van der Meer, et al. Standards Track [Page 3] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 with due consideration to congestion control.  Another solution that
 may be appropriate for some applications is to carry RTP over TCP
 (such as in RFC 2326 [8], section 10.12).  At the network layer,
 resource allocation or preferential service may be available to
 reduce the probability of loss.  For a general description of methods
 to repair streaming media, see RFC 2354 [9].
 Though the RTP payload format defined in this document is capable of
 transporting any MPEG-4 stream, other, more specific, formats may
 exist, such as RFC 3016 [12] for transport of MPEG-4 video (ISO/IEC
 14496 [1] part 2).
 Configuration of the payload is provided to accommodate the
 transportation of any MPEG-4 stream at any possible bit rate.
 However, for a specific MPEG-4 elementary stream typically only very
 few configurations are needed.  So as to allow for the design of
 simplified, but dedicated receivers, this specification requires that
 specific modes be defined for transport of MPEG-4 streams.  This
 document defines modes for MPEG-4 CELP and AAC streams, as well as a
 generic mode that can be used to transport any MPEG-4 stream.  In the
 future, new RFCs are expected to specify additional modes for the
 transportation of MPEG-4 streams.
 The RTP payload format defined in this document specifies carriage of
 system-related information that is often equivalent to the
 information that may be contained in the MPEG-4 Sync Layer (SL) as
 defined in MPEG-4 Systems [1].  This document does not prescribe how
 to transcode or map information from the SL to fields defined in the
 RTP payload format.  Such processing, if any, is left to the
 discretion of the application.  However, to anticipate the need for
 the transportation of any additional system-related information in
 the future, an auxiliary field can be configured that may carry any
 such data.
 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
 "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this
 document are to be interpreted as described in BCP 14, RFC 2119 [4].

2. Carriage of MPEG-4 Elementary Streams over RTP

2.1. Signaling by MIME Format Parameters

 With this payload format, a single MPEG-4 elementary stream can be
 transported.  Information on the type of MPEG-4 stream carried in the
 payload is conveyed by MIME format parameters, as in an SDP [5]
 message or by other means (see section 4).  These MIME format
 parameters specify the configuration of the payload.  To allow for
 simplified and dedicated receivers, a MIME format parameter is

van der Meer, et al. Standards Track [Page 4] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 available to signal a specific mode of using this payload.  A mode
 definition MAY include the type of MPEG-4 elementary stream, as well
 as the applied configuration, so as to avoid the need for receivers
 to parse all MIME format parameters.  The applied mode MUST be
 signaled.

2.2. MPEG Access Units

 For carriage of compressed audio-visual data, MPEG defines Access
 Units.  An MPEG Access Unit (AU) is the smallest data entity to which
 timing information is attributed.  In the case of audio, an Access
 Unit may represent an audio frame and in the case of video, a
 picture.  MPEG Access Units are octet-aligned by definition.  If, for
 example, an audio frame is not octet-aligned, up to 7 zero-padding
 bits MUST be inserted at the end of the frame to achieve the octet-
 aligned Access Units, as required by the MPEG-4 specification.
 MPEG-4 decoders MUST be able to decode AUs in which such padding is
 applied.
 Consistent with the MPEG-4 specification, this document requires that
 each MPEG-4 part 2 video Access Unit include all the coded data of a
 picture, any video stream headers that may precede the coded picture
 data, and any video stream stuffing that may follow it, up to but not
 including the startcode indicating the start of a new video stream or
 the next Access Unit.

2.3. Concatenation of Access Units

 Frequently it is possible to carry multiple Access Units in one RTP
 packet.  This is particularly useful for audio; for example, when AAC
 is used for encoding a stereo signal at 64 kbits/sec, AAC frames
 contain on average, approximately 200 octets.  On a LAN with a 1500
 octet MTU, this would allow an average of 7 complete AAC frames to be
 carried per RTP packet.
 Access Units may have a fixed size in octets, but a variable size is
 also possible.  To facilitate parsing in the case of multiple
 concatenated AUs in one RTP packet, the size of each AU is made known
 to the receiver.  When concatenating in the case of a constant AU
 size, this size is communicated "out of band" through a MIME format
 parameter.  When concatenating in case of variable size AUs, the RTP
 payload carries "in band" an AU size field for each contained AU.
 In combination with the RTP payload length, the size information
 allows the RTP payload to be split by the receiver back into the
 individual AUs.

van der Meer, et al. Standards Track [Page 5] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 To simplify the implementation of RTP receivers, it is required that
 when multiple AUs are carried in an RTP packet, each AU MUST be
 complete, i.e., the number of AUs in an RTP packet MUST be integral.
 In addition, an AU MUST NOT be repeated in other RTP packets; hence
 repetition of an AU is only possible when using a duplicate RTP
 packet.

2.4. Fragmentation of Access Units

 MPEG allows for very large Access Units.  Since most IP networks have
 significantly smaller MTU sizes, this payload format allows for the
 fragmentation of an Access Unit over multiple RTP packets.  Hence,
 when an IP packet is lost after IP-level fragmentation, only an AU
 fragment may get lost instead of the entire AU.  To simplify the
 implementation of RTP receivers, an RTP packet SHALL either carry one
 or more complete Access Units or a single fragment of one AU, i.e.,
 packets MUST NOT contain fragments of multiple Access Units.

2.5. Interleaving

 When an RTP packet carries a contiguous sequence of Access Units, the
 loss of such a packet can result in a "decoding gap" for the user.
 One method of alleviating this problem is to allow for the Access
 Units to be interleaved in the RTP packets.  For a modest cost in
 latency and implementation complexity, significant error resiliency
 to packet loss can be achieved.
 To support optional interleaving of Access Units, this payload format
 allows for index information to be sent for each Access Unit.  After
 informing receivers about buffer resources to allocate for de-
 interleaving, the RTP sender is free to choose the interleaving
 pattern without propagating this information a priori to the
 receiver(s).  Indeed, the sender could dynamically adjust the
 interleaving pattern based on the Access Unit size, error rates, etc.
 The RTP receiver does not need to know the interleaving pattern used;
 it only needs to extract the index information of the Access Unit and
 insert the Access Unit into the appropriate sequence in the decoding
 or rendering queue.  An example of interleaving is given below.

van der Meer, et al. Standards Track [Page 6] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 For example, if we assume that an RTP packet contains 3 AUs, and that
 the AUs are numbered 0, 1, 2, 3, 4, and so forth, and if an
 interleaving group length of 9 is chosen, then RTP packet(i) contains
 the following AU(n):
    RTP packet(0):  AU(0),  AU(3),  AU(6)
    RTP packet(1):  AU(1),  AU(4),  AU(7)
    RTP packet(2):  AU(2),  AU(5),  AU(8)
    RTP packet(3):  AU(9),  AU(12), AU(15)
    RTP packet(4):  AU(10), AU(13), AU(16)  Etc.

2.6. Time Stamp Information

 The RTP time stamp MUST carry the sampling instant of the first AU
 (fragment) in the RTP packet.  When multiple AUs are carried within
 an RTP packet, the time stamps of subsequent AUs can be calculated if
 the frame period of each AU is known.  For audio and video, this is
 possible if the frame rate is constant.  However, in some cases it is
 not possible to make such a calculation (for example, for variable
 frame rate video, or for MPEG-4 BIFS streams carrying composition
 information).  To support such cases, this payload format can be
 configured to carry a time stamp in the RTP payload for each
 contained Access Unit.  A time stamp MAY be conveyed in the RTP
 payload only for non-first AUs in the RTP packet, and SHALL NOT be
 conveyed for the first AU (fragment), as the time stamp for the first
 AU in the RTP packet is carried by the RTP time stamp.
 MPEG-4 defines two types of time stamps: the composition time stamp
 (CTS) and the decoding time stamp (DTS).  The CTS represents the
 sampling instant of an AU, and hence the CTS is equivalent to the RTP
 time stamp.  The DTS may be used in MPEG-4 video streams that use
 bi-directional coding, i.e., when pictures are predicted in both
 forward and backward direction by using either a reference picture in
 the past, or a reference picture in the future.  The DTS cannot be
 carried in the RTP header.  In some cases, the DTS can be derived
 from the RTP time stamp using frame rate information; this requires
 deep parsing in the video stream, which may be considered
 objectionable.  If the video frame rate is variable, the required
 information may not even be present in the video stream.  For both
 reasons, the capability has been defined to optionally carry the DTS
 in the RTP payload for each contained Access Unit.
 To keep the coding of time stamps efficient, each time stamp
 contained in the RTP payload is coded as a difference.  For the CTS,
 the offset from the RTP time stamps is provided, and for the DTS, the
 offset from the CTS.

van der Meer, et al. Standards Track [Page 7] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

2.7. State Indication of MPEG-4 System Streams

 ISO/IEC 14496-1 defines states for MPEG-4 system streams.  So as to
 convey state information when transporting MPEG-4 system streams,
 this payload format allows for the optional carriage in the RTP
 payload of the stream state for each contained Access Unit.  Stream
 states are used to signal "crucial" AUs that carry information whose
 loss cannot be tolerated and are also useful when repeating AUs
 according to the carousel mechanism defined in ISO/IEC 14496-1.

2.8. Random Access Indication

 Random access to the content of MPEG-4 elementary streams may be
 possible at some but not all Access Units.  To signal Access Units
 where random access is possible, a random access point flag can
 optionally be carried in the RTP payload for each contained Access
 Unit.  Carriage of random access points is particularly useful for
 MPEG-4 system streams in combination with the stream state.

2.9. Carriage of Auxiliary Information

 This payload format defines a specific field to carry auxiliary data.
 The auxiliary data field is preceded by a field that specifies the
 length of the auxiliary data, so as to facilitate the skipping of
 data without parsing it.  The coding of the auxiliary data is not
 defined in this document; instead, the format, meaning and signaling
 of auxiliary information is expected to be specified in one or more
 future RFCs.  Auxiliary information MUST NOT be transmitted until its
 format, meaning and signaling have been specified and its use has
 been signaled.  Receivers that have knowledge of the auxiliary data
 MAY decode the auxiliary data, but receivers without knowledge of
 such data MUST skip the auxiliary data field.

2.10. MIME Format Parameters and Configuring Conditional Fields

 To support the features described in the previous sections, several
 fields are defined for carriage in the RTP payload.  However, their
 use strongly depends on the type of MPEG-4 elementary stream that is
 carried.  Sometimes a specific field is needed with a certain length,
 while in other cases such a field is not needed.  To be efficient in
 either case, the fields to support these features are configurable by
 means of MIME format parameters.  In general, a MIME format parameter
 defines the presence and length of the associated field.  A length of
 zero indicates absence of the field.  As a consequence, parsing of
 the payload requires knowledge of MIME format parameters.  The MIME
 format parameters are conveyed to the receiver via SDP [5] messages,
 as specified in section 4.4.1, or through other means.

van der Meer, et al. Standards Track [Page 8] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

2.11. Global Structure of Payload Format

 The RTP payload following the RTP header, contains three octet-
 aligned data sections, of which the first two MAY be empty, see
 Figure 1.
       +---------+-----------+-----------+---------------+
       | RTP     | AU Header | Auxiliary | Access Unit   |
       | Header  | Section   | Section   | Data Section  |
       +---------+-----------+-----------+---------------+
                 <----------RTP Packet Payload----------->
          Figure 1: Data sections within an RTP packet
 The first data section is the AU (Access Unit) Header Section, that
 contains one or more AU-headers; however, each AU-header MAY be
 empty, in which case the entire AU Header Section is empty.  The
 second section is the Auxiliary Section, containing auxiliary data;
 this section MAY also be configured empty.  The third section is the
 Access Unit Data Section, containing either a single fragment of one
 Access Unit or one or more complete Access Units.  The Access Unit
 Data Section MUST NOT be empty.

2.12. Modes to Transport MPEG-4 Streams

 While it is possible to build fully configurable receivers capable of
 receiving any MPEG-4 stream, this specification also allows for the
 design of simplified, but dedicated receivers, that are for example,
 capable of receiving only one type of MPEG-4 stream.  This is
 achieved by requiring that specific modes be defined in order to use
 this specification.  Each mode may define constraints for transport
 of one or more types of MPEG-4 streams, for instance on the payload
 configuration.
 The applied mode MUST be signaled.  Signaling the mode is
 particularly important for receivers that are only capable of
 decoding one or more specific modes.  Such receivers need to
 determine whether the applied mode is supported, so as to avoid
 problems with processing of payloads that are beyond the capabilities
 of the receiver.
 In this document several modes are defined for the transportation of
 MPEG-4 CELP and AAC streams, as well as a generic mode that can be
 used for any MPEG-4 stream.  In the future, new RFCs may specify
 other modes of using this specification.  However, each mode MUST be
 in full compliance with this specification (see section 3.3.7).

van der Meer, et al. Standards Track [Page 9] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

2.13. Alignment with RFC 3016

 This payload can be configured as nearly identical to the payload
 format defined in RFC 3016 [12] for the MPEG-4 video configurations
 recommended in RFC 3016.  Hence, receivers that comply with RFC 3016
 can decode such RTP payload, provided that additional packets
 containing video decoder configuration (VO, VOL, VOSH) are inserted
 in the stream, as required by RFC 3016 [12].  Conversely, receivers
 that comply with the specification in this document SHOULD be able to
 decode payloads, names and parameters defined for MPEG-4 video in RFC
 3016 [12].  In this respect, it is strongly RECOMMENDED that the
 implementation provide the ability to ignore "in band" video decoder
 configuration packets that may be found in streams conforming to the
 RFC 3016 video payload.
 Note the "out of band" availability of the video decoder
 configuration is optional in RFC 3016 [12].  To achieve maximum
 interoperability with the RTP payload format defined in this
 document, applications that use RFC 3016 to transport MPEG-4 video
 (part 2) are recommended to make the video decoder configuration
 available as a MIME parameter.

3. Payload Format

3.1. Usage of RTP Header Fields and RTCP

 Payload Type (PT): The assignment of an RTP payload type for this
    packet format is outside the scope of this document; it is
    specified by the RTP profile under which this payload format is
    used, or signaled dynamically out-of-band (e.g., using SDP).
 Marker (M) bit: The M bit is set to 1 to indicate that the RTP packet
    payload contains either the final fragment of a fragmented Access
    Unit or one or more complete Access Units.
 Extension (X) bit: Defined by the RTP profile used.
 Sequence Number: The RTP sequence number SHOULD be generated by the
    sender in the usual manner with a constant random offset.
 Timestamp: Indicates the sampling instant of the first AU contained
    in the RTP payload.  This sampling instant is equivalent to the
    CTS in the MPEG-4 time domain.  When using SDP, the clock rate of
    the RTP time stamp MUST be expressed using the "rtpmap" attribute.
    If an MPEG-4 audio stream is transported, the rate SHOULD be set
    to the same value as the sampling rate of the audio stream.  If an
    MPEG-4 video stream is transported, it is RECOMMENDED that the
    rate be set to 90 kHz.

van der Meer, et al. Standards Track [Page 10] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 In all cases, the sender SHALL make sure that RTP time stamps are
 identical only if the RTP time stamp refers to fragments of the same
 Access Unit.
 According to RFC 3550 [2] (section 5.1), it is RECOMMENDED that RTP
 time stamps start at a random value for security reasons.  This is
 not an issue for synchronization of multiple RTP streams.  However,
 when streams from multiple sources are to be synchronized (for
 example one stream from local storage, another from an RTP streaming
 server), synchronization may become impossible if the receiver only
 knows the original time stamp relationships.  In such cases the time
 stamp relationship required for obtaining synchronization may be
 provided by out of band means.  The format of such information, as
 well as methods to convey such information, are beyond the scope of
 this specification.
 SSRC: set as described in RFC 3550 [2].
 CC and CSRC fields are used as described in RFC 3550 [2].
 RTCP SHOULD be used as defined in RFC 3550 [2].  Note that time
 stamps in RTCP Sender Reports may be used to synchronize multiple
 MPEG-4 elementary streams and also to synchronize MPEG-4 streams with
 non-MPEG-4 streams, in case the delivery of these streams uses RTP.

3.2. RTP Payload Structure

3.2.1. The AU Header Section

 When present, the AU Header Section consists of the AU-headers-length
 field, followed by a number of AU-headers, see Figure 2.
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+-+
    |AU-headers-length|AU-header|AU-header|      |AU-header|padding|
    |                 |   (1)   |   (2)   |      |   (n)   | bits  |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+-+
                 Figure 2: The AU Header Section
 The AU-headers are configured using MIME format parameters and MAY be
 empty.  If the AU-header is configured empty, the AU-headers-length
 field SHALL NOT be present and consequently the AU Header Section is
 empty.  If the AU-header is not configured empty, then the AU-
 headers-length is a two octet field that specifies the length in bits
 of the immediately following AU-headers, excluding the padding bits.
 Each AU-header is associated with a single Access Unit (fragment)
 contained in the Access Unit Data Section in the same RTP packet.

van der Meer, et al. Standards Track [Page 11] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 For each contained Access Unit (fragment), there is exactly one AU-
 header.  Within the AU Header Section, the AU-headers are bit-wise
 concatenated in the order in which the Access Units are contained in
 the Access Unit Data Section.  Hence, the n-th AU-header refers to
 the n-th AU (fragment).  If the concatenated AU-headers consume a
 non-integer number of octets, up to 7 zero-padding bits MUST be
 inserted at the end in order to achieve octet-alignment of the AU
 Header Section.

3.2.1.1. The AU-header

 Each AU-header may contain the fields given in Figure 3.  The length
 in bits of the fields, with the exception of the CTS-flag, the
 DTS-flag and the RAP-flag fields, is defined by MIME format
 parameters; see section 4.1.  If a MIME format parameter has the
 default value of zero, then the associated field is not present.  The
 number of bits for fields that are present and that represent the
 value of a parameter MUST be chosen large enough to correctly encode
 the largest value of that parameter during the session.
 If present, the fields MUST occur in the mutual order given in Figure
 3.  In the general case, a receiver can only discover the size of an
 AU-header by parsing it since the presence of the CTS-delta and DTS-
 delta fields is signaled by the value of the CTS-flag and DTS-flag,
 respectively.
    +---------------------------------------+
    |     AU-size                           |
    +---------------------------------------+
    |     AU-Index / AU-Index-delta         |
    +---------------------------------------+
    |     CTS-flag                          |
    +---------------------------------------+
    |     CTS-delta                         |
    +---------------------------------------+
    |     DTS-flag                          |
    +---------------------------------------+
    |     DTS-delta                         |
    +---------------------------------------+
    |     RAP-flag                          |
    +---------------------------------------+
    |     Stream-state                      |
    +---------------------------------------+
 Figure 3: The fields in the AU-header.  If used, the AU-Index field
           only occurs in the first AU-header within an AU Header
           Section; in any other AU-header, the AU-Index-delta field
           occurs instead.

van der Meer, et al. Standards Track [Page 12] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 AU-size: Indicates the size in octets of the associated Access Unit
    in the Access Unit Data Section in the same RTP packet.  When the
    AU-size is associated with an AU fragment, the AU size indicates
    the size of the entire AU and not the size of the fragment.  In
    this case, the size of the fragment is known from the size of the
    AU data section.  This can be exploited to determine whether a
    packet contains an entire AU or a fragment, which is particularly
    useful after losing a packet carrying the last fragment of an AU.
 AU-Index: Indicates the serial number of the associated Access Unit
    (fragment).  For each (in decoding order) consecutive AU or AU
    fragment, the serial number is incremented by 1.  When present,
    the AU-Index field occurs in the first AU-header in the AU Header
    Section, but MUST NOT occur in any subsequent (non-first) AU-
    header in that Section.  To encode the serial number in any such
    non-first AU-header, the AU-Index-delta field is used.
 AU-Index-delta: The AU-Index-delta field is an unsigned integer that
    specifies the serial number of the associated AU as the difference
    with respect to the serial number of the previous Access Unit.
    Hence, for the n-th (n>1) AU, the serial number is found from:
    AU-Index(n) = AU-Index(n-1) + AU-Index-delta(n) + 1
    If the AU-Index field is present in the first AU-header in the AU
    Header Section, then the AU-Index-delta field MUST be present in
    any subsequent (non-first) AU-header.  When the AU-Index-delta is
    coded with the value 0, it indicates that the Access Units are
    consecutive in decoding order.  An AU-Index-delta value larger
    than 0 signals that interleaving is applied.
 CTS-flag: Indicates whether the CTS-delta field is present.  A value
    of 1 indicates that the field is present, a value of 0 indicates
    that it is not present.
    The CTS-flag field MUST be present in each AU-header if the length
    of the CTS-delta field is signaled to be larger than zero.  In
    that case, the CTS-flag field MUST have the value 0 in the first
    AU-header and MAY have the value 1 in all non-first AU-headers.
    The CTS-flag field SHOULD be 0 for any non-first fragment of an
    Access Unit.

van der Meer, et al. Standards Track [Page 13] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 CTS-delta: Encodes the CTS by specifying the value of CTS as a 2's
    complement offset (delta) from the time stamp in the RTP header of
    this RTP packet.  The CTS MUST use the same clock rate as the time
    stamp in the RTP header.
 DTS-flag: Indicates whether the DTS-delta field is present.  A value
    of 1 indicates that DTS-delta is present, a value of 0 indicates
    that it is not present.
    The DTS-flag field MUST be present in each AU-header if the length
    of the DTS-delta field is signaled to be larger than zero.  The
    DTS-flag field MUST have the same value for all fragments of an
    Access Unit.
 DTS-delta: Specifies the value of the DTS as a 2's complement offset
    (delta) from the CTS.  The DTS MUST use the same clock rate as the
    time stamp in the RTP header.  The DTS-delta field MUST have the
    same value for all fragments of an Access Unit.
 RAP-flag: When set to 1, indicates that the associated Access Unit
    provides a random access point to the content of the stream.  If
    an Access Unit is fragmented, the RAP flag, if present, MUST be
    set to 0 for each non-first fragment of the AU.
 Stream-state:  Specifies the state of the stream for an AU of an
    MPEG-4 system stream; each state is identified by a value of a
    modulo counter.  In ISO/IEC 14496-1, MPEG-4 system streams use the
    AU_SequenceNumber to signal stream states.  When the stream state
    changes, the value of the stream-state MUST be incremented by one.
    Note: no relation is required between stream-states of different
    streams.

3.2.2. The Auxiliary Section

 The Auxiliary Section consists of the auxiliary-data-size field
 followed by the auxiliary-data field.  Receivers MAY (but are not
 required to) parse the auxiliary-data field; to facilitate skipping
 of the auxiliary-data field by receivers, the auxiliary-data-size
 field indicates the length in bits of the auxiliary-data.  If the
 concatenation of the auxiliary-data-size and the auxiliary-data
 fields consume a non-integer number of octets, up to 7 zero padding
 bits MUST be inserted immediately after the auxiliary data in order
 to achieve octet-alignment.  See Figure 4.

van der Meer, et al. Standards Track [Page 14] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+
    | auxiliary-data-size   | auxiliary-data       |padding bits |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+
         Figure 4: The fields in the Auxiliary Section
 The length in bits of the auxiliary-data-size field is configurable
 by a MIME format parameter; see section 4.1.  The default length of
 zero indicates that the entire Auxiliary Section is absent.
 auxiliary-data-size: specifies the length in bits of the immediately
    following auxiliary-data field;
 auxiliary-data: the auxiliary-data field contains data of a format
    not defined by this specification.

3.2.3. The Access Unit Data Section

 The Access Unit Data Section contains an integer number of complete
 Access Units or a single fragment of one AU.  The Access Unit Data
 Section is never empty.  If data of more than one Access Unit is
 present, then the AUs are concatenated into a contiguous string of
 octets.  See Figure 5.  The AUs inside the Access Unit Data Section
 MUST be in decoding order, though not necessarily contiguous in the
 case of interleaving.
 The size and number of Access Units SHOULD be adjusted such that the
 resulting RTP packet is not larger than the path MTU.  To handle
 larger packets, this payload format relies on lower layers for
 fragmentation, which may result in reduced performance.
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |AU(1)                                                          |
    +                                                               |
    |                                                               |
    |               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |               |AU(2)                                          |
    +-+-+-+-+-+-+-+-+                                               |
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               | AU(n)                         |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |AU(n) continued|
    |-+-+-+-+-+-+-+-+
      Figure 5: Access Unit Data Section; each AU is octet-aligned.

van der Meer, et al. Standards Track [Page 15] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 When multiple Access Units are carried, the size of each AU MUST be
 made available to the receiver.  If the AU size is variable, then the
 size of each AU MUST be indicated in the AU-size field of the
 corresponding AU-header.  However, if the AU size is constant for a
 stream, this mechanism SHOULD NOT be used; instead, the fixed size
 SHOULD be signaled by the MIME format parameter "constantSize"; see
 section 4.1.
 The absence of both AU-size in the AU-header and the constantSize
 MIME format parameter indicates the carriage of a single AU
 (fragment), i.e., that a single Access Unit (fragment) is transported
 in each RTP packet for that stream.

3.2.3.1. Fragmentation

 A packet SHALL carry either one or more complete Access Units, or a
 single fragment of an Access Unit.  Fragments of the same Access Unit
 have the same time stamp but different RTP sequence numbers.  The
 marker bit in the RTP header is 1 on the last fragment of an Access
 Unit, and 0 on all other fragments.

3.2.3.2. Interleaving

 Unless prohibited by the signaled mode, a sender MAY interleave
 Access Units.  Receivers that are capable of receiving modes that
 support interleaving MUST be able to decode interleaved Access Units.
 When a sender interleaves Access Units, it needs to provide
 sufficient information to enable a receiver to unambiguously
 reconstruct the original order, even in the case of out-of-order
 packets, packet loss or duplication.  The information that senders
 need to provide depends on whether or not the Access Units have a
 constant time duration.  Access Units have a constant time duration,
 if:
 TS(i+1) - TS(i) = constant
     for any i, where:
        i indicates the index of the AU in the original order, and
        TS(i) denotes the time stamp of AU(i)
 The MIME parameter "constantDuration" SHOULD be used to signal that
 Access Units have a constant time duration; see section 4.1.
 If the "constantDuration" parameter is present, the receiver can
 reconstruct the original Access Unit timing based solely on the RTP
 timestamp and AU-Index-delta.  Accordingly, when transmitting Access
 Units of constant duration, the AU-Index, if present, MUST be set to

van der Meer, et al. Standards Track [Page 16] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 the value 0.  Receivers of constant duration Access Units MUST use
 the RTP timestamp to determine the index of the first AU in the RTP
 packet.  The AU-Index-delta header and the signaled
 "constantDuration" are used to reconstruct AU timing.
 If the "constantDuration" parameter is not present, then senders MAY
 signal AUs of constant duration by coding the AU-Index with zero in
 each RTP packet.  In the absence of the constantDuration parameter
 receivers MUST conclude that the AUs have constant duration if the
 AU-index is zero in two consecutive RTP packets.
 When transmitting Access Units of variable duration, then the
 "constantDuration" parameter MUST NOT be present, and the transmitter
 MUST use the AU-Index to encode the index information required for
 re-ordering, and the receiver MUST use that value to determine the
 index of each AU in the RTP packet.  The number of bits of the AU-
 Index field MUST be chosen so that valid index information is
 provided at the applied interleaving scheme, without causing problems
 due to roll-over of the AU-Index field.  In addition, the CTS-delta
 MUST be coded in the AU header for each non-first AU in the RTP
 packet, so that receivers can place the AUs correctly in time.
 When interleaving is applied, a de-interleave buffer is needed in
 receivers to put the Access Units in their correct logical
 consecutive decoding order.  This requires the computation of the
 time stamp for each Access Unit.  In case of a constant time duration
 per Access Unit, the time stamp of the i-th access unit in an RTP
 packet with RTP time stamp T is calculated as follows:
 Timestamp[0] = T
 Timestamp[i, i > 0] = T +(Sum(for k=1 to i of (AU-Index-delta[k]
                       + 1))) * access-unit-duration
 When AU-Index-delta is always 0, this reduces to T + i * (access-
 unit-duration).  This is the non-interleaved case, where the frames
 are consecutive in decoding order.  Note that the AU-Index field
 (present for the first Access Unit) is indeed not needed in this
 calculation.

3.2.3.3. Constraints for Interleaving

 The size of the packets should be suitably chosen to be appropriate
 to both the path MTU and the capacity of the receiver's de-interleave
 buffer.  The maximum packet size for a session SHOULD be chosen to
 not exceed the path MTU.

van der Meer, et al. Standards Track [Page 17] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 To allow receivers to allocate sufficient resources for de-
 interleaving, senders MUST provide the information to receivers as
 specified in this section.
 AUs enter the decoder in decoding order.  The de-interleave buffer is
 used to re-order a stream of interleaved AUs back into decoding
 order.  When interleaving is applied, the decoding of "early" AUs has
 to be postponed until all AUs that precede it in decoding order are
 present.  Therefore, these "early" AUs are stored in the de-
 interleave buffer.  As an example in Figure 6, the interleaving
 pattern from section 2.5 is considered.
                           +--+--+--+--+--+--+--+--+--+--+--+-
 Interleaved AUs           | 0| 3| 6| 1| 4| 7| 2| 5| 8| 9|12|..
                           +--+--+--+--+--+--+--+--+--+--+--+-
 Storage of "early" AUs         3  3  3  3  3  3
                                   6  6  6  6  6  6
                                         4  4  4
                                            7  7  7
                                                          12 12
 Figure 6: Storage of "early" AUs in the de-interleave buffer per
           interleaved AU.
 AU(3) is to be delivered to the decoder after AU(0), AU(1) and AU(2);
 of these AUs, AU(2) arrives from the network last and hence AU(3)
 needs to be stored until AU(2) is present in the pattern.  Similarly,
 AU(6) is to be stored until AU(5) is present, while AU(4) and AU(7)
 are to be stored until AU(2) and AU(5) are present, respectively.
 Note that the fullness of the de-interleave buffer varies in time.
 In Figure 6, the de-interleave buffer contains at most 4, but often
 less AUs.
 So as to give a rough indication of the resources needed in the
 receiver for de-interleaving, the maximum displacement in time of an
 AU is defined.  For any AU(j) in the pattern, each AU(i) with i<j
 that is not yet present can be determined.  The maximum displacement
 in time of an AU is the maximum difference between the time stamp of
 an AU in the pattern and the time stamp of the earliest AU that is
 not yet present.  In other words, when considering a sequence of
 interleaved AUs, then:
 Maximum displacement = max{TS(i) - TS(j)}
     for any i and any j>i, where:
        i and j indicate the index of the AU in the interleaving
              pattern, and
        TS denotes the time stamp of the AU.

van der Meer, et al. Standards Track [Page 18] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 As an example in Figure 7, the interleaving pattern from section 2.5
 is considered.  For each AU in the pattern, the index is given of the
 earliest of any earlier AUs not yet present.  Hence for each AU(n) in
 the interleaving pattern the smallest index k (with k<n) of not yet
 delivered AUs is indicated.  A "-" indicates that all previous AUs
 are present.  If the AU period is constant, the maximum displacement
 equals 5 AU periods, as found for AU(6) and AU(7).
                               +--+--+--+--+--+--+--+--+--+--+--+-
 Interleaved AUs               | 0| 3| 6| 1| 4| 7| 2| 5| 8| 9|12|..
                               +--+--+--+--+--+--+--+--+--+--+--+-
 Earliest not yet present AU     -  1  1  -  2  2  -  -  -  - 10
 Figure 7: For each AU in the interleaving pattern, the earliest of
           any earlier AUs not yet present
 When interleaving, senders MUST signal the maximum displacement in
 time during the session via the MIME format parameter
 "maxDisplacement"; see section 4.1.
 An estimate of the size of the de-interleave buffer is found by
 multiplying the maximum displacement by the maximum bit rate:
 size(de-interleave buffer) = {(maxDisplacement) * Rate(max)} / (RTP
                              clock frequency),
     where:
        Rate(max) is the maximum bit-rate of the transported stream.
 Note that receivers can derive Rate(max) from the MIME format
 parameters streamType, profile-level-id, and config.
 However, this calculation estimates the size of the de-interleave
 buffer and the required size may differ from the calculated value.
 If this calculation under-estimates the size of the
 de-interleave buffer, then senders, when interleaving, MUST signal a
 size of the de-interleave buffer via the MIME format parameter
 "de-interleaveBufferSize"; see section 4.1.  If the calculation
 over-estimates the size of the de-interleave buffer, then senders,
 when interleaving, MAY signal a size of the de-interleave buffer via
 the MIME format parameter "de-interleaveBufferSize".

van der Meer, et al. Standards Track [Page 19] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 The signaled size of the de-interleave buffer MUST be large enough to
 contain all "early" AUs at any point in time during the session.
 That is:
 minimum de-interleave buffer size = max [sum {if TS(i) > TS(j) then
                                     AU-size(i) else 0}]
     for any j and any i<j, where:
        i and j indicate the index of an AU in the interleaving
              pattern,
        TS(i) denotes the time stamp of AU(i), and
        AU-size(i) denotes the size of AU(i) in number of octets.
 If the "de-interleaveBufferSize" parameter is present, then the
 applied buffer for de-interleaving in a receiver MUST have a size
 that is at least equal to the signaled size of the de-interleave
 buffer, else a size that is at least equal to the calculated size of
 the de-interleave buffer.
 No matter what interleaving scheme is used, the scheme must be
 analyzed to calculate the applicable maxDisplacement value, as well
 as the required size of the de-interleave buffer.  Senders SHOULD
 signal values that are not larger than the strictly required values;
 if larger values are signaled, the receiver will buffer excessively.
 Note that for low bit-rate material, the applied interleaving may
 make packets shorter than the MTU size.

3.2.3.4. Crucial and Non-Crucial AUs with MPEG-4 System Data

 Some Access Units with MPEG-4 system data, called "crucial" AUs,
 carry information whose loss cannot be tolerated, either in the
 presentation or in the decoder.  At each crucial AU in an MPEG-4
 system stream, the stream state changes.  The stream-state MAY remain
 constant at non-crucial AUs.  In ISO/IEC 14496-1, MPEG-4 system
 streams use the AU_SequenceNumber to signal stream states.
 Example: Given three AUs, AU1 = "Insertion of node X", AU2 = "Set
 position of node X", AU3 = "Set position of node X".  AU1 is crucial,
 since if it is lost, AU2 cannot be executed.  However, AU2 is not
 crucial, since AU3 can be executed even if AU2 is lost.
 When a crucial AU is (possibly) lost, the stream is corrupted.  For
 example, when an AU is lost and the stream state has changed at the
 next received AU, then it is possible that the lost AU was crucial.
 Once corrupted, the stream remains corrupted until the next random
 access point.  Note that loss of non-crucial AUs does not corrupt the
 stream.  When a decoder starts receiving a stream, the decoder MUST

van der Meer, et al. Standards Track [Page 20] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 consider the stream corrupted until an AU is received that provides a
 random access point.
 An AU that provides a random access point, as signaled by the RAP-
 flag, may or may not be crucial.  Non-crucial RAP AUs provide a
 "repeated" random access point for use by decoders that recently
 joined the stream or that need to re-start decoding after a stream
 corruption.  Non-crucial RAP AUs MUST include all updates since the
 last crucial RAP AU.
 Upon receiving AUs, decoders are to react as follows:
 a) if the RAP-flag is set to 1 and the stream-state changes, then the
    AU is a crucial RAP AU, and the AU MUST be decoded.
 b) if the RAP-flag is set to 1 and the stream state does not change,
    then the AU is a non-crucial RAP AU, and the receiver SHOULD
    decode it if the stream is corrupted.  Otherwise, the decoder MUST
    ignore the AU.
 c) if the RAP-flag is set to 0, then the AU MUST be decoded, unless
    the stream is corrupted, in which case the AU MUST be ignored.

3.3. Usage of this Specification

3.3.1. General

 Usage of this specification requires definition of a mode.  A mode
 defines how to use this specification, as deemed appropriate.
 Senders MUST signal the applied mode via the MIME format parameter
 "mode", as specified in section 4.1.  This specification defines a
 generic mode that can be used for any MPEG-4 stream, as well as
 specific modes for the transportation of MPEG-4 CELP and MPEG-4 AAC
 streams, defined in ISO/IEC 14496-3 [1].
 When use of this payload format is signaled using SDP [5], an
 "rtpmap" attribute is part of that signaling.  The same requirements
 apply for the rtpmap attribute in any mode compliant to this
 specification.  The general form of an rtpmap attribute is:
 a=rtpmap:<payload type> <encoding name>/<clock rate>[/<encoding
           parameters>]
 For audio streams, <encoding parameters> specifies the number of
 audio channels: 2 for stereo material (see RFC 2327 [5]) and 1 for
 mono.  Provided no additional parameters are needed, this parameter
 may be omitted for mono material, hence its default value is 1.

van der Meer, et al. Standards Track [Page 21] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

3.3.2. The Generic Mode

 The generic mode can be used for any MPEG-4 stream.  In this mode, no
 mode-specific constraints are applied; hence, in the generic mode,
 the full flexibility of this specification can be exploited.  The
 generic mode is signaled by mode=generic.
 An example is given below for the transportation of a BIFS-Anim
 stream.  In this example carriage of multiple BIFS-Anim Access Units
 is allowed in one RTP packet.  The AU-header contains the AU-size
 field, the CTS-flag and, if the CTS flag is set to 1, the CTS-delta
 field.  The number of bits of the AU-size and the CTS-delta fields
 are 10 and 16, respectively.  The AU-header also contains the RAP-
 flag and the Stream-state of 4 bits.  This results in an AU-header
 with a total size of two or four octets per BIFS-Anim AU.  The RTP
 time stamp uses a 1 kHz clock.  Note that the media type name is
 video, because the BIFS-Anim stream is part of an audio-visual
 presentation.  For conventions on media type names, see section 4.1.
 In detail:
 m=video 49230 RTP/AVP 96
 a=rtpmap:96 mpeg4-generic/1000
 a=fmtp:96 streamtype=3; profile-level-id=1807; mode=generic;
 objectType=2; config=0842237F24001FB400094002C0; sizeLength=10;
 CTSDeltaLength=16; randomAccessIndication=1;
 streamStateIndication=4
 Note: The a=fmtp line has been wrapped to fit the page, it comprises
 a single line in the SDP file.
 The hexadecimal value of the "config" parameter is the
 BIFSConfiguration() as defined in ISO/IEC 14496-1.  The
 BIFSConfiguration() specifies that the BIFS stream is a BIFS-Anim
 stream.  For the description of MIME parameters, see section 4.1.

3.3.3. Constant Bit-rate CELP

 This mode is signaled by mode=CELP-cbr.  In this mode, one or more
 complete CELP frames of fixed size can be transported in one RTP
 packet; interleaving MUST NOT be used with this mode.  The RTP
 payload consists of one or more concatenated CELP frames, each of
 equal size.  CELP frames MUST NOT be fragmented when using this mode.
 Both the AU Header Section and the Auxiliary Section MUST be empty.
 The MIME format parameter constantSize MUST be provided to specify
 the length of each CELP frame.

van der Meer, et al. Standards Track [Page 22] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 For example:
 m=audio 49230 RTP/AVP 96
 a=rtpmap:96 mpeg4-generic/16000/1
 a=fmtp:96 streamtype=5; profile-level-id=14; mode=CELP-cbr; config=
 440E00; constantSize=27; constantDuration=240
 Note: The a=fmtp line has been wrapped to fit the page, it comprises
 a single line in the SDP file.
 The hexadecimal value of the "config" parameter is the
 AudioSpecificConfig()as defined in ISO/IEC 14496-3.
 AudioSpecificConfig() specifies a mono CELP stream with a sampling
 rate of 16 kHz at a fixed bitrate of 14.4 kb/s and 6 sub-frames per
 CELP frame.  For the description of MIME parameters, see section 4.1.

3.3.4. Variable Bit-rate CELP

 This mode is signaled by mode=CELP-vbr.  With this mode, one or more
 complete CELP frames of variable size can be transported in one RTP
 packet with OPTIONAL interleaving.  In this mode, the largest
 possible value for AU-size is greater than the maximum CELP frame
 size. Because CELP frames are very small, there is no support for
 fragmentation of CELP frames.  Hence, CELP frames MUST NOT be
 fragmented when using this mode.
 In this mode, the RTP payload consists of the AU Header Section,
 followed by one or more concatenated CELP frames.  The Auxiliary
 Section MUST be empty.  For each CELP frame contained in the payload,
 there MUST be a one octet AU-header in the AU Header Section to
 provide:
 a) the size of each CELP frame in the payload and
 b) index information for computing the sequence (and hence timing) of
    each CELP frame.
 Transport of CELP frames requires that the AU-size field be coded
 with 6 bits.  Therefore, in this mode 6 bits are allocated to the
 AU-size field, and 2 bits to the AU-Index(-delta) field.  Each AU-
 Index field MUST be coded with the value 0.  In the AU Header
 Section, the concatenated AU-headers are preceded by the 16-bit AU-
 headers-length field, as specified in section 3.2.1.
 In addition to the required MIME format parameters, the following
 parameters MUST be present: sizeLength, indexLength, and
 indexDeltaLength.  CELP frames always have a fixed duration per
 Access Unit; when interleaving in this mode, this specific duration

van der Meer, et al. Standards Track [Page 23] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 MUST be signaled by the MIME format parameter constantDuration.  In
 addition, the parameter maxDisplacement MUST be present when
 interleaving.
 For example:
 m=audio 49230 RTP/AVP 96
 a=rtpmap:96 mpeg4-generic/16000/1
 a=fmtp:96 streamtype=5; profile-level-id=14; mode=CELP-vbr; config=
 440F20; sizeLength=6; indexLength=2; indexDeltaLength=2;
 constantDuration=160; maxDisplacement=5
 Note: The a=fmtp line has been wrapped to fit the page; it comprises
 a single line in the SDP file.
 The hexadecimal value of the "config" parameter is the
 AudioSpecificConfig() as defined in ISO/IEC 14496-3.
 AudioSpecificConfig() specifies a mono CELP stream with a sampling
 rate of 16 kHz, at a bitrate that varies between 13.9 and 16.2 kb/s
 and with 4 sub-frames per CELP frame.  For the description of MIME
 parameters, see section 4.1.

3.3.5. Low Bit-rate AAC

 This mode is signaled by mode=AAC-lbr.  This mode supports the
 transportation of one or more complete AAC frames of variable size.
 In this mode, the AAC frames are allowed to be interleaved and hence
 receivers MUST support de-interleaving.  The maximum size of an AAC
 frame in this mode is 63 octets.  AAC frames MUST NOT be fragmented
 when using this mode.  Hence, when using this mode, encoders MUST
 ensure that the size of each AAC frame is at most 63 octets.
 The payload configuration in this mode is the same as in the variable
 bit-rate CELP mode as defined in 3.3.4.  The RTP payload consists of
 the AU Header Section, followed by concatenated AAC frames.  The
 Auxiliary Section MUST be empty.  For each AAC frame contained in the
 payload, the one octet AU-header MUST provide:
 a) the size of each AAC frame in the payload and
 b) index information for computing the sequence (and hence timing) of
    each AAC frame.
 In the AU-header Section, the concatenated AU-headers MUST be
 preceded by the 16-bit AU-headers-length field, as specified in
 section 3.2.1.

van der Meer, et al. Standards Track [Page 24] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 In addition to the required MIME format parameters, the following
 parameters MUST be present: sizeLength, indexLength, and
 indexDeltaLength.  AAC frames always have a fixed duration per Access
 Unit; when interleaving in this mode, this specific duration MUST be
 signaled by the MIME format parameter constantDuration.  In addition,
 the parameter maxDisplacement MUST be present when interleaving.
 For example:
 m=audio 49230 RTP/AVP 96
 a=rtpmap:96 mpeg4-generic/22050/1
 a=fmtp:96 streamtype=5; profile-level-id=14; mode=AAC-lbr; config=
 1388; sizeLength=6; indexLength=2; indexDeltaLength=2;
 constantDuration=1024; maxDisplacement=5
 Note: The a=fmtp line has been wrapped to fit the page; it comprises
 a single line in the SDP file.
 The hexadecimal value of the "config" parameter is the
 AudioSpecificConfig(), as defined in ISO/IEC 14496-3.
 AudioSpecificConfig() specifies a mono AAC stream with a sampling
 rate of 22.05 kHz.  For the description of MIME parameters, see
 section 4.1.

3.3.6. High Bit-rate AAC

 This mode is signaled by mode=AAC-hbr.  This mode supports the
 transportation of variable size AAC frames.  In one RTP packet,
 either one or more complete AAC frames are carried, or a single
 fragment of an AAC frame is carried.  In this mode, the AAC frames
 are allowed to be interleaved and hence receivers MUST support de-
 interleaving.  The maximum size of an AAC frame in this mode is 8191
 octets.
 In this mode, the RTP payload consists of the AU Header Section,
 followed by either one AAC frame, several concatenated AAC frames or
 one fragmented AAC frame.  The Auxiliary Section MUST be empty.  For
 each AAC frame contained in the payload, there MUST be an AU-header
 in the AU Header Section to provide:
 a) the size of each AAC frame in the payload and
 b) index information for computing the sequence (and hence timing) of
    each AAC frame.
 To code the maximum size of an AAC frame requires 13 bits.
 Therefore, in this configuration 13 bits are allocated to the AU-
 size, and 3 bits to the AU-Index(-delta) field.  Thus, each AU-header

van der Meer, et al. Standards Track [Page 25] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 has a size of 2 octets.  Each AU-Index field MUST be coded with the
 value 0.  In the AU Header Section, the concatenated AU-headers MUST
 be preceded by the 16-bit AU-headers-length field, as specified in
 section 3.2.1.
 In addition to the required MIME format parameters, the following
 parameters MUST be present: sizeLength, indexLength, and
 indexDeltaLength.  AAC frames always have a fixed duration per Access
 Unit; when interleaving in this mode, this specific duration MUST be
 signaled by the MIME format parameter constantDuration.  In addition,
 the parameter maxDisplacement MUST be present when interleaving.
 For example:
 m=audio 49230 RTP/AVP 96
 a=rtpmap:96 mpeg4-generic/48000/6
 a=fmtp:96 streamtype=5; profile-level-id=16; mode=AAC-hbr;
 config=11B0; sizeLength=13; indexLength=3;
 indexDeltaLength=3; constantDuration=1024
 Note: The a=fmtp line has been wrapped to fit the page; it comprises
 a single line in the SDP file.
 The hexadecimal value of the "config" parameter is the
 AudioSpecificConfig(), as defined in ISO/IEC 14496-3.
 AudioSpecificConfig() specifies a 5.1 channel AAC stream with a
 sampling rate of 48 kHz.  For the description of MIME parameters, see
 section 4.1.

3.3.7. Additional Modes

 This specification only defines the modes specified in sections 3.3.2
 through 3.3.6.  Additional modes are expected to be defined in future
 RFCs.  Each additional mode MUST be in full compliance with this
 specification.
 Any new mode MUST be defined such that an implementation including
 all the features of this specification can decode the payload format
 corresponding to this new mode.  For this reason, a mode MUST NOT
 specify new default values for MIME parameters.  In particular, MIME
 parameters that configure the RTP payload MUST be present (unless
 they have the default value), even if its presence is redundant in
 case the mode assigns a fixed value to a parameter.  A mode may
 additionally define that some MIME parameters are required instead of
 optional, that some MIME parameters have fixed values (or ranges),
 and that there are rules restricting its usage.

van der Meer, et al. Standards Track [Page 26] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

4. IANA Considerations

 This section describes the MIME types and names associated with this
 payload format.  Section 4.1 registers the MIME types, as per RFC
 2048 [3].
 This format may require additional information about the mapping to
 be made available to the receiver.  This is done using parameters
 described in the next section.

4.1. MIME Type Registration

 MIME media type name: "video" or "audio" or "application"
 "video" MUST be used for MPEG-4 Visual streams (ISO/IEC 14496-2) or
 MPEG-4 Systems streams (ISO/IEC 14496-1) that convey information
 needed for an audio/visual presentation.
 "audio" MUST be used for MPEG-4 Audio streams (ISO/IEC 14496-3) or
 MPEG-4 Systems streams that convey information needed for an audio
 only presentation.
 "application" MUST be used for MPEG-4 Systems streams (ISO/IEC
 14496-1) that serve purposes other than audio/visual presentation,
 e.g., in some cases when MPEG-J (Java) streams are transmitted.
 Depending on the required payload configuration, MIME format
 parameters may need to be available to the receiver.  This is done
 using the parameters described in the next section.  There are
 required and optional parameters.
 Optional parameters are of two types: general parameters and
 configuration parameters.  The configuration parameters are used to
 configure the fields in the AU Header section and in the auxiliary
 section.  The absence of any configuration parameter is equivalent to
 the associated field set to its default value, which is always zero.
 The absence of all configuration parameters results in a default
 "basic" configuration with an empty AU-header section and an empty
 auxiliary section in each RTP packet.
 MIME subtype name: mpeg4-generic
 Required parameters:
 MIME format parameters are not case dependent; for clarity however,
 both upper and lower case are used in the names of the parameters
 described in this specification.

van der Meer, et al. Standards Track [Page 27] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

    streamType:
    The integer value that indicates the type of MPEG-4 stream that is
    carried; its coding corresponds to the values of the streamType,
    as defined in Table 9 (streamType Values) in ISO/IEC 14496-1.
    profile-level-id:
    A decimal representation of the MPEG-4 Profile Level indication.
    This parameter MUST be used in the capability exchange or session
    set-up procedure to indicate the MPEG-4 Profile and Level
    combination of which the relevant MPEG-4 media codec is capable.
    For MPEG-4 Audio streams, this parameter is the decimal value from
       Table 5 (audioProfileLevelIndication Values) in ISO/IEC 14496-
       1, indicating which MPEG-4 Audio tool subsets are required to
       decode the audio stream.
    For MPEG-4 Visual streams, this parameter is the decimal value
       from Table G-1 (FLC table for profile and level indication) of
       ISO/IEC 14496-2 [1], indicating which MPEG-4 Visual tool
       subsets are required to decode the visual stream.
    For BIFS streams, this parameter is the decimal value obtained
       from (SPLI + 256*GPLI), where:
       SPLI is the decimal value from Table 4 in ISO/IEC 14496-1 with
                the applied sceneProfileLevelIndication;
       GPLI is the decimal value from Table 7 in ISO/IEC 14496-1 with
          the applied graphicsProfileLevelIndication.
    For MPEG-J streams, this parameter is the decimal value from table
       13 (MPEGJProfileLevelIndication) in ISO/IEC 14496-1, indicating
       the profile and level of the MPEG-J stream.
    For OD streams, this parameter is the decimal value from table 3
       (ODProfileLevelIndication) in ISO/IEC 14496-1, indicating the
       profile and level of the OD stream.
    For IPMP streams, this parameter has either the decimal value 0,
       indicating an unspecified profile and level, or a value larger
       than zero, indicating an MPEG-4 IPMP profile and level as
       defined in a future MPEG-4 specification.
    For Clock Reference streams and Object Content Info streams, this
       parameter has the decimal value zero, indicating that profile
       and level information is conveyed through the OD framework.

van der Meer, et al. Standards Track [Page 28] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

    config:
    A hexadecimal representation of an octet string that expresses the
    media payload configuration.  Configuration data is mapped onto
    the hexadecimal octet string in an MSB-first basis.  The first bit
    of the configuration data SHALL be located at the MSB of the first
    octet.  In the last octet, if necessary to achieve octet-
    alignment, up to 7 zero-valued padding bits shall follow the
    configuration data.
    For MPEG-4 Audio streams, config is the audio object type specific
       decoder configuration data AudioSpecificConfig(), as defined in
       ISO/IEC 14496-3.  For Structured Audio, the
       AudioSpecificConfig() may be conveyed by other means, not
       defined by this specification.  If the AudioSpecificConfig() is
       conveyed by other means for Structured Audio, then the config
       MUST be a quoted empty hexadecimal octet string, as follows:
       config="".
       Note that a future mode of using this RTP payload format for
       Structured Audio may define such other means.
    For MPEG-4 Visual streams, config is the MPEG-4 Visual
       configuration information as defined in subclause 6.2.1, Start
       codes of ISO/IEC 14496-2.  The configuration information
       indicated by this parameter SHALL be the same as the
       configuration information in the corresponding MPEG-4 Visual
       stream, except for first-half-vbv-occupancy and latter-half-
       vbv-occupancy, if it exists, which may vary in the repeated
       configuration information inside an MPEG-4 Visual stream (See
       6.2.1 Start codes of ISO/IEC 14496-2).
    For BIFS streams, this is the BIFSConfig() information as defined
       in ISO/IEC 14496-1.  Version 1 of BIFSConfig is defined in
       section 9.3.5.2, and version 2 is defined in section 9.3.5.3.
       The MIME format parameter objectType signals the version of
       BIFSConfig.
    For IPMP streams, this is either a quoted empty hexadecimal octet
       string, indicating the absence of any decoder configuration
       information (config=""), or the IPMPConfiguration() as will be
       defined in a future MPEG-4 IPMP specification.
    For Object Content Info (OCI) streams, this is the
       OCIDecoderConfiguration() information of the OCI stream, as
       defined in section 8.4.2.4 in ISO/IEC 14496-1.

van der Meer, et al. Standards Track [Page 29] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

    For OD streams, Clock Reference streams and MPEG-J streams, this
       is a quoted empty hexadecimal octet string (config=""), as no
       information on the decoder configuration is required.
    mode:
    The mode in which this specification is used.  The following modes
    can be signaled:
    mode=generic,
    mode=CELP-cbr,
    mode=CELP-vbr,
    mode=AAC-lbr and
    mode=AAC-hbr.
    Other modes are expected to be defined in future RFCs.  See also
    section 3.3.7 and 4.2 of RFC 3640.
 Optional general parameters:
    objectType:
    The decimal value from Table 8 in ISO/IEC 14496-1, indicating the
    value of the objectTypeIndication of the transported stream.  For
    BIFS streams, this parameter MUST be present to signal the version
    of BIFSConfiguration().  Note that objectTypeIndication may signal
    a non-MPEG-4 stream and that the RTP payload format defined in
    this document may not be suitable for carrying a stream that is
    not defined by MPEG-4.  The objectType parameter SHOULD NOT be set
    to a value that signals a stream that cannot be carried by this
    payload format.
    constantSize:
    The constant size in octets of each Access Unit for this stream.
    The constantSize and the sizeLength parameters MUST NOT be
    simultaneously present.
    constantDuration:
    The constant duration of each Access Unit for this stream,
    measured with the same units as the RTP time stamp.
    maxDisplacement:
    The decimal representation of the maximum displacement in time of
    an interleaved AU, as defined in section 3.2.3.3, expressed in
    units of the RTP time stamp clock.
    This parameter MUST be present when interleaving is applied.

van der Meer, et al. Standards Track [Page 30] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

    de-interleaveBufferSize:
    The decimal representation in number of octets of the size of the
    de-interleave buffer, described in section 3.2.3.3.  When
    interleaving, this parameter MUST be present if the calculation of
    the de-interleave buffer size given in 3.2.3.3 and based on
    maxDisplacement and rate(max) under-estimates the size of the
    de-interleave buffer.  If this calculation does not under-estimate
    the size of the de-interleave buffer, then the
    de-interleaveBufferSize parameter SHOULD NOT be present.
 Optional configuration parameters:
    sizeLength:
    The number of bits on which the AU-size field is encoded in the
    AU-header.  The sizeLength and the constantSize parameters MUST
    NOT be simultaneously present.
    indexLength:
    The number of bits on which the AU-Index is encoded in the first
    AU-header.  The default value of zero indicates the absence of the
    AU-Index field in each first AU-header.
    indexDeltaLength:
    The number of bits on which the AU-Index-delta field is encoded in
    any non-first AU-header.  The default value of zero indicates the
    absence of the AU-Index-delta field in each non-first AU-header.
    CTSDeltaLength:
    The number of bits on which the CTS-delta field is encoded in the
    AU-header.
    DTSDeltaLength:
    The number of bits on which the DTS-delta field is encoded in the
    AU-header.
    randomAccessIndication:
    A decimal value of zero or one, indicating whether the RAP-flag is
    present in the AU-header.  The decimal value of one indicates
    presence of the RAP-flag, the default value zero indicates its
    absence.
    streamStateIndication:
    The number of bits on which the Stream-state field is encoded in
    the AU-header.  This parameter MAY be present when transporting
    MPEG-4 system streams, and SHALL NOT be present for MPEG-4 audio
    and MPEG-4 video streams.

van der Meer, et al. Standards Track [Page 31] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

    auxiliaryDataSizeLength:
    The number of bits that is used to encode the auxiliary-data-size
    field.
 Applications MAY use more parameters, in addition to those defined
 above.  Each additional parameter MUST be registered with IANA to
 ensure that there is not a clash of names.  Each additional parameter
 MUST be accompanied by a specification in the form of an RFC, MPEG
 standard, or other permanent and readily available reference (the
 "Specification Required" policy defined in RFC 2434 [6]).  Receivers
 MUST tolerate the presence of such additional parameters, but these
 parameters SHALL NOT impact the decoding of receivers that comply
 with this specification.
 Encoding considerations:
 This MIME subtype is defined for RTP transport only.  System
 bitstreams MUST be generated according to MPEG-4 Systems
 specifications (ISO/IEC 14496-1).  Video bitstreams MUST be generated
 according to MPEG-4 Visual specifications (ISO/IEC 14496-2).  Audio
 bitstreams MUST be generated according to MPEG-4 Audio specifications
 (ISO/IEC 14496-3).  The RTP packets MUST be packetized according to
 the RTP payload format defined in RFC 3640.
 Security considerations:
 As defined in section 5 of RFC 3640.
 Interoperability considerations:
 MPEG-4 provides a large and rich set of tools for the coding of
 visual objects.  For effective implementation of the standard,
 subsets of the MPEG-4 tool sets have been provided for use in
 specific applications.  These subsets, called 'Profiles', limit the
 size of the tool set a decoder is required to implement.  In order to
 restrict computational complexity, one or more 'Levels' are set for
 each Profile.  A Profile@Level combination allows:
     .  a codec builder to implement only the subset of the standard
        he needs, while maintaining interworking with other MPEG-4
        devices that implement the same combination, and
     .  checking whether MPEG-4 devices comply with the standard
        ('conformance testing').
 A stream SHALL be compliant with the MPEG-4 Profile@Level specified
 by the parameter "profile-level-id".  Interoperability between a
 sender and a receiver is achieved by specifying the parameter
 "profile-level-id" in MIME content.  In the capability
 exchange/announcement procedure, this parameter may mutually be set
 to the same value.

van der Meer, et al. Standards Track [Page 32] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 Published specification:
 The specifications for MPEG-4 streams are presented in ISO/IEC
 14496-1, 14496-2, and 14496-3.  The RTP payload format is described
 in RFC 3640.
 Applications which use this media type:
 Multimedia streaming and conferencing tools.
 Additional information: none
 Magic number(s): none
 File extension(s):
 None.  A file format with the extension .mp4 has been defined for
 MPEG-4 content but is not directly correlated with this MIME type for
 which the sole purpose is RTP transport.
 Macintosh File Type Code(s): none
 Person & email address to contact for further information:
 Authors of RFC 3640, IETF Audio/Video Transport working group.
 Intended usage: COMMON
 Author/Change controller:
 Authors of RFC 3640, IETF Audio/Video Transport working group.

4.2. Registration of Mode Definitions with IANA

 This specification can be used in a number of modes.  The mode of
 operation is signaled using the "mode" MIME parameter, with the
 initial set of values specified in section 4.1.  New modes may be
 defined at any time, as described in section 3.3.7.  These modes MUST
 be registered with IANA, to ensure that there is not a clash of
 names.
 A new mode registration MUST be accompanied by a specification in the
 form of an RFC, MPEG standard, or other permanent and readily
 available reference (the "Specification Required" policy defined in
 RFC 2434 [6]).

4.3. Concatenation of Parameters

 Multiple parameters SHOULD be expressed as a MIME media type string,
 in the form of a semicolon-separated list of parameter=value pairs
 (for parameter usage examples see sections 3.3.2 up to 3.3.6).

van der Meer, et al. Standards Track [Page 33] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

4.4. Usage of SDP

4.4.1. The a=fmtp Keyword

 It is assumed that one typical way to transport the above-described
 parameters associated with this payload format is via an SDP message
 [5] for example transported to the client in reply to an RTSP
 DESCRIBE [8] or via SAP [11].  In that case, the (a=fmtp) keyword
 MUST be used as described in RFC 2327 [5], section 6, the syntax then
 being:
 a=fmtp:<format> <parameter name>=<value>[; <parameter name>=<value>]

5. Security Considerations

 RTP packets using the payload format defined in this specification
 are subject to the security considerations discussed in the RTP
 specification [2].  This implies that confidentiality of the media
 streams is achieved by encryption.  Because the data compression used
 with this payload format is applied end-to-end, encryption may be
 performed on the compressed data so there is no conflict between the
 two operations.  The packet processing complexity of this payload
 type (i.e., excluding media data processing) does not exhibit any
 significant non-uniformity in the receiver side to cause a denial-
 of-service threat.
 However, it is possible to inject non-compliant MPEG streams (Audio,
 Video, and Systems) so that the receiver/decoder's buffers are
 overloaded, which might compromise the functionality of the receiver
 or even crash it.  This is especially true for end-to-end systems
 like MPEG, where the buffer models are precisely defined.
 MPEG-4 Systems support stream types including commands that are
 executed on the terminal, like OD commands, BIFS commands, etc. and
 programmatic content like MPEG-J (Java(TM) Byte Code) and MPEG-4
 scripts.  It is possible to use one or more of the above in a manner
 non-compliant to MPEG to crash the receiver or make it temporarily
 unavailable.  Senders that transport MPEG-4 content SHOULD ensure
 that such content is MPEG compliant, as defined in the compliance
 part of IEC/ISO 14496 [1].  Receivers that support MPEG-4 content
 should prevent malfunctioning of the receiver in case of non MPEG
 compliant content.
 Authentication mechanisms can be used to validate the sender and the
 data to prevent security problems due to non-compliant malignant
 MPEG-4 streams.

van der Meer, et al. Standards Track [Page 34] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 In ISO/IEC 14496-1, a security model is defined for MPEG-4 Systems
 streams carrying MPEG-J access units that comprise Java(TM) classes
 and objects.  MPEG-J defines a set of Java APIs and a secure
 execution model.  MPEG-J content can call this set of APIs and
 Java(TM) methods from a set of Java packages supported in the
 receiver within the defined security model.  According to this
 security model, downloaded byte code is forbidden to load libraries,
 define native methods, start programs, read or write files, or read
 system properties. Receivers can implement intelligent filters to
 validate the buffer requirements or parametric (OD, BIFS, etc.) or
 programmatic (MPEG-J, MPEG-4 scripts) commands in the streams.
 However, this can increase the complexity significantly.
 Implementors of MPEG-4 streaming over RTP who also implement MPEG-4
 scripts (subset of ECMAScript) MUST ensure that the action of such
 scripts is limited solely to the domain of the single presentation in
 which they reside (thus disallowing session to session communication,
 access to local resources and storage, etc).  Though loading static
 network-located resources (such as media) into the presentation
 should be permitted, network access by scripts MUST be restricted to
 such a (media) download.

6. Acknowledgements

 This document evolved into RFC 3640 after several revisions.  Thanks
 to contributions from people in the ISMA forum, the IETF AVT Working
 Group and the 4-on-IP ad-hoc group within MPEG.  The authors wish to
 thank all people involved, particularly Andrea Basso, Stephen Casner,
 M. Reha Civanlar, Carsten Herpel, John Lazaro, Zvi Lifshitz, Young-
 kwon Lim, Alex MacAulay, Bill May, Colin Perkins, Dorairaj V and
 Stephan Wenger for their valuable comments and support.

van der Meer, et al. Standards Track [Page 35] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

APPENDIX: Usage of this Payload Format

Appendix A. Interleave Analysis

A. Examples of Delay Analysis with Interleave

A.1. Introduction

 Interleaving issues are discussed in this appendix.  Some general
 notes are provided on de-interleaving and error concealment, while a
 number of interleaving patterns are examined, in particular for
 determining the size of the de-interleave buffer and the maximum
 displacement of access units in time.  In these examples, the maximum
 displacement is cited in terms of an access unit count, for ease of
 reading.  In actual streams, it is signaled in units of the RTP time
 stamp clock.

A.2. De-interleaving and Error Concealment

 This appendix does not describe any details on de-interleaving and
 error concealment, as the control of the AU decoding and error
 concealment process has little to do with interleaving.  If the next
 AU to be decoded is present and there is sufficient storage available
 for the decoded AU, then decode it immediately.  If not, wait.  When
 the decoding deadline is reached (i.e., the time when decoding must
 begin in order to be completed by the time the AU is to be
 presented), or if the decoder is some hardware that presents a
 constant delay between initiation of decoding of an AU and
 presentation of that AU, then decoding must begin at that deadline
 time.
 If the next AU to be decoded is not present when the decoding
 deadline is reached, then that AU is lost so the receiver must take
 whatever error concealment measures are deemed appropriate.  The
 play-out delay may need to be adjusted at that point (especially if
 other AUs have also missed their deadline recently).  Or, if it was a
 momentary delay, and maintaining the latency is important, then the
 receiver should minimize the glitch and continue processing with the
 next AU.

A.3. Simple Group Interleave

A.3.1. Introduction

 An example of regular interleave is when packets are formed into
 groups.  If the 'stride' of the interleave (the distance between
 interleaved AUs) is N, packet 0 could contain AU(0), AU(N), AU(2N),
 and so on; packet 1 could contain AU(1), AU(1+N), AU(1+2N), and so

van der Meer, et al. Standards Track [Page 36] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 on.  If there are M access units in a packet, then there are M*N
 access units in the group.
 An example with N=M=3 follows; note that this is the same example as
 given in section 2.5 and that a fixed time duration per Access Unit
 is assumed:
 Packet   Time stamp   Carried AUs      AU-Index, AU-Index-delta
 P(0)     T[0]         0, 3, 6          0, 2, 2
 P(1)     T[1]         1, 4, 7          0, 2, 2
 P(2)     T[2]         2, 5, 8          0, 2, 2
 P(3)     T[9]         9,12,15          0, 2, 2
 In this example, the AU-Index is present in the first AU-header and
 coded with the value 0, as required for fixed duration AUs.  The
 position of the first AU of each packet within the group is defined
 by the RTP time stamp, while the AU-Index-delta field indicates the
 position of subsequent AUs relative to the first AU in the packet.
 All AU-Index-delta fields are coded with the value N-1, equal to 2 in
 this example.  Hence the RTP time stamp and the AU-Index-delta are
 used to reconstruct the original order.  See also section 3.2.3.2.

A.3.2. Determining the De-interleave Buffer Size

 For the regular pattern as in this example, Figure 6 in section
 3.2.3.3 shows that the de-interleave buffer stores at most 4 AUs.  A
 de-interleaveBufferSize value that is at least equal to the total
 number of octets of any 4 "early" AUs that are stored at the same
 time may be signaled.

A.3.3. Determining the Maximum Displacement

 For the regular pattern as in this example, Figure 7 in section 3.3
 shows that the maximum displacement in time equals 5 AU periods.
 Hence, the minimum maxDisplacement value that must be signaled is 5
 AU periods.  In case each AU has the same size, this maxDisplacement
 value over-estimates the de-interleave buffer size with one AU.
 However, note that in case of variable AU sizes, the total size of
 any 4 "early" AUs that must be stored at the same time may exceed
 maxDisplacement times the maximum bitrate, in which case the de-
 interleaveBufferSize must be signaled.

van der Meer, et al. Standards Track [Page 37] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

A.4. More Subtle Group Interleave

A.4.1. Introduction

 Another example of forming packets with group interleave is given
 below.  In this example, the packets are formed such that the loss of
 two subsequent RTP packets does not cause the loss of two subsequent
 AUs.  Note that in this example, the RTP time stamps of packet 3 and
 packet 4 are earlier than the RTP time stamps of packets 1 and 2,
 respectively; a fixed time duration per Access Unit is assumed.
 Packet   Time stamp   Carried AUs      AU-Index, AU-Index-delta
 0        T[0]         0,  5            0, 4
 1        T[2]         2,  7            0, 4
 2        T[4]         4,  9            0, 4
 3        T[1]         1,  6            0, 4
 4        T[3]         3,  8            0, 4
 5        T[10]       10, 15            0, 4
 and so on ..
 In this example, the AU-Index is present in the first AU-header and
 coded with the value 0, as required for AUs with a fixed duration.
 To reconstruct the original order, the RTP time stamp and the AU-
 Index-delta (coded with the value 4) are used.  See also section
 3.2.3.2.

A.4.2. Determining the De-interleave Buffer Size

 From Figure 8, it can be to determined that at most 5 "early" AUs are
 to be stored.  If the AUs are of constant size, then this value
 equals 5 times the AU size.  The minimum size of the de-interleave
 buffer equals the maximum total number of octets of the "early" AUs
 that are to be stored at the same time.  This gives the minimum value
 of the de-interleaveBufferSize that may be signaled.
                            +--+--+--+--+--+--+--+--+--+--+
 Interleaved AUs            | 0| 5| 2| 7| 4| 9| 1| 6| 3| 8|
                            +--+--+--+--+--+--+--+--+--+--+
                              -  -  5  -  5  -  2  7  4  9
                                          7     4  9  5
 "Early" AUs                                    5     6
                                                7     7
                                                9     9
 Figure 8: Storage of "early" AUs in the de-interleave buffer per
           interleaved AU.

van der Meer, et al. Standards Track [Page 38] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

A.4.3. Determining the Maximum Displacement

 From Figure 9, it can be seen that the maximum displacement in time
 equals 8 AU periods.  Hence the minimum maxDisplacement value to be
 signaled is 8 AU periods.
                                  +--+--+--+--+--+--+--+--+--+--+
 Interleaved AUs                  | 0| 5| 2| 7| 4| 9| 1| 6| 3| 8|
                                  +--+--+--+--+--+--+--+--+--+--+
 Earliest not yet present AU        -  1  1  1  1  1  -  3  -  -
 Figure 9: For each AU in the interleaving pattern, the earliest of
           any earlier AUs not yet present
 In case each AU has the same size, the found maxDisplacement value
 over-estimates the de-interleave buffer size with three AUs.
 However, in case of variable AU sizes, the total size of any 5
 "early" AUs stored at the same time may exceed maxDisplacement times
 the maximum bitrate, in which case de-interleaveBufferSize must be
 signaled.

A.5. Continuous Interleave

A.5.1. Introduction

 In continuous interleave, once the scheme is 'primed', the number of
 AUs in a packet exceeds the 'stride' (the distance between them).
 This shortens the buffering needed, smoothes the data-flow, and gives
 slightly larger packets -- and thus lower overhead -- for the same
 interleave.  For example, here is a continuous interleave also over a
 stride of 3 AUs, but with 4 AUs per packet, for a run of 20 AUs.
 This shows both how the scheme 'starts up' and how it finishes.  Once
 again, the example assumes fixed time duration per Access Unit.
 Packet   Time-stamp   Carried AUs         AU-Index, AU-Index-delta
 0        T[0]                      0      0
 1        T[1]                  1   4      0  2
 2        T[2]              2   5   8      0  2  2
 3        T[3]          3   6   9  12      0  2  2  2
 4        T[7]          7  10  13  16      0  2  2  2
 5        T[11]        11  14  17  20      0  2  2  2
 6        T[15]        15  18              0  2
 7        T[19]        19                  0
 In this example, the AU-Index is present in the first AU-header and
 coded with the value 0, as required for AUs with a fixed duration.
 To reconstruct the original order, the RTP time stamp and the

van der Meer, et al. Standards Track [Page 39] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

 AU-Index-delta (coded with the value 2) are used.  See also 3.2.3.2.
 Note that this example has RTP time-stamps in increasing order.

A.5.2. Determining the De-interleave Buffer Size

 For this example the de-interleave buffer size can be derived from
 Figure 10.  The maximum number of "early" AUs is 3.  If the AUs are
 of constant size, then the de-interleave buffer size equals 3 times
 the AU size.  Compared to the example in A.2, for constant size AUs
 the de-interleave buffer size is reduced from 4 to 3 times the AU
 size, while maintaining the same 'stride'.
                      +--+--+--+--+--+--+--+--+--+--+--+--+--+--+-
 Interleaved AUs      | 0| 1| 4| 2| 5| 8| 3| 6| 9|12| 7|10|13|16|
                      +--+--+--+--+--+--+--+--+--+--+--+--+--+--+-
                        -  -  -  4  -  -  4  8  -  -  8 12  -  -
                                          5           9
 "Early" AUs                              8          12
 Figure 10: Storage of "early" AUs in the de-interleave buffer per
            interleaved AU.

A.5.3. Determining the Maximum Displacement

 For this example, the maximum displacement has a value of 5 AU
 periods.  See Figure 11.  Compared to the example in A.2, the maximum
 displacement does not decrease, though in fact less de-interleave
 buffering is required.
                      +--+--+--+--+--+--+--+--+--+--+--+--+--+--+-
 Interleaved AUs      | 0| 1| 4| 2| 5| 8| 3| 6| 9|12| 7|10|13|16|
                      +--+--+--+--+--+--+--+--+--+--+--+--+--+--+-
 Earliest not yet
      present AU        -  -  2  -  3  3  -  -  7  7  -  - 11 11
 Figure 11: For each AU in the interleaving pattern, the earliest of
            any earlier AUs not yet present

van der Meer, et al. Standards Track [Page 40] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

References

Normative References

 [1]  ISO/IEC International Standard 14496 (MPEG-4); "Information
      technology - Coding of audio-visual objects", January 2000
 [2]  Schulzrinne, H., Casner, S., Frederick, R. and V. Jacobson,
      "RTP:  A Transport Protocol for Real-Time Applications", RFC
      3550, July 2003.
 [3]  Freed, N., Klensin, J. and J. Postel, "Multipurpose Internet
      Mail Extensions (MIME) Part Four: Registration Procedures", BCP
      13, RFC 2048, November 1996.
 [4]  Bradner, S., "Key words for use in RFCs to Indicate Requirement
      Levels", BCP 14, RFC 2119, March 1997.
 [5]  Handley, M. and V. Jacobson, "SDP: Session Description
      Protocol", RFC 2327, April 1998.
 [6]  Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA
      Considerations Section in RFCs", BCP 26, RFC 2434, October 1998.

Informative References

 [7]  Hoffman, D., Fernando, G., Goyal, V. and M. Civanlar, "RTP
      Payload Format for MPEG1/MPEG2 Video", RFC 2250, January 1998.
 [8]  Schulzrinne, H., Rao, A. and R. Lanphier, "Real-Time Session
      Protocol (RTSP)", RFC 2326, April 1998.
 [9]  Perkins, C. and O. Hodson, "Options for Repair of Streaming
      Media", RFC 2354, June 1998.
 [10] Schulzrinne, H. and J. Rosenberg, "An RTP Payload Format for
      Generic Forward Error Correction", RFC 2733, December 1999.
 [11] Handley, M., Perkins, C. and E. Whelan, "Session Announcement
      Protocol", RFC 2974, October 2000.
 [12] Kikuchi, Y., Nomura, T., Fukunaga, S., Matsui, Y. and H. Kimata,
      "RTP Payload Format for MPEG-4 Audio/Visual Streams", RFC 3016,
      November 2000.

van der Meer, et al. Standards Track [Page 41] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

Authors' Addresses

 Jan van der Meer
 Philips Electronics
 Prof Holstlaan 4
 Building WAH-1
 5600 JZ Eindhoven
 Netherlands
 EMail: jan.vandermeer@philips.com
 David Mackie
 Apple Computer, Inc.
 One Infinite Loop, MS:302-3KS
 Cupertino  CA 95014
 EMail: dmackie@apple.com
 Viswanathan Swaminathan
 Sun Microsystems Inc.
 2600 Casey Avenue
 Mountain View, CA 94043
 EMail: viswanathan.swaminathan@sun.com
 David Singer
 Apple Computer, Inc.
 One Infinite Loop, MS:302-3MT
 Cupertino  CA 95014
 EMail: singer@apple.com
 Philippe Gentric
 Philips Electronics
 51 rue Carnot
 92156 Suresnes
 France
 EMail: philippe.gentric@philips.com

van der Meer, et al. Standards Track [Page 42] RFC 3640 Transport of MPEG-4 Elementary Streams November 2003

Full Copyright Statement

 Copyright (C) The Internet Society (2003).  All Rights Reserved.
 This document and translations of it may be copied and furnished to
 others, and derivative works that comment on or otherwise explain it
 or assist in its implementation may be prepared, copied, published
 and distributed, in whole or in part, without restriction of any
 kind, provided that the above copyright notice and this paragraph are
 included on all such copies and derivative works.  However, this
 document itself may not be modified in any way, such as by removing
 the copyright notice or references to the Internet Society or other
 Internet organizations, except as needed for the purpose of
 developing Internet standards in which case the procedures for
 copyrights defined in the Internet Standards process must be
 followed, or as required to translate it into languages other than
 English.
 The limited permissions granted above are perpetual and will not be
 revoked by the Internet Society or its successors or assignees.
 This document and the information contained herein is provided on an
 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement

 Funding for the RFC Editor function is currently provided by the
 Internet Society.

van der Meer, et al. Standards Track [Page 43]

/data/webs/external/dokuwiki/data/pages/rfc/rfc3640.txt · Last modified: 2003/11/26 20:21 by 127.0.0.1

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki