]> Orange

mohamed.boucadair@orange.com

Huawei

benoit.claise@huawei.com

Operations and Management OPSAWG IPFIX This document specifies new IP Flow Information Export (IPFIX) Information Elements (IEs) to solve issues with existing ipv6ExtensionHeaders and tcpOptions IPFIX IEs, especially the ability to export any observed IPv6 extension headers or TCP options. Discussion Venues Discussion of this document takes place on the Operations and Management Area Working Group Working Group mailing list (opsawg@ietf.org), which is archived at . Source for this draft and an issue tracker can be found at .

Introduction This document specifies new IP Flow Information Export (IPFIX) Information Elements (IEs) to solve a set of issues encountered with the specifications of ipv6ExtensionHeaders (to export IPv6 extension headers) and tcpOptions (to export TCP options) IEs . More details about these issues are provided in the following sub-sections.

Issues with ipv6ExtensionHeaders Information Element The specification of the ipv6ExtensionHeaders IPFIX IE (64) does not:

• Cover the full extension headers' range ().
• Specify the procedure to follow when all bits are exhausted.
• Specify a means to export the order and the number of occurrences of a given extension header.
• Specify how to automatically update the IANA IPFIX registry () when a new value is assigned in the IPv6 Extension Header Types registry . Only a frozen set of extension headers can be exported using the ipv6ExtensionHeaders IE. For example, the ipv6ExtensionHeaders IE can't report some IPv6 EHs, specifically EHs for Host Identity Protocol (139), Shim6 Protocol (140) or extension headers for experimentation and testing.
• Specify whether the exported values match the full enclosed values or only up to a limit imposed by hardware or software (e.g., ). Note that some implementations may not be able to export all observed extension headers in a Flow because of a hardware or software limit (see, e.g., ). The specification of the ipv6ExtensionHeaders Information Element does not discuss whether it covers all enclosed extension headers or only up to a limit.
• Specify how to report the length of IPv6 extension headers.
• Optimize the encoding.
• Explain the reasoning for reporting values which do not correspond to extension headers (e.g., "Unknown Layer 4 header" or "Payload compression header").
• Specify how to report extension header chains or aggregate extension headers length.

addresses these issues. This specification deprecates the ipv6ExtensionHeaders IPFIX IE in favor of the new IEs defined in this document.

Issues with tcpOptions Information Element The specification of the tcpOptions IPFIX IE (209) does not:

• Describe how some observed TCP options in a Flow can be exported using IPFIX. Only TCP options having a Kind <= 63 can be exported in a tcpOptions IE.
• Allow reporting the observed Experimental Identifiers (ExIDs) that are carried in shared Experimental TCP options (Kind=253 or 254) .
• Optimize the encoding.

addresses these issues. This specification deprecates the tcpOptions IE in favor of the new IEs defined in this document.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses the IPFIX-specific terminology (Information Element, Template Record, Flow, etc.) defined in . As in , these IPFIX-specific terms have the first letter of a word capitalized. Also, the document uses the terms defined in IPv6 and TCP specifications. In addition, the document makes use of the following term:

Extension header chain:

Refers to the chain of extension headers that are present in an IPv6 packet.

This term should not be confused with the IPv6 header chain, which includes the IPv6 header, zero or more IPv6 extension headers, and zero or a single Upper-Layer Header.

Flow with varying extension header chain:

Refers to a Flow where distinct extension header chains are observed. Concretely, different packets in such a Flow will have a different sequence of extension header type codes.

Information Elements for IPv6 Extension Headers

ipv6ExtensionHeaderType Information Element

Name:

ipv6ExtensionHeaderType

ElementID:

TBD1

Description:

Type of an IPv6 extension header observed in packets of this Flow.

Abstract Data Type:

unsigned8

Data Type Semantics:

identifier

Additional Information:

See for assigned extension header types.

See for the general definition of IPv6 extension headers.

Reference:

This-Document

ipv6ExtensionHeaderCount Information Element

Name:

ipv6ExtensionHeaderCount

ElementID:

TBD2

Description:

The number of consecutive occurrences of the same extension header type in a Flow.

The type of the extension header is provided in the ipv6ExtensionHeaderType Information Element.

Abstract Data Type:

unsigned8

Data Type Semantics:

totalCounter

Additional Information:

See for assigned extension header types.

See for the general definition of IPv6 extension headers.

Reference:

This-Document

ipv6ExtensionHeadersFull Information Element

Name:

ipv6ExtensionHeadersFull

ElementID:

TBD3

Description:

IPv6 extension headers observed in packets of this Flow. The information is encoded in a set of bit fields. For each IPv6 extension header, there is a bit in this set. The bit is set to 1 if any observed packet of this Flow contains the corresponding IPv6 extension header. Otherwise, if no observed packet of this Flow contains the respective IPv6 extension header, the value of the corresponding bit is 0.

The IPv6 extension header associated with each bit is provided in [NEW_IPFIX_IPv6EH_SUBREGISTRY]. Bit 0 corresponds to the least-significant bit in the ipv6ExtensionHeadersFull IE while bit 255 corresponds to the most-significant bit of the IE. In doing so, few octets will be needed to encode common IPv6 extension headers when observed in a Flow.

The "No Next Header" (59) value () is used if there is no upper-layer header in an IPv6 packet. Even if the value is not considered as an extension header as such, the corresponding bit is set in the ipv6ExtensionHeadersFull IE whenever that value is encountered in the Flow.

Extension headers observed in a Flow with varying extension header chain MAY be aggregated in one single ipv6ExtensionHeadersFull Information Element or be exported in separate ipv6ExtensionHeadersFull IEs, one for each extension header chain.

Abstract Data Type:

unsigned256

Data Type Semantics:

flags

Additional Information:

See the assigned bits to each IPv6 extension header type in [NEW_IPFIX_IPv6EH_SUBREGISTRY].

See for assigned extension header types.

See for the general definition of IPv6 extension headers.

Reference:

This-Document

• Note to the RFC Editor: Please replace [NEW_IPFIX_IPv6EH_SUBREGISTRY] with the link to the "ipv6ExtensionHeaders Bits" registry ().

ipv6ExtensionHeaderTypeCountList Information Element

Name:

ipv6ExtensionHeaderTypeCountList

ElementID:

TBD4

Description:

As per , IPv6 nodes must accept and attempt to process extension headers occurring any number of times in the same packet. This Information Element echoes the order of extension headers and number of consecutive occurrences of the same extension header type in a Flow.

This Information Element is a subTemplateList of ipv6ExtensionHeaderType and ipv6ExtensionHeaderCount Information Elements.

Each header chain in Flow with varying extension header chain MUST be exported in a separate IE.

The same extension header type may appear several times in an ipv6ExtensionHeaderTypeCountList Information Element. For example, if an IPv6 packet of a Flow includes a Hop-by-Hop Options header, a Destination Options header, a Fragment header, and Destination Options header, the ipv6ExtensionHeaderTypeCountList Information Element will report:

• the count of Hop-by-Hop Options header,
• the occurrences of the Destination Options header that are observed before the Fragment header,
• the occurrences of the Fragment header, and
• the occurrences of the Destination Options header that are observed right after the Fragment header.

Abstract Data Type:

subTemplateList

Data Type Semantics:

list

Additional Information:

See the assigned IPv6 extension header types in .

See for the general definition of IPv6 extension headers.

Reference:

This-Document

ipv6ExtensionHeadersLimit Information Element

Name:

ipv6ExtensionHeadersLimit

ElementID:

TBD5

Description:

When set to "false", this Information Element indicates that the exported extension headers information (e.g., ipv6ExtensionHeadersFull or ipv6ExtensionHeaderTypeCountList) does not match the full enclosed extension headers, but only up to a limit that is typically set by hardware or software.

When set to "true", this Information Element indicates that the exported extension header information matches the full enclosed extension headers.

Abstract Data Type:

boolean

Data Type Semantics:

default

Additional Information:

See for the general definition of IPv6 extension headers.

See for an example of IPv6 packet processing due to limits on extension headers.

Reference:

This-Document

ipv6ExtensionHeadersChainLength Information Element

Name:

ipv6ExtensionHeadersChainLength

ElementID:

TBD6

Description:

In theory, there are no limits on the number of IPv6 extension headers that may be present in a packet other than the path MTU. However, it was regularly reported that IPv6 packets with extension headers are often dropped in the Internet.

As discussed in , some hardware devices implement a parsing buffer of a fixed size to process packets, including all the headers. When the aggregate length of headers of an IPv6 packet exceeds that size, the packet will be discarded or deferred to a slow path.

The ipv6ExtensionHeadersChainLength IE is used to report, in octets, the length of an extension header chain observed in a Flow. The length is the sum of the length of all extension headers of the chain. Exporting such information might help identifying root causes of performance degradation, including packet drops.

Each header chain length of a Flow with varying extension header chain MUST be exported in a separate ipv6ExtensionHeadersChainLength IE.

Abstract Data Type:

unsigned32

Data Type Semantics:

identifier

Units:

octets

Additional Information:

See for the general definition of IPv6 extension headers.

See for an overview of operational implications of IPv6 packets with extension headers.

Reference:

This-Document

ipv6ExtensionHeaderChainLengthList Information Element

Name:

ipv6ExtensionHeaderChainLengthList

ElementID:

TBD7

Description:

This Information Element is used to report the chains and their length as observed in a Flow with varying extension header chain.

This Information Element is a subTemplateList of ipv6ExtensionHeadersFull and ipv6ExtensionHeadersChainLength Information Elements.

If several extension header chains are observed in a Flow, each header chain MUST be exported in a separate ipv6ExtensionHeaderChainLengthList IE.

Abstract Data Type:

subTemplateList

Data Type Semantics:

list

Additional Information:

See the assigned IPv6 extension header types in .

See for the general definition of IPv6 extension headers.

Reference:

This-Document

Information Elements for TCP Options

tcpOptionsFull Information Element This section specifies a new IE to cover the full TCP options range.

Name:

tcpOptionsFull

ElementID:

TBD8

Description:

TCP options in packets of this Flow. The information is encoded in a set of bit fields. For each TCP option, there is a bit in this set. The bit is set to 1 if any observed packet of this Flow contains the corresponding TCP option. Otherwise, if no observed packet of this Flow contains the respective TCP option, the value of the corresponding bit is 0.

Options are mapped to bits according to their option numbers. TCP option Kind 0 corresponds to the least-significant bit in the tcpOptionsFull IE while Kind 255 corresponds to the most-significant bit of the IE. This approach allows an observer to export any observed TCP option even if it does support that option and without requiring updating a mapping table.

Abstract Data Type:

unsigned256

Data Type Semantics:

flags

Additional Information:

See the assigned TCP option Kinds at .

See for the general definition of TCP options.

Reference:

This-Document

tcpSharedOptionExID16 Information Element

Name:

tcpSharedOptionExID16

ElementID:

TBD9

Description:

Any observed 2-byte Experiments IDs (ExIDs) in a shared TCP option (Kind=253 or 254) in a Flow. The information is encoded in a set of 16-bit fields. Each 16-bit field carries an observed 2-byte ExID in a shared option.

Abstract Data Type:

octetArray

Data Type Semantics:

identifier

Additional Information:

See assigned ExIDs at .

See for the general definition of TCP options.

See for the shared use of experimental TCP Options.

Reference:

This-Document

tcpSharedOptionExID32 Information Element

Name:

tcpSharedOptionExID32

ElementID:

TBD10

Description:

Any observed 4-byte Experiments IDs (ExIDs) in a shared TCP option (Kind=253 or 254) in a Flow. The information is encoded in a set of 32-bit fields. Each 32-bit field carries an observed 4-byte ExID in a shared option.

Abstract Data Type:

octetArray

Data Type Semantics:

identifier

Additional Information:

See assigned ExIDs at .

See for the general definition of TCP options.

See for the shared use of experimental TCP Options.

Reference:

This-Document

Operational Considerations

IPv6 Extension Headers The value of ipv6ExtensionHeadersFull IE should be encoded in fewer octets as per the guidelines in . If an implementation determines that an observed packet of a Flow includes an extension header (including an extension header that it does not support), then the exact observed code of that extension header MUST be echoed in the ipv6ExtensionHeaderTypeCountList IE (). How an implementation disambiguates between unknown upper-layer protocols vs. extension headers is not IPFIX-specific. Readers may refer, for example, to for a behavior of an intermediate node that encounters an unknown Next Header type. It is out of the scope of this document to discuss those considerations. The ipv6ExtensionHeadersFull Information Element SHOULD NOT be exported if ipv6ExtensionHeaderTypeCountList Information Element is also present because of the overlapping scopes between these two IEs. If both IEs are present, then ipv6ExtensionHeaderTypeCountList Information Element takes precedence. The ipv6ExtensionHeadersLimit IE () may or may not be present when the ipv6ExtensionHeadersChainLength IE () is also present as these IEs are targeting distinct properties of extension headers handling.

TCP Options The value of tcpOptionsFull IE should be encoded in fewer octets as per the guidelines in . Implementations of tcpSharedOptionExID16 and tcpSharedOptionExID32 IEs are assumed to be provided with a list of valid Experiment IDs . How that list is maintained is implementation-specific. Absent that list, an implementation can't autonomously determine whether an ExID is present and, if so, whether it is 2- or 4-byte length. If a TCP Flow contains packets with a mix of 2-byte and 4-byte Experiment IDs, the same Template Record is used with both tcpSharedOptionExID16 and tcpSharedOptionExID32 IEs.

Examples This section provides a few examples to illustrate the use of some IEs defined in this document.

IPv6 Extension Headers provides an example of reported values in an ipv6ExtensionHeadersFull IE for an IPv6 Flow in which only the IPv6 Destination Options (0) header is observed. One octet is sufficient to report these observed options. Concretely, the ipv6ExtensionHeadersFull IE will be set to 0x01. The bits are set following the table provided in .

A First Example of Extension Headers

provides another example of reported values in an ipv6ExtensionHeadersFull IE for an IPv6 Flow in which the Destination Options (0), IPv6 Hop-by-Hop Options (1), and Routing (5) headers are observed. One octet is sufficient to report these observed options. Concretely, the ipv6ExtensionHeadersFull IE will be set to 0x23.

A Second Example of Extension Headers

TCP Options Given TCP Kind allocation practices and the option mapping defined in , fewer octets are likely to be used for Flows with common TCP options. shows an example of reported values in a tcpOptionsFull IE for a TCP Flow in which End of Option List (0), Maximum Segment Size (2), and Window Scale (3) options are observed. One octet is sufficient to report these observed options. Concretely, the tcpOptionsFull IE will be set to 0x0D.

An Example of TCP Options

Let us consider a TCP Flow in which shared options with ExIDs 0x0348 (HOST_ID) , 0x454E (TCP-ENO) , and 0xE2D4C3D9 (Shared Memory communications over RMDA protocol) are observed. As shown in , two TCP shared IEs will be used to report these observed ExIDs:

1. The tcpSharedOptionExID16 IE set to 0x0348454E to report observed 2-byte ExIDs: HOST_ID and TCP-ENO ExIDs.
2. The tcpSharedOptionExID32 IE set to 0xE2D4C3D9 to report the only observed 4-byte ExID.

Example of TCP Shared IEs

Security Considerations IPFIX security considerations are discussed in . ipv6ExtensionHeadersChainLength and ipv6ExtensionHeadersLimit IEs can be exploited by an unauthorized observer as a means to deduce the processing capabilities of nodes. discusses the required measures to guarantee the integrity and confidentiality of the exported information. This document does not add new security considerations for exporting IEs other than those already discussed in .

IANA Considerations

Deprecate ipv6ExtensionHeaders and tcpOptions Information Elements This document requests IANA to update the "IPFIX Information Elements" registry under the "IP Flow Information Export (IPFIX) Entities" registry group as follows:

• Update the ipv6ExtensionHeaders IE (64) entry by marking it as deprecated in favor of the ipv6ExtensionHeadersFull IE defined in this document. This note should also be echoed in the "Additional Information" of the ipv6ExtensionHeaders IE.
• Update the tcpOptions IE (209) entry by marking it as deprecated in favor of the tcpOptionsFull IE defined in this document. This note should also be echoed in the "Additional Information" of the tcpOptions IE.

IANA is also requested to update the reference of ipv6ExtensionHeaders IE (64) and tcpOptions IE (209) to point to this document.

New IPFIX Information Elements This document requests IANA to add the following new IPFIX IEs to the "IPFIX Information Elements" registry under the "IP Flow Information Export (IPFIX) Entities" registry group : New IPFIX Information Elements

Value	Name	Reference
TBD1	ipv6ExtensionHeader	of This-Document
TBD2	ipv6ExtensionHeaderCount	of This-Document
TBD3	ipv6ExtensionHeadersFull	of This-Document
TBD4	ipv6ExtensionHeaderTypeCountList	of This-Document
TBD5	ipv6ExtensionHeadersLimit	of This-Document
TBD6	ipv6ExtensionHeadersChainLength	of This-Document
TBD7	ipv6ExtensionHeaderChainLengthList	of This-Document
TBD7	tcpOptionsFull	of This-Document
TBD8	tcpSharedOptionExID16	of This-Document
TBD9	tcpSharedOptionExID32	of This-Document

New IPFIX Information Element Data Type This document requests IANA to add the following new abstract data type to the "IPFIX Information Element Data Types" registry under the "IP Flow Information Export (IPFIX) Entities" registry group : New IPFIX Information Element Data Type

Value	Description	Reference
TBD10	unsigned256	This-Document

The type "unsigned256" represents a non-negative integer value in the range of '0' to '2^256 - 1'. Similar to , this type MUST be encoded using the default canonical format in network byte order. Reduced-Size encoding () applies to this data type.

IPFIX Subregistry for IPv6 Extension Headers This document requests IANA to create a new registry entitled "ipv6ExtensionHeaders Bits" under the IANA IPFIX registry group . When a new code is assigned to an IPv6 EH in , the next available free bit is selected by IANA for this EH from "ipv6ExtensionHeaders Bits" registry and the registry is updated with the details that mirror the assigned EH. The "Label" mirrors the "keyword" of an EH as indicated in , while the "Protocol Number" mirrors the "Protocol Number" in . IANA is requested to add the following note to :

• Note:

  When a new code is assigned to an IPv6 Extension Header, the next available free bit in [NEW_IPFIX_IPv6EH_SUBREGISTRY] is selected for this new Extension Header. [NEW_IPFIX_IPv6EH_SUBREGISTRY] is updated accordingly. Modifications to existing registrations must be mirrored in [NEW_IPFIX_IPv6EH_SUBREGISTRY].

• Note to the RFC Editor: Please replace [NEW_IPFIX_IPv6EH_SUBREGISTRY] with the link used by IANA for this new registry.

Otherwise, the registration policy for the registry is Expert Review (). See more details in .

Initial Values The initial values of this registry are provided in . Initial Values of the IPv6 Extension Headers IPFIX Subregistry

Bit	Label	Protocol Number	Description	Reference
0	DST	60	Destination Options for IPv6	This-Document
1	HOP	0	IPv6 Hop-by-Hop Options	This-Document
2	NoNxt	59	No Next Header for IPv6	This-Document
3	UNK		Unknown Layer 4 header (compressed, encrypted, not supported)	This-Document
4	FRA0	44	Fragment header - first fragment	This-Document
5	RH	43	Routing header	This-Document
6	FRA1	44	Fragmentation header - not first fragment	This-Document
7 to 11			Unassigned
12	MOB	135	Mobility Header	This-Document
13	ESP	50	Encapsulating Security Payload	This-Document
14	AH	51	Authentication Header	This-Document
15			Unassigned
16	HIP	139	Host Identity Protocol	This-Document
17	SHIM6	140	Shim6 Protocol	This-Document
18		253	Use for experimentation and testing	This-Document
19		254	Use for experimentation and testing	This-Document
20 to 255			Unassigned

Guidelines for the Designated Experts It is suggested that multiple designated experts be appointed for registry change requests. Criteria that should be applied by the designated experts include determining whether the proposed registration duplicates existing entries and whether the registration description is clear and fits the purpose of this registry. Within the review period, the designated experts will either approve or deny the registration request, communicating this decision to the IANA. Denials should include an explanation and, if applicable, suggestions as to how to make the request successful.

References Normative References IANA IANA IANA IANA IANA This document specifies the IP Flow Information Export (IPFIX) protocol, which serves as a means for transmitting Traffic Flow information over the network. In order to transmit Traffic Flow information from an Exporting Process to a Collecting Process, a common representation of flow data and a standard means of communicating them are required. This document describes how the IPFIX Data and Template Records are carried over a number of transport protocols from an IPFIX Exporting Process to an IPFIX Collecting Process. This document obsoletes RFC 5101. This document specifies version 6 of the Internet Protocol (IPv6). It obsoletes RFC 2460. This document describes how the experimental TCP option codepoints can concurrently support multiple TCP extensions, even within the same connection, using a new IANA TCP experiment identifier. This approach is robust to experiments that are not registered and to those that do not use this sharing mechanism. It is recommended for all new TCP options that use these codepoints. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document specifies the Transmission Control Protocol (TCP). TCP is an important transport-layer protocol in the Internet protocol stack, and it has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFCs 1011 and 1122, and it should be considered as a replacement for the portions of those documents dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. This document defines the data types and management policy for the information model for the IP Flow Information Export (IPFIX) protocol. This information model is maintained as the IANA "IPFIX Information Elements" registry, the initial contents of which were defined by RFC 5102. This information model is used by the IPFIX protocol for encoding measured traffic information and information related to the traffic Observation Point, the traffic Metering Process, and the Exporting Process. Although this model was developed for the IPFIX protocol, it is defined in an open way that allows it to be easily used in other protocols, interfaces, and applications. This document obsoletes RFC 5102. Many protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA). To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry. This is the third edition of this document; it obsoletes RFC 5226. Informative References Network nodes may discard packets if they are unable to process protocol headers of packets due to processing constraints or limits. When such packets are dropped, the sender receives no indication, so it cannot take action to address the cause of discarded packets. This specification defines several new ICMPv6 errors that can be sent by a node that discards packets because it is unable to process the protocol headers. A node that receives such an ICMPv6 error may use the information to diagnose packet loss and may modify what it sends in future packets to avoid subsequent packet discards. SiPanda This specification defines various limits that may be applied to receiving, sending, and otherwise processing packets that contain IPv6 extension headers. The need for such limits is pragmatic to facilitate interoperability amongst hosts and routers in the presence of extension headers, thereby increasing the feasibility of deployment of extension headers. The limits described herein establish the minimum baseline of support for use of extension headers on the Internet. If it is known that all communicating parties for a particular communication, including destination hosts and any routers in the path, are capable of supporting more than the baseline then these default limits may be freely exceeded. This document summarizes the operational implications of IPv6 extension headers specified in the IPv6 protocol specification (RFC 8200) and attempts to analyze reasons why packets with IPv6 extension headers are often dropped in the public Internet. Recent RFCs have discussed issues with host identification in IP address-sharing systems, such as address/prefix-sharing devices and application-layer proxies. Potential solutions for revealing a host identifier in shared address deployments have also been discussed. This memo describes the design, deployment, and privacy considerations for one such solution in operational use on the Internet today that uses a TCP option to transmit a host identifier. Despite growing adoption of TLS, a significant fraction of TCP traffic on the Internet remains unencrypted. The persistence of unencrypted traffic can be attributed to at least two factors. First, some legacy protocols lack a signaling mechanism (such as a STARTTLS command) by which to convey support for encryption, thus making incremental deployment impossible. Second, legacy applications themselves cannot always be upgraded and therefore require a way to implement encryption transparently entirely within the transport layer. The TCP Encryption Negotiation Option (TCP-ENO) addresses both of these problems through a new TCP option kind providing out-of-band, fully backward-compatible negotiation of encryption. This document describes IBM's Shared Memory Communications over RDMA (SMC-R) protocol. This protocol provides Remote Direct Memory Access (RDMA) communications to TCP endpoints in a manner that is transparent to socket applications. It further provides for dynamic discovery of partner RDMA capabilities and dynamic setup of RDMA connections, as well as transparent high availability and load balancing when redundant RDMA network paths are available. It maintains many of the traditional TCP/IP qualities of service such as filtering that enterprise users demand, as well as TCP socket semantics such as urgent data.

Acknowledgments Thanks to Paul Aitken and Eric Vyncke for the reviews and comments. Special thanks to Andrew Feren for sharing data about scans of IPFIX data he collected. Thanks to Wesley Eddy for the tsvart review, Yingzhen Qu for the opsdir review, Dirk Von Hugo for intdir review, Joel Halpern for the genart review, and Tero Kivinen for the secdir review. Thanks to Thomas Graf for the Shepherd review. Thanks to Mahesh Jethanandani for the AD review.