IPv6 Options for In Situ Operations, Administration, and Maintenance (IOAM) Thoughtspot
3rd Floor, Indiqube Orion 24th Main Rd, Garden Layout, HSR Layout Bangalore Karnataka 560 102 India shwetha.bhandari@thoughtspot.com
Cisco Systems, Inc.
Hansaallee 249, 3rd Floor Duesseldorf 40549 Germany fbrockne@cisco.com
tsv ippm In situ Operations, Administration, and Maintenance (IOAM) records operational and telemetry information in the packet while the packet traverses a path between two points in the network. This document outlines how IOAM Data-Fields are encapsulated in IPv6.
Status of This Memo This is an Internet Standards Track document. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at .
Copyright Notice Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents () in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.
Table of Contents
  • .  Introduction
  • .  Conventions
    • .  Requirements Language
    • .  Abbreviations
  • .  In situ OAM Metadata Transport in IPv6
  • .  IOAM Deployment in IPv6 Networks
    • .  Considerations for IOAM Deployment and Implementation in IPv6 Networks
    • .  IOAM-Domains Bounded by Hosts
    • .  IOAM-Domains Bounded by Network Devices
  • .  Security Considerations
    • .  Applicability of Authentication Header (AH)
  • .  IANA Considerations
  • .  References
    • .  Normative References
    • .  Informative References
  • Acknowledgements
  • Contributors
  • Authors' Addresses
Introduction In situ Operations, Administration, and Maintenance (IOAM) records operational and telemetry information in the packet while the packet traverses a path between two points in the network. IOAM concepts and associated nomenclature as well as IOAM Data-Fields are defined in . This document outlines how IOAM Data-Fields are encapsulated in IPv6 and discusses deployment requirements for networks that use IPv6-encapsulated IOAM Data-Fields. The terms "encapsulation" and "decapsulation" are used in this document in the same way as in : An IOAM encapsulating node incorporates one or more IOAM Option-Types into packets that IOAM is enabled for.
Conventions
Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.
Abbreviations Abbreviations used in this document:
E2E:
Edge-to-Edge
IOAM:
In situ Operations, Administration, and Maintenance as defined in
OAM:
Operations, Administration, and Maintenance
POT:
Proof of Transit
In situ OAM Metadata Transport in IPv6 IOAM in IPv6 is used to enhance diagnostics of IPv6 networks. It complements other mechanisms designed to enhance diagnostics of IPv6 networks, such as the "IPv6 Performance and Diagnostic Metrics (PDM) Destination Option" described in . At the time this document was written, several implementations of IOAM for IPv6 exist, e.g., IOAM for IPv6 in the Linux Kernel (supported from Kernel version 5.15 onward, IPv6 IOAM in Linux Kernel) and IOAM for IPv6 in Vector Packet Processing (VPP). IOAM Data-Fields can be encapsulated with two types of extension headers in IPv6 packets -- either the hop-by-hop options header or the destination options header. Multiple options with the same option type MAY appear in the same hop-by-hop options or destination options header with distinct content. An IPv6 packet carrying IOAM data in an extension header can have other extension headers, compliant with .
IPv6 Hop-by-Hop and Destination Option Format for Carrying IOAM Data-Fields +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Option-Type | Opt Data Len | Reserved | IOAM Opt-Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ | | | . . I . . O . . A . . M . . . . Option Data . O . . P . . T . . I . . O . . N | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+
Option-Type:
8-bit option type identifier as defined in .
Opt Data Len:
8-bit unsigned integer. Length of this option, in octets, not including the first 2 octets.
Reserved:
8-bit field MUST be set to zero by the source.
IOAM Option-Type:
Abbreviated to "IOAM Opt-Type" in the diagram above: 8-bit field as defined in .
Option Data:
Variable-length field. The data is specific to the Option-Type, as detailed below.
Pre-allocated Trace Option:
The IOAM Pre-allocated Trace Option-Type, defined in , is represented as an IPv6 option in the hop-by-hop extension header:
Option-Type:
0x31 (8-bit identifier of the IPv6 Option-Type for IOAM).
IOAM Type:
IOAM Pre-allocated Trace Option-Type.
Proof of Transit Option-Type:
The IOAM POT Option-Type, defined in , is represented as an IPv6 option in the hop-by-hop extension header:
Option-Type:
0x31 (8-bit identifier of the IPv6 Option-Type for IOAM).
IOAM Type:
IOAM POT Option-Type.
Edge-to-Edge Option:
The IOAM E2E Option, defined in , is represented as an IPv6 option in destination extension header:
Option-Type:
0x11 (8-bit identifier of the IPv6 Option-Type for IOAM).
IOAM Type:
IOAM E2E Option-Type.
Direct Export (DEX) Option:
The IOAM Direct Export Option-Type, defined in , is represented as an IPv6 option in the hop-by-hop extension header:
Option-Type:
0x11 (8-bit identifier of the IPv6 Option-Type for IOAM).
IOAM Type:
IOAM Direct Export (DEX) Option-Type.
All the IOAM IPv6 options defined here have alignment requirements. Specifically, they all require alignment on multiples of 4 bytes. This ensures that fields specified in are aligned at a multiple-of-4 offset from the start of the hop-by-hop and destination options header. IPv6 options can have a maximum length of 255 octets. Consequently, the total length of IOAM Option-Types including all data fields is also limited to 255 octets when encapsulated into IPv6.
IOAM Deployment in IPv6 Networks
Considerations for IOAM Deployment and Implementation in IPv6 Networks IOAM deployments in IPv6 networks MUST take the following considerations and requirements into account.
  1. IOAM MUST be deployed in an IOAM-Domain. An IOAM-Domain is a set of nodes that use IOAM. An IOAM-Domain is bounded by its perimeter or edge. The set of nodes forming an IOAM-Domain may be connected to the same physical infrastructure (e.g., a service provider's network). They may also be remotely connected to each other (e.g., an enterprise VPN or an overlay). It is expected that all nodes in an IOAM-Domain are managed by the same administrative entity. Please refer to for more details on IOAM-Domains.
  2. Implementations of IOAM MUST ensure that the addition of IOAM Data-Fields does not alter the way routers forward packets or the forwarding decisions they make. Packets with added IOAM information must follow the same path within the domain as an identical packet without IOAM information would, even in the presence of Equal-Cost Multipath (ECMP). This behavior is important for deployments where IOAM Data-Fields are only added "on-demand". Implementations of IOAM MUST ensure that ECMP behavior for packets with and without IOAM Data-Fields is the same. In order for IOAM to work in IPv6 networks, IOAM MUST be explicitly enabled per interface on every node within the IOAM-Domain. Unless a particular interface is explicitly enabled (i.e., explicitly configured) for IOAM, a router MUST ignore IOAM Options.
  3. In order to maintain the integrity of packets in an IOAM-Domain, the Maximum Transmission Unit (MTU) of transit routers and switches must be configured to a value that does not lead to an "ICMP Packet Too Big" error message being sent to the originator and the packet being dropped. The PMTU tolerance range must be identified, and IOAM encapsulation operations or data field insertion must not exceed this range. Control of the MTU is critical to the proper operation of IOAM. The PMTU tolerance must be identified through configuration, and IOAM operations must not exceed the packet size beyond PMTU.
  4. precludes insertion of IOAM data directly into the original IPv6 header of in-flight packets. IOAM deployments that do not encapsulate/decapsulate IOAM on the host but desire to encapsulate/decapsulate IOAM on transit nodes MUST add an additional IPv6 header to the original packet. IOAM data is added to this additional IPv6 header.
IOAM-Domains Bounded by Hosts For deployments where the IOAM-Domain is bounded by hosts, hosts will perform the operation of IOAM Data-Field encapsulation and decapsulation, i.e., hosts will place the IOAM Data-Fields directly in the IPv6 header or remove the IOAM Data-Fields directly from the IPv6 header. IOAM data is carried in IPv6 packets as hop-by-hop or destination options as specified in this document.
IOAM-Domains Bounded by Network Devices For deployments where the IOAM-Domain is bounded by network devices, network devices such as routers form the edge of an IOAM-Domain. Network devices will perform the operation of IOAM Data-Field encapsulation and decapsulation. Network devices will encapsulate IOAM Data-Fields in an additional, outer, IPv6 header that carries the IOAM Data-Fields.
Security Considerations This document describes the encapsulation of IOAM Data-Fields in IPv6. For general IOAM security considerations, see . Security considerations of the specific IOAM Data-Fields for each case (i.e., Trace, POT, and E2E) are also described and defined in . As this document describes new options for IPv6, the security considerations of and apply. From a network-protection perspective, there is an assumed trust model such that any node that adds IOAM to a packet, removes IOAM from a packet, or modifies IOAM Data-Fields of a packet is assumed to be allowed to do so. By default, packets that include IPv6 extension headers with IOAM information MUST NOT be leaked through the boundaries of the IOAM-Domain. IOAM-Domain boundary routers MUST filter any incoming traffic from outside the IOAM-Domain that contains IPv6 extension headers with IOAM information. IOAM-Domain boundary routers MUST also filter any outgoing traffic leaving the IOAM-Domain that contains IPv6 extension headers with IOAM information. In the general case, an IOAM node only adds, removes, or modifies an IPv6 extension header with IOAM information, if the directive to do so comes from a trusted source and the directive is validated. Problems may occur if the above behaviors are not implemented or if the assumed trust model is violated (e.g., through a security breach). In addition to the security considerations discussed in , the security considerations associated with IPv6 extension headers listed in apply.
Applicability of Authentication Header (AH) The network devices in an IOAM-Domain are trusted to add, update, and remove IOAM options according to the constraints specified in . IOAM-Domain does not rely on the AH as defined in to secure IOAM options. The use of IOAM options with AH and its processing are not defined in this document. Future documents may define the use of IOAM with AH and its processing.
IANA Considerations IANA has assigned the IPv6 Option-Types from the "Destination Options and Hop-by-Hop Options" subregistry of "Internet Protocol Version 6 (IPv6) Parameters" .
Hex Value Binary Value Description Reference
act chg rest
0x11 00 0 10001 IOAM Destination Option and IOAM Hop-by-Hop Option RFC 9486
0x31 00 1 10001 IOAM Destination Option and IOAM Hop-by-Hop Option RFC 9486
References Normative References Key words for use in RFCs to Indicate Requirement Levels In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Data Fields for In Situ Operations, Administration, and Maintenance (IOAM) In situ Operations, Administration, and Maintenance (IOAM) collects operational and telemetry information in the packet while the packet traverses a path between two points in the network. This document discusses the data fields and associated data types for IOAM. IOAM-Data-Fields can be encapsulated into a variety of protocols, such as Network Service Header (NSH), Segment Routing, Generic Network Virtualization Encapsulation (Geneve), or IPv6. IOAM can be used to complement OAM mechanisms based on, e.g., ICMP or other types of probe packets. In Situ Operations, Administration, and Maintenance (IOAM) Direct Exporting In situ Operations, Administration, and Maintenance (IOAM) is used for recording and collecting operational and telemetry information. Specifically, IOAM allows telemetry data to be pushed into data packets while they traverse the network. This document introduces a new IOAM option type (denoted IOAM-Option-Type) called the "IOAM Direct Export (DEX) Option-Type". This Option-Type is used as a trigger for IOAM data to be directly exported or locally aggregated without being pushed into in-flight data packets. The exporting method and format are outside the scope of this document. Informative References Record Route for IPv6 (RR6) Hop-by-Hop Option Extension NEC Corporation This document proposes a 'Record Route for IPv6 (RR6)' mechanism. It is composed of one new IPv6 hop-by-hop option (RR6 option) extension. The RR6 mechanism is redesigned to establish an optimized record route mechanism for IPv6. Two types of new features are introduced. 1. In order to prevent scalability problems and to make the mechanism flexible, the beginning and ending points of the data recording range are controlled by using their specifiers. 2. In order to record long IPv6 addresses efficiently, a simple address compression method is introduced. Since the compression is achieved by utilizing characteristics of recorded IPv6 addresses effectively, the mechanism is efficient and optimized for IPv6. Work in Progress IP Authentication Header This document describes an updated version of the IP Authentication Header (AH), which is designed to provide authentication services in IPv4 and IPv6. This document obsoletes RFC 2402 (November 1998). [STANDARDS-TRACK] Internet Protocol, Version 6 (IPv6) Specification This document specifies version 6 of the Internet Protocol (IPv6). It obsoletes RFC 2460. IPv6 Performance and Diagnostic Metrics (PDM) Destination Option To assess performance problems, this document describes optional headers embedded in each packet that provide sequence numbers and timing information as a basis for measurements. Such measurements may be interpreted in real time or after the fact. This document specifies the Performance and Diagnostic Metrics (PDM) Destination Options header. The field limits, calculations, and usage in measurement of PDM are included in this document. Operational Implications of IPv6 Packets with Extension Headers This document summarizes the operational implications of IPv6 extension headers specified in the IPv6 protocol specification (RFC 8200) and attempts to analyze reasons why packets with IPv6 extension headers are often dropped in the public Internet.
Acknowledgements The authors would like to thank , , , , , , , , , , , , , and for the comments and advice. For the IPv6 encapsulation, this document leverages concepts described in . The authors would like to acknowledge the work done by the author and people involved in writing it.
Contributors This document was the collective effort of several authors. The text and content were contributed by the editors and the coauthors listed below. Cisco Systems, Inc.
7200-11 Kit Creek Road Research Triangle Park NC 27709 United States of America cpignata@cisco.com
RtBrick Inc.
hannes@rtbrick.com
john@leddy.net
JP Morgan Chase
25 Bank Street London E14 5JP United Kingdom stephen.youell@jpmorgan.com
Huawei Network.IO Innovation Lab
Israel tal.mizrahi.phd@gmail.com
Mellanox Technologies, Inc.
350 Oakmead Parkway, Suite 100 Sunnyvale CA 94085 United States of America avivk@mellanox.com
Mellanox Technologies, Inc.
350 Oakmead Parkway, Suite 100 Sunnyvale CA 94085 United States of America gbarak@mellanox.com
Facebook
1 Hacker Way Menlo Park CA 94025 United States of America petr@fb.com
Barefoot Networks, an Intel company
4750 Patrick Henry Drive Santa Clara CA 95054 United States of America mickey.spiegel@intel.com
Kaloom
suresh@kaloom.com
Cisco Systems, Inc.
7200 Kit Creek Road Research Triangle Park NC 27709 United States of America rajiva@cisco.com
PO BOX 521 Heidelberg VIC 3084 Australia markzzzsmith+id@gmail.com
Authors' Addresses Thoughtspot
3rd Floor, Indiqube Orion 24th Main Rd, Garden Layout, HSR Layout Bangalore Karnataka 560 102 India shwetha.bhandari@thoughtspot.com
Cisco Systems, Inc.
Hansaallee 249, 3rd Floor Duesseldorf 40549 Germany fbrockne@cisco.com