Deterministic Networking (DetNet): Packet Ordering FunctionEricssonMagyar Tudosok krt. 11.BudapestHungary1117balazs.a.varga@ericsson.comEricssonMagyar Tudosok krt. 11.BudapestHungary1117janos.farkas@ericsson.comBelden Electronics GmbHStuttgarter Strasse 45-51.NeckartenzlingenGermany72654Stephan.Kehrer@belden.comBelden Electronics GmbHStuttgarter Strasse 45-51.NeckartenzlingenGermany72654Tobias.Heer@belden.com
RTG
DetNet
The replication and elimination functions of the Deterministic Networking
(DetNet) architecture can result in out-of-order packets, which is not
acceptable for some time-sensitive applications. The Packet Ordering
Function (POF) algorithms described in this document enable restoration
of the correct packet order when the replication and elimination
functions are used in DetNet networks. The POF only provides ordering within
the latency bound of a DetNet flow; it does not provide any additional
reliability.
Status of This Memo
This document is not an Internet Standards Track specification; it is
published for informational purposes.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are candidates for any level of Internet
Standard; see Section 2 of RFC 7841.
Information about the current status of this document, any
errata, and how to provide feedback on it may be obtained at
.
Copyright Notice
Copyright (c) 2024 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
() in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Revised BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Revised BSD License.
Table of Contents
. Introduction
. Terminology
. Terms Used in This Document
. Abbreviations
. Requirements for POF Implementations
. POF Algorithms
. Prerequisites and Assumptions
. POF Building Blocks
. The Basic POF Algorithm
. The Advanced POF Algorithm
. Further Enhancements of the POF Algorithms
. Selecting and Using the POF Algorithms
. Control and Management Plane Parameters for POF
. Security Considerations
. IANA Considerations
. References
. Normative References
. Informative References
Acknowledgements
Authors' Addresses
Introduction defines the Packet Replication
Function (PRF) and Packet Elimination Function (PEF) in DetNet for
achieving extremely low packet loss. The PRF and PEF provide service
protection for DetNet flows. This service protection method relies on
copies of the same packet sent over multiple maximally disjoint paths and
uses sequencing information to eliminate duplicates. A possible
implementation of the PRF and PEF is described in , and the related YANG model is
defined in .
In general, use of per-packet replication and elimination functions can
result in out-of-order delivery of packets, which is not acceptable for
some deterministic applications. Correcting packet order is not a trivial
task; therefore, details of a Packet Ordering Function (POF) are
specified in this document.
defines the external observable result of a POF (i.e., that
packets are reordered) but does not specify any implementation details.
So far in packet networks, out-of-order delivery situations have been handled
at higher OSI layers at the endpoints/hosts (e.g., in the TCP stack when
packets are sent to the application layer) and not within a network in nodes
acting at the Layer 2 or Layer 3 OSI layers.
shows a DetNet flow on which Packet
Replication, Elimination, and Ordering Functions (PREOF)
are applied during forwarding from source to destination.
PREOF Scenario in a DetNet Network
+------------+
+-----------E1----+ | |
+----+ | | +---R3---+ | +----+
|src |------R1 +---+ | E3----O1---+ dst|
+----+ | | E2-------+ +----+
+-------R2 |
+-----------------+
R: replication point (PRF)
E: elimination point (PEF)
O: ordering function (POF)
In general, the use of PREOF requires sequencing information
to be included in the packets of a DetNet compound flow. This can be
done by adding a sequence number as part of DetNet encapsulation
. Sequencing information is typically added once,
at or close to the source.
It is important to note that different applications can react differently to out-of-order
delivery. A single out-of-order packet (e.g., packet order #1, #3, #2,
#4, #5) is interpreted by some application as a single error, but
other applications treat it as three errors in a row.
For example, in industrial scenarios,
three errors in a row is a typical error threshold and can cause the
application to stop (e.g., go to a fail-safe state).
The POF ensures in-order delivery for packets within
the latency bound of the DetNet flow. The POF does not correct
errors in the packet flow (e.g., duplicate packets or packets that are too late).
TerminologyTerms Used in This Document
This document uses the terminology established in the DetNet architecture
; the reader is assumed
to be familiar with that document and its terminology.
Abbreviations
The following abbreviations are used in this document:
DetNet
Deterministic Networking
PEF
Packet Elimination Function
POF
Packet Ordering Function
PREOF
Packet Replication, Elimination, and Ordering Functions
PRF
Packet Replication Function
Requirements for POF Implementations
The requirements for POF implementations are:
To solve the out-of-order delivery problem of the replication
and elimination functions of DetNet networks.
To consider the delay bound requirement of a DetNet flow.
To be simple and to require only a minimum set of states,
configuration parameters, and resources per DetNet flow in network
nodes.
To add minimal or no delay to the forwarding process
of packets.
To not require synchronization between PREOF nodes.
Some aspects are explicitly out of scope for a POF:
To eliminate the delay variation caused by the packet ordering.
Dealing with delay variation is a DetNet forwarding sub-layer
target, and it can be achieved, for example, by placing a de-jitter
buffer or flow regulator (e.g., shaping) function after the POF.
POF AlgorithmsPrerequisites and Assumptions
The POF algorithms discussed in this document make some assumptions and
trade-offs regarding the characteristics of the sequence of received
packets. In particular, the algorithms assume that a
PEF is performed on the incoming packets
before they are handed to the POF. Hence, the sequence
of incoming packets can be out-of-order or incomplete but cannot
contain duplicate packets. However, the PREOF run
independently without any state exchange required between the
PEF and the POF or the PRF and the POF. Error cases in which
duplicate packets are presented to the POF can lead to out-of-order delivery of duplicate packets and to increased delays.
The algorithms further require that the delay difference between two
replicated packets that arrive at the PEF before the POF is bounded and
known. Error cases that violate this condition (e.g., a packet that
arrives later than this bound) will result in out-of-order packets.
The algorithms also make some trade-offs. For simplicity, it is designed
to allow for some out-of-order packets directly after
initialization. If this is not acceptable,
provides an alternative initialization scheme
that prevents out-of-order packets in the initialization phase.
POF Building Blocks
The method described in this document provides a POF for DetNet networks. The
configuration parameters of the POF can be derived when engineering the
DetNet flow through the network.
The POF method is provided via the following:
Delay calculator:
Calculates buffering time for out-of-order packets.
Buffering time considers (i) the delay
difference of paths used for forwarding the replicated packets
and (ii) the bounded delay requirement of the given DetNet flow.
Conditional delay buffer:
Used for buffering the out-of-order packets of a
DetNet flow for a given time.
Note: The conditional delay buffer of the POF increases the burstiness of the
traffic as it only adds delay for some of the packets.
shows the building blocks of a
possible POF implementation.
POF Building Blocks
+------------+ +--------------+
| Delay calc | | Conditional |
+--| for packet >--->>---| Delay Buffer >--+
| +------------+ +--------------+ |
| |
+------^--------+ |
->>--| POF selector >---------------------------------+-->>----
| (Flow ident.) |
+---------------+
->>- packet flow
The Basic POF Algorithm
The basic POF algorithm delays all out-of-order packets until all
previous packets arrive or a given time ("POFMaxDelay") elapses. The
basic POF algorithm works as follows:
The sequence number of the last forwarded packet ("POFLastSent") is
stored for each DetNet flow.
The sequence number (seq_num) of a received packet is compared to
that of the last forwarded one ("POFLastSent").
If (seq_num <= POFLastSent + 1)
Then the packet is forwarded without buffering, and "POFLastSent"
is updated (POFLastSent = seq_num).
Else, the received packet is buffered.
A buffered packet is forwarded from the buffer when its seq_num
becomes equal to "POFLastSent + 1" OR a predefined time ("POFMaxDelay")
elapses.
When a packet is forwarded from the buffer, "POFLastSent" is
updated with its seq_num (POFLastSent = seq_num).
Notes:
The difference between sequence numbers in consecutive packets is bounded
due to the history window of the elimination function before the POF.
Therefore, "<=" can be evaluated despite the circular
sequence number space. A possible implementation of the PEF and
the impact of the history window are described in .
The basic POF algorithm can be extended to cope with multiple failure scenarios
(i.e., simultaneous packet loss and out-of-order packets) when the expiration
of the timer for a packet with sequence number N triggers the release of some
packets with a sequence number smaller than N.
The state used by the basic POF algorithm (i.e., "POFLastSent") needs
initialization and maintenance. This works as follows:
The next received packet is forwarded and the "POFLastSent"
updated when the POF is reset OR no packet is received
for a predefined time ("POFTakeAnyTime").
The reset of the POF erases all packets from the time-based
buffer used by the POF.
The basic POF algorithm has two parameters to engineer:
"POFMaxDelay", which cannot be smaller than the delay
difference of the paths used by the flow.
"POFTakeAnyTime", which is calculated based on several factors,
for example, the settings of the elimination function(s) relating
to RECOVERY_TIMEOUT before the POF, the flow characteristics
(e.g., inter-packet time), and the delay difference of the paths
used by the flow.
Design of these parameters is out of scope for this document.
Note: Multiple network failures can impact the POF
(e.g., complete outage of all redundant paths).
The basic POF algorithm increases the delay of packets with maximum
"POFMaxDelay" time. In-order packets are not delayed. This basic
POF method can be applied in all network scenarios where the remaining
delay budget of a flow at the POF point is larger than "POFMaxDelay"
time.
shows the delay budget
situation at the POF point.
Delay Budget Situation at the POF Point
Path delay
difference
/-------------/
<- path with min delay -> /-- remaining delay budget --/
|-----------------------|-------------|----------------------------|
0 t1 t2 T
<-------- path with max delay -------->
/-------------------- delay budget at POF point -------------------/
The Advanced POF Algorithm
In network scenarios where the remaining delay budget of a flow at the
POF point is smaller than "POFMaxDelay" time, the basic method needs
extensions.
The issue is that packets on the longest path cannot be buffered in order
to keep the delay budget of the flow. It must be noted that such a packet
(i.e., forwarded over the longest path) needs no buffering as it is the
last chance to deliver a packet with a given sequence number. This is
because all replicas already arrived via a shorter path(s).
The advanced POF algorithm requires extensions of the basic POF algorithm:
to identify the received packet's path at the POF location and
to make the value of "POFMaxDelay" for buffered packets path
dependent ("POFMaxDelay_i", where "i" notes the path the packet
has used).
The advanced POF algorithm identifies the path of a given packet and uses this
information to select the predefined time ("POFMaxDelay_i") to apply for the
buffered packet. So, in the advanced POF algorithm, "POFMaxDelay" is an
array that contains the predefined and path-specific buffering time for each
redundant path of a flow. Values in the "POFMaxDelay" array are engineered
to fulfill the delay budget requirement.
Design of these parameters is out of scope for this document.
Note: For the advanced POF algorithm, the path-dependent delays
might result in multiple packets triggering the "maximum delay
reached" at exactly the same time. The transmission order of
these packets should be done in their seq_num order.
The method for identifying the packet's path at the POF location
depends on the network scenario.
It can be implemented via
various techniques, for example, using ingress interface information,
encoding the path in the packet itself (e.g., replication functions
set a different FlowID per member flow at their egress and such
a FlowID is used to identify the path of a packet at the POF), or
other means.
Methods for identifying the packet's path are out of scope
for this document.
Note: When using the advanced POF algorithm, it might be
advantageous to combine PEF and POF locations in the DetNet network, as
this can simplify the method used for identifying the packet's path
at the POF location.
Further Enhancements of the POF Algorithms
POF algorithms can be further enhanced by distinguishing the case of
initialization from normal operation at the price of more states and
more sophisticated implementation. Such enhancements could, for example,
react better after some failure scenarios (e.g., complete outage of all
paths of a DetNet flow) and can be dependent on the PEF implementation.
The challenge for POF initialization is that it is not known whether the
first received packet is in-order or out-of-order (for example, after a
reset).
The original initialization (see ) considers the
first packet as in-order, so out-of-order packet(s) during
"POFMaxTime"/"POFMaxTime_path_i" time -- after the first packet is
received -- cannot be corrected.
The motivation behind such an initialization
is simplicity of POF implementation.
A possible enhancement of POF initialization works as follows:
After a reset, all received packets are buffered with their
predefined timer ("POFMaxTime"/"POFMaxTime_path_i").
No basic or advanced POF rules are applied until the first timer
expires.
When the first timer expires, the packet with lowest seq_num in the
buffer is selected and forwarded, and "POFLastSent" is set with its
seq_num.
The basic or advanced POF rules are applied for the packet(s) in the
buffer and the subsequently received packets.
Selecting and Using the POF Algorithms
The selection of the POF algorithm depends on the network scenario and
the remaining delay budget of a flow. Using the POF algorithms and calculating their
parameters require proper design. Knowing the path delay difference is
essential for the POF algorithms described here. Failure scenarios
breaking the design assumptions can impact the result of the POF (e.g.,
packet received out of the expected worst-case delay window
-- calculated based on the path delay difference -- can result in unwanted
out-of-order delivery).
In DetNet scenarios, there is always an elimination function before the
POF (therefore, duplicates are not considered by the POF). Implementing
them together in the same node allows the POF to consider PEF events/states
during the reordering. For example, under normal circumstances, the
difference between sequence numbers in consecutive packets is bounded due
to the history window of the PEF. However, in some scenarios (e.g., reset of
sequence number), the difference can be much larger than the size of the
history window.
Control and Management Plane Parameters for POFPOF algorithms require the following parameters to be set:
Basic POF
"POFMaxDelay"
"POFTakeAnyTime"
Advanced POF
"POFMaxDelay_i" for each possible path i
"POFTakeAnyTime"
Configuration(s) related to network path identification
Note: In a proper design, "POFTakeAnyTime" is always larger than "POFMaxDelay".
Security Considerations
PREOF-related security considerations (including POF) are described in
. There are no
additional POF-related security considerations originating from this
document.
IANA ConsiderationsThis document has no IANA actions.
ReferencesNormative ReferencesDeterministic Networking ArchitectureThis document provides the overall architecture for Deterministic Networking (DetNet), which provides a capability to carry specified unicast or multicast data flows for real-time applications with extremely low data loss rates and bounded latency within a network domain. Techniques used include 1) reserving data-plane resources for individual (or aggregated) DetNet flows in some or all of the intermediate nodes along the path of the flow, 2) providing explicit routes for DetNet flows that do not immediately change with the network topology, and 3) distributing data from DetNet flow packets over time and/or space to ensure delivery of each packet's data in spite of the loss of a path. DetNet operates at the IP layer and delivers service over lower-layer technologies such as MPLS and Time- Sensitive Networking (TSN) as defined by IEEE 802.1.Deterministic Networking (DetNet) Security ConsiderationsA DetNet (deterministic network) provides specific performance guarantees to its data flows, such as extremely low data loss rates and bounded latency (including bounded latency variation, i.e., "jitter"). As a result, securing a DetNet requires that in addition to the best practice security measures taken for any mission-critical network, additional security measures may be needed to secure the intended operation of these novel service properties.This document addresses DetNet-specific security considerations from the perspectives of both the DetNet system-level designer and component designer. System considerations include a taxonomy of relevant threats and attacks, and associations of threats versus use cases and service properties. Component-level considerations include ingress filtering and packet arrival-time violation detection.This document also addresses security considerations specific to the IP and MPLS data plane technologies, thereby complementing the Security Considerations sections of those documents.Informative ReferencesIEEE Standard for Local and metropolitan area networks -- Frame Replication and Elimination for ReliabilityIEEEIEEE Standard for Local and metropolitan area networks -- Frame Replication and Elimination for Reliability - Amendment 1: Information Model, YANG Data Model, and Management Information Base ModuleIEEEAcknowledgements
Authors extend their appreciation to ,
, , , , , , , and for their insightful comments and productive discussion that
helped to improve the document.
Authors' AddressesEricssonMagyar Tudosok krt. 11.BudapestHungary1117balazs.a.varga@ericsson.comEricssonMagyar Tudosok krt. 11.BudapestHungary1117janos.farkas@ericsson.comBelden Electronics GmbHStuttgarter Strasse 45-51.NeckartenzlingenGermany72654Stephan.Kehrer@belden.comBelden Electronics GmbHStuttgarter Strasse 45-51.NeckartenzlingenGermany72654Tobias.Heer@belden.com