Precision Availability Metrics (PAMs) for Services Governed by Service Level Objectives (SLOs)Ericssongregimirsky@gmail.comEricssonjoel.halpern@ericsson.comZTE Corp.xiao.min2@zte.com.cnludwig@clemm.orgFuturewei2330 Central ExpresswaySanta ClaraCA95050United States of Americastrazpdj@gmail.comInria and University of Luxembourg615 Rue du Jardin BotaniqueVillers-les-Nancy54600Francejerome.francois@inria.fr
TSV
ippmIPPMPerformance Measurement
This document defines a set of metrics for networking services with
performance requirements expressed as Service Level Objectives (SLOs).
These metrics, referred to as "Precision Availability Metrics (PAMs)",
are useful for defining and monitoring SLOs.
For example, PAMs can be used by providers and/or customers of an RFC 9543 Network Slice Service
to assess whether the service is provided in compliance with its defined SLOs.
Status of This Memo
This document is not an Internet Standards Track specification; it is
published for informational purposes.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are candidates for any level of Internet
Standard; see Section 2 of RFC 7841.
Information about the current status of this document, any
errata, and how to provide feedback on it may be obtained at
.
Copyright Notice
Copyright (c) 2024 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
() in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Revised BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Revised BSD License.
Table of Contents
. Introduction
. Conventions
. Terminology
. Acronyms
. Precision Availability Metrics
. Introducing Violated Intervals
. Derived Precision Availability Metrics
. PAM Configuration Settings and Service Availability
. Statistical SLO
. Other Expected PAM Benefits
. Extensions and Future Work
. IANA Considerations
. Security Considerations
. Informative References
Acknowledgments
Contributors
Authors' Addresses
Introduction
Service providers and users often need to assess the quality with which network services are being delivered.
In particular, in cases where service-level guarantees are documented (including their companion metrology) as part of a
contract established between the customer and the service provider, and Service Level Objectives (SLOs) are defined,
it is essential to provide means to verify that what has been delivered complies with what has been possibly negotiated
and (contractually) defined between the customer and the service provider.
Examples of SLOs would be target values for the maximum packet delay
(one-way and/or round-trip) or maximum packet loss ratio that would be deemed acceptable.
More generally, SLOs can be used to characterize the ability of a particular set of nodes to communicate
according to certain measurable expectations. Those expectations can include but are not limited to aspects
such as latency, delay variation, loss, capacity/throughput, ordering, and fragmentation.
Whatever SLO parameters are chosen and whichever way service-level parameters are being measured,
Precision Availability Metrics indicate whether or not a given service has been available according to expectations at all times.
Several metrics (often documented in the IANA "Performance Metrics" registry
according to and ) can be used to characterize the service quality, expressing
the perceived quality of delivered networking services versus their SLOs.
Of concern is not so much the absolute service level (for example, actual latency experienced)
but whether the service is provided in compliance with the negotiated and eventually contracted service levels.
For instance, this may include whether the experienced packet delay falls within
an acceptable range that has been contracted for the service.
The specific quality of service depends on the SLO or a set thereof for a given service that is in effect.
Non-compliance to an SLO might result in the degradation of the quality of experience for gamers
or even jeopardize the safety of a large geographical area.
The same service level may be deemed acceptable for one application, while
unacceptable for another, depending on the needs of the application. Hence,
it is not sufficient to measure service levels per se over time; the quality
of the service being contextually provided (e.g., with the applicable SLO in
mind) must be also assessed. However, at this point, there are no standard
metrics that can be used to account for the quality with which services are
delivered relative to their SLOs or to determine whether their SLOs are
being met at all times. Such metrics and the instrumentation to support
them are essential for various purposes, including monitoring (to ensure
that networking services are performing according to their objectives) as
well as accounting (to maintain a record of service levels delivered, which
is important for the monetization of such services as well as for the
triaging of problems).
The current state-of-the-art of metrics include, for example,
interface metrics that can be used to obtain statistical data on traffic volume and
behavior that can be observed at an interface . However, they are agnostic of actual service levels and not specific to
distinct flows. Flow records maintain statistics
about flows, including flow volume and flow duration, but again, they
contain very little information about service levels, let
alone whether the service levels delivered meet their respective targets, i.e., their associated SLOs.
This specification introduces a new set of metrics, Precision Availability Metrics (PAMs), aimed at capturing
service levels for a flow, specifically the degree to
which the flow complies with the SLOs that are in effect.
PAMs can be used to assess whether a service is provided in compliance with its defined SLOs.
This information can be used in multiple ways, for example,
to optimize service delivery, take timely counteractions in the event of service degradation,
or account for the quality of services being delivered.
Availability is discussed in .
In this document, the term "availability" reflects that
a service that is characterized by its SLOs is considered unavailable whenever those SLOs are violated,
even if basic connectivity is still working. "Precision" refers to services
whose service levels are governed by SLOs and must be delivered precisely
according to the associated quality and performance requirements. It should be noted that precision
refers to what is being assessed, not the mechanism used to measure it. In other words,
it does not refer to the precision of the mechanism with which actual service levels are measured.
Furthermore, the precision, with respect to the delivery of an SLO, particularly applies when a metric value
approaches the specified threshold levels in the SLO.
The specification and implementation of methods
that provide for accurate measurements are separate topics independent of the definition of
the metrics in which the results of such measurements would be expressed.
Likewise, Service Level Expectations (SLEs), as defined in ,
are outside the scope of this document.
ConventionsTerminology
In this document, SLA and SLO are used as defined in .
The reader may refer to
for an applicability example of these concepts in the context of RFC 9543 Network Slice Services.
Acronyms
IPFIX
IP Flow Information Export
PAM
Precision Availability Metric
SLA
Service Level Agreement
SLE
Service Level Expectation
SLO
Service Level Objective
SVI
Severely Violated Interval
SVIR
Severely Violated Interval Ratio
SVPC
Severely Violated Packets Count
VFI
Violation-Free Interval
VI
Violated Interval
VIR
Violated Interval Ratio
VPC
Violated Packets Count
Precision Availability MetricsIntroducing Violated Intervals
When analyzing the availability metrics of a service between two measurement points,
a time interval as the unit of PAMs needs to be selected. In ,
a time interval of one second is used. That is reasonable, but some services may require different granularity (e.g., decamillisecond).
For that reason, the time interval in PAMs is viewed as a variable parameter, though constant for a particular measurement session.
Furthermore, for the purpose of PAMs, each time interval is classified as either Violated Interval (VI),
Severely Violated Interval (SVI), or Violation-Free Interval (VFI). These are defined as follows:
VI is a time interval during which at least one of the performance
parameters degraded below its configurable optimal threshold.
SVI is a time interval during which at least one of the performance
parameters degraded below its configurable critical threshold.
Consequently, VFI is a time interval during which all performance parameters are
at or better than their respective pre-defined optimal levels.
The monitoring of performance parameters to determine the quality of an interval
is performed between the elements of the network that are identified in the SLO corresponding to the performance parameter.
Mechanisms for setting levels of a threshold of an SLO are outside the scope of this document.
From the definitions above, a set of basic metrics can be defined that count the number of time intervals that fall into each category:
VI count
SVI count
VFI count
These count metrics are essential in calculating respective ratios (see )
that can be used to assess the instability of a service.
Beyond accounting for violated intervals, it is sometimes beneficial to
maintain counts of packets for which a performance threshold is violated. For
example, this allows for distinguishing between cases in which violated
intervals are caused by isolated violation occurrences (such as a sporadic
issue that may be caused by a temporary spike in a queue depth along the
packet's path) or by broad violations across multiple packets (such as a
problem with slow route convergence across the network or more foundational
issues such as insufficient network resources). Maintaining such counts and
comparing them with the overall amount of traffic also facilitate assessing
compliance with statistical SLOs (see ). For these reasons, the following
additional metrics are defined:
VPC (Violated Packets Count)
SVPC (Severely Violated Packets Count)
Derived Precision Availability Metrics
A set of metrics can be created based on PAMs as introduced in this document.
In this document, these metrics are referred to as "derived PAMs".
Some of these metrics are modeled after Mean Time Between Failure (MTBF) metrics; a
"failure" in this context refers to a failure to deliver a service according to its SLO.
Time since the last violated interval (e.g., since last violated ms or
since last violated second).
This parameter is suitable for monitoring the current compliance status of the service, e.g., for trending analysis.
Number of packets since the last violated packet. This parameter is
suitable for the monitoring of the current compliance status of the service.
Mean time between VIs (e.g., between violated milliseconds or between violated seconds). This parameter is the
arithmetic mean of time between consecutive VIs.
Mean packets between VIs. This parameter is the arithmetic
mean of the number of SLO-compliant packets between consecutive VIs.
It is another variation of MTBF in a service setting.
An analogous set of metrics can be produced for SVI:
Time since the last SVI (e.g., since last violated ms or since last violated second). This parameter is suitable
for the monitoring of the current compliance status of the service.
Number of packets since the last severely violated packet. This parameter is
suitable for the monitoring of the current compliance status of the service.
Mean time between SVIs (e.g., between severely violated
milliseconds or between severely violated seconds). This parameter is the
arithmetic mean of time between consecutive SVIs.
Mean packets between SVIs. This parameter is the arithmetic
mean of the number of SLO-compliant packets between consecutive SVIs.
It is another variation of "MTBF" in a service setting.
To indicate a historic degree of precision availability, additional derived PAMs can be defined as follows:
Violated Interval Ratio (VIR) is the ratio of the summed numbers of VIs and SVIs to the total number of time unit intervals in a
time of the availability periods during a fixed measurement session.
Severely Violated Interval Ratio (SVIR) is the ratio of SVIs to the total number of time unit intervals in a time of the availability periods
during a fixed measurement session.
PAM Configuration Settings and Service Availability
It might be useful for a service provider to determine the current condition of the service for which
PAMs are maintained. To facilitate this, it is conceivable to complement PAMs with a state model.
Such a state model can be used to indicate whether a service is currently considered as available or unavailable
depending on the network's recent ability to provide service without incurring intervals during which violations occur.
It is conceivable to define such a state model in which transitions occur per some predefined PAM settings.
While the definition of a service state model is outside the scope of this document, this section provides
some considerations for how such a state model and accompanying configuration settings could be defined.
For example, a state model could be defined by a Finite State Machine featuring two states:
"available" and "unavailable". The initial state could be "available". A service could subsequently be deemed as "unavailable"
based on the number of successive interval violations that have been experienced up to the particular observation time moment.
To return to a state of "available", a number of intervals without violations would need to be observed.
The number of successive intervals with violations, as well as the
number of successive intervals that are free of violations, required
for a state to transition to another state is defined by a configuration setting.
Specifically, the following configuration parameters are defined:
Unavailability threshold:
The number of successive intervals during which a violation occurs to transition to an unavailable state.
Availability threshold:
The number of successive intervals during which
no violations must occur to allow transition to an available state from a
previously unavailable state.
Additional configuration parameters could be defined to account for the severity of violations. Likewise, it is conceivable to define
configuration settings that also take VIR and SVIR into account.
Statistical SLO
It should be noted that certain SLAs may be
statistical, requiring the service levels of packets in a
flow to adhere to specific distributions. For example, an SLA might
state that any given SLO applies to at least a certain percentage of
packets, allowing for a certain level of, for example,
packet loss and/or exceeding packet delay threshold to take place.
Each such event, in that case, does not necessarily constitute an
SLO violation. However, it is still useful to maintain those
statistics, as the number of out-of-SLO packets still matters when
looked at in proportion to the total number of packets.
Along that vein, an SLA might establish a multi-tiered SLO of, say, end-to-end
latency (from the lowest to highest tier) as follows:
not to exceed 30 ms for any packet;
not to exceed 25 ms for 99.999% of packets; and
not to exceed 20 ms for 99% of packets.
In that case, any individual packet with a latency greater than 20 ms latency
and lower than 30 ms cannot be considered an SLO violation in itself, but compliance with
the SLO may need to be assessed after the fact.
To support statistical SLOs more directly requires
additional metrics, for example, metrics that represent histograms for
service-level parameters with buckets corresponding to individual
SLOs. Although the definition of histogram metrics is outside the scope of this document
and could be considered for future work (see ), for the example just given, a histogram
for a particular flow could be maintained with four buckets: one
containing the count of packets within 20 ms, a second with a count of
packets between 20 and 25 ms (or simply all within 25 ms), a third with
a count of packets between 25 and 30 ms (or merely all packets within
30 ms), and a fourth with a count of anything beyond (or simply a total
count). Of course, the number of buckets and the boundaries between
those buckets should correspond to the needs of the SLA associated with the application,
i.e., to the specific guarantees and SLOs that were
provided.
Other Expected PAM Benefits
PAMs provide several benefits with other, more conventional performance metrics.
Without PAMs, it would be possible to conduct ongoing measurements of service levels,
maintain a time series of service-level records, and then assess compliance with specific
SLOs after the fact. However, doing so would require the collection of vast amounts of data
that would need to be generated, exported, transmitted, collected, and stored.
In addition, extensive post-processing would be required to compare that data against SLOs
and analyze its compliance. Being able to perform these tasks at scale
and in real time would present significant additional challenges.
Adding PAMs allows for a more compact expression of service-level compliance.
In that sense, PAMs do not simply represent raw data but expresses actionable information.
In conjunction with proper instrumentation, PAMs can thus help avoid expensive post-processing.
Extensions and Future Work
The following is a list of items that are outside the scope of this specification but will be useful extensions and opportunities for future work:
A YANG data model will allow PAMs to be incorporated into monitoring applications based on the YANG, NETCONF, and RESTCONF frameworks.
In addition, a YANG data model will enable the configuration and retrieval of PAM-related settings.
A set of IPFIX Information Elements will allow PAMs to be associated with flow records and exported as part of flow data,
for example, for processing by accounting applications that assess compliance of delivered services with quality guarantees.
Additional second-order metrics, such as "longest disruption of service time" (measuring consecutive time units with SVIs),
can be defined and would be deemed useful by some users. At the same time, such metrics can be computed in a straightforward manner
and will be application specific in many cases. For this reason, such metrics are omitted here in order to not overburden this specification.
Metrics can be defined to represent histograms for service-level parameters with buckets corresponding to individual SLOs.
IANA ConsiderationsThis document has no IANA actions.Security Considerations
Instrumentation for metrics that are used to assess compliance with
SLOs constitutes an attractive target for an attacker. By interfering
with the maintenance of such metrics, services could be falsely
identified as complying (when they are not) or vice versa
(i.e., flagged as being non-compliant when indeed they are). While this
document does not specify how networks should be instrumented to
maintain the identified metrics, such instrumentation needs to be
adequately secured to ensure accurate measurements and prohibit
tampering with metrics being kept.
Where metrics are being defined relative to an SLO, the configuration
of those SLOs needs to be adequately secured. Likewise, where
SLOs can be adjusted, the correlation between any metric instance
and a particular SLO must be unambiguous. The same service levels that constitute
SLO violations for one flow and should be maintained as part of
the "violated time units" and related metrics
may be compliant for another flow. In cases when it is
impossible to tie together SLOs and PAMs, it is
preferable to merely maintain statistics about service levels
delivered (for example, overall histograms of end-to-end
latency) without assessing which constitute violations.
By the same token, the definition of what constitutes a
"severe" or a "significant" violation depends on configuration settings or
context. The configuration of such settings or context needs to be
specially secured. Also, the configuration must be bound to
the metrics being maintained. Thus, it will be clear which configuration setting
was in effect when those metrics were being assessed. An attacker
that can tamper with such configuration settings will render the
corresponding metrics useless (in the best case) or misleading (in
the worst case).
Informative ReferencesPerformance MetricsIANAEnd-to-end error performance parameters and objectives for international, constant bit-rate digital paths and connectionsITU-TThe Interfaces Group MIBThis memo discusses the 'interfaces' group of MIB-II, especially the experience gained from the definition of numerous media-specific MIB modules for use in conjunction with the 'interfaces' group for managing various sub-layers beneath the internetwork-layer. It specifies clarifications to, and extensions of, the architectural issues within the MIB-II model of the 'interfaces' group. [STANDARDS-TRACK]Terminology for Policy-Based ManagementThis document is a glossary of policy-related terms. It provides abbreviations, explanations, and recommendations for use of these terms. The intent is to improve the comprehensibility and consistency of writing that deals with network policy, particularly Internet Standards documents (ISDs). This memo provides information for the Internet community.Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow InformationThis document specifies the IP Flow Information Export (IPFIX) protocol, which serves as a means for transmitting Traffic Flow information over the network. In order to transmit Traffic Flow information from an Exporting Process to a Collecting Process, a common representation of flow data and a standard means of communicating them are required. This document describes how the IPFIX Data and Template Records are carried over a number of transport protocols from an IPFIX Exporting Process to an IPFIX Collecting Process. This document obsoletes RFC 5101.Information Model for IP Flow Information Export (IPFIX)This document defines the data types and management policy for the information model for the IP Flow Information Export (IPFIX) protocol. This information model is maintained as the IANA "IPFIX Information Elements" registry, the initial contents of which were defined by RFC 5102. This information model is used by the IPFIX protocol for encoding measured traffic information and information related to the traffic Observation Point, the traffic Metering Process, and the Exporting Process. Although this model was developed for the IPFIX protocol, it is defined in an open way that allows it to be easily used in other protocols, interfaces, and applications. This document obsoletes RFC 5102.IP Connectivity Provisioning Profile (CPP)This document describes the Connectivity Provisioning Profile (CPP) and proposes a CPP template to capture IP/MPLS connectivity requirements to be met within a service delivery context (e.g., Voice over IP or IP TV). The CPP defines the set of IP transfer parameters to be supported by the underlying transport network together with a reachability scope and bandwidth/capacity needs. Appropriate performance metrics, such as one-way delay or one-way delay variation, are used to characterize an IP transfer service. Both global and restricted reachability scopes can be captured in the CPP.Such a generic CPP template is meant to (1) facilitate the automation of the service negotiation and activation procedures, thus accelerating service provisioning, (2) set (traffic) objectives of Traffic Engineering functions and service management functions, and (3) improve service and network management systems with 'decision- making' capabilities based upon negotiated/offered CPPs.A YANG Data Model for Interface ManagementThis document defines a YANG data model for the management of network interfaces. It is expected that interface-type-specific data models augment the generic interfaces data model defined in this document. The data model includes definitions for configuration and system state (status information and counters for the collection of statistics).The YANG data model in this document conforms to the Network Management Datastore Architecture (NMDA) defined in RFC 8342.This document obsoletes RFC 7223.Registry for Performance MetricsThis document defines the format for the IANA Registry of Performance
Metrics. This document also gives a set of guidelines for Registered
Performance Metric requesters and reviewers.Initial Performance Metrics Registry EntriesThis memo defines the set of initial entries for the IANA Registry of
Performance Metrics. The set includes UDP Round-Trip Latency and
Loss, Packet Delay Variation, DNS Response Latency and Loss, UDP
Poisson One-Way Delay and Loss, UDP Periodic One-Way Delay and Loss,
ICMP Round-Trip Latency and Loss, and TCP Round-Trip Delay and Loss.A Framework for Network Slices in Networks Built from IETF TechnologiesThis document describes network slicing in the context of networks built from IETF technologies. It defines the term "IETF Network Slice" to describe this type of network slice and establishes the general principles of network slicing in the IETF context.The document discusses the general framework for requesting and operating IETF Network Slices, the characteristics of an IETF Network Slice, the necessary system components and interfaces, and the mapping of abstract requests to more specific technologies. The document also discusses related considerations with monitoring and security.This document also provides definitions of related terms to enable consistent usage in other IETF documents that describe or use aspects of IETF Network Slices.Acknowledgments
The authors greatly appreciate review and comments by and .
ContributorsChina Mobile32 XuanWuMenXi StreetBeijing100053Chinahanliuyan@chinamobile.comOrange35000 RennesFrancemohamed.boucadair@orange.comOld Dog ConsultingUnited Kingdomadrian@olddog.co.ukAuthors' AddressesEricssongregimirsky@gmail.comEricssonjoel.halpern@ericsson.comZTE Corp.xiao.min2@zte.com.cnludwig@clemm.orgFuturewei2330 Central ExpresswaySanta ClaraCA95050United States of Americastrazpdj@gmail.comInria and University of Luxembourg615 Rue du Jardin BotaniqueVillers-les-Nancy54600Francejerome.francois@inria.fr