Control plane metrics

Monitor the kgateway control plane with Prometheus metrics.

By default, the kgateway control plane exposes metrics in Prometheus format. You can use these metrics to monitor the health and performance of your gateway environment. For more information about how metrics are implemented, refer to the kgateway project developer docs.

View control plane metrics

The following steps show you how to quickly view the metrics endpoint of the control plane deployment. To integrate the metrics into your observability stack, see the OpenTelemetry guide.

Port-forward the control plane deployment on port 9092.

kubectl -n kgateway-system port-forward deployment/kgateway 9092

Open your browser to the metrics endpoint: http://localhost:9092/metrics.

Example output:

# HELP kgateway_controller_reconciliations_total Total controller reconciliations
# TYPE kgateway_controller_reconciliations_total counter
kgateway_controller_reconciliations_total{controller="gateway",result="success"} 1
kgateway_controller_reconciliations_total{controller="gatewayclass",result="success"} 2
kgateway_controller_reconciliations_total{controller="gatewayclass-provisioner",result="success"} 2

Control plane metrics reference

Review the following table to understand more about each metric.

Helpful terms:

Controller: A Kubernetes controller that reconciles resources as part of the kgateway control plane deployment.
Resource: A Kubernetes object that is managed by a controller of the control plane.
Snapshot: A complete, point-in-time representation of the current state of resources that the controller builds and serves to a gateway proxy via the Envoy extensible Discovery Service (XDS) API.
Status write: A status write is an attempt to save the status of a resource. The result label has one of the following values:
- written when the status was saved.
- unchanged when the live status already matched.
- conflict or not_found when the write hit a conflict or the resource no longer existed (the control plane tries again when it processes the resource next).
- error when the write failed after all retries.
Snapshot sync: A snapshot sync roughly represents the time it takes for a resource change to become effective in the gateway proxies.
Transform: The process of the control plane converting high-level resources or intermediate representations (IR) into lower-level representations into the structure that the XDS API expects for a snapshot.

Name	Type	Labels	Help
kgateway_build_info	gauge	version, revision, goversion, goos, goarch	Build information of the running controller. The series always has value 1.
kgateway_controller_reconcile_duration_seconds	histogram	controller, name, namespace	Reconcile duration for controller
kgateway_controller_reconciliations_total	counter	controller, name, namespace, result	Total number of controller reconciliations
kgateway_ec2_discovery_endpoints_active	gauge	namespace, name	Current number of active Envoy endpoints discovered for an EC2 Backend
kgateway_ec2_discovery_error_state	gauge	namespace, name	Whether the most recent EC2 discovery poll for a Backend failed (1) or succeeded (0)
kgateway_ec2_discovery_poll_duration_seconds	histogram	namespace, name, result	Duration of EC2 endpoint discovery polls per Backend
kgateway_ec2_discovery_poll_total	counter	namespace, name, result, reason	Total number of EC2 endpoint discovery refresh attempts per Backend
kgateway_envoy_xds_rejects_active	gauge	gateway_namespace, gateway_name, type_url	Number of connected stream and resource type pairs with unresolved xDS rejections
kgateway_envoy_xds_rejects_total	counter	gateway_namespace, gateway_name, type_url	Total xDS rejection episodes by stream and resource type; repeated NACKs count once until a request without an error
kgateway_nack_resends_suppressed_total	counter	typeURLLabel	Number of times a NACKed xDS response was not re-sent because the snapshot had not changed
kgateway_perclient_bounded_publishes_total	counter	gatewayLabel, namespaceLabel, mode	Withheld or held per-client publications released by a time bound before every referenced cluster was ready, by mode
kgateway_perclient_defers_total	counter	gatewayLabel, namespaceLabel, reason	Per-client xDS snapshots built while a referenced cluster was not ready, by reason
kgateway_perclient_inconsistent_snapshots_total	counter		Per-client xDS snapshots that failed the pre-publication consistency check; any increment is a bug
kgateway_perclient_reclaimed_total	counter		Per-client xDS snapshot cache entries reclaimed after their client left and stayed gone for the grace period
kgateway_perclient_withheld_clients	gauge	gatewayLabel, namespaceLabel, state	Clients whose first publish on their connection is withheld until referenced clusters are ready, by state
kgateway_reconnect_responses_total	counter	typeURLLabel	Number of reconnecting proxies’ first endpoint, route or secret requests whose held version was cleared, so the current or next snapshot answers them even at that version
kgateway_resources_managed	gauge	namespace, parent, resource	Current number of resources managed
kgateway_resources_updates_dropped_total	counter		Total number of xds_snapshot sync metrics updates dropped. If this metric is ever greater than 0, the xds_snapshot sync metrics should be considered invalid until process restart
kgateway_routing_domains	gauge	namespace, gateway, port	Number of domains per listener
kgateway_routing_replacements_total	counter	gateway_namespace, gateway, error_type	Number of routes, virtual hosts, or route configurations replaced with a synthetic 500 direct response due to invalid configuration detected during translation.
kgateway_status_writes_total	counter	kind, result	Total number of status write attempts by resource kind and result. result="error" counts writes that failed after every retry.
kgateway_translator_translation_duration_seconds	histogram	name, namespace, translator	Translation duration
kgateway_translator_translations_total	counter	name, namespace, translator, result	Total number of translations
kgateway_validation_cache_hits_total	counter	caller	Total number of Envoy validation cache hits.
kgateway_validation_cache_misses_total	counter	caller	Total number of Envoy validation cache misses.
kgateway_validation_calls_total	counter	caller	Total number of Envoy validation requests.
kgateway_validation_duration_seconds	histogram	caller, result	Duration of Envoy validation requests.
kgateway_validation_invalid_xds_total	counter	caller	Total number of Envoy invalid-xDS validation results.
kgateway_validation_invocation_errors_total	counter	caller	Total number of Envoy validation invocation errors.
kgateway_validation_mode	gauge	mode, validator_mode	Configured validation mode. The active mode series has value 1.
kgateway_validation_valid_total	counter	caller	Total number of successful Envoy validation results.
kgateway_xds_auth_rq_failure_total	counter		Total number of failed xDS auth requests
kgateway_xds_auth_rq_success_total	counter		Total number of successful xDS auth requests
kgateway_xds_auth_rq_total	counter		Total number of xDS auth requests
kgateway_xds_cluster_scoping_disabled_gateways	gauge	gatewayLabel, namespaceLabel	Gateways with a connected proxy group whose CDS is unscoped because its configuration selects a destination at request time
kgateway_xds_cluster_scoping_dropped_clusters	gauge	gatewayLabel, namespaceLabel	Most backend clusters withheld from any of a gateway’s connected proxy groups because their configuration references none of them
kgateway_xds_cluster_scoping_transitions_total	counter	gatewayLabel, namespaceLabel, transition	Clusters whose emitted-set change needed a grace window, per proxy group, by transition
kgateway_xds_snapshot_deferred_clients	gauge	gateway, namespace	Connected xDS clients whose snapshot is currently withheld because per-client inputs are not ready
kgateway_xds_snapshot_resources	gauge	gateway, namespace, resource	Current number of resources in XDS snapshot
kgateway_xds_snapshot_sync_duration_seconds	histogram	gateway, namespace	Duration of time for a gateway resource update to be synced in an XDS snapshot
kgateway_xds_snapshot_syncs_total	counter	gateway, namespace	Total number of XDS snapshot syncs
kgateway_xds_snapshot_transform_duration_seconds	histogram	gateway, namespace	XDS snapshot transform duration
kgateway_xds_snapshot_transforms_total	counter	gateway, namespace, result	Total number of XDS snapshot transforms

Monitor NACKs

Monitor and troubleshoot Envoy proxy configuration rejections with control plane metrics.

About NACKs

A NACK (negative acknowledgement) occurs when an Envoy proxy rejects a configuration update from the kgateway control plane. NACKs typically indicate configuration issues that prevent the proxy from applying the desired routing, policy, or listener settings.

Typically, the kgateway control plane reports errors during translation, in which the control plane immediately rejects invalid configuration in custom resources and records errors in custom resource statuses. However, some errors can only be caught at configuration time, in which the control plane cannot determine the configuration failure until an Envoy proxy in the data plane rejects the configuration.

To track these NACKs, the kgateway control plane exposes Prometheus metrics that monitor the health of config synchronization between the control plane and Envoy proxies.

For more general information about kgateway control plane metrics, see Control plane metrics.

Common causes of NACKs include:

Invalid Envoy configuration that passes control plane validation but is rejected by the Envoy proxy at runtime.
Invalid TLS certificates.

You can access the kgateway control plane metrics endpoint to view NACK metrics that track configuration rejections from Envoy proxies.

View the NACK metrics

Port-forward the control plane deployment on port 9092.

kubectl -n kgateway-system port-forward deployment/kgateway 9092

Open the kgateway control plane metrics endpoint.

Search for the kgateway_envoy_xds_rejects_total and kgateway_envoy_xds_rejects_active metrics.

Note

The metrics only appear after at least one NACK has been reported. If no NACKs have occurred, these metrics are not present in the metrics output.

Example output:

# HELP kgateway_envoy_xds_rejects_total Total number of xDS responses rejected by envoy proxy
# TYPE kgateway_envoy_xds_rejects_total counter
kgateway_envoy_xds_rejects_total{gateway_namespace="default",gateway_name="gw",type_url="envoy.config.route.v3.RouteConfiguration"} 1
# HELP kgateway_envoy_xds_rejects_active Number of xDS responses currently rejected by envoy proxy
# TYPE kgateway_envoy_xds_rejects_active gauge
kgateway_envoy_xds_rejects_active{gateway_namespace="default",gateway_name="gw",type_url="envoy.config.route.v3.RouteConfiguration"} 0

Both metrics are labeled by gateway_namespace, gateway_name, and type_url. The type_url label identifies the Envoy resource type that was rejected, such as envoy.config.route.v3.RouteConfiguration.

Metric	Type	Description
`kgateway_envoy_xds_rejects_total`	Counter	The cumulative total number of xDS responses rejected by Envoy proxies. This value only increases over time.
`kgateway_envoy_xds_rejects_active`	Gauge	The current number of active xDS rejections. This value increases when a new NACK occurs and decreases when the proxy successfully applies a subsequent configuration update (ACK). A value of `0` indicates no active rejections.

You can use these metrics to configure alerts to notify you when NACKs occur so you can quickly investigate and resolve configuration issues.

For more information about how to set up the observability stack, see the OpenTelemetry stack guide.

OTel stack Gateway proxy metrics

Was this page helpful?