Red Hat Connectivity Link

Connectivity Link tracing

Connectivity Link supports tracing at both the control plane and data-plane levels.

Connectivity Link exports control-plane traces to your OpenTelemetry Collector so that you can observe reconciliation loops and internal operations. This is useful for debugging controller behavior, understanding operator performance, and tracking policy lifecycle events.

Data-plane tracing traces actual user requests through the gateway and policy enforcement components. You can see request flows through Istio, Authorino, Limitador, and the wasm-shim module. Data-plane tracing is useful for debugging request-level issues and policy enforcement.

To use tracing, you must configure both types of tracing. You must configure the kuadrant custom resource (CR) for the data plane. For control plane tracing, you must configure each operating component separately, such as the kuadrant-operator, authorino-operator, and limitador-operator deployments. This configuration sends traces to the same collector, providing a complete view of your Connectivity Link system from policy reconciliation to request processing.

Correlating control plane and data plane traces

Even though control plane and data plane traces are separate, you can correlate them. For example, create a RateLimitPolicy to understand how traces work together to show all events.

Create a RateLimitPolicy at 15:30:00, then view the control plane trace to see the following events:
- Policy reconciliation completed at 15:30:05.
- Limitador configuration updated.
- wasm-shim configuration updated.
Next, send a test request at 15:30:10, then view data plane trace to see the following events:
- Request processed through the wasm-shim module.
- Rate limit check sent to Limitador.
- Response returned.

You can use a similar pattern of action for any events that you want to correlate manually. This type of correlation is useful in development environments.

Control-plane tracing environment variables

You can enable control tracing in Connectivity Link by setting OpenTelemetry environment variables in the deployment. The method for setting the variables depends on your deployment approach, for example, whether you used the Operator Lifecycle Manager (OLM) or YAML manifests.

Control plane traces appear under the service name kuadrant-operator in the Grafana dashboard.

Table 9. Available OpenTelemetry environment variables
Variable	Description	Default
`OTEL_EXPORTER_OTLP_ENDPOINT`	OTLP collector endpoint URL, for example, `rpc://tempo.tempo.svc.cluster.local:4317`. The following supported endpoint URL schemes are: `rpc://host:port` → gRPC OTLP Insecure: `http://host:port` → HTTP OTLP Secure: `https://host:port` → HTTPS OTLP	Tracing disabled
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`	Override endpoint specifically for traces	Uses `OTEL_EXPORTER_OTLP_ENDPOINT`
`OTEL_EXPORTER_OTLP_INSECURE`	Use insecure connection to collector; set to `false` when used in production with TLS	`true`
`OTEL_SERVICE_NAME`	Service name for traces	`kuadrant-operator`
`OTEL_SERVICE_VERSION`	Service version for telemetry data	Empty

Configure data plane tracing in Connectivity Link

Enable data plane tracing in OpenShift Service Mesh with the kuadrant CR. You must perform these steps on each OpenShift Container Platform cluster that you want to use Connectivity Link on.

Prerequisites

You installed Connectivity Link.
You are logged into a running OpenShift Container Platform cluster as an administrator.
You have Red Hat OpenShift Distributed Tracing Platform installed and configured to support OpenTelemetry.
You installed a compatible version of OpenShift Service Mesh.

Procedure

Enable tracing in OpenShift Service Mesh by configuring your Telemetry custom resource (CR) as follows:

Example OpenShift Service Mesh Telemetry CR with tracing

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: gateway-system
spec:
  tracing:
  - providers:
    - name: tempo-otlp
    randomSamplingPercentage: 100
# ...

Apply the configuration by running the following command:
```
$ oc apply -f mesh-default.yaml
```

Configure a tracing extension provider for OpenShift Service Mesh in your Istio CR by adding a list value to the spec.values.meshConfig.extensionProviders parameter. Ensure that you also add the otel port and service information:

Example Istio CR with tracing extension provider

apiVersion: operator.istio.io/v1alpha1
kind: Istio
metadata:
  name: default
spec:
  namespace: gateway-system
  values:
    meshConfig:
      defaultConfig:
        tracing: {}
      enableTracing: true
      extensionProviders:
      - name: tempo-otlp
        opentelemetry:
          port: 4317
          service: tempo.tempo.svc.cluster.local
# ...

Important

If you are setting the controller manually, you must set the OpenTelemetry protocol (OTLP) in the Service CR port name and appProtocol fields. For example, when using gRPC, the port name should begin with grpc- or the appProtocol should be grpc:

kind: Service
apiVersion: v1
metadata:
  name: otel-collector
spec:
  ports:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
      appProtocol: grpc
  selector:
    app: otel-collector
# ...

Apply the configuration by running the following command:
```
$ oc apply -f istio.yaml
```

Optional. If you want to collect Authorino and Limitador traces in a different location than your Kaudrant traces, complete the following steps:

Enable request tracing in your Authorino custom resource (CR) and send authentication and authorization traces to the central collector as follows:
Example Authorino CR with request tracing
```
apiVersion: operator.authorino.kuadrant.io/v1beta1
kind: Authorino
metadata:
  name: authorino
spec:
  tracing:
    endpoint: rpc://authorino-collector:4317
    insecure: true
# ...
```
Set insecure to true to skip TLS certificate verification in development environments. Set to false for production environments.
Apply the configuration by running the following command:
```
$ oc apply -f authorino.yaml
```
Enable request tracing in your Limitador CR and send rate limit traces to the central collector as follows:
Example Limitador CR with request tracing
```
apiVersion: limitador.kuadrant.io/v1alpha1
kind: Limitador
metadata:
  name: limitador
spec:
  tracing:
    endpoint: rpc://limitador-collector:4317
    insecure: true
# ...
```
Set insecure to true to skip TLS certificate verification in development environments. Set to false for production environments.

Apply the configuration by running the following command:

$ oc apply -f limitador.yaml

Important

Trace IDs do not propagate to WebAssembly modules in OpenShift Service Mesh. This means that requests passed to Limitador do not have the relevant parent trace ID. However, if the trace initiation point is outside OpenShift Service Mesh, the parent trace ID is available to Limitador and included in traces. This impacts correlating traces from Limitador with traces from Authorino, the gateway, and any other components in the request path.

Configure data-plane tracing in the Kuadrant CR by providing the collector endpoint as shown in the following example:

Example kuadrant CR

apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
  namespace: kuadrant-system
spec:
  observability:
    dataPlane:
      defaultLevels:
        - debug: "true"
      httpHeaderIdentifier: x-request-id
    tracing:
      defaultEndpoint: rpc://tempo.tempo.svc.cluster.local:4317
      insecure: true

spec.observability.dataPlane.defaultLevels: Set this value to enable trace filtering at the required level, for example debug: "true" for debug-level trace filtering.
spec.observability.tracing.defaultEndpoint: The URL of the tracing collector backend, such as the OTLP endpoint. Use rpc://, gRPC OTLP, port 4317, for full compatibility across all components.

spec.observability.tracing.insecure: Set to true to skip TLS certificate verification in development environments. Set to false for production environments.

Important

Point to the collector service, such as the Distributed Tracing Platform, not the query service. The collector receives traces from your applications. The query service is only for viewing traces in the GUI.

Apply the configuration by running the following command:
```
$ oc apply -f kuadrant.yaml
```

Verification

Verify that the CR applied successfully by listing the objects of that Kind by running the following command:
```
$ oc get kuadrant
```

Data-plane tracing configuration fields

You can enable data-plane tracing in OpenShift Service Mesh by configuring tracing with the Kuadrant custom resource (CR).

The Kuadrant CR directs traces to a central collector for improved observability and troubleshooting. To view traces, first configure defaultLevels in the Kuadrant CR, then view traces in your tracing GUI, such as Grafana.

Data-Plane Configuration Fields

defaultLevels: Controls the OpenTelemetry trace filtering level for the wasm-shim module. This determines which trace spans are exported to your tracing collector.
- Supported levels from the highest to lowest verbosity: DEBUG, INFO, WARN, ERROR. Default value is WARN.
- Priority means that the highest level set wins: DEBUG > INFO > WARN > ERROR.
httpHeaderIdentifier: Specifies the HTTP header name used to correlate requests in traces, such as x-request-id.

Important

The defaultLevels configuration controls trace span filtering sent to your observability backend, such as Distributed Tracing Platform, not the verbosity of logs appearing in gateway pod output.

To see debug logs in gateway pods, you must configure the Envoy log level separately.