Red Hat Connectivity Link

Troubleshoot by using traces and logs

You can use a tracing user interface such Grafana to search for OpenShift Service Mesh and Connectivity Link trace information by trace ID. You can get the trace ID from logs, or from a header in a sample request that you want to troubleshoot.

You can also search for recent traces, filtering by the service that you want to focus on.

If you centrally aggregate logs by using tools such as Grafana Loki and Promtail, you can jump between trace information and the relevant logs for that service.

By using a combination of tracing and logs, you can visualize and troubleshoot request timing issues and narrow down to specific services. This method gives you even more insight a more complete picture of your user traffic when you combine it with Connectivity Link metrics and dashboards.

View rate-limit logging with trace IDs

You can enable request logging with trace IDs to get more information about requests when you use the Limitador component of Connectivity Link for rate limiting. To do this, you must increase the log level.

Prerequisites

You installed Connectivity Link.
You have administrator access to a running OpenShift Container Platform cluster.
You configured Grafana dashboards.
You have Red Hat OpenShift Distributed Tracing Platform installed and configured to support OpenTelemetry.

Procedure

Set the verbosity to 3 or higher in your Limitador custom resource (CR) as follows:

Example Limitador CR

apiVersion: limitador.kuadrant.io/v1alpha1
kind: Limitador
metadata:
  name: limitador
spec:
  verbosity: 3

Example log entry with the traceparent field holding the trace ID

"Request received: Request { metadata: MetadataMap { headers: {"te": "trailers", "grpc-timeout": "5000m", "content-type": "application/grpc", "traceparent": "00-4a2a933a23df267aed612f4694b32141-00f067aa0ba902b7-01", "x-envoy-internal": "true", "x-envoy-expected-rq-timeout-ms": "5000"} }, message: RateLimitRequest { domain: "default/toystore", descriptors: [RateLimitDescriptor { entries: [Entry { key: "limit.general_user__f5646550", value: "1" }, Entry { key: "metadata.filter_metadata.envoy\\.filters\\.http\\.ext_authz.identity.userid", value: "alice" }], limit: None }], hits_addend: 1 }, extensions: Extensions }"

Configure access logs

You can configure Envoy access logs in OpenShift Service Mesh so that you can to track a single request across multiple services and components by using a unique identifier.

Prerequisites

You installed Connectivity Link.
You have a running OpenShift Container Platform cluster.
You have administrator access to the OpenShift Container Platform cluster.

Procedure

Enable mesh-wide, default-format access logs by using the Istio Telemetry API. Use the following example as a starting point:
Example Telemetry API config
```
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: access-logs
  namespace: gateway-system
spec:
  accessLogging:
    - providers:
      - name: envoy
```
You might also use the istio-system as your namespace, depending on your setup.

For better parsing and integration with log aggregation systems, enable JSON-formatted access logs. Only log errors as shown in the following example:

Example JSON config

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: access-logs-json
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy
    filter:
      expression: "response.code >= 400"

To enable logging for a specific workload and add filtering, use the following example:

Example JSON workload config

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: selective-logging
  namespace: my-app-ns
spec:
  selector:
    matchLabels:
      app: productpage
  accessLogging:
    - providers:
        - name: access-logs-json
      filter:
        expression: "response.code >= 400"

Tip	The `expression` field uses Common Expression Language (CEL). You can use CEL-based filters to avoid excessive and meaningless logs.

If you are using the Sail Operator, check which Istio Operator is active in your cluster by running the following command:
```
$ oc get istio -A
```
The expected output is a list of your mesh deployments, such as default, prod-mesh and their current status.

Configure the Istio mesh with a custom access log provider to enable JSON encoding:

apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
spec:
  namespace: istio-system
  values:
    meshConfig:
      accessLogFile: /dev/stdout
      accessLogEncoding: JSON
      accessLogFormat: |
        {
          "start_time": "%START_TIME%",
          "method": "%REQ(:METHOD)%",
          "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
          "protocol": "%PROTOCOL%",
          "response_code": "%RESPONSE_CODE%",
          "response_flags": "%RESPONSE_FLAGS%",
          "bytes_received": "%BYTES_RECEIVED%",
          "bytes_sent": "%BYTES_SENT%",
          "duration": "%DURATION%",
          "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
          "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%",
          "user_agent": "%REQ(USER-AGENT)%",
          "request_id": "%REQ(X-REQUEST-ID)%",
          "authority": "%REQ(:AUTHORITY)%",
          "upstream_host": "%UPSTREAM_HOST%",
          "upstream_cluster": "%UPSTREAM_CLUSTER%",
          "route_name": "%ROUTE_NAME%"
        }

Next steps

Filter your access logs to focus on the errors you need to see.
Enable request, log, and tracing correlation.

Filter access logs

You can filter your access logs to reduce extra messages and focus on the issues and errors that are relevant to your use case.

Prerequisites

You installed Connectivity Link.
You have a running OpenShift Container Platform cluster.
You have administrator access to the OpenShift Container Platform cluster.
You enabled access logs.

Procedure

Configure your Telemetry custom resource (CR) to only log errors by using the following example:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: access-logs-errors-only
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy
      filter:
        expression: "response.code >= 400"

Configure your Telemetry custom resource (CR) to only log specific routes by using the following example:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: access-logs-api-only
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy
      filter:
        expression: 'request.url_path.startsWith("/api/")'

Configure your Telemetry custom resource (CR) to exclude health checks by using the following example:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: access-logs-no-healthz
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy
      filter:
        expression: '!request.url_path.startsWith("/healthz")'

Common access log format variables

You can quickly set up Envoy logs by using the most common format variables so that you get exactly the data you want.

Example configuration snippet

envoyFileAccessLog:
  path: /dev/stdout
  logFormat:
    text: "[%START_TIME%] %REQ(X-REQUEST-ID)% %RESP(HEADER)% %RESPONSE_FLAGS%\n"

Table 10. Common Envoy access log format variables
Variable	Description
`%START_TIME%`	Request start time
`%REQ(HEADER)%`	Request header value, such as `%REQ(X-REQUEST-ID)%`
`%RESP(HEADER)%`	Response header value
`%PROTOCOL%`	Protocol, such as HTTP/1.1, HTTP/2
`%RESPONSE_CODE%`	HTTP response code
`%RESPONSE_FLAGS%`	Response flags indicating issues, such as UH, UF
`%BYTES_RECEIVED%`	Bytes received from client
`%BYTES_SENT%`	Bytes sent to client
`%DURATION%`	Total request duration in milliseconds
`%UPSTREAM_HOST%`	Upstream host address
`%UPSTREAM_CLUSTER%`	Upstream cluster name
`%ROUTE_NAME%`	Route name that matched