Red Hat Connectivity Link

Perform two-layer validation with the X.509 format

You can use the X.509 format for your certificates in many scenarios, including high-security environments.

See the following examples:

Zero-trust architecture: You need cryptographic proof of identity for every request.
Compliance requirements: Security standards, such as PCI DSS, HIPAA, and FedRAMP, mandate mutual TLS (mTLS).
Machine-to-machine communication: Services authenticate with long-lived certificates from your Public Key Infrastructure (PKI). If you are using mTLS, you are using PKI. PKI issues, distributes, and renews your root certificate authorities (CAs).
Multi-CA trust: You have different clients that are issued certificates by different CAs and you need fine-grained control over which CAs are trusted for which routes.
No shared secrets: You want to avoid rotating and distributing API keys or tokens.

When you use X.509 authentication in Connectivity Link, you implement the following two-step validation model:

Step 1. Cryptographic proof:
- During the initial TLS handshake, the client presents a certificate that the Gateway object validates against your trusted CA certificates. This cryptographically verifies that the client is who they say they are because they have the private key and your certificate is trusted.
- Invalid, expired, or untrusted certificates are rejected at the TLS layer before reaching the application.
- The Gateway object sets the x-forwarded-client-cert (XFCC) header with the certificate details so that incoming XFCC headers from clients are sanitized.
Step 2. Context-aware validation:
- Authorino validates certificates extracted from the XFCC header. This means that you can apply fine-grained rules and manage trust across multiple CAs by using simple label selectors.
- Certificate attributes extract identity details directly from client certificates. You can use these details to create precise access rules, or pass them along to your backend services as custom headers, ensuring that your applications always know exactly who is calling them.

Both layers must succeed for a request to authenticate.

Prepare certificate authorities and client certificates

To set up authentication for clients with provider-specific TLS validation, you must first have certificate authorities (CAs) and client certificates.

You can use the following procedure to manually generate resources, or you can provision certificates by using an existing certificate manager or other automated process. Whichever method you select, you must complete the following tasks to use X.509 validation with Connectivity Link:

Include the extendedKeyUsage=clientAuth parameter in your client certificates.
Mount the root CA certificate into the Gateway object pods. This enables network-level verification. The Gateway object requires the CA bundle to allow the TLS connection to establish.
Store the CA used to sign the client certificate in a Secret and label the Secret for discovery by Authorino. This enables policy-level verification. Authorino requires the CA bundle to perform deep packet inspection of the identity. Then Authorino can grant or deny the specific client’s access a specific API.

Prerequisites

You are logged into OpenShift Container Platform as a cluster administrator.
You installed Connectivity Link.

Procedure

On your local host, generate your certificate authority (CA) by running the following command:

$ openssl req -x509 -sha512 -nodes \
  -days 365 \
  -newkey rsa:4096 \
  -subj "/CN=Test CA/O=Example Org/C=US" \
  -addext basicConstraints=CA:TRUE \
  -addext keyUsage=digitalSignature,keyCertSign \
  -keyout /tmp/ca.key \
  -out /tmp/ca.crt

On your local host, create X.509 v3 extensions for your client certificate by running the following command:

$ cat > /tmp/x509v3.ext << EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage=digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
extendedKeyUsage=clientAuth
EOF

Generate the client certificate on your local host by running the following commands:
1. Generate your Private Key. This is the secret file that stays on the client. Run the following command:
  $ openssl genrsa -out /tmp/client.key 4096
2. Create a Certificate Signing Request (CSR). The following command uses your private key and identity information to create a request for a certificate:
  $ openssl req -new \ -subj "/CN=test-client/O=Example Org/C=US" \ -key /tmp/client.key \ -out /tmp/client.csr
3. Sign the certificate. This command takes your request, looks at your internal CA, and issues the final Public Certificate, the client.crt:
  $ openssl x509 -req -sha512 \ -days 365 \ -CA /tmp/ca.crt \ -CAkey /tmp/ca.key \ -CAcreateserial \ -extfile /tmp/x509v3.ext \ -in /tmp/client.csr \ -out /tmp/client.crt
If you do not have a CA that you use to issue and sign client certificates, create the self-signed CA certificate in your cluster by running the following commands:
1. Create the ConfigMap for Gateway TLS validation by running the following command:
  $ oc create configmap gateway-client-ca \ -n <gateway_system> \ --from-file=ca.crt=/tmp/ca.crt
  - Replace <gateway_system> with the namespace where your Gateway object is applied.
  - The ConfigMap must be in the same namespace as the Gateway object.
2. Create the Authorino validation Secret by running the following command:
  $ oc create secret generic authorino-ca-validation \ -n kuadrant-system \ --from-file=ca.crt=/tmp/ca.crt
  The Secret must be in the same namespace as Authorino.
3. Label the Secret so that Authorino can find its trusted anchors by running the following command:
  $ oc label secret authorino-ca-validation \ -n kuadrant-system \ authorino.kuadrant.io/managed-by=authorino \ app.kubernetes.io/name=trusted-client

Create a Gateway object for use with the X.509 format

To use the X.509 format for certificates, after you have your certificate authorities (CAs) and client certificates set up, you must create a Gateway object without front-end TLS validation and associated required resources.

Not configuring front-end TLS in your Gateway passes authentication to your two-step X.509-based validation.

You create the following custom resources by using the following procedure:

A Gateway object configured for use with the X.509 certificate format.
A cert-manager Issuer: Self-signed certificate issuer for the Gateway object’s server certificate.
A TLSPolicy CR: Kuadrant policy to manage the Gateway object’s server TLS certificate.

Prerequisites

You are logged into OpenShift Container Platform as a cluster administrator.
You installed Connectivity Link using Istio as your Gateway object controller.
You created your CAs and X.509-formatted certificates.

Procedure

Create the cert-manager Issuer custom resource (CR) by using the following example:
```
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: gateway-cert-issuer
  namespace: <gateway_system>
spec:
  selfSigned: {}
```
- Replace <gateway_system> with the namespace of your Gateway object.
Apply the cert-manager Issuer object by running the following command:
```
$ oc apply - f gateway-cert-issuer.yaml
```

Create the TLSPolicy object by using the following example:

apiVersion: kuadrant.io/v1
kind: TLSPolicy
metadata:
  name: mtls-gateway-tls
  namespace: <gateway_system>
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: mtls-gateway
    sectionName: https
  issuerRef:
    group: cert-manager.io
    kind: Issuer
    name: gateway-cert-issuer

Replace the value of the metadata.namespace parameter with the namespace where your Gateway object is applied.

Apply the TLSPolicy object by running the following command:
```
$ oc apply -f mtls-gateway-tls.yaml
```

On your local host, create an infrastructure customization ConfigMap CR to inject the client CA volume mounts into the managed Istio proxy pods by using the following example:

Example ConfigMap CR for use with an X.509 Gateway object

apiVersion: v1
kind: ConfigMap
metadata:
  name: mtls-gateway-config
  namespace: <gateway_system>
data:
  deployment: |
    spec:
      template:
        spec:
          volumes:
          - name: client-ca
            configMap:
              name: gateway-client-ca
          containers:
          - name: istio-proxy
            volumeMounts:
            - name: client-ca
              mountPath: /etc/certs
              readOnly: true

Replace the value of the metadata.namespace parameter with the namespace where your Gateway object is applied.

Apply the infrastructure customization ConfigMap CR by running the following command:
```
$ oc apply -f mtls-gateway-config.yaml
```

Create the mTLS Gateway object by using the following example:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mtls-gateway
  namespace: <gateway_system>
spec:
  gatewayClassName: openshift-default
  infrastructure:
    parametersRef:
      group: ""
      kind: ConfigMap
      name: mtls-gateway-config
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    hostname: "<gateway_hostname>"
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - name: <gateway_name>-<listener_name>-tls
        kind: Secret

Replace the value of the metadata.namespace parameter with the namespace where your Gateway object is applied.
Replace the value of the spec.listeners.hostname parameter with the name of your Gateway object hostname.
Replace spec.listeners.tls.certificateRefs.name with the name of your Gateway object and the name of your mTLS listener in the <gateway_name>-<listener_name>-tls format.

Apply the mTLS Gateway CR by running the following command:
```
$ oc apply -f mtls-gateway.yaml
```

Create an EnvoyFilter CR to configure mutual TLS validation by using the following example:

Example mutual TLS EnvoyFilter CR

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: mtls-validation
  namespace: <gateway_system>
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: mtls-gateway
  configPatches:
  - applyTo: FILTER_CHAIN
    match:
      context: GATEWAY
      listener:
        portNumber: 443
    patch:
      operation: MERGE
      value:
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
            commonTlsContext:
              validationContext:
                trustedCa:
                  filename: /etc/certs/ca.crt
            requireClientCertificate: true

Replace the value of the metadata.namespace parameter with the namespace where your Gateway object is applied.

Apply the EnvoyFilter CR by running the following command:
```
$ oc apply -f mtls-envoyfilter.yaml
```

Install a sample application for X.509 testing

You can test your end-to-end flow of authenticating requests by X.509 client certificates by deploying a sample application before you roll out X.509 to a production environment.

Prerequisites

You are logged into OpenShift Container Platform as a cluster administrator.
You installed Connectivity Link.
You created your CAs and X.509-formatted certificates.
You created Gateway, TLSPolicy, and cert-manager Issuer objects.

Procedure

Deploy the sample application by using the following resources:

Example httpbin application

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: httpbin
        image: quay.io/kuadrant/httpbin@sha256:2865a9ac6596340135fc3cd82cbc543c4050124afb10361e36f727abe4f71900
        imagePullPolicy: IfNotPresent
        securityContext:
          runAsUser: 1000
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - ALL
        ports:
        - containerPort: 80
        volumeMounts:
        - name: tmp
          mountPath: /tmp
      volumes:
      - name: tmp
        emptyDir: {}

Apply the application objects by running the following command:
```
$ oc apply -f httpbin.yaml
```

Create an HTTPRoute custom resource for X.509 certificates

To use the X.509 format for certificate, you must create an HTTPRoute custom resource (CR) that references your Gateway object.

Note	If you require a `GRPCRoute` CR for your use-case, see "Creating a GRPCRoute custom resource" for an example object and current support status.

Prerequisites

You are logged into OpenShift Container Platform as a cluster administrator.
You installed Connectivity Link using Istio as your Gateway object controller.
You created your CAs and X.509-formatted certificates.
You created your Gateway object.

Procedure

Create your HTTPRoute CR by using the following example:
```
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: <httpbin_route>
  namespace: <default>
  labels:
    app: <httpbin>
spec:
  parentRefs:
  - name: mtls-gateway
    namespace: <gateway_system>
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      port: 80
```
- Replace the metadata.name parameter value with the name of your application route.
- Replace the metadata.namespace parameter with the namespace where your Gateway object is applied.
- Replace the metadata.labels.app value with the name of the application.
- Replace the spec.parentRefs.namespace value with the namespace where your Gateway object is applied.
Apply your HTTPRoute custom resource by running the following command:
```
$ oc apply -f <app_http_route.yaml>
```
Replace <app_http_route.yaml> with the name of your HTTPRoute YAML.

Create an AuthPolicy custom resource for X.509 certificates

To use the X.509 format for certificate, you must create an AuthPolicy custom resource (CR) that completes critical functions of your two-step validation.

When you use the following procedure, you create an AuthPolicy CR that implements the following steps:

Extracts the certificate from the x-forwarded-client-cert header.
Validates the certificate chain against CA certificates labeled app.kubernetes.io/name: trusted-client.
Enforces authorization based on a certificate organization attribute.
Injects certificate attributes into request headers for fine-grained access control.

Prerequisites

You are logged into OpenShift Container Platform as a cluster administrator.
You installed Connectivity Link using Istio as your Gateway object controller.
You created your CAs and X.509-formatted certificates.
You created your Gateway object.
You created your route.

Procedure

Create your X.509 AuthPolicy CR by using the following example:

Example X.509 AuthPolicy CR

apiVersion: kuadrant.io/v1
kind: AuthPolicy
metadata:
  name: x509-auth-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: httpbin-route
  rules:
    authentication:
      "x509-client-cert":
        x509:
          source:
            xfccHeader: "x-forwarded-client-cert"
          selector:
            matchLabels:
              app.kubernetes.io/name: trusted-client
    authorization:
      "certificate-attributes":
        patternMatching:
          patterns:
          - predicate: "size(auth.identity.Organization) > 0 && auth.identity.Organization[0] == 'Example Org'"
    response:
      success:
        headers:
          "x-client-common-name":
            plain:
              expression: auth.identity.CommonName
          "x-client-org":
            plain:
              expression: auth.identity.Organization[0]

The spec.rules.authentication."x509-client-cert".x509 parameter causes the policy to extract the certificate from the X-Forwarded-Client-Cert standard (XFCC) header. The Gateway object populates this header after validating the client certificate.
The spec.rules.authentication."x509-client-cert".x509.source.xfccHeader parameter value is the name of the XFCC header. This is the default header name.
Set the spec.rules.response.success.headers."x-client-common-name".plain.expression parameter value to ensure that the policy extracts the common name (CN) from the certificate.
Set the spec.rules.response.success.headers"x-client-org".plain.expression parameter value to ensure that the policy extracts the organization (O) from the certificate.

Apply your AuthPolicy custom resource by running the following command:
```
$ oc apply -f <app_authpolicy.yaml>
```
Replace <app_authpolicy.yaml> with the name of your AuthPolicy YAML.

Verify X.509 layer 1 authentication

To test your layer 1 X.509 authentication, you can run test scenarios to check that connections are rejected.

Prerequisites

You are logged into OpenShift Container Platform as a cluster administrator.
You installed Connectivity Link using Istio as your Gateway object controller.
You created your CAs and X.509-formatted certificates.
You created your Gateway object.
You created your route.
You installed the httpbin sample application.

Procedure

Retrieve the Gateway object hostname by running the following command:
```
$ export GATEWAY_HOSTNAME=$(oc get gateway mtls-gateway -n <gateway_system> -o jsonpath='{.status.addresses[0].value}')
```
Replace <gateway_system> with the namespace where your Gateway object is applied.

Test with a valid certificate by running the following command:

$ curl -ik https://httpbin.${GATEWAY_HOSTNAME}/get \
  --cert /tmp/client.crt \
  --key /tmp/client.key

Example output

HTTP 200

Verify that the TLS handshake fails when there is no certificate by running the following command:

$ curl -ik https://httpbin.${GATEWAY_HOSTNAME}/get

Example output

curl: (56) OpenSSL SSL_read: OpenSSL/3.6.2: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

Verify that the TLS handshake fails when there is an untrusted certificate by running the following commands:

Generate an untrusted certificate by running the following command:

$ openssl req -x509 -sha512 -nodes \
  -days 365 \
  -newkey rsa:4096 \
  -subj "/CN=Untrusted CA/O=Untrusted Org/C=US" \
  -keyout /tmp/untrusted-ca.key \
  -out /tmp/untrusted-ca.crt

Generate an untrusted private key by running the following command:
```
$ openssl genrsa -out /tmp/untrusted-client.key 4096
```

Create a Certificate Signing Request (CSR) for the untrusted client by running the following command:

$ openssl req -new \
  -subj "/CN=untrusted-client/O=Untrusted Org/C=US" \
  -key /tmp/untrusted-client.key \
  -out /tmp/untrusted-client.csr

Sign the request using your untrusted authority to issue the client certificate by running the following command:

$ openssl x509 -req -sha512 \
  -days 365 \
  -CA /tmp/untrusted-ca.crt \
  -CAkey /tmp/untrusted-ca.key \
  -CAcreateserial \
  -extfile /tmp/x509v3.ext \
  -in /tmp/untrusted-client.csr \
  -out /tmp/untrusted-client.crt

Test the untrusted certificate by running the following command:

$ curl -ik https://httpbin.${GATEWAY_HOSTNAME}/get \
  --cert /tmp/untrusted-client.crt \
  --key /tmp/untrusted-client.key

Example output

curl: (56) OpenSSL SSL_read: OpenSSL/3.6.2: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0

Verify X.509 layer 2 authentication

To test that your X.509 authentication is working as intended, you can run a test scenario and check that your layer 2 policies reject connections. In this test, the client uses a certificate signed by your trusted certificate authority (CA), so the network handshake succeeds.

However, because the client certificate has the wrong attributes, it fails at the policy layer through the Authorino Operator.

With the final command of this test, you take the following actions:

The handshake, layer 1: Use curl to present the certificate. Envoy checks its mounted /etc/certs directory, sees that this certificate was signed by your legitimate ca.crt, and allows the TLS connection to establish.
The policy evaluation, layer 2 processing: Envoy routes the HTTP request metadata and the client certificate details to the Authorino Operator.
The rejection, layer 2 failure: The Authorino Operator evaluates the request against your configured AuthPolicy custom resource (CR), which does not allow O=Unauthorized Org and the authentication fails.

Prerequisites

You are logged into OpenShift Container Platform as a cluster administrator.
You installed Connectivity Link using Istio as your Gateway object controller.
You created your CAs and X.509-formatted certificates.
You created your Gateway object.
You created your route.
You installed the httpbin sample application.

Procedure

Retrieve the gateway hostname by running the following command:
```
$ export GATEWAY_HOSTNAME=$(oc get gateway mtls-gateway -n <gateway_system> -o jsonpath='{.status.addresses[0].value}')
```
Replace <gateway_system> with the namespace where your Gateway object is applied.

Verify a connection failure when there is a valid certificate, but wrong attributes by running the following commands:

Generate a certificate with a different organization by running the following command:
```
$ openssl genrsa -out /tmp/unauthorized-client.key 4096
```

Create a Certificate Signing Request (CSR) with unauthorized attributes by running the following command:

$ openssl req -new \
  -subj "/CN=unauthorized-client/O=Unauthorized Org/C=US" \
  -key /tmp/unauthorized-client.key \
  -out /tmp/unauthorized-client.csr

Sign the request using your trusted certificate authority by running the following command:

$ openssl x509 -req -sha512 \
  -days 365 \
  -CA /tmp/ca.crt \
  -CAkey /tmp/ca.key \
  -CAcreateserial \
  -extfile /tmp/x509v3.ext \
  -in /tmp/unauthorized-client.csr \
  -out /tmp/unauthorized-client.crt

Test the unauthorized certificate by running the following command:

$ curl -ik https://httpbin.${GATEWAY_HOSTNAME}/get \
  --cert /tmp/unauthorized-client.crt \
  --key /tmp/unauthorized-client.key

Example output

HTTP 403

Troubleshoot X.509 authentication

If requests are either bypassing your X.509 authentication, or not accessing your backend as expected, you can diagnose the cause by taking steps that are based on the feedback being displayed.

Prerequisites

You are logged into OpenShift Container Platform as a cluster administrator.
You installed Connectivity Link.
You created your CAs and X.509-formatted certificates.
You created your Gateway object.
You created your route.

Procedure

If the TLS handshake succeeds without client certificate, the Envoy filter might not be properly applied. Troubleshoot this situation by taking the following steps:
1. Verify that the EnvoyFilter exists by running the following command:
  $ oc get envoyfilter mtls-validation -n <gateway_system>
  Replace <gateway_system> with the namespace where your Gateway object is applied.
2. Check that the targetRef in the Envoy filter points to the Gateway object by running the following command:
  $ oc get envoyfilter mtls-validation -n <gateway_system> -o yaml | grep -A 3 targetRefs
  Replace <gateway_system> with the namespace where your Gateway object is applied.
3. Check that the Envoy configuration has the correct configuration by running the following command:
  $ oc exec -n <gateway_system> \ $(oc get pod -n <gateway_system> -l gateway.networking.k8s.io/gateway-name=mtls-gateway -o jsonpath='{.items[0].metadata.name}') \ -- curl -s localhost:15000/config_dump?include_eds | grep -A 20 validation_context
  Replace <gateway_system> with the namespace where your Gateway object is applied.
If the Gateway object logs show failed to load trusted CA, the certificate authority (CA) might not be mounted. Use the following steps to check the CA:
1. Verify that the ConfigMap exists by running the following command:
  $ oc get configmap gateway-client-ca -n <gateway_system>
  Replace <gateway_system> with the namespace where your Gateway object is applied.
2. Check that the volume mount in the Gateway pod by running the following command:
  $ oc describe pod -n <gateway_system> -l gateway.networking.k8s.io/gateway-name=mtls-gateway | grep -A 5 Mounts
  Replace <gateway_system> with the namespace where your Gateway object is applied.
3. Verify that the CA file exists in the Gateway pod by running the following command:
  $ oc exec -n <gateway_system> \ $(oc get pod -n <gateway_system> -l gateway.networking.k8s.io/gateway-name=mtls-gateway -o jsonpath='{.items[0].metadata.name}') \ -- cat /etc/certs/ca.crt
  Replace <gateway_system> with the namespace where your Gateway object is applied.
If Authorino rejects requests with a certificate not found error, the XFCC header might not be set correctly. Check the configuration and logs to help resolve this issue as follows:
1. Verify the forward_client_cert_details configuration by running the following command:
  $ oc exec -n <gateway_system> \ $(oc get pod -n <gateway_system> -l gateway.networking.k8s.io/gateway-name=mtls-gateway -o jsonpath='{.items[0].metadata.name}') \ -- curl -s localhost:15000/config_dump?include_eds | grep -i forward_client_cert
  Replace <gateway_system> with the namespace where your Gateway object is applied.
2. Check the Envoy access logs for the XFCC header by running the following command:
  $ oc logs -n <gateway_system> -l gateway.networking.k8s.io/gateway-name=mtls-gateway | grep XFCC
  Replace <gateway_system> with the namespace where your Gateway object is applied.