Red Hat Connectivity Link

Token-based rate limiting with TokenRateLimitPolicy

As an application developer, you can use a TokenRateLimitPolicy custom resource (CR) to enforce rate limits based on token consumption rather than the number of requests.

This policy extends the Envoy Rate Limit Service (RLS) protocol with automatic token usage extraction. It is particularly useful for protecting Large Language Model (LLM) APIs, where the cost and resource usage correlate more closely with the number of tokens processed.

TokenRateLimitPolicy counts tokens by extracting usage metrics in the body of the artificial intelligence (AI) inference API call, allowing for finer-grained control over API usage based on actual workload.

How token rate limiting works

The TokenRateLimitPolicy object tracks cumulative token usage per client. Before forwarding a request, it checks whether the client has already exceeded their limit. After the upstream responds, the policy extracts the actual token cost and updates the client’s counter.

For example:

On an incoming request, the gateway evaluates the matching rules and predicates from the TokenRateLimitPolicy resources.
If the request matches, the gateway prepares the necessary rate limit descriptors and monitors the response.
After receiving the response, the gateway extracts the usage.total_tokens field from the JSON response body.
The gateway then sends a RateLimitRequest to the Limitador Operator (Limitador), including the actual token count as a hits_addend.
Limitador tracks the cumulative token usage and responds to the gateway with OK or OVER_LIMIT.

Key features and use cases

Enforces limits based on token usage by extracting the usage.total_tokens field from an OpenAI-style inference JSON response body.
Suitable for consumption-based APIs such as LLMs where the cost is based on token counts.
Allows defining different limits based on criteria such as user identity, API endpoints, or HTTP methods.
Works with AuthPolicy CRs to apply specific limits to authenticated users or groups.
Inherits functionalities from RateLimitPolicy CRs, including defining multiple limits with different durations and using Redis for shared counters in multicluster environments.

Integrating with AuthPolicy

You can combine TokenRateLimitPolicy CRs with AuthPolicy CRs to apply token limits based on authenticated user identity. When an AuthPolicy CR successfully authenticates a request, it injects identity information that is used by the TokenRateLimitPolicy CR to select the appropriate limit.

For example, you can define different token limits for users belonging to differently tiered groups, identified using claims in a JWT-validated by the AuthPolicy CR.

Configure token-based rate limiting for LLM APIs

You can protect Large Language Model (LLM) APIs deployed on OpenShift Container Platform by configuring a TokenRateLimitPolicy custom resource (CR) that you integrate with an AuthPolicy object for user-specific limits.

Prerequisites

You installed Connectivity Link on the OpenShift Container Platform cluster you are working with.
Gateway and HTTPRoute objects are both configured to expose your service.
You created and applied an AuthPolicy custom resource (CR) that overrides the default deny-all setting.
If you are running in a multicluster setup, or requiring persistent counters, Redis is configured for the Limitador Operator component.
Your configured your upstream service to return an OpenAI-compatible JSON response containing a usage.total_tokens field in the response body.

Procedure

Create a TokenRateLimitPolicy YAML, for example, tokenratelimitpolicy.yaml that includes the following information:

Example TokenRateLimitPolicy YAML

apiVersion: kuadrant.io/v1alpha1
kind: TokenRateLimitPolicy
metadata:
  name: llm-protection
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: ai-gateway
  limits:
    free-users:
      rates:
        - limit: 10000
          window: 24h
      when:
        - predicate: request.path == "/v1/chat/completions"
        - predicate: |
            auth.identity.groups.split(",").exists(g, g == "free")
      counters:
        - expression: auth.identity.userid
    pro-users:
      rates:
        - limit: 100000
          window: 24h
      when:
        - predicate: request.path == "/v1/chat/completions"
        - predicate: |
            auth.identity.groups.split(",").exists(g, g == "pro")
      counters:
        - expression: auth.identity.userid

Choose a filename for the policy that makes sense in your environment.
The spec.limits.free-users.rates.limit parameter is set for 10,000 tokens per day for free tier.
The spec.limits.free-users.when.predicate parameter is set to inference traffic only.
The spec.limits.pro-users.rates.limit parameter is set for 100,000 per day for the pro user.
The spec.limits.pro-users.when.predicate parameter is set to inference traffic only.
The predicates must match each other in each section.

Apply the TokenRateLimitPolicy policy by running the following command:
```
$ oc apply -f <tokenratelimitpolicy.yaml> -n <gateway_namespace>
```
- Replace <tokenratelimitpolicy.yaml> with the filename you used.
- Replace <gateway_namespace> with your gateway namespace.
Check the status of the policy to ensure it was accepted and enforced on the target Gateway object. Look for conditions with type: Accepted and type: Enforced with status: "True".
```
$ oc get <tokenratelimitpolicy.yaml> llm-protection -n <gateway_namespace> -o jsonpath='{.status.conditions}'
```
Replace <tokenratelimitpolicy.yaml> with the filename you used. Replace <gateway_namespace> with your Gateway object namespace.
Send requests to your API endpoint, including the required authentication details, by running the following command:
```
$ curl -H "Authorization: <auth_policy>" \
     -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}' \
     <api_endpoint>
```
Replace <auth_policy> and <api_endpoint> with your values.

Verification

Ensure that your upstream service responds with an OpenAI-compatible JSON body containing the usage.total_tokens field.
Requests made when the client is within their token limits should receive a 200 OK response or other success status and their token counter is updated.
Requests made when the client has already exceeded their token limits should receive a 429 Too Many Requests response.