Set up AI Gateway

Configure your Helm chart installation to use AI Gateway. Then, use a custom GatewayParameters resource to set up AI Gateway.

Before you begin

Get started to install the Kubernetes Gateway API CRDs and kgateway.

Enable the AI extension

Configure your kgateway Helm chart installation to use AI Gateway.

1. Upgrade kgateway with the AI Gateway extension enabled. Note: To use AI Gateway with agentgateway, you must also enable agentgateway.

   ⚠️

   If you use a different version or extra Helm settings such as in a -f values.yaml file, update the following command accordingly.
   KgatewayAgentgateway

   helm upgrade -i -n kgateway-system kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
        --set gateway.aiExtension.enabled=true \
        --version v$NEW_VERSION

   Agentgateway supports the routing rules and policies that are native to the Kubernetes Gateway API, such as basic matching, rewrites, retries, and timeouts. Note that agentgateway does not yet support kgateway TrafficPolicies. Any guide that relies on TrafficPolicies will not work with the agentgateway data plane. Instead, use the kgateway Envoy-based data plane. For AI gateway use cases, this limitation means the following:

   • ✅ AI routing with agentgateway to all supported cloud LLM providers, including those that are compatible with the OpenAI API.
   • ✅ Function calling.
   • ❌ Model failover.
   • ❌ Prompt guards and enrichment, which use TrafficPolicies.
   • ❌ AI observability metrics, which rely on Envoy dynamic metadata.

   helm upgrade -i -n kgateway-system kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
        --set gateway.aiExtension.enabled=true \
        --set agentGateway.enabled=true \
        --version v$NEW_VERSION

2. Verify that your Helm installation is updated.

   helm get values kgateway -n kgateway-system -o yaml

   Example output:

   KgatewayAgentgateway

   
   gateway:
     aiExtension:
       enabled: true

   
   agentgateway:
     enabled: true
   gateway:
     aiExtension:
       enabled: true

Create an AI Gateway

1. Create a GatewayParameters resource which enables an AI extension for the Gateway.

   Cloud ProviderLocal Environment

   For AI services in a cloud provider, use a LoadBalancer service. This way, the AI Gateway can be accessed from outside the cluster.

   kubectl apply -f- <<EOF
   apiVersion: gateway.kgateway.dev/v1alpha1
   kind: GatewayParameters
   metadata:
     name: ai-gateway
     namespace: kgateway-system
     labels:
       app: ai-gateway
   spec:
     kube:
       aiExtension:
         enabled: true
         ports:
         - name: ai-monitoring
           containerPort: 9092
         image:
           registry: cr.kgateway.dev/kgateway-dev
           repository: kgateway-ai-extension
           tag: v2.1.0-main
       service:
         type: LoadBalancer
   EOF

   For local environments such as Ollama, use a NodePort service. This way, the AI Gateway can be accessed locally.

   kubectl apply -f- <<EOF
   apiVersion: gateway.kgateway.dev/v1alpha1
   kind: GatewayParameters
   metadata:
     name: ai-gateway
     namespace: kgateway-system
     labels:
       app: ai-gateway
   spec:
     kube:
       aiExtension:
         enabled: true
         ports:
         - name: ai-monitoring
           containerPort: 9092
         image:
           registry: cr.kgateway.dev/kgateway-dev
           repository: kgateway-ai-extension
           tag: v2.1.0-main
       service:
         type: NodePort
   EOF

2. Create a Gateway that uses the default GatewayClass and the AI-enabled GatewayParameters resource you created in the previous step. To use the default Envoy-based kgateway proxy, set the gatewayClassName to kgateway. To use agentgateway, set the gatewayClassName to agentgateway.

   KgatewayAgentgateway

   kubectl apply -f- <<EOF
   kind: Gateway
   apiVersion: gateway.networking.k8s.io/v1
   metadata:
     name: ai-gateway
     namespace: kgateway-system
     labels:
       app: ai-gateway
   spec:
     gatewayClassName: kgateway
     infrastructure:
       parametersRef:
         name: ai-gateway
         group: gateway.kgateway.dev
         kind: GatewayParameters
     listeners:
     - protocol: HTTP
       port: 8080
       name: http
       allowedRoutes:
         namespaces:
           from: All
   EOF

   kubectl apply -f- <<EOF
   kind: Gateway
   apiVersion: gateway.networking.k8s.io/v1
   metadata:
     name: ai-gateway
     namespace: kgateway-system
     labels:
       app: ai-gateway
   spec:
     gatewayClassName: agentgateway
     infrastructure:
       parametersRef:
         name: ai-gateway
         group: gateway.kgateway.dev
         kind: GatewayParameters
     listeners:
     - protocol: HTTP
       port: 8080
       name: http
       allowedRoutes:
         namespaces:
           from: All
   EOF

3. Verify that the AI Gateway is created.

   • Gateway: Note that it might take a few minutes for an address to be assigned.
   • Deployment: The pod has two containers: kgateway-proxy and kgateway-ai-extension.

   kubectl get gateway,pods -l app.kubernetes.io/name=ai-gateway -A

   Example output:

   NAMESPACE         NAME                                           CLASS      ADDRESS       PROGRAMMED   AGE
   kgateway-system   gateway.gateway.networking.k8s.io/ai-gateway   kgateway   xx.xx.xx.xx   True         13s
   
   NAMESPACE         NAME                              READY   STATUS             RESTARTS   AGE
   kgateway-system   pod/ai-gateway-6f4786fcb6-gqhlm   2/2     Running   0          13s

   If you see an error, check the logs of the kgateway-ai-extension container.

   kubectl logs -l app.kubernetes.io/app=ai-gateway -n kgateway-system -c kgateway-ai-extension

Next

• For OpenAI: Continue with the Authenticate to the LLM guide.
• For other cloud LLM providers such as Gemini: Review the Cloud LLM providers guide for provider-specific setup examples.
• For local LLM providers such as Ollama: Set up Ollama as a local LLM provider.