OpenAI

Configure OpenAI as an LLM provider in agentgateway.

Before you begin

Set up an agentgateway proxy.

Set up access to OpenAI

1. Create an API key to access the OpenAI API. If you use another AI provider, create an API key for that provider’s AI instead, and be sure to modify the example commands in these tutorials to use your provider’s AI API instead.

2. Save the API key in an environment variable.

   export OPENAI_API_KEY=<insert your API key>

3. Create a Kubernetes secret to store your AI API key.

   kubectl apply -f- <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: openai-secret
     namespace: kgateway-system
   type: Opaque
   stringData:
     Authorization: $OPENAI_API_KEY
   EOF

4. Create an AgentgatewayBackend resource to configure an LLM provider that references the AI API key secret.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayBackend
   metadata:
     name: openai
     namespace: kgateway-system
   spec:
     ai:
       provider:
         openai:
           model: gpt-3.5-turbo  # Optional: specify default model
        # host: api.openai.com  # Optional: custom host if needed
        # port: 443  # Optional: custom port
     policies:
       auth:
         secretRef:
           name: openai-secret
   EOF

   Review the following table to understand this configuration. For more information, see the API reference.

   Setting	Description
   ai.provider.openai	Define the OpenAI provider.
   openai.model	The OpenAI model to use, such as gpt-3.5-turbo.
   policies.auth	Configure the authentication token for OpenAI API. The example refers to the secret that you previously created.

5. Create an HTTPRoute resource that routes incoming traffic to the AgentgatewayBackend. The following example sets up a route on the /openai path to the AgentgatewayBackend that you previously created. The URLRewrite filter rewrites the path from /openai to the path of the API in the LLM provider that you want to use, /v1/chat/completions.

   kubectl apply -f- <<EOF
   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: openai
     namespace: kgateway-system
   spec:
     parentRefs:
       - name: agentgateway
         namespace: kgateway-system
     rules:
     - matches:
       - path:
           type: PathPrefix
           value: /openai
       backendRefs:
       - name: openai
         namespace: kgateway-system
         group: agentgateway.dev
         kind: AgentgatewayBackend
   EOF

6. Send a request to the LLM provider API. Verify that the request succeeds and that you get back a response from the chat completion API.

   Cloud Provider LoadBalancerPort-forward for local testing

   curl "$INGRESS_GW_ADDRESS/openai" -H content-type:application/json  -d '{
      "model": "",
      "messages": [
        {
          "role": "system",
          "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
        },
        {
          "role": "user",
          "content": "Compose a poem that explains the concept of recursion in programming."
        }
      ]
    }' | jq

   curl "localhost:8080/openai" -H content-type:application/json  -d '{
      "model": "gpt-3.5-turbo",
      "messages": [
        {
          "role": "system",
          "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
        },
        {
          "role": "user",
          "content": "Compose a poem that explains the concept of recursion in programming."
        }
      ]
    }' | jq

   Example output:

   {
     "id": "chatcmpl-AEHYs2B0XUlEioCduH1meERmMwBGF",
     "object": "chat.completion",
     "created": 1727967462,
     "model": "gpt-3.5-turbo-0125",
     "choices": [
       {
         "index": 0,
         "message": {
           "role": "assistant",
           "content": "In the world of code, a method elegant and rare,\nKnown as recursion, a loop beyond compare.\nLike a mirror reflecting its own reflection,\nIt calls upon itself with deep introspection.\n\nA function that calls itself with artful grace,\nDividing a problem into a smaller space.\nLike a nesting doll, layers deep and profound,\nIt solves complex tasks, looping around.\n\nWith each recursive call, a step is taken,\nTowards solving the problem, not forsaken.\nA dance of self-replication, a mesmerizing sight,\nUnraveling complexity with power and might.\n\nBut beware of infinite loops, a perilous dance,\nWithout a base case, it's a risky chance.\nFor recursion is a waltz with a delicate balance,\nInfinite beauty, yet a risky dalliance.\n\nSo embrace the concept, in programming's domain,\nLet recursion guide you, like a poetic refrain.\nA magical loop, a recursive song,\nIn the symphony of code, where brilliance belongs.",
           "refusal": null
         },
         "logprobs": null,
         "finish_reason": "stop"
       }
     ],
     "usage": {
       "prompt_tokens": 39,
       "completion_tokens": 200,
       "total_tokens": 239,
       "prompt_tokens_details": {
         "cached_tokens": 0
       },
       "completion_tokens_details": {
         "reasoning_tokens": 0
       }
     },
     "system_fingerprint": null
   }

Next steps

• Want to use other endpoints than chat completions, such as embeddings or models? Check out the multiple endpoints guide.

• Explore other guides for LLM consumption, such as function calling, model failover, and prompt guards.