Anthropic

Configure Anthropic (Claude) as an LLM provider in agentgateway.

Before you begin

Set up access to Anthropic

Get an API key to access the Anthropic API.

Save the API key in an environment variable.

export ANTHROPIC_API_KEY=<insert your API key>

Create a Kubernetes secret to store your Anthropic API key.

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: anthropic-secret
  namespace: kgateway-system
type: Opaque
stringData:
  Authorization: $ANTHROPIC_API_KEY
EOF

Create a resource to configure an LLM provider that references the Anthropic API key secret.

kubectl apply -f- <<EOF
apiVersion: gateway.kgateway.dev/v1alpha1
kind: 
metadata:
  name: anthropic
  namespace: kgateway-system
spec:
  type: AI
  ai:
    llm:
      anthropic:
        authToken:
          kind: SecretRef
          secretRef:
            name: anthropic-secret
        model: "claude-3-opus-20240229"
        apiVersion: "2023-06-01"
EOF

Review the following table to understand this configuration. For more information, see the API reference.

Setting	Description
`type`	Set to `AI` to configure this for an AI provider.
`ai`	Define the AI backend configuration. The example uses Anthropic (`spec.ai.llm.anthropic`).
`authToken`	Configure the authentication token for Anthropic API. The example refers to the secret that you previously created. The token is automatically sent in the `x-api-key` header.
`model`	Optional: Override the model name, such as `claude-3-opus-20240229`. If unset, the model name is taken from the request.
`apiVersion`	Optional: A version header to pass to the Anthropic API. For more information, see the Anthropic API versioning docs.

Create an HTTPRoute resource that routes incoming traffic to the . The following example sets up a route on the /anthropic path. Note that kgateway automatically rewrites the endpoint to the Anthropic /v1/messages endpoint.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: anthropic
  namespace: kgateway-system
spec:
  parentRefs:
    - name: agentgateway
      namespace: kgateway-system
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /anthropic
    backendRefs:
    - name: anthropic
      namespace: kgateway-system
      group: gateway.kgateway.dev
      kind: 
EOF

Send a request to the LLM provider API. Note that Anthropic uses the /v1/messages endpoint format instead of /v1/chat/completions. Verify that the request succeeds and that you get back a response from the API.

curl "$INGRESS_GW_ADDRESS/anthropic" -H content-type:application/json  -d '{
   "model": "",
   "messages": [
     {
       "role": "user",
       "content": "Explain how AI works in simple terms."
     }
   ]
 }' | jq

curl "localhost:8080/anthropic" -H content-type:application/json  -d '{
   "model": "",
   "messages": [
     {
       "role": "user",
       "content": "Explain how AI works in simple terms."
     }
   ]
 }' | jq

Example output:

{
  "model": "claude-3-opus-20240229",
  "usage": {
    "prompt_tokens": 16,
    "completion_tokens": 318,
    "total_tokens": 334
  },
  "choices": [
    {
      "message": {
        "content": "Artificial Intelligence (AI) is a field of computer science that focuses on creating machines that can perform tasks that typically require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. Here's a simple explanation of how AI works:\n\n1. Data input: AI systems require data to learn and make decisions. This data can be in the form of images, text, numbers, or any other format.\n\n2. Training: The AI system is trained using this data. During training, the system learns to recognize patterns, relationships, and make predictions based on the input data.\n\n3. Algorithms: AI uses various algorithms, which are sets of instructions or rules, to process and analyze the data. These algorithms can be simple or complex, depending on the task at hand.\n\n4. Machine Learning: A subset of AI, machine learning, enables the system to automatically learn and improve from experience without being explicitly programmed. As the AI system is exposed to more data, it can refine its algorithms and become more accurate over time.\n\n5. Output: Once the AI system has processed the data, it generates an output. This output can be a prediction, a decision, or an action, depending on the purpose of the AI system.\n\nAI can be categorized into narrow (weak) AI and general (strong) AI. Narrow AI is designed to perform a specific task, such as playing chess or recognizing speech, while general AI aims to have human-like intelligence that can perform any intellectual task.",
        "role": "assistant"
      },
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "id": "msg_01PbaJfDHnjEBG4BueJNR2ff",
  "created": 1764627002,
  "object": "chat.completion"
}

Next steps

Explore other guides for LLM consumption, such as function calling, model failover, and prompt guards.

Amazon Bedrock Azure OpenAI