For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Reduce duplicated authentication, TLS, and tunnel settings between an AI provider group and other backends that reach the same endpoint.

Reduce duplication between an AI provider group and other backends that connect to the same endpoint, without changing how the provider group load balances or fails over.

About sharing connection settings

When you load balance across multiple providers, each provider entry in a priority group is its own set of connection settings: hostname, port, authentication, TLS, and any proxy tunnel. If you also need to reach one of those same endpoints for a different purpose — for example, a dedicated route that scrapes a per-instance metrics endpoint — you end up creating a second AgentgatewayBackend with its own copy of those settings.

There is no way to reference an existing AgentgatewayBackend from inside an AI provider group. The custom provider type’s backendRef field targets only a Service or InferencePool, not another AgentgatewayBackend. This is intentional: a provider group is designed to describe each provider’s endpoint directly, not to compose other backend resources.

You can still avoid duplicating the authentication, TLS, and tunnel settings. Each provider inside an AI provider group has a name, and policies can target that specific provider by name using sectionName. An AgentgatewayPolicy with a backend section can list multiple targetRefs at once, so a single policy can attach to one provider inside the aggregate AgentgatewayBackend and to a separate AgentgatewayBackend that reaches the same endpoint. Both then share the same auth, tls, and tunnel configuration from one place.

Sharing settings between an AI provider group and a backend does not change the AI provider group itself. In the following example, the llm-providers backend still has the same three providers in the same priority group. The priority group behaves the same way whether or not the providers share connection settings.

For example, load balancing within the group still uses the Power of Two Choices algorithm. Failover, retries, and eviction still follow the priority group structure, not how connection settings are attached. Each provider keeps its own name, model, and API paths, so telemetry still attributes requests to the correct provider. The dedicated backend and route for the metrics endpoint keep scraping that instance independently, because only its connection settings moved into the shared policy.

Before you begin

1. Set up an agentgateway proxy.
2. Set up API access to each LLM provider that you want to use.
3. Understand how load balancing across multiple providers works, since this guide builds on that setup.

Share settings between a provider group and a dedicated backend

The following steps set up an AI provider group with an on-premises instance and two individually addressable cloud instances, plus a separate AgentgatewayBackend that reaches one of the cloud instances for its own dedicated route. A shared AgentgatewayPolicy keeps the authentication, TLS, and tunnel settings for that cloud instance in one place.

1. Create the aggregate AgentgatewayBackend with an AI provider group. Each cloud provider uses the custom provider type so you can declare explicit API paths. Leave auth, tls, and tunnel off of the providers that you plan to share settings for — the policy in step 3 supplies them.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayBackend
   metadata:
     name: llm-providers
     namespace: agentgateway-system
   spec:
     ai:
       groups:
         - providers:
             - name: on-prem-instance
               custom:
                 model: custom-model
                 formats:
                   - type: Completions
                     path: /api/v1/chat/completions
               host: on-prem-instance.internal
               port: 443
               policies:
                 auth:
                   secretRef:
                     name: on-prem-instance-secret
             - name: cloud-instance-1
               custom:
                 model: custom-model
                 formats:
                   - type: Completions
                     path: /api/v1/chat/completions
               host: cloud-instance-1.example.com
               port: 443
             - name: cloud-instance-2
               custom:
                 model: custom-model
                 formats:
                   - type: Completions
                     path: /api/v1/chat/completions
               host: cloud-instance-2.example.com
               port: 443
   EOF

2. Create a dedicated AgentgatewayBackend for the same cloud instance, used by a separate route, such as one that scrapes a per-instance metrics endpoint. This example uses the openai provider type with /metrics configured as passthrough, matching a common pattern for a non-inference route that reaches the same host. Leave auth, tls, and tunnel off here too.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayBackend
   metadata:
     name: cloud-instance-1-metrics
     namespace: agentgateway-system
   spec:
     ai:
       provider:
         openai:
           model: custom-model
         host: cloud-instance-1.example.com
         port: 443
     policies:
       ai:
         routes:
           "/metrics": "Passthrough"
   EOF

3. Create an AgentgatewayPolicy that targets both the cloud-instance-1 provider inside the aggregate AgentgatewayBackend (using sectionName) and the dedicated metrics AgentgatewayBackend. The backend section on the policy supplies the shared auth, tls, and tunnel settings to both targets.

   kubectl apply -f- <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayPolicy
   metadata:
     name: cloud-instance-1-connection
     namespace: agentgateway-system
   spec:
     targetRefs:
       - group: agentgateway.dev
         kind: AgentgatewayBackend
         name: llm-providers
         sectionName: cloud-instance-1
       - group: agentgateway.dev
         kind: AgentgatewayBackend
         name: cloud-instance-1-metrics
     backend:
       auth:
         secretRef:
           name: cloud-instance-1-secret
       tls:
         sni: cloud-instance-1.example.com
       tunnel:
         backendRef:
           group: agentgateway.dev
           kind: AgentgatewayBackend
           name: forward-proxy
           port: 3128
   EOF

   Field	Description
   targetRefs[0].sectionName	Selects the cloud-instance-1 provider by name inside the llm-providers AI provider group. Without a sectionName, the policy would apply to every provider in the group, including on-prem-instance and cloud-instance-2.
   targetRefs[1]	The separate AgentgatewayBackend used for the dedicated metrics route. A policy’s targetRefs can list multiple targets, so one policy applies to both.
   backend.auth.secretRef	The Secret with credentials for cloud-instance-1, now defined once instead of on each backend.
   backend.tls.sni	The TLS Server Name Indication to use when connecting to cloud-instance-1, shared by both targets.
   backend.tunnel.backendRef	Routes both targets’ connections through the same forward proxy AgentgatewayBackend, such as for an HTTP CONNECT proxy. See Tunnel through a proxy for how to set up the proxy backend itself.

   Repeat step 3 for cloud-instance-2 and any other cloud instance that needs its own dedicated backend, each with its own policy and secret.

Cleanup

kubectl delete AgentgatewayPolicy cloud-instance-1-connection -n agentgateway-system
kubectl delete AgentgatewayBackend cloud-instance-1-metrics -n agentgateway-system
kubectl delete AgentgatewayBackend llm-providers -n agentgateway-system

Next steps

• Review Targeting and merging for how inline AI policies on a backend merge with attached AgentgatewayPolicy resources.
• Set up a tunnel through a proxy if your provider connections need to go through a forward proxy.
• Configure failover with priority groups for high availability.