Introduction

This is a how-to guide to setup AgentGateway and get your first AI route working with OpenAI. We’ll walk through the complete setup from scratch - creating a Kubernetes cluster, installing AgentGateway, and connecting it to OpenAI’s API.

What is AgentGateway?

AgentGateway is an open source, AI-native data plane built in Rust for connecting, securing, and observing AI traffic. Originally created by Solo.io and now a Linux Foundation project, it acts as a purpose-built proxy layer between your applications and AI services like LLMs, MCP tool servers, and other AI agents.

How to: Building Agentgateway to support Multi-LLM providers.

Agentgateway makes it simple to route traffic to multiple LLM providers through a single gateway using the Kubernetes Gateway API. This guide walks through setting up agentgateway OSS on a local Kind cluster with xAI, Anthropic, and OpenAI backends, all routed through a listener named llm-providers.

One of the most common patterns in AI-native infrastructure is routing traffic to multiple LLM providers behind a single entry point. Whether you’re comparing models, building failover strategies, or just want a unified API across providers, agentgateway gives you a clean Kubernetes-native way to do it using the Gateway API and AgentgatewayBackend custom resources.

ingress-nginx retirement

In November 2025 the Kubernetes community announced the retirement of the venerable ingress-nginx Ingress controller.

Ingress-nginx was one of the first options available in Kubernetes for configuring ingress traffic to workloads hosted on Kubernetes. The story of ingress and traffic management has evolved steadily over the years, culminating in the Kubernetes Gateway API specification, which offers a more robust and complete solution to the problem.

Support for ingress-nginx is scheduled to end in March.

MCP servers are cropping up all over the enterprise like weeds in a nice lawn. And just like weeds, this can cause problems. MCP servers should be secured, but how? The official spec says use OAuth, but that is not appropriate within an enterprise organization.

On a recent LinkedIn post I made the point:

Any internal enterprise MCP client / AI agent that communicates to an [remote] MCP server should be secured with enterprise SSO. If the agent is acting autonomously, then agent identity should be enforced. But that is for a different post… (check out my 5 part series on Entra Agent ID).

Hosted LLM providers like OpenAI and Anthropic have rate limiting capabilities centered around requests per minute (RPM) and tokens per minute (TPM). These are specified at the “organization” level and “project” level. API keys are associated with an organization or project, and each call is subject to token or rate limit restrictions. The actual limits are set by the provider, not you. They will depend on what tier you pay for, but generally not directly configurable. For example:

Today at Open Source Summit Europe, the Linux Foundation accepted agentgateway, a new open source AI-native project created by Solo.io. Agentgateway provides drop-in security, observability, and governance for agent-to-agent and agent-to-tool communication and supports leading interoperable protocols, including Agent2Agent (A2A) and Model Context Protocol (MCP).

Building the Go-To AI Gateway

There are many gateways available today, but most were designed before the rise of AI agents and struggle to support modern AI protocols without major rearchitecture. As a company with deep expertise in Envoy, we initially considered it as the foundation for agentgateway. However, we quickly realized that supporting modern agent protocols like A2A and MCP would require a significant re-architecture of Envoy itself.

In June 2025, the MCP community updated the specification to alleviate some of the concerns from the previous version regarding MCP Authorization. However, this update introduces new concerns, especially around enterprise usage. Nevertheless, many public facing MCP clients (Claude, VS Code, etc) do implement the MCP Authorization spec, and many public facing MCP services are expecting this.

In agentgateway, we are trying to make this easier for those building MCP servers. In recent builds, we’ve introduced a way to configure mcpAuthentication which leverages an external OAuth provider, specifically implementing the MCP server side of the MCP Authorization spec for you via configuration.

Today, we’re excited to share the next major milestone: agentgateway is now a full-featured, AI-native gateway that combines deep MCP and A2A protocol awareness, robust traffic policy controls, inference gateway support, Kubernetes Gateway API support, and unified access to major LLMs, all purpose-built with Rust for real-world agentic systems.

Back when we first introduced agentgateway, it was designed to fill a critical gap in the AI stack: enabling structured, secure, and scalable communication between agents, tools, and LLMs using protocols like MCP and A2A. Solo.io wrote about this in a blog post where we explained why traditional API gateways fall short in agentic environments. Since then, the project has grown tremendously.