Ollama

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
  port: 3000
  models:
  - name: "*"
    provider: openAI
    params:
      hostOverride: "localhost:11434"

agentgateway -f config.yaml

curl http://localhost:3000/v1/chat/completions \
  -H "content-type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [
      {"role": "user", "content": "Hello! Tell me about Ollama in one sentence."}
    ]
  }' | jq

{"model":"llama3.2","usage":{"prompt_tokens":14,"completion_tokens":323,"total_tokens":337},"choices":[{"message":{"content":"<think>\nOkay, user just asked for a one-sentence explanation of Ollama. That's pretty concise and specific—no fluff allowed here. They're probably either testing my knowledge or genuinely need a quick definition without jargon overload.\n\nHmm, judging by the tone, they might be evaluating if I can give crisp technical explanations. Since they didn't specify their familiarity level, a neutral layman-to-engineer explanation would work best. \n\nOllama is known for being a wrapper around LLMs that handles infrastructure stuff invisibly to end users. But how to phrase it in one sentence without sounding like \"magic\"? Need to balance clarity and technical accuracy...\n\n*Brainstorming:*\nOption 1: Focus on what it does (local, multi-model inference) + benefit aspect\nOption 2: Contrast with alternatives (\"handles all the heavy lifting\")\nOption 3: Mention its approach-to-problem innovation\n\nGoing with Option 1 feels safest since non-engineers might struggle with \"wrapper\" terminology. Also emphasizing accessibility (\"you\") helps bridge technical and casual users. Should keep it under 50 characters for social media-readiness.\n\n...wait, is this going to feel too basic? No—simple beats vague when someone asks explicitly. Final polish: add asterisk as signal that I can expand explanation if they want more depth.\n</think>\nOllama lets you run large language models (LLMs) like Llama and Mistral on your device by handling all the infrastructure work for local multi-model inference.\n\n*(Let me know if you'd like a longer explanation!)*","role":"assistant"},"index":0,"finish_reason":"stop"}],"id":"chatcmpl-738","object":"chat.completion","created":1773934551,"system_fingerprint":"fp_ollama"}