Bring Your Own LLM

TruGen lets you replace the default LLM with any OpenAI-compatible endpoint. Everything else (STT, TTS, avatar rendering, tools, knowledge base) keeps working. Use this for compliance, fine-tuned models, or your own inference infrastructure.

Why bring your own LLM

Benefit	Description
Customisation	Use a domain-specific fine-tune for better vertical accuracy
Data residency	Keep sensitive data inside your own infrastructure or private cloud
Compliance	Meet regulatory requirements around auditing and data handling
Cost control	Choose your own provider or run on your own hardware

Requirements

Your endpoint must implement the OpenAI Chat Completions specification:

Endpoint: POST /chat/completions
Payload: accepts a messages: [] array
Streaming: Server-Sent Events (SSE) with data: framing

Optional: required only for specific features:

Function calling: needed if the agent uses tools
Vision: needed for multimodal inputs

Configure via the Developer Portal

Open your agent

Navigate to Create New Agent or an existing agent in app.trugen.ai.

Open Agent Config

Find the LLM section.

Select Bring Your Own LLM

In the Provider dropdown, select Bring your own LLM.

Enter your endpoint

Provide your API URL, Model Name, and API Token. Save.

Bring Your Own LLM configuration in Developer Studio — Developer Portal: Custom LLM configuration

Configure via API

Set the LLM config’s provider to "custom":

"llm": {
  "provider": "custom",
  "model": "MODEL_NAME",
  "url": "https://your-inference.example.com/v1",
  "token": "YOUR_ENDPOINT_TOKEN"
}

That’s the only change needed. The rest of your agent configuration is unchanged. Full request example:

curl --request POST \
  --url https://api.trugen.ai/v1/ext/agent \
  --header 'Content-Type: application/json' \
  --header 'x-api-key: YOUR_API_KEY' \
  --data '{
    "agent_name": "Custom LLM Agent",
    "agent_system_prompt": "You are a helpful assistant.",
    "avatars": [
      {
        "avatar_key_id": "665a1170",
        "llm": {
          "provider": "custom",
          "model": "your-fine-tune-v1",
          "url": "https://your-inference.example.com/v1",
          "token": "YOUR_ENDPOINT_TOKEN"
        }
      }
    ]
  }'

Endpoint contract

Your /chat/completions handler receives:

POST /chat/completions
{
  "model": "your-fine-tune-v1",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user",   "content": "What are your business hours?" }
  ],
  "stream": true,
  "tools": [ /* ... */ ]
}

And must respond with an SSE stream:

data: {"id":"…","object":"chat.completion.chunk","choices":[{"delta":{"content":"We"},"index":0}]}

data: {"id":"…","object":"chat.completion.chunk","choices":[{"delta":{"content":"'re open"},"index":0}]}

data: {"id":"…","object":"chat.completion.chunk","choices":[{"delta":{"content":" Monday to Friday."},"index":0,"finish_reason":"stop"}]}

data: [DONE]

Keep-alive during long generations

Long generations without any output can trigger idle timeouts. Emit a comment line every 10 seconds:

: keep-alive

Function calling

If your endpoint supports OpenAI-style function calling, return tool calls in the delta:

{
  "choices": [
    {
      "delta": {
        "tool_calls": [
          {
            "index": 0,
            "id": "call_abc",
            "function": {
              "name": "check_order_status",
              "arguments": "{\"orderId\": \"ORD-9821\"}"
            }
          }
        ]
      }
    }
  ]
}

TruGen invokes the tool, sends the result back as a follow-up messages[].role: "tool" turn, and continues generation.

Fallback to a hosted model

If your custom endpoint errors or times out, TruGen can fall back to a hosted model automatically:

"llm": {
  "provider": "custom",
  "model": "your-fine-tune-v1",
  "url": "https://your-inference.example.com/v1",
  "token": "YOUR_ENDPOINT_TOKEN",
  "fallback_provider": "groq",
  "fallback_model": "openai/gpt-oss-120b"
}

Strongly recommended for production: a single infrastructure hiccup on your side shouldn’t kill live sessions.

Latency expectations

Custom LLMs add network round-trip time between TruGen’s infrastructure and yours. Aim for:

Time to first token: under 500 ms
Endpoint region: as close to your TruGen region as possible
Network latency to TruGen edge: under 50 ms

If your endpoint is in a different continent, latency will noticeably degrade the conversation.

What’s next

Available LLMs

Compare hosted providers if BYO isn’t a hard requirement.

Prompting strategies

Write prompts that work well for real-time voice.

Tools

Enable function calling on your custom LLM.

Introduction

Models

AI Teammates (Avatars)

Text to Video

Why bring your own LLM

Requirements

Configure via the Developer Portal

Configure via API

Endpoint contract

Keep-alive during long generations

Function calling

Fallback to a hosted model

Latency expectations

What’s next

Available LLMs

Prompting strategies

Tools

​Why bring your own LLM

​Requirements

​Configure via the Developer Portal

​Configure via API

​Endpoint contract

​Keep-alive during long generations

​Function calling

​Fallback to a hosted model

​Latency expectations

​What’s next

Available LLMs

Prompting strategies

Tools

Why bring your own LLM

Requirements

Configure via the Developer Portal

Configure via API

Endpoint contract

Keep-alive during long generations

Function calling

Fallback to a hosted model

Latency expectations

What’s next