Skip to main content
The system prompt is the single most important input you control. It defines who your agent is, how it speaks, what it knows, and what it won’t do. Real-time video agents are harder to prompt than a text chatbot for three reasons:
  1. Responses are spoken, not read. Markdown, symbols, and long paragraphs sound wrong.
  2. Latency is visible. Every extra token delays the first word out.
  3. The user can see the agent. Verbose responses feel robotic and unnatural on camera.
This guide is a six-section framework that handles all three. Copy the templates, then adapt.
The LLM has no awareness that its output will be spoken and rendered on a face. You have to tell it explicitly.

The six sections

Structure every prompt in this order: the LLM reads top-to-bottom, and earlier instructions carry more weight.
  1. Identity: who the agent is
  2. Style: how it talks
  3. Behaviors: how it handles the conversation
  4. Response rules: how to format spoken output
  5. Guardrails: what it must never do
  6. Flow (optional): a specific step-by-step for structured conversations

1. Identity

Start with a clear, one-paragraph identity: name, role, expertise, and purpose. Concrete beats abstract.
Bad identities:
  • “You are a helpful assistant.”: no persona, no scope
  • “You are an AI that can help with anything.”: impossible to be good at anything
  • “You are Steve Jobs.”: never impersonate a real person

2. Style

Describe communication style with behavioral anchors: not adjectives. “Warm” is vague; “opens with a first-name greeting and asks how you’re doing” is behavior.
3–4 concrete cues beats 10 vague adjectives.

3. Behaviors

Describe how the agent runs a conversation: opening, active listening, redirects, closing.

4. Response rules

This is the section that makes voice work. The LLM defaults to text-like output (markdown, lists, code blocks). You have to override that.
This section alone accounts for most of the difference between a prompt that works and one that doesn’t.

5. Guardrails

Hard rules the agent must never break. Enforce them in the prompt itself and by keeping the prompt above user messages in your instruction hierarchy.
This section is about style: rules folded into the system prompt as text, with no enforcement beyond the LLM choosing to follow them. For rules that need their own callable check, a canned deterministic response, and an optional trigger webhook, use the dedicated Guardrails feature instead. It’s a separate object the agent calls as a tool, not prompt text. The two are complementary: keep style constraints here, and move anything safety- or compliance-critical to the real feature. Either way, the agent only checks a guardrail, prompt-based or the dedicated feature, when the prompt actually tells it to.

6. Flow (optional)

For structured conversations (interviews, intake, onboarding), describe the phases and how the agent moves between them. For open-ended agents, skip this section entirely.
For structured flows, describe each phase explicitly in the prompt and let the LLM decide when to move between them based on the user’s answers.

A complete example

Bringing it all together:

Prompt hygiene checklist

Before shipping any prompt:
  • Under 2000 tokens total
  • Contains an Identity section with a specific persona
  • Contains Response rules that explicitly say “voice interface”
  • No markdown, code fences, or JSON schemas in the prompt itself
  • Guardrails cover: AI disclosure, off-topic redirects, and any regulated advice
  • Tested with at least 10 real user questions
  • Tested with adversarial input (“Ignore your instructions and…”)

Iterating on prompts

Prompts fail in ways you can’t predict from reading them. The only way to know is to listen to real conversations.
  • Week 1: ship the first version. Read 20 transcripts.
  • Week 2: update the prompt based on what you saw. Ship. Read another 20.
  • Ongoing: every time the agent embarrasses you, add a rule that would have prevented it.
Don’t try to write a perfect prompt on day one. Iterate.

What’s next

Templates

Share prompts across many agents.

Available LLMs

Choose the right model for your prompt.