Code Mode for In-App SaaS Agents: Prompt Caching & Cost Math
How using code mode prompt caching inside a SaaS product's internal AI assistant stabilizes tool schemas, boosts cache hit rates, and keeps large payloads out of context.
On this page · 9 sections
Overview
Most write-ups on MCP code mode focus on external developer clients connecting to a remote server. Inside a real B2B SaaS product, there is an even bigger payoff that almost nobody talks about: running code mode for your **own in-app AI assistant** so your prompt cache hit rate stays near its ceiling and your per-user LLM cost stops scaling linearly with your feature count.
When a SaaS app grows to 150+ actions across sending, campaigns, audiences, inbox triage, prospecting, and API keys, exposing every JSON Schema on every turn silently breaks unit economics. Here is why static tool lists destroy prompt caching, how a two-tool search_tools + run_program design fixes it in production, and how to enforce strict workspace security inside a QuickJS sandbox.
Why 150 tool definitions quietly wreck prompt caching and unit economics
Modern LLM providers discount cached input tokens by 75% to 90% — **only when the prompt prefix is byte-identical**. The moment a SaaS team tries to be clever by dynamically filtering tool definitions per route, per plan, or per user role (if (user.isAdmin) tools.push(...)), every subtle permutation invalidates the cache prefix for the entire tool block. On the flip side, if you keep all 150+ JSON Schemas pinned on every request so they cache, you still burn 25,000+ tokens on the first turn, dilute tool-selection accuracy, and pay full price on every multi-step loop where raw API arrays get echoed back into conversation history.
The bottleneck is not just tool-definition size — it is intermediate data. Ask an in-app assistant *"Which of my last 10 campaigns had a bounce rate above 2%?"* under classic tool calling, and the model makes 11 sequential round trips, stuffing every JSON payload into the context window even though the final answer is two lines long.
| Dimension | Classic 150-tool JSON Schemas | Two-tool Code Mode (`search_tools` + `run_program`) |
|---|---|---|
| System + tools prefix | 20k–35k tokens, fragile if filtered per route/role | 2 compact tool definitions, byte-identical across every user |
| Prompt cache hit rate | Drops whenever dynamic tool subsets shift | Stable prefix across tenants, roles, and pages |
| Multi-step read queries | N sequential LLM turns; raw JSON enters context | 1 sandboxed JS function; only the aggregated summary returns |
| Write / send safety | Easy to over-expose destructive tools by accident | Sandbox allowlists read-only tools only; writes require direct confirmation |
The two-tool split: progressive discovery + QuickJS sandbox
In Mailbase's own AI & MCP layer, the entire surface collapses into two tools with an unchanging prefix:
**1. search_tools(query, family, detail)** handles progressive disclosure. A call with detail: "names" returns only tool names; "descriptions" adds titles, read-only/destructive flags, and whether the sandbox may call it; "full" returns the Zod-derived JSON Schema for only the few tools the model actually needs right now. Zero workspace data is touched.
**2. run_program(code)** executes a short async JavaScript body inside an isolated **QuickJS** sandbox — no network, no filesystem, no timers, no environment variables — bounded by hard wall-clock, CPU-time, memory, max-tool-call, and max-output limits. Inside the guest, await mailbase.<tool_name>(args) bridges back to the host, so the model can run Promise.all(campaigns.slice(0, 5).map((c) => mailbase.get_campaign_stats({ id: c.id }))) and return a 200-byte summary instead of 40 KB of raw JSON.
Enforcing multi-tenant permissions inside a code-mode bridge
A common objection from backend engineers is security: *if the model writes JavaScript, can it bypass RBAC or fire emails in a loop?* Not if the host bridge refuses to be a shortcut.
Every mailbase.<tool>() call from QuickJS must pass through the exact same guarded handler as a direct call: the host checks a static **read-only allowlist** derived from tool annotations (write and send tools are rejected inside run_program and can only be invoked directly where the UI asks the user to confirm), validates arguments against the tool's Zod schema, verifies the caller's OAuth/session scopes, checks the member's current workspace role, and scrubs unexpected server errors before anything returns to the guest.
For SaaS teams operating on tight margins, the result is double leverage: you keep one cached, byte-identical prompt prefix across your entire user base while keeping multi-step analytics and inbox queries fast, deterministic, and isolated per tenant.
Common mistakes
- Skipping SPF, DKIM, and DMARC, or assuming they're a one-time setup.
- Sending real volume from a brand-new, un-warmed domain.
- Reusing a stale list without re-verifying, so bounces spike.
- Ignoring complaint rate until a single bad campaign sinks the domain.
Sources and further reading
Official docs for current setup details, pricing, and API behavior — verify specifics there, since they change.
Related guides
More on code mode prompt caching and the surrounding deliverability workflow: