DeliverabilityUpdated · 7 min read

Code Mode for In-App SaaS Agents: Prompt Caching & Cost Math

How using code mode prompt caching inside a SaaS product's internal AI assistant stabilizes tool schemas, boosts cache hit rates, and keeps large payloads out of context.

MT
Mailbase Team
code mode prompt caching
On this page · 9 sections

Overview

Most write-ups on MCP code mode focus on external developer clients connecting to a remote server. Inside a real B2B SaaS product, there is an even bigger payoff that almost nobody talks about: running code mode for your **own in-app AI assistant** so your prompt cache hit rate stays near its ceiling and your per-user LLM cost stops scaling linearly with your feature count.

When a SaaS app grows to 150+ actions across sending, campaigns, audiences, inbox triage, prospecting, and API keys, exposing every JSON Schema on every turn silently breaks unit economics. Here is why static tool lists destroy prompt caching, how a two-tool search_tools + run_program design fixes it in production, and how to enforce strict workspace security inside a QuickJS sandbox.

Why 150 tool definitions quietly wreck prompt caching and unit economics

Modern LLM providers discount cached input tokens by 75% to 90% — **only when the prompt prefix is byte-identical**. The moment a SaaS team tries to be clever by dynamically filtering tool definitions per route, per plan, or per user role (if (user.isAdmin) tools.push(...)), every subtle permutation invalidates the cache prefix for the entire tool block. On the flip side, if you keep all 150+ JSON Schemas pinned on every request so they cache, you still burn 25,000+ tokens on the first turn, dilute tool-selection accuracy, and pay full price on every multi-step loop where raw API arrays get echoed back into conversation history.

The bottleneck is not just tool-definition size — it is intermediate data. Ask an in-app assistant *"Which of my last 10 campaigns had a bounce rate above 2%?"* under classic tool calling, and the model makes 11 sequential round trips, stuffing every JSON payload into the context window even though the final answer is two lines long.

DimensionClassic 150-tool JSON SchemasTwo-tool Code Mode (`search_tools` + `run_program`)
System + tools prefix20k–35k tokens, fragile if filtered per route/role2 compact tool definitions, byte-identical across every user
Prompt cache hit rateDrops whenever dynamic tool subsets shiftStable prefix across tenants, roles, and pages
Multi-step read queriesN sequential LLM turns; raw JSON enters context1 sandboxed JS function; only the aggregated summary returns
Write / send safetyEasy to over-expose destructive tools by accidentSandbox allowlists read-only tools only; writes require direct confirmation

The two-tool split: progressive discovery + QuickJS sandbox

In Mailbase's own AI & MCP layer, the entire surface collapses into two tools with an unchanging prefix:

**1. search_tools(query, family, detail)** handles progressive disclosure. A call with detail: "names" returns only tool names; "descriptions" adds titles, read-only/destructive flags, and whether the sandbox may call it; "full" returns the Zod-derived JSON Schema for only the few tools the model actually needs right now. Zero workspace data is touched.

**2. run_program(code)** executes a short async JavaScript body inside an isolated **QuickJS** sandbox — no network, no filesystem, no timers, no environment variables — bounded by hard wall-clock, CPU-time, memory, max-tool-call, and max-output limits. Inside the guest, await mailbase.<tool_name>(args) bridges back to the host, so the model can run Promise.all(campaigns.slice(0, 5).map((c) => mailbase.get_campaign_stats({ id: c.id }))) and return a 200-byte summary instead of 40 KB of raw JSON.

Enforcing multi-tenant permissions inside a code-mode bridge

A common objection from backend engineers is security: *if the model writes JavaScript, can it bypass RBAC or fire emails in a loop?* Not if the host bridge refuses to be a shortcut.

Every mailbase.<tool>() call from QuickJS must pass through the exact same guarded handler as a direct call: the host checks a static **read-only allowlist** derived from tool annotations (write and send tools are rejected inside run_program and can only be invoked directly where the UI asks the user to confirm), validates arguments against the tool's Zod schema, verifies the caller's OAuth/session scopes, checks the member's current workspace role, and scrubs unexpected server errors before anything returns to the guest.

For SaaS teams operating on tight margins, the result is double leverage: you keep one cached, byte-identical prompt prefix across your entire user base while keeping multi-step analytics and inbox queries fast, deterministic, and isolated per tenant.

Common mistakes

  • Skipping SPF, DKIM, and DMARC, or assuming they're a one-time setup.
  • Sending real volume from a brand-new, un-warmed domain.
  • Reusing a stale list without re-verifying, so bounces spike.
  • Ignoring complaint rate until a single bad campaign sinks the domain.

Sources and further reading

Official docs for current setup details, pricing, and API behavior — verify specifics there, since they change.

Related guides

More on code mode prompt caching and the surrounding deliverability workflow:

Questions

About this page.

Why does code mode improve prompt cache hit rates in a SaaS app?
Because instead of injecting dynamically filtered subsets of 100+ tool schemas per page or user role (which invalidates the cached prefix), every user shares the exact same two-tool prefix (search_tools + run_program). Only the few schemas needed for the current task load on demand.
Is running LLM-generated JavaScript safe in a multi-tenant SaaS?
Yes, when isolated properly: run the guest code in a locked-down QuickJS sandbox with zero network/filesystem/env access and strict CPU, memory, and wall-clock caps, allowlist only read-only tools inside the sandbox, and run every bridged tool call through the same auth, scope, and workspace guards as a normal API call.
What happens when the AI assistant needs to send an email or modify data?
Write and send actions are explicitly blocked inside run_program. The model must call those tools directly outside the sandbox so the application can show a human confirmation step before anything leaves.
Does code mode work for both MCP clients and an in-app assistant?
Yes. Exposing the same search_tools and run_program registry to both your internal product assistant and your external MCP server means you maintain one guarded catalog and get prompt-cache savings on both surfaces.