Banyan — Cost & Token Optimization (roadmap)

Cost & token optimization

How Banyan intends to reduce the number of tokens a workload needs, and the cost of the tokens it still needs.

Roadmap, not shipped. Every technique below is marked with its status. Nothing marked PLANNED is implemented — these are established techniques with published figures from the tools that report them, listed so the plan is concrete and reviewable. The percentages are those publishers' numbers, not measurements on Banyan. When one ships, it will carry a figure measured on our own hardware, or no figure at all.

Token gateway

Technique	Reported saving
Context compression PLANNED Strip redundancy from payloads — repeated build logs, verbose JSON arrays, boilerplate — before the prompt reaches the model.	20–95% fewer input tokens Headroom (open source), publisher's figures
Semantic caching PLANNED Embed each query, look for a near-identical earlier request in a vector index, and return the stored answer instead of re-running inference. Fits Banyan's batch and overflow workloads well — they repeat far more than interactive chat does.	Up to 100% on an exact repeat hit LiteLLM / Portkey / Helicone, publishers' figures
Scrape cleanup PLANNED Extract readable text from fetched pages, discarding markup, scripts and styling before they enter the context window.	Up to 90% fewer prompt tokens on RAG and web-search tasks text-extraction utilities, publishers' figures

Technique

Reported saving

Context compression PLANNED

Strip redundancy from payloads — repeated build logs, verbose JSON arrays, boilerplate — before the prompt reaches the model.

20–95% fewer input tokens

Headroom (open source), publisher's figures

Semantic caching PLANNED

Embed each query, look for a near-identical earlier request in a vector index, and return the stored answer instead of re-running inference.

Fits Banyan's batch and overflow workloads well — they repeat far more than interactive chat does.

Up to 100% on an exact repeat hit

LiteLLM / Portkey / Helicone, publishers' figures

Scrape cleanup PLANNED

Extract readable text from fetched pages, discarding markup, scripts and styling before they enter the context window.

Up to 90% fewer prompt tokens on RAG and web-search tasks

text-extraction utilities, publishers' figures

Agent architecture

Technique	Reported saving
Semantic tool selection PLANNED Keep tool schemas in a local vector store and inject only the handful relevant to the current step, instead of the whole toolset on every call.	Up to 90% smaller agent prompts published agent-framework benchmarks
State offloading PLANNED Move conversation history out of the active context into SQLite or Redis, retrieving only the turns a step actually needs.	Scales with session length no single published figure
Compaction triggers PLANNED Watch context length; past a threshold, summarise history with a small local model and drop the raw transcript.	30–60% on long sessions published agent-framework benchmarks

Technique

Reported saving

Semantic tool selection PLANNED

Keep tool schemas in a local vector store and inject only the handful relevant to the current step, instead of the whole toolset on every call.

Up to 90% smaller agent prompts

published agent-framework benchmarks

State offloading PLANNED

Move conversation history out of the active context into SQLite or Redis, retrieving only the turns a step actually needs.

Scales with session length

no single published figure

Compaction triggers PLANNED

Watch context length; past a threshold, summarise history with a small local model and drop the raw transcript.

30–60% on long sessions

published agent-framework benchmarks

Prompt engineering

Technique	Reported saving
Prompt dieting PLANNED Rewrite system prompts into dense declarative form — no pleasantries, no restated edge cases.	30–50% fewer input tokens widely reported; varies by prompt
Strict output capping PARTIAL Bound generation with max_tokens and instruct the model to answer in fragments rather than essays. max_tokens is already honoured end to end. The prompt-side discipline is not automated.	Large — output tokens cost 3–4x input on our own rate card Banyan rate card (this one is arithmetic, not a claim)

Technique

Reported saving

Prompt dieting PLANNED

Rewrite system prompts into dense declarative form — no pleasantries, no restated edge cases.

30–50% fewer input tokens

widely reported; varies by prompt

Strict output capping PARTIAL

Bound generation with max_tokens and instruct the model to answer in fragments rather than essays.

max_tokens is already honoured end to end. The prompt-side discipline is not automated.

Large — output tokens cost 3–4x input on our own rate card

Banyan rate card (this one is arithmetic, not a claim)