How to Reduce LLM Token Costs for AI Agent Memory

TL;DR:

Six techniques that compound: token budgeting (−75% prompt tokens), hierarchical summarization (−59%), Ebbinghaus eviction (−59%), embedding quantization (4× storage), Jaccard self-curation, and hot/cold caching (83% RAM reduction). Stack them in the right order, and you cut memory costs by 3–4× without touching your model.

Switching from naive file-based memory injection to retrieval-based memory can meaningfully reduce prompt tokens on small stores, though savings vary based on store size and query patterns. In the companion comparator, retrieval cut a 24-entry Hermes memory prompt from 594 tokens to 166 tokens on the same query. If you want the full breakdown of why that works and how to set it up, that is covered in the latest blog.

Retrieval-based memory is not the end of the optimization story. As your agent runs longer, a different set of failure modes starts appearing:

• Stale entries inflating prompt costs as the store grows
• Duplicate memories degrading retrieval precision over time
• Vector indexes consuming RAM on constrained hardware
• Retrieval latency accumulating across thousands of daily calls

These six techniques are independent layers you can stack on top of your existing retrieval architecture. Each targets a specific failure mode, and each is measured with the right metric for what it actually optimizes:

• Prompt-shaping techniques (1–3) are verified with API token counts against the same 24-entry store
• Storage and infrastructure techniques (4–6) use storage math, similarity scans, and cache proxies

Naive Injection vs Retrieval-Based Memory

This is the floor on a small 24-entry store. Savings will vary at larger store sizes. The six techniques below push costs lower or address failure modes that token count alone does not capture.

Quick Summary

Techniques 1–3 are verified with real API token counts using openai/gpt-4o-mini, on same 24-entry store and the same query. Techniques 4–6 use storage math, similarity scans, and cache proxies as noted:

These percentages should not be added together. Each technique was measured as an alternative memory-shaping strategy against the same naive baseline, not as a cumulative pipeline.

How to Reproduce These Numbers?

The token counts for Techniques 1–3 come from verify_advanced_api.py, a dedicated verification script in the repo that calls OpenRouter with each prompt variant and reads back real usage.prompt_tokens.

To run it against your own memory directory:

python verify_advanced_api.py \
 --memory-dir examples/hermes-memory \
 --user-id hermes-advanced-api-1 \
 --memory-limit 10 \
 --advanced-budget-tokens 80

The script seeds Mem0 with your Hermes entries, retrieves memories for the query, builds four prompt variants (naive baseline, token budgeting, hierarchical summary, Ebbinghaus filtered), sends each to the same model, and prints the token counts side by side. The numbers in this article came from running it on a 24-entry store with openai/gpt-4o-mini.

Where to set the budget:

Start at 500–800 tokens for typical personal assistant or in-home agents. For multi-domain agents handling varied query types, instrument your p95 memory token usage over a week and set the budget at 120 percent of that value. This gives you a ceiling that handles outlier queries without over-constraining normal ones.

Techniques Details

1. Token Budgeting (−75% Prompt Tokens)

Token budgeting is the highest-leverage technique in this list and the simplest to implement.

Measured result: We found that budget=80, selected=3 memories, overflow=7, led to prompt tokens=149, which is a 75% reduction compared to the naive baseline of 600.

def budget_memories(memories: list[str], token_budget: int, model_hint: str) -> BudgetedContext:
    selected: list[str] = []
    used = 0
    overflow = 0
    for memory in memories:
        tokens = estimate_tokens(memory, model_hint)
        if used + tokens <= token_budget:
            selected.append(memory)
            used += tokens
        else:
            overflow += 1
    context = "\n".join(f"- {m}" for m in selected)
    if overflow:
        context += ("\n\n[Note: {overflow} additional relevant memories were omitted to stay within the memory token budget.]")
    if not context:
        context = "- No relevant Mem0 memories found."
    return BudgetedContext(memories=selected, context=context, tokens_used=used, overflow_count=overflow)

2. Hierarchical Summarization (−59% Prompt Tokens)

Measured result (real API token count): Reduction of prompt tokens from 600 to just 247 prompt tokens (−59%).

def build_hierarchical_summary(entries: list[tuple[str, dict[str, str]]]) -> str:
    by_source: dict[str, list[str]] = {}
    for text, metadata in entries:
        by_source.setdefault(metadata["source"], []).append(text)
    session_summaries = []
    for source, texts in by_source.items():
        important = sorted(texts, key=entry_importance, reverse=True)[:4]
        session_summaries.append(f"{source}: " + " ".join(important))
    long_horizon = ("Long-horizon summary: user values concise, reversible home-automation recommendations; nighttime lighting should be dim; safety-critical devices should not be changed without explicit approval.")
    return "\n".join([*session_summaries, long_horizon])

3. Importance-Based Eviction with Ebbinghaus Decay (−59% Prompt Tokens)

Measured result (real API token count): 9 of 24 entries retained and prompt tokens reduced from 600 to 249 (−59%).

def evict_with_decay(entries: list[tuple[str, dict[str, str]]], threshold: float, now: datetime) -> list[tuple[str, dict[str, str]]]:
    retained = []
    for index, (text, metadata) in enumerate(entries):
        age_days = 3 + (index % 8) * 6
        access_count = (4 if any(t in text.lower() for t in ["porch","outside","lights"]) else index % 3)
        score = ebbinghaus_score(importance=entry_importance(text), created_at=now - timedelta(days=age_days), access_count=access_count, now=now)
        if score > threshold:
            retained.append((text, metadata))
    return retained

4. Embedding Quantization (4× Storage Reduction)

Measured result: We found that float32 requires 49,152 bytes for 24 entries, while int8 requires just 12,288 bytes, which is 4.0× smaller.

5. Self-Curation with Jaccard Similarity (Retrieval Precision)

Measured result: 1 merge candidate found with similarity ≥ 0.30, score = 0.44.

6. Hybrid Hot/Cold Caching (83% RAM Reduction)

Measured result: 4/24 hot entries with a simulated hit rate of 40.0%, and a RAM reduction proxy of 83.3%.

How to Stack These Based on the Failure Mode

Not all six techniques belong in every deployment. Here is how to decide which ones to add first:

• If prompt token cost is climbing week over week, start with token budgeting (Technique 1).

• If you are running on constrained hardware like a Raspberry Pi or homelab with limited RAM, prioritize embedding quantization (Technique 4) and hot/cold caching (Technique 6) first.

• If retrieval precision is degrading as your store grows, schedule a weekly self-curation pass (Technique 5).

What to Do Next

You have two paths from here:

Build it yourself: Clone the GitHub repo and run --run-advanced to reproduce all 6 benchmarks locally.

Skip the infrastructure: Start free on Mem0 and get all techniques in place without writing the optimization layer yourself.

Frequently Asked Questions

Q. Do these techniques work with agents other than Hermes?

Yes. The underlying patterns such as token budgeting, summarization, eviction, quantization, deduplication, and caching apply to any agent framework with a persistent memory store.

Q. Which technique should I add first?

Token budgeting. It has the highest measured impact, the lowest implementation effort, and no risk of accidentally removing important memories.