mem0/integrations/mem0-agent/eval/README.md

# Write-gate evaluation

The v1 plugin had no way to measure extraction quality, so it degraded for three months
without anyone noticing. When the corpus was finally audited, 20.5% of it was
near-duplicate heartbeats — the largest single cluster was 119 near-identical
training-progress frames — and organic searches had fallen from 257/month to 75/month.
The memory got noisy, people stopped trusting it, and nothing in the system said so.

This directory is the instrument that would have said so. It scores the write gate
against a labeled fixture set and fails a build when the score regresses.

The gate has two halves, and each mode measures one of them:

| Half | Where it lives | Measured by |
| --- | --- | --- |
| Local trigger rules — mechanical noise never leaves the machine | `src/mem0_agent/triggers.py` | `--offline` |
| Custom instructions — the platform stores nothing from a narration/activity/repo-file window | `src/mem0_agent/config/project_config.py` (`INSTRUCTIONS`) | `--live` |

## Running it

### Offline (default, no network, no credentials)

```bash
cd integrations/mem0-agent
PYTHONPATH=src python3 eval/run.py --offline
PYTHONPATH=src python3 eval/run.py --offline --level aggressive   # sweep a capture level
PYTHONPATH=src python3 eval/run.py --offline --check              # exit non-zero on regression
```

Runs all fixtures through `mem0_agent.triggers.classify` and scores the decisions
against the labels. It costs nothing, so it belongs in CI on every commit that touches
`triggers.py`.

`triggers.py` is imported lazily. If it is missing or its `classify()` signature is
unreadable, the run reports the reason and scores nothing rather than crashing — but
`--check` then exits non-zero, because a gate that cannot measure must not pass.

### Live (writes to a scratch project)

```bash
export MEM0_API_KEY=...
PYTHONPATH=src python3 eval/run.py --live \
  --project-id proj_SCRATCH --org-id org_YOURS --cleanup --check
```

Replays every `exclude` and `extract` fixture through `mem0_agent.api.Api` with
`infer=True`, waits for extraction, reads back, and scores what the platform actually
stored. This is the only way to test the custom instructions; they are a prompt, and
prompts are not unit-testable.

Safety and correctness properties, each of which exists because of something that went
wrong before:

- **`--project-id` and `--org-id` are required and have no defaults.** A live run writes
  real memories. v1's benchmark data ended up in the production project because a
  harness defaulted to whatever the API key resolved to. Point these at a scratch
  project.
- **Every fixture gets its own user id**, `eval-<runid>-<fixture id>`, under app id
  `eval-<runid>`. Results are attributable to a fixture and a run, and anything left
  behind is trivially findable.
- **It polls; it does not sleep.** Extraction landed anywhere between 20s and 5min
  during validation, so any fixed wait is either wrong or wasteful. `--timeout`
  (default 360s) bounds the wait; `--min-settle` (default 60s) is the minimum time
  before a zero read is allowed to count as suppression, since "nothing stored" and
  "not stored yet" look identical.
- **It scores `metadata.type`, never `categories`.** Platform categorization lags ~3.9h
  at the median and is 0% for memories under an hour old, so a category-based score
  would read zero on a fresh run and tell you nothing.
- **`--cleanup` deletes everything the run wrote.** Without it the report prints the
  `app_id` and user-id prefix needed to clean up by hand.

One honesty caveat on the live `type_match` metric: `metadata.type` is stamped by the
*client* at write time (from `triggers.classify` when available, otherwise from the
fixture's `expect_type`). So `type_match` measures that the type survives the round
trip, and — when the classifier is present — that the classifier chose the right one.
It is not the platform's independent opinion. The report records `stamped_by` per
fixture so a reader can tell which case they are looking at.

Both modes write `eval/last_report.json`: every score, every per-fixture row, the
thresholds in force, and the pass/fail verdict.

## The fixture set

`fixtures.py` holds the labeled windows. Each entry is exactly:

```python
{"id": str, "window": [{"role", "content"}, ...], "label": "drop"|"exclude"|"extract",
 "expect_type": str | None, "note": str}
```

| Label | Contract | Enforced by |
| --- | --- | --- |
| `drop` | Must never be sent to the platform at all | client trigger rules |
| `exclude` | May be sent, but the platform must store nothing from it | custom instructions |
| `extract` | Must produce ≥1 memory, ideally of `expect_type` | both halves |

The `drop` and `exclude` windows use the wording of the audited v1 corpus: the
epoch/loss/ETA training heartbeat, the `N of M chunks processed (X%)` frame, the
markdown-ingest `pid`/`elapsed` line, monitoring status lines, tool-only turns,
assistant narration, file-modification lists, `CLAUDE.md`/README excerpts, one-off
directives (`you do it`), and session-only status (`at 12:45:24 the job was wrapping
up`). Several noise classes appear two or three times with only the numbers changed —
that is what a near-duplicate cluster actually looks like, and a gate that catches the
first frame but not the third has learned the numbers rather than the shape.

The `extract` block covers all five durable types plus four **mixed** windows, where a
durable fact sits between two progress lines. Mixed windows are the most informative
fixtures in the set: they fail in both directions. A gate too eager to drop heartbeats
destroys the fact along with them; a gate too eager to store keeps the heartbeats.

Print coverage at any time:

```bash
python3 eval/fixtures.py
```

## Thresholds

`--check` exits non-zero when a gated metric falls below its floor:

| Metric | Floor | Why |
| --- | --- | --- |
| `hard_drop_recall` | 0.95 | Fraction of `drop` fixtures the client never sends. This is the number that was silently 0 in v1. |
| `extract_recall` | 0.80 | Fraction of `extract` fixtures that survive. Offline: flagged for capture. Live: ≥1 memory stored. A gate that stores nothing scores perfectly on noise. |

Both floors must hold; they measure opposite failure directions and either one alone is
trivially gamed.

Other metrics are reported but not gated, because they diagnose rather than decide:

- `hard_drop_explicit` — how much of the noise containment comes from a real drop rule
  rather than from no flag rule happening to match. `capture.py` only forwards windows
  whose action is `flag`, so a `skip` does contain the noise — but only until someone
  adds a flag rule that matches it. A gap between `hard_drop_recall` and
  `hard_drop_explicit` is a list of heartbeats held back by luck.
- `hard_drop_precision` — of everything hard-dropped, how much was safe to drop. This is
  where over-broad drop rules show up, and it is the metric the mixed fixtures exist to
  move. A hard drop on an `extract` window is the one unrecoverable error: the memory is
  gone and nothing logs a miss. Hard-dropping an `exclude` window is *not* counted
  against this score — those are meant to be discarded, and discarding them locally is
  simply cheaper than having the platform do it.
- `extract_skipped` vs `extract_hard_dropped` — same lost memory, different repair. A
  skip means a flag rule is missing; a hard drop means a drop rule is too greedy.
- `flag_precision`, `type_accuracy`, `type_coverage`, `exclude_suppression`,
  `noise_leak_count` — the rest of the picture.

`noise_leak_count` deserves a note: it regex-matches stored memory text for heartbeat
markers (`epoch`, `ETA`, `N% complete`, `chunks processed`, `pid NNNN`, …). Any hit
means the class that ate 20.5% of the v1 corpus has found a new way through, even if
every count-based score looks fine.

Raising a floor is cheap and should be done once a score has held above the new bar for
a while. Lowering one is a decision that belongs in a PR description, next to the reason.

## Baseline (design validation)

Established against the live platform during the v2 design phase, using the same
fixture wording:

- **Exclude classes: 5 of 6 suppressed by the custom instructions alone.** Training
  heartbeats, chunk-progress frames, file-modification lists, session-only status, and
  assistant narration all stored nothing.
- **The repo-file class is client-side only.** A pasted `CLAUDE.md` excerpt produced
  three confident "preference" memories. This is not fixable by prompting: a pasted
  convention and a stated convention are textually identical, so the extractor is right
  to store it and the client must never send it. `triggers.py` owns this rule, and it is
  mandatory rather than tunable.
- **Extract classes: 5 of 5 captured** — preference, decision, convention, insight and
  runbook — plus the mixed window, where the bastion-host fact was stored and neither
  surrounding progress line was.
- **Categories were empty on every memory** at read time, which is what fixed the read
  path on `metadata.type` and this harness with it.

Write latency was 0.38–0.51s for the fire-and-forget `add`; extraction landed 20s–5min
later. That gap is the whole reason the live mode polls.

## Adding a fixture

1. Append it to `_DROP`, `_EXCLUDE` or `_EXTRACT` in `fixtures.py` with a fresh `id`.
2. Use real wording. A fixture invented to be easy to classify measures nothing —
   prefer a window copied from an actual session or an audit.
3. Write a `note` saying which failure the fixture guards. Six months from now that
   sentence is the only thing standing between a red score and someone "fixing" it by
   deleting the fixture.
4. `extract` fixtures need an `expect_type` from
   `mem0_agent.config.project_config.TYPES`; `drop` and `exclude` fixtures must have
   `expect_type = None`.
5. Run `PYTHONPATH=src python3 -m pytest tests/test_fixtures.py -q` — it enforces the
   schema, unique ids, valid labels and types, and that every durable type is covered.
6. Re-run `--offline` and, when the change touches the instructions, `--live`.

Adding a fixture usually lowers a score. That is the point: the score was previously
measuring a smaller world.