Compare commits

..

1 Commits

Author SHA1 Message Date
kartik-mem0 4284ef0d2a feat(integrations): add Hermes provider with shared memory utilities 2026-09-18 13:21:43 +05:30
128 changed files with 4627 additions and 8310 deletions
+1 -1
View File
@@ -12,7 +12,7 @@
"name": "mem0",
"source": "./integrations/claude-code-plugin",
"description": "Cross-session memory and token savings for coding agents.",
"version": "0.3.2"
"version": "0.3.1"
}
]
}
+1 -1
View File
@@ -12,7 +12,7 @@
"name": "mem0",
"source": "./integrations/cursor-plugin",
"description": "Cross-session memory and token savings for coding agents.",
"version": "0.3.2"
"version": "0.3.1"
}
]
}
@@ -32,9 +32,6 @@ jobs:
- name: Type check
run: bun run type-check
- name: Test
run: bun test
- name: Build
run: bun run build
+1 -1
View File
@@ -5,7 +5,7 @@
{
"id": "mem0",
"displayName": "Mem0",
"version": "0.3.2",
"version": "0.3.1",
"description": "Cross-session memory and token savings for coding agents.",
"homepage": "https://mem0.ai",
"keywords": ["memory", "personalization", "mcp", "semantic-search"],
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@mem0/cli",
"version": "0.2.14",
"version": "0.2.13",
"description": "The official CLI for mem0 — the memory layer for AI agents",
"type": "module",
"bin": {
+1 -2
View File
@@ -31,8 +31,7 @@ export class PlatformBackend implements Backend {
this.headers = {
Authorization: `Token ${config.apiKey}`,
"Content-Type": "application/json",
"X-Mem0-Source": "CLI",
"X-Mem0-Client": `mem0-cli-node/${CLI_VERSION}`,
"X-Mem0-Source": "cli",
"X-Mem0-Client-Language": "node",
"X-Mem0-Client-Version": CLI_VERSION,
};
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "mem0-cli"
version = "0.2.13"
version = "0.2.12"
description = "The official CLI for mem0 — the memory layer for AI agents"
readme = "README.md"
license = "Apache-2.0"
+1 -1
View File
@@ -1,3 +1,3 @@
"""mem0 CLI — the command-line interface for the mem0 memory layer."""
__version__ = "0.2.13"
__version__ = "0.2.12"
+1 -2
View File
@@ -27,8 +27,7 @@ class PlatformBackend(Backend):
headers={
"Authorization": f"Token {config.api_key}",
"Content-Type": "application/json",
"X-Mem0-Source": "CLI",
"X-Mem0-Client": f"mem0-cli-python/{__version__}",
"X-Mem0-Source": "cli",
"X-Mem0-Client-Language": "python",
"X-Mem0-Client-Version": __version__,
},
+32 -93
View File
@@ -7,14 +7,6 @@ mode: "wide"
<Tabs>
<Tab title="Python">
<Update label="2026-09-18" description="v2.1.0">
**Improvements:**
- **Client:** Requests now carry three surface-identity headers so the platform can tell which product made a call. `X-Mem0-Source` names the surface and `X-Application` the host app it runs inside, both set-once so a wrapper that already declared its identity keeps it. `X-Mem0-Client` is append-only and carries `name/version` per layer, outermost first, so a plugin calling this SDK reports the whole chain rather than only the last speaker. `MEM0_SOURCE`, `MEM0_APPLICATION` and `MEM0_CLIENT_STACK` set them from the environment for wrappers that cannot pass options ([#7326](https://github.com/mem0ai/mem0/pull/7326))
- **Client:** The client stack is bounded by dropping whole entries rather than slicing characters, and this SDK's own entry is the reserved one. Truncating the joined string could sever an identifier mid-name and the platform parsed the fragment as a real client ([#7326](https://github.com/mem0ai/mem0/pull/7326))
</Update>
<Update label="2026-09-02" description="v2.0.20">
**Improvements:**
@@ -1235,14 +1227,6 @@ See the [OSS v2 to v3 migration guide](https://docs.mem0.ai/migration/oss-v2-to-
<Tab title="TypeScript">
<Update label="2026-09-18" description="v3.2.0">
**Improvements:**
- **Client:** Requests now carry `X-Mem0-Source`, `X-Application` and `X-Mem0-Client`, matching the Python SDK. The first two are set-once so an outer wrapper keeps its identity; the third is append-only and reports the whole layer chain. Read from `MEM0_SOURCE`, `MEM0_APPLICATION` and `MEM0_CLIENT_STACK` when set ([#7326](https://github.com/mem0ai/mem0/pull/7326))
- **Client:** The SDK version in `X-Mem0-Client` is injected at build time rather than hardcoded, so it cannot go stale at the next release ([#7326](https://github.com/mem0ai/mem0/pull/7326))
</Update>
<Update label="2026-09-02" description="v3.1.8">
**Improvements:**
@@ -1880,13 +1864,6 @@ See the [OSS v2 to v3 migration guide](https://docs.mem0.ai/migration/oss-v2-to-
<Tab title="CLI">
<Update label="2026-09-18" description="Python v0.2.13 / Node v0.2.14">
**Improvements:**
- **Client:** Requests now carry the three surface-identity headers (`X-Mem0-Source`, `X-Application`, `X-Mem0-Client`) introduced in the Python and TypeScript SDKs, so the platform can attribute calls made through the CLI to the correct surface and version ([#7326](https://github.com/mem0ai/mem0/pull/7326))
</Update>
<Update label="2026-08-24" description="Python v0.2.12 / Node v0.2.13">
**New Features:**
@@ -2099,7 +2076,7 @@ A full-featured command-line interface for Mem0, available in both Python and No
- New Git repository writes use a hash of the remote identity in `agent_id`. Search and explicit shared-memory deletion include both current and legacy repository IDs within the repository's `app_id`. Existing memories are not rewritten. Legacy IDs retain their original ambiguity for matching owner/repository names on different Git hosts.
**Packaging:**
- Claude Code, Cursor, Codex, Kimi, Antigravity, and the portable Python bundle are versioned at `0.3.1`. OpenCode and DeepSeek Harness are `0.3.0`; Pi Agent is `0.3.0`; OpenClaw is `1.1.0`. Each host's changes and upgrade considerations are listed in its tab.
- Claude Code, Cursor, Codex, Kimi, Antigravity, and the portable Python bundle are versioned at `0.3.1`. OpenCode, Pi Agent, and DeepSeek Harness are `0.3.0`; OpenClaw is `1.1.0`. Each host's changes and upgrade considerations are listed in its tab.
- Python and TypeScript CI run their respective runtime suites. Package checks build the installable artifacts, check generated-file consistency, and reject TypeScript output that still imports monorepo source.
[#7203](https://github.com/mem0ai/mem0/pull/7203)
@@ -2394,14 +2371,9 @@ Initial release of the Mem0 plugin for Claude Code and Cursor, followed by Codex
<Tab title="Claude Code">
<Update label="2026-09-18" description="Claude Code plugin v0.3.2">
<Update label="Unreleased" description="Sidekick availability">
**Improvements:**
- **Telemetry:** `PLUGIN_VERSION` bumped to `0.3.2`. The `mem0-plugin/<version>` wire header and `plugin_version` telemetry field now reflect the fixes from #7322 through #7358 ([#7373](https://github.com/mem0ai/mem0/pull/7373))
- **Telemetry:** Events are no longer delivered twice, no longer lose parked events on flush, and now attribute each event to the plugin that produced it ([#7323](https://github.com/mem0ai/mem0/pull/7323), [#7324](https://github.com/mem0ai/mem0/pull/7324), [#7358](https://github.com/mem0ai/mem0/pull/7358))
**Changes:**
- **Sidekick:** Sidekick is now available only in Claude Code, with Sonnet, worktree isolation, and parent memories.
Sidekick is now available only in Claude Code, with Sonnet, worktree isolation, and parent memories.
</Update>
@@ -2424,14 +2396,9 @@ Initial release of the Mem0 plugin for Claude Code and Cursor, followed by Codex
<Tab title="Cursor">
<Update label="2026-09-18" description="Cursor plugin v0.3.2">
<Update label="Unreleased" description="Sidekick availability">
**Improvements:**
- **Telemetry:** `PLUGIN_VERSION` bumped to `0.3.2`. The `mem0-plugin/<version>` wire header and `plugin_version` telemetry field now reflect the fixes from #7322 through #7358 ([#7373](https://github.com/mem0ai/mem0/pull/7373))
- **Telemetry:** Events are no longer delivered twice, no longer lose parked events on flush, and now attribute each event to the plugin that produced it ([#7323](https://github.com/mem0ai/mem0/pull/7323), [#7324](https://github.com/mem0ai/mem0/pull/7324), [#7358](https://github.com/mem0ai/mem0/pull/7358))
**Changes:**
- **Sidekick:** Removes Sidekick and its start/stop hooks. Memory capture, search, and six skills remain available.
Removes Sidekick and its start/stop hooks. Memory capture, search, and six skills remain available.
</Update>
@@ -2454,14 +2421,9 @@ Initial release of the Mem0 plugin for Claude Code and Cursor, followed by Codex
<Tab title="Codex">
<Update label="2026-09-18" description="Codex plugin v0.3.2">
<Update label="Unreleased" description="Sidekick availability">
**Improvements:**
- **Telemetry:** `PLUGIN_VERSION` bumped to `0.3.2`. The `mem0-plugin/<version>` wire header and `plugin_version` telemetry field now reflect the fixes from #7322 through #7358 ([#7373](https://github.com/mem0ai/mem0/pull/7373))
- **Telemetry:** Events are no longer delivered twice, no longer lose parked events on flush, and now attribute each event to the plugin that produced it ([#7323](https://github.com/mem0ai/mem0/pull/7323), [#7324](https://github.com/mem0ai/mem0/pull/7324), [#7358](https://github.com/mem0ai/mem0/pull/7358))
**Changes:**
- **Sidekick:** Renames shared tracking to use subagent terminology. Native subagent memory support remains available.
Renames shared tracking to use subagent terminology. Native subagent memory support remains available.
</Update>
@@ -2481,16 +2443,32 @@ Initial release of the Mem0 plugin for Claude Code and Cursor, followed by Codex
</Tab>
<Tab title="OpenCode">
<Tab title="Agent Plugins v1">
<Update label="2026-09-18" description="OpenCode plugin v0.4.0">
<Update label="Unreleased" description="Sidekick availability">
**Changes:**
- **Telemetry:** The PostHog `source` tag changed from the literal `"plugin"` to `OPENCODE_PLUGIN`, and `project_hash` is now salted. Saved PostHog insights filtering on `source = "plugin"` will stop matching new events; historical data is unaffected ([#7322](https://github.com/mem0ai/mem0/pull/7322))
- **Config:** A new `keyFingerprint` key appears in the install-count deduplication logic; installs are now counted once per key rather than on every activation ([#7325](https://github.com/mem0ai/mem0/pull/7325))
Sidekick is available only in Claude Code, not in the portable package.
</Update>
<Update label="2026-09-08" description="Portable Mem0 plugin v0.3.1">
**Added:**
- One portable package at `integrations/mem0-agent-plugin/`, using the Agent Plugins 1.0.0 root `plugin.json`, `mcp.json`, and fixed `skills/` locations.
- Ships a local, read-only `search_memories` server and the six shared memory skills. Uses `PLUGIN_ROOT` for bundled files and `PLUGIN_DATA` for persistent plugin state; all package files remain inside the installable directory.
**Packaging:**
- Generated from the shared Python runtime and skill templates. Builds validate the manifest, MCP configuration, skills, and generated-file consistency.
- Host lifecycle hooks and native Sidekick declarations remain in the native plugin packages; the portable package does not provide automatic lifecycle capture or host-specific subagent isolation. Its bundled remember skill cannot persist a new memory on its own because the portable package has no capture hooks or write tool.
[#7203](https://github.com/mem0ai/mem0/pull/7203)
</Update>
</Tab>
<Tab title="OpenCode">
<Update label="2026-09-08" description="OpenCode plugin v0.3.0">
**Changed:**
@@ -2589,14 +2567,9 @@ Initial release of the Mem0 plugin for Claude Code and Cursor, followed by Codex
<Tab title="Antigravity">
<Update label="2026-09-18" description="Antigravity plugin v0.3.2">
<Update label="Unreleased" description="Sidekick availability">
**Improvements:**
- **Telemetry:** `PLUGIN_VERSION` bumped to `0.3.2`. The `mem0-plugin/<version>` wire header and `plugin_version` telemetry field now reflect the fixes from #7322 through #7358 ([#7373](https://github.com/mem0ai/mem0/pull/7373))
- **Telemetry:** Events are no longer delivered twice, no longer lose parked events on flush, and now attribute each event to the plugin that produced it ([#7323](https://github.com/mem0ai/mem0/pull/7323), [#7324](https://github.com/mem0ai/mem0/pull/7324), [#7358](https://github.com/mem0ai/mem0/pull/7358))
**Changes:**
- **Sidekick:** Removes Sidekick. Memory capture, search, and six skills remain available.
Removes Sidekick. Memory capture, search, and six skills remain available.
</Update>
@@ -2692,14 +2665,9 @@ Existing memories written by the previous versions are not rewritten. If your me
<Tab title="Kimi">
<Update label="2026-09-18" description="Kimi Code plugin v0.3.2">
<Update label="Unreleased" description="Sidekick availability">
**Improvements:**
- **Telemetry:** `PLUGIN_VERSION` bumped to `0.3.2`. The `mem0-plugin/<version>` wire header and `plugin_version` telemetry field now reflect the fixes from #7322 through #7358 ([#7373](https://github.com/mem0ai/mem0/pull/7373))
- **Telemetry:** Events are no longer delivered twice, no longer lose parked events on flush, and now attribute each event to the plugin that produced it ([#7323](https://github.com/mem0ai/mem0/pull/7323), [#7324](https://github.com/mem0ai/mem0/pull/7324), [#7358](https://github.com/mem0ai/mem0/pull/7358))
**Changes:**
- **Sidekick:** Removes Sidekick and its start/stop hooks. Memory capture, recall, and six skills remain available.
Removes Sidekick and its start/stop hooks. Memory capture, recall, and six skills remain available.
</Update>
@@ -2737,14 +2705,6 @@ Existing memories written by the previous versions are not rewritten. If your me
<Tab title="OpenClaw">
<Update label="2026-09-18" description="openclaw-mem0 v1.2.0">
**Changes:**
- **Config:** Added `keyFingerprint` to the config schema for install-count deduplication; installs are now counted once per key rather than on every activation ([#7325](https://github.com/mem0ai/mem0/pull/7325))
- **Telemetry:** Events are no longer delivered twice, and the `plugin_version` field now reflects the plugin that produced the event ([#7323](https://github.com/mem0ai/mem0/pull/7323), [#7324](https://github.com/mem0ai/mem0/pull/7324), [#7358](https://github.com/mem0ai/mem0/pull/7358))
</Update>
<Update label="2026-09-08" description="openclaw-mem0 v1.1.0">
**Changed:**
@@ -3033,13 +2993,6 @@ Existing memories written by the previous versions are not rewritten. If your me
<Tab title="Pi Agent">
<Update label="2026-09-18" description="Pi Agent plugin v0.3.1">
**Improvements:**
- **Telemetry:** Events are no longer delivered twice, no longer lose parked events, and now attribute each event to the plugin that produced it. The `plugin_version` wire field reflects the fixed release ([#7323](https://github.com/mem0ai/mem0/pull/7323), [#7324](https://github.com/mem0ai/mem0/pull/7324), [#7358](https://github.com/mem0ai/mem0/pull/7358))
</Update>
<Update label="2026-09-08" description="Pi Agent plugin v0.3.0">
**Changed:**
@@ -3153,13 +3106,6 @@ Existing memories written by the previous versions are not rewritten. If your me
<Tab title="DeepSeek Harness">
<Update label="2026-09-18" description="deepseek-plugin v0.3.1">
**Improvements:**
- **Telemetry:** Rebuild with the fixed shared telemetry core from `agent-plugin-core`. Events are no longer delivered twice, no longer lose parked events on flush, and now attribute each event to the plugin that produced it ([#7323](https://github.com/mem0ai/mem0/pull/7323), [#7324](https://github.com/mem0ai/mem0/pull/7324), [#7358](https://github.com/mem0ai/mem0/pull/7358))
</Update>
<Update label="2026-09-08" description="deepseek-plugin v0.3.0">
**Added:**
@@ -3204,13 +3150,6 @@ Existing memories written by the previous versions are not rewritten. If your me
<Tab title="Vercel AI SDK">
<Update label="2026-09-18" description="Vercel AI SDK v3.0.3">
**Improvements:**
- **Client:** Inherits the three surface-identity headers (`X-Mem0-Source`, `X-Application`, `X-Mem0-Client`) from the TypeScript SDK bump, so platform calls made through the Vercel AI SDK provider are now correctly attributed ([#7326](https://github.com/mem0ai/mem0/pull/7326))
</Update>
<Update label="2026-08-24" description="Vercel AI SDK v3.0.2">
**Security:**
-30
View File
@@ -172,36 +172,6 @@ claude plugin update mem0@mem0-plugins --scope user
| Sidekick won't start | Must be in a Git repo. Check that your Claude Code version supports plugin agents and worktrees. |
| Remove the plugin | `claude plugin uninstall mem0@mem0-plugins` |
## Telemetry
The plugin sends usage events (which hook ran, timing, result counts, failure
types) so Mem0 can see what's used and what's breaking.
These events are **not anonymous**. When an API key is configured, which
installing the plugin requires, they are sent under your Mem0 account email,
the same way the Python SDK and the CLI attribute theirs. Without a key they
are sent under a random per-machine id.
Each event carries the event name, the plugin version, the harness it ran in,
your OS and Python version, and per-event properties describing what happened:
timings, counts, coarse outcome and failure labels, and which model was
configured. Repository and session identifiers are hashed with a random salt
generated on your machine, so they cannot be linked back to a repository name
or path.
The exact set is enforced in code rather than by this list: every property is
filtered through a denylist of sensitive keys and credential-shaped values are
redacted before anything is sent.
Prompts, memory text, queries, file paths, repository names, and API keys are
never sent.
Turn it off:
```bash
export MEM0_TELEMETRY=false
```
<CardGroup cols={2}>
<Card title="Mem0 MCP Setup" icon="puzzle-piece" href="/platform/mem0-mcp">
Detailed MCP configuration for all clients
+1 -1
View File
@@ -114,7 +114,7 @@ Both tools also accept optional per-call `userId`, `agentId`, and `runId` params
## Telemetry
Writes are tagged `source="DEEPSEEK_HARNESS"` so Mem0 can attribute usage to this integration. Usage events include operation names, durations, result counts, and coarse failure kinds. They are **not anonymous**: when an API key is configured they are sent under your Mem0 account email, the same way the SDK attributes its own. Queries, memory text, entity IDs, and API keys are never included. Set `MEM0_TELEMETRY=false` to opt out.
Writes are tagged `source="DEEPSEEK_HARNESS"` so Mem0 can attribute usage to this integration. Anonymous usage events include operation names, durations, result counts, and coarse failure kinds. Queries, memory text, entity IDs, and API keys are never included. Set `MEM0_TELEMETRY=false` to opt out.
<Note>
This plugin is a developer preview and tracks the evolving DeepSeek Harness plugin API.
+49 -6
View File
@@ -11,6 +11,25 @@ You can run Mem0 in three ways:
- **Self-hosted server mode**: point the plugin at a Mem0 server you run yourself (the Docker-shipped server). The plugin only talks HTTP to your server.
- **OSS mode**: run Mem0 in-process with your own LLM, embedder, and vector store. No Mem0 server required.
## Requirements and compatibility
Use Python 3.11+ and a recent Hermes release. The host contract was checked against
Hermes `v0.21.3` (`v2026.9.14`) and main at
`c62bd9f2078a946108f1c9d9b24bf118963277ef`. Earlier host versions have not been validated.
<Warning>
Both checked Hermes revisions still bundle Mem0. Hermes gives bundled memory providers
precedence over a same-named user plugin. Installing or enabling this plugin does not
replace that bundled copy. Use the isolated preview instructions in the
[plugin README](https://github.com/mem0ai/mem0/tree/main/integrations/hermes-plugin#local-worktree-preview)
to try this version in a separate Hermes checkout. Once Hermes removes its bundled Mem0
provider, the user-installed copy loads directly.
</Warning>
Keep `memory.provider: mem0`, `$HERMES_HOME/mem0.json`, and your existing `MEM0_*` variables.
There is no memory migration. An explicitly configured user ID continues to share memories
across hosts; otherwise the provider uses the gateway user ID, then `hermes-user`.
## How It Works
Hermes runs a built-in memory system (file-based `MEMORY.md` and `USER.md`) alongside one external provider. When Mem0 is active, it works additively with the built-in system at two points in every conversation turn.
@@ -21,7 +40,9 @@ When you send a message, Hermes searches your stored memories for the current qu
### 2. Background fact extraction (sync)
Once the model finishes, Hermes sends the `(user message, assistant response)` pair to Mem0 in a background thread. Mem0 extracts facts automatically (for example, "user prefers Python" or "user works at Acme Corp"), so you never have to tell it what to remember. Each write is tagged with the gateway channel it came from.
Once the model finishes, Hermes sends the `(user message, assistant response)` pair to Mem0 in a background thread. Mem0 extracts facts automatically (for example, "user prefers Python" or "user works at Acme Corp"), so you never have to tell it what to remember. Each write is tagged with the gateway channel it came from and the Hermes session as top-level `run_id`. Recall remains scoped to the user across sessions.
The plugin uses the shared `agent-plugin-core` redaction and token-aware batching. Full non-empty user and assistant text is preserved after known-secret redaction; oversized messages are split across requests instead of truncated. Raw tool results and the full historical transcript are not captured. Pattern-based redaction cannot recognize every possible secret.
## Agent Tools
@@ -30,7 +51,7 @@ When Mem0 is active, the model gets four tools it can call during a conversation
| Tool | Description | Parameters |
|------|-------------|------------|
| `mem0_search` | Semantic search by meaning, ranked by relevance | `query` (required), `top_k` (default 10, max 50), `rerank` (default `false`, Platform mode only) |
| `mem0_add` | Store a fact verbatim, with no LLM extraction | `content` (required) |
| `mem0_add` | Store a fact after known-secret redaction, with no LLM extraction | `content` (required) |
| `mem0_update` | Update a memory's text by ID | `memory_id`, `text` (both required) |
| `mem0_delete` | Delete a memory by ID | `memory_id` (required) |
@@ -43,7 +64,27 @@ curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scri
source ~/.bashrc
```
The `mem0ai` package is installed automatically when you enable the Mem0 provider, so there is no manual pip step. OSS providers may need extra packages (for example `qdrant-client`, `psycopg2-binary`, or `ollama`), which the setup flow installs for you when you pick them.
After this integration is published to the Mem0 repository:
```bash
hermes plugins install mem0ai/mem0/integrations/hermes-plugin
hermes plugins enable mem0
hermes memory setup
hermes memory status
```
For unpublished worktree changes, use the README's local installation instructions instead.
The bundled-provider precedence described above applies to both installation methods.
Recent Hermes development versions install the plugin's declared dependencies automatically.
On Hermes `v0.21.3`, install them into the environment that runs Hermes:
```bash
HERMES_REPO=/path/to/hermes-agent
uv pip install --python "$HERMES_REPO/.venv/bin/python" 'mem0ai>=2.0.10,<3' 'httpx>=0.27,<1'
```
OSS providers may need extra packages such as `qdrant-client`, `psycopg2-binary`, or `ollama`, which the setup flow installs when you select them.
## Platform Setup
@@ -196,9 +237,10 @@ Behavioral settings live in `~/.hermes/mem0.json` and are written for you by `he
| `mode` | `platform` | `platform` (Mem0 Cloud) or `oss` (self-managed, in-process). Self-hosted server routing is set via `host` |
| `host` | none | Self-hosted Mem0 server URL. When set, the plugin talks HTTP to your server instead of the cloud |
| `api_key` | none | Mem0 Platform API key, or the admin key of a self-hosted server. Stored in `.env` as `MEM0_API_KEY` |
| `user_id` | `hermes-user` | Identifier that scopes memories. See cross-channel behavior below |
| `user_id` | Gateway user ID, then `hermes-user` | Identifier that scopes memories. See cross-channel behavior below |
| `agent_id` | `hermes` | Agent identifier attached to writes |
| `rerank` | `false` | Rerank search results for relevance (Platform mode only) |
| `sync_max_chars` | Uncapped for platform/HTTP; `450` for OSS | Positive maximum per chunk; longer text is split without dropping its tail. `0` disables the character cap |
### Cross-channel memories
@@ -213,7 +255,8 @@ Either way, every write is tagged with `metadata.channel` (for example `telegram
- **Circuit breaker**: if Mem0 fails five times in a row, Hermes pauses calls for two minutes, then retries. The agent keeps working without memory during that window. Expected client errors, like a 404 on a missing memory id, do not count toward tripping the breaker.
- **Non-blocking**: fact extraction runs in a background daemon thread, and current-turn recall waits at most 3 seconds, so a slow or failed call never blocks your conversation.
- **Thread-safe**: the client uses lazy initialization with locking, and the background sync and recall threads are guarded so concurrent gateway messages cannot produce duplicate memories.
- **Capture queue**: overlapping turns are queued in memory instead of skipped while an earlier turn is syncing. Failed requests are logged without durable retries. Shutdown waits at most five seconds; a process exit can lose pending turns.
- **Existing collections**: an OSS embedding-dimension mismatch fails initialization without deleting vectors. Restore the original embedding configuration or select a new collection name.
## Troubleshooting
@@ -246,7 +289,7 @@ curl http://localhost:11434/api/tags
### Memories not appearing
- `mem0_add` stores text verbatim with no extraction. Ordinary conversation turns are extracted automatically by the background sync.
- `mem0_add` stores text after known-secret redaction with no extraction. Ordinary conversation turns are extracted automatically by the background sync.
- Search is semantic, so try a broader query.
- Confirm `user_id` is the same across sessions (check `~/.hermes/mem0.json`).
+1 -3
View File
@@ -481,9 +481,7 @@ Plugin config is stored in `~/.openclaw/openclaw.json` with file permissions `0o
### Telemetry
Usage telemetry (PostHog) is enabled by default to help improve the plugin. No conversation content or memory values are included, only event counts (recall, capture, tool usage, CLI commands).
These events are **not anonymous**. OpenClaw does not send your account email the way the SDK does, but it does send an unsalted SHA-256 hash of it, falling back to a hash of the API key and then to a random per-machine id. Mem0 holds the email the hash is derived from, so the hash identifies your account rather than concealing it. The first run that resolves an account also emits a PostHog `$identify`, which permanently merges any earlier random id into that identity.
Anonymous usage telemetry (PostHog) is enabled by default to help improve the plugin. No conversation content or memory values are included, only event counts (recall, capture, tool usage, CLI commands).
To opt out, set the environment variable:
+1 -1
View File
@@ -256,7 +256,7 @@ If the user is on a pre-current major (Python < 2, TS < 3, or a Platform call st
- [Agno](https://docs.mem0.ai/integrations/agno) [Platform]: Use when the user is on Agno.
- [Camel AI](https://docs.mem0.ai/integrations/camel-ai) [Both]: Use when the user is on Camel AI.
- [ChatDev](https://docs.mem0.ai/integrations/chatdev) [Platform]: Use when the user is on ChatDev.
- [Hermes](https://docs.mem0.ai/integrations/hermes) [Both]: Use when the user is on Hermes.
- [Hermes Agent](https://docs.mem0.ai/integrations/hermes) [Both]: Use when adding the native Mem0 memory provider to Hermes Agent, preserving existing configuration, or checking bundled-provider precedence.
- [Pi Agent](https://docs.mem0.ai/integrations/pi-agent) [Platform]: Use when adding automatic capture, prompt recall, scoped memory, and six memory commands to Pi Agent.
- [DeepSeek Harness](https://docs.mem0.ai/integrations/deepseek-plugin) [Platform]: Use when adding automatic recall, completed-turn capture, and native search/add tools to DeepSeek Harness.
- [OpenAI Agents SDK](https://docs.mem0.ai/integrations/openai-agents-sdk) [Platform]: Use when the user is on the OpenAI Agents SDK.
+2 -43
View File
@@ -7,6 +7,7 @@ Agent and editor integrations. Most packages are self-contained; coding-agent pl
| `vercel-ai-sdk/` | `@mem0/vercel-ai-provider` | tsup (CJS+ESM) | ESLint + Prettier | jest + vitest (edge/node) |
| `openclaw/` | `@mem0/openclaw-mem0` | tsup (ESM) | none | vitest |
| `agent-plugin-core/` | Shared Python/TypeScript behavior, skill templates, builds, and conformance | Python build script | ruff + tsc | pytest + node:test |
| `hermes-plugin/` | Native Hermes memory provider; generated shared redaction and batching | Python build script | ruff | pytest with `--confcutdir=integrations/hermes-plugin/tests` |
| `mem0-agent-plugin/` | One portable Agent Plugins v1 package | Python | ruff | shared conformance |
| `claude-code-plugin/`, `cursor-plugin/`, `codex-plugin/`, `kimi-plugin/`, `antigravity-plugin/` | Self-contained native plugins generated from the shared Python core | Python | ruff | pytest |
| `opencode-plugin/` | `@mem0/opencode-plugin` (Bun/TypeScript) | tsup (via Bun) | tsc | bun test |
@@ -43,54 +44,13 @@ Run the type check after every TypeScript change: `pnpm run typecheck` or `tsc -
- **`vercel-ai-sdk/`** wraps the Vercel AI SDK through a `createMem0` provider. Integrations for AI-SDK repos go through this wrapper, not raw `MemoryClient`.
- **`agent-plugin-core/`** owns the shared Python memory runtime, TypeScript lifecycle utilities, skill templates, builds, and conformance runner. Claude Code is the behavioral source of truth. Native manifests and adapters live in sibling plugin directories; do not hand-edit their generated `core/` or `skills/` trees. Build and validation details are in [`agent-plugin-core/README.md`](agent-plugin-core/README.md).
- **`hermes-plugin/`** preserves the upstream Hermes memory-provider API, setup, and cloud/HTTP/OSS backends. Only `core/message_utils.py` is generated; no MCP server or generic skills are installed. Its offline tests use a separate pytest invocation with `--confcutdir=integrations/hermes-plugin/tests` to avoid importing the host entry point during collection.
- **`opencode-plugin/`** is a Bun/TypeScript plugin for OpenCode (`@mem0/opencode-plugin` on npm). It registers Mem0 memory tools as an OpenCode plugin with its own skills and telemetry.
- **`openclaw/`**, **`pi-agent-plugin/`**, **`deepseek-plugin/`** are editor and agent plugins with the same shape. `deepseek-plugin/` registers Mem0 search/add tools as a native DeepSeek Harness (Cordis) plugin.
- **`n8n-nodes-mem0/`** is an n8n community node: add, search, get, update, delete.
- **`zapier-mem0/`** is a Zapier Platform CLI app: add, search, get, delete. It deploys to Zapier, not npm, so it is **not** in the release router. Deploy it with `gh workflow run zapier-mem0-cd.yml --ref main` (needs the `ZAPIER_DEPLOY_KEY` secret).
- **`mem0-strands/`** is a native Strands `MemoryStore` (Python, published to PyPI as `mem0-strands`). It plugs into the Strands `MemoryManager` for automatic recall and server-side extraction, over the hosted Mem0 platform or self-hosted Mem0 OSS. The package lives under `mem0-strands/python/`.
## Surface attribution
Every integration tells the Mem0 platform which surface it is. Three headers,
and the rules on them are what keep one layer from erasing another:
| Header | Carries | Rule |
|--------|---------|------|
| `X-Mem0-Source` | one canonical source value | **set-once** — write only if absent |
| `X-Application` | the host app it runs inside | **set-once** — write only if absent |
| `X-Mem0-Client` | `name/version`, outermost first | **append-only** — add yourself, never replace |
Set-once means check-then-set, never assignment. An integration that wraps the
SDK is the outermost layer and sets the source; the SDK underneath defers to it.
Assignment is exactly how every agent plugin came to be indistinguishable from
every other one at the platform.
How to declare it from an integration, in order of preference:
1. Send the headers yourself, if you make the HTTP call directly.
2. Pass `source` in the call options, if you go through an SDK.
3. Set `MEM0_SOURCE` / `MEM0_APPLICATION` / `MEM0_CLIENT_STACK` in the
environment before constructing the client. The SDKs read these and defer to
anything already present.
Append-only applies where a stack can actually form: an SDK handed a client that
already carries `X-Mem0-Client` appends itself rather than replacing. An SDK
constructed with no outer context simply reports itself, which is correct — it
is the outermost layer in that process.
The backend recognizes a fixed list of source values and buckets everything else
into `OTHERS`. A new value has to land in the platform's `EventSource` enum, so
do not invent one without that change going in too.
`X-Application` is allowlisted the same way, and this one has a rule of its own:
**omit the header when you do not know the host.** A value outside the allowlist
is discarded server-side, so guessing produces an event that claims an
attribution we do not actually have. The portable bundle is the case that
matters. It runs in whatever editor a user drops it into, so its build leaves
`PLATFORM_APPLICATION` empty and `memory_core` sends no header at all, while the
native bundles each name the host they were generated for. If you add a build
target, decide which of those two it is.
## Adding an integration
1. For a native coding-agent host, add `integrations/<name>-plugin/` with `plugin-build.json`, its manifest, and a thin adapter, then generate its shared runtime. Portable clients use the single `mem0-agent-plugin/` package. Independent TypeScript integrations stay self-contained and import shared lifecycle behavior from `agent-plugin-core/typescript/`.
@@ -101,4 +61,3 @@ target, decide which of those two it is.
5. If it is a Claude Code or editor marketplace plugin, register the generated native bundle path in the applicable marketplace files. Preserve the existing public plugin name.
6. Document it under `docs/integrations/` and add the page to `docs/docs.json` and `docs/llms.txt`.
7. Add rows to the table above and to the CI/CD tables in [`../.github/AGENTS.md`](../.github/AGENTS.md).
8. Send the three headers in [Surface attribution](#surface-attribution), and land the matching `EventSource` value on the platform in the same week. Until it exists, your traffic reports as `OTHERS`.
+9 -5
View File
@@ -18,10 +18,11 @@ integrations/
├── cursor-plugin/ # Native Cursor package and adapter
├── codex-plugin/ # Native Codex package and adapter
├── kimi-plugin/ # Native Kimi package and adapter
└── antigravity-plugin/ # Native Antigravity package and adapter
├── antigravity-plugin/ # Native Antigravity package and adapter
└── hermes-plugin/ # Native Hermes provider; shared message helpers only
```
Each native directory owns only its manifest, native hooks or adapter, tests, and `plugin-build.json`. Its `core/` and `skills/` directories are generated from this module. They are committed because clients install a self-contained plugin directory and the Agent Plugins specification forbids package files from resolving outside the plugin root.
Each native directory owns its manifest, native hooks or adapter, tests, and `plugin-build.json`. Hermes also retains its upstream setup wizard and backend adapters. Its `core/` and `skills/` directories are generated from this module. They are committed because clients install a self-contained plugin directory and the Agent Plugins specification forbids package files from resolving outside the plugin root.
Sidekick belongs only to Claude Code. Its agent definition is in `claude-code-plugin/agents/sidekick.md`; its hooks are in `claude-code-plugin/adapters/claude/hook.py`. Other plugins must not register Sidekick. The shared core handles memory and native subagent tracking for Claude Code and Codex.
@@ -29,7 +30,7 @@ TypeScript integrations (`openclaw`, `opencode-plugin`, `pi-agent-plugin`, and `
## Shared memory behavior
The six Python packages use the same `search_memories` MCP tool and six skill templates. Native hooks collect conversations and flush them to Mem0 in the background. The portable package uses the Agent Plugins v1 layout so compatible hosts can load its MCP server and skills. It has no lifecycle hooks or flush worker; its bundled `remember` skill assumes automatic capture and cannot save a memory on its own.
The six MCP-based Python packages use the same `search_memories` MCP tool and six skill templates. Native hooks collect conversations and flush them to Mem0 in the background. The portable package uses the Agent Plugins v1 layout so compatible hosts can load its MCP server and skills. It has no lifecycle hooks or flush worker; its bundled `remember` skill assumes automatic capture and cannot save a memory on its own.
Python search accepts `query`, `top_k`, `category`, `scope`, and optional `run_id`:
@@ -49,6 +50,8 @@ TypeScript hosts reuse redaction and lifecycle utilities but retain their own to
For installation, follow the host guides: [Claude Code](../../docs/integrations/claude-code.mdx), [Cursor](../../docs/integrations/cursor.mdx), [Codex](../../docs/integrations/codex.mdx), [Kimi](../../docs/integrations/kimi.mdx), and [Antigravity](../../docs/integrations/antigravity.mdx).
Hermes bundles only `python/message_utils.py`, sharing the same secret redaction and lossless token batching. It retains Hermes user-wide recall, four native tools, three backend modes, and legacy configuration. Completed turns are queued in memory, with session `run_id` on writes; network failures are logged and are not durably retried. Explicit `sync_max_chars` values split text instead of truncating it. It does not install the generic MCP skills, repository scopes, or Sidekick.
## Build and verify
From the repository root:
@@ -58,7 +61,7 @@ python3.11 -m venv /tmp/mem0-agent-plugins
/tmp/mem0-agent-plugins/bin/pip install \
-r integrations/agent-plugin-core/requirements-dev.txt
for host in claude-code cursor codex kimi antigravity; do
for host in claude-code cursor codex kimi antigravity hermes; do
/tmp/mem0-agent-plugins/bin/python \
integrations/agent-plugin-core/build/build.py "$host" \
--kind native --check
@@ -99,7 +102,7 @@ Do not put a real key in source files, command history shared with others, or pu
For another native Python host:
1. Add `integrations/<host>-plugin/` with its native manifest and the smallest adapter that translates host events.
2. Add `plugin-build.json` declaring the plugin-root variable and runtime files.
2. Add `plugin-build.json` declaring the plugin-root variable and runtime files. Native providers can select `native.pythonFiles` and set `native.skills: false` when they expose host-native tools instead of MCP.
3. Add one adapter contract test.
4. Register the host in `build/build.py` and `conformance/run.py`.
5. Run `--sync`, `--check`, and the conformance command above.
@@ -117,6 +120,7 @@ For a TypeScript host, import the shared lifecycle modules directly and keep onl
| Codex | Native prompt and final-response fields | Structured failure indicators when present; otherwise unknown | Parent context; native agent ID |
| Kimi | Prompt hooks and completed v2 wire output | Native success/failure hooks | No plugin subagent hooks or agent declaration |
| Antigravity | Incremental completed transcript messages, including later prompts | Native tool errors | No plugin subagent hooks or agent declaration |
| Hermes | Completed turns via `sync_turn`, split without truncation | No separate tool-evidence capture | No plugin subagent hooks or agent declaration |
| Portable v1 | Explicit memory skills | No native lifecycle hooks | No native subagent declaration |
Python status uses `subagent_runs` and `last_subagent`. Legacy SQLite names and event handling keep existing records and running workers compatible.
+24 -43
View File
@@ -25,6 +25,7 @@ NATIVE_PLUGINS = {
"codex": INTEGRATIONS_ROOT / "codex-plugin",
"kimi": INTEGRATIONS_ROOT / "kimi-plugin",
"antigravity": INTEGRATIONS_ROOT / "antigravity-plugin",
"hermes": INTEGRATIONS_ROOT / "hermes-plugin",
}
PROTECTED_OUTPUTS = {
REPOSITORY_ROOT,
@@ -81,38 +82,6 @@ def replace_output(staged: Path, output: Path) -> Path:
return output
def _render_harness_id(host: str, *, portable: bool = False) -> str:
"""Emit core/_harness_id.py for one host.
Carries both vocabularies from a single definition: the PostHog `source` tag
and the platform's X-Mem0-Source / X-Application pair. Keeping them together
is what stops the two from drifting into separate vocabularies for the same
thing.
The portable bundle runs in whatever editor a user drops it into, so it does
not know its host and must not guess one. HARNESS_ID stays "coding-agent",
which is true and useful for grouping in PostHog, but PLATFORM_APPLICATION is
left empty: X-Application names a real host app, is checked against an
allowlist server-side, and a value that is always discarded is worse than no
value -- it reads like an attribution we have and do not.
"""
tag = host.upper().replace("-", "_") + "_PLUGIN"
application = "" if portable else host
return (
'"""Generated by integrations/agent-plugin-core/build/build.py. Do not edit."""\n'
"\n"
f'HARNESS_ID = "{host}"\n'
f'SOURCE_TAG = "{tag}"\n'
"\n"
"# Platform-side vocabulary (mem0_event.source + X-Application). The whole\n"
"# plugin family is one source; which editor it runs in is the application.\n"
"# An empty application means the host is unknown, and memory_core omits\n"
"# the header entirely rather than sending a placeholder.\n"
'PLATFORM_SOURCE = "MEM0_PLUGIN"\n'
f'PLATFORM_APPLICATION = "{application}"\n'
)
def _bundle_python(
staged: Path,
host: str,
@@ -120,19 +89,26 @@ def _bundle_python(
*,
plugin_data: str = "",
portable: bool = False,
python_files: list[str] | None = None,
skills: bool = True,
) -> None:
core = staged / "core"
core.mkdir()
for source in sorted((CORE_ROOT / "python").glob("*.py")):
if portable and source.name in {"flush_worker.py", "hook_runner.py"}:
continue
shutil.copy2(source, core / source.name)
# Generated per host so identity does not depend on an entrypoint remembering
# to call telemetry.init(). mcp_server.py and the detached telemetry.py sender
# never did, which is how MCP searches reported harness=generic and every
# batch they drained was labelled MEM0_PLUGIN regardless of the real host.
(core / "_harness_id.py").write_text(_render_harness_id(host, portable=portable), encoding="utf-8")
if python_files is None:
python_files = sorted(source.name for source in (CORE_ROOT / "python").glob("*.py"))
if not isinstance(python_files, list) or any(
not isinstance(name, str) or Path(name).name != name or Path(name).suffix != ".py" for name in python_files
):
raise ValueError("pythonFiles must be a list of Python source filenames")
_copy_declared_files(
core,
CORE_ROOT / "python",
{name: name for name in python_files if not portable or name not in {"flush_worker.py", "hook_runner.py"}},
)
if not isinstance(skills, bool):
raise ValueError("native skills must be a boolean")
if not skills:
return
values = {
"PLUGIN_ROOT": plugin_root,
@@ -188,6 +164,8 @@ def _build_native(host: str, source_root: Path, staged: Path, descriptor: dict)
host,
native["pluginRoot"],
plugin_data=str(native.get("pluginData") or ""),
python_files=native.get("pythonFiles"),
skills=native.get("skills", True),
)
_copy_declared_files(staged, source_root, native.get("files", {}))
@@ -259,7 +237,10 @@ def sync_generated(host: str, kind: str) -> Path:
with tempfile.TemporaryDirectory(prefix=f"mem0-sync-{host}-") as temporary:
generated = build(host, kind, Path(temporary) / "bundle")
for directory in ("core", "skills"):
replace_output(generated / directory, target / directory)
if (generated / directory).is_dir():
replace_output(generated / directory, target / directory)
elif (target / directory).exists():
shutil.rmtree(target / directory)
return target
@@ -13,10 +13,9 @@ import time
from pathlib import Path
from typing import Any
CORE_ROOT = Path(__file__).resolve().parents[1]
REPOSITORY_ROOT = CORE_ROOT.parents[1]
PYTHON_HOSTS = ("claude-code", "cursor", "codex", "kimi", "antigravity")
PYTHON_HOSTS = ("claude-code", "cursor", "codex", "kimi", "antigravity", "hermes")
GROUPS = (
"python-bundles",
"python-tests",
@@ -30,9 +29,14 @@ LIVE_GROUP = "live-platform"
sys.path.insert(0, str(CORE_ROOT))
sys.path.insert(0, str(CORE_ROOT / "python"))
from memory_core import redact # noqa: E402
from build.build import build # noqa: E402
from conformance.artifacts import TYPESCRIPT_ARTIFACTS, verify_artifact as _typescript_artifact_check # noqa: E402
from conformance.artifacts import ( # noqa: E402
TYPESCRIPT_ARTIFACTS,
)
from conformance.artifacts import ( # noqa: E402
verify_artifact as _typescript_artifact_check,
)
from message_utils import redact # noqa: E402
def _package_directories() -> dict[str, Path]:
@@ -61,6 +65,14 @@ def _runtime_commands() -> dict[str, list[list[str]]]:
"-q",
"--ignore=integrations/agent-plugin-core/tests/test_conformance.py",
"--ignore=integrations/claude-code-plugin/tests/integration",
],
[
sys.executable,
"-m",
"pytest",
"integrations/hermes-plugin/tests",
"--confcutdir=integrations/hermes-plugin/tests",
"-q",
]
],
"typescript-core": [["pnpm", "test"], ["pnpm", "typecheck"]],
@@ -290,11 +290,6 @@ def run(
if args.plugin_data_dir:
os.environ[data_dir_env] = args.plugin_data_dir
# Snapshot BEFORE anything writes to the data dir: cache_plugin_api_key
# writes `api-key` and EvidenceStore creates `evidence.sqlite3`, so asking
# after them always saw content and every fresh install reported an upgrade.
data_dir_was_empty = telemetry.data_dir_was_empty()
cache_plugin_api_key()
if args.action == "session-start":
clear_stale_api_key_cache()
@@ -310,19 +305,8 @@ def run(
return 0
if args.action == "session-start":
# Claims the marker atomically and says which event to record, so a
# second session starting alongside this one cannot record it too.
first_event = telemetry.claim_install(was_empty=data_dir_was_empty)
if first_event == "install":
if telemetry.is_first_run():
telemetry.record("install")
elif first_event == "upgrade":
# First run after a build that never wrote the marker; the
# predecessor version was never recorded anywhere.
telemetry.record("upgrade", from_version="pre-0.3")
else:
previous = telemetry.claim_version_change()
if previous:
telemetry.record("upgrade", from_version=previous)
recovered = recover_pending_handoffs()
record_session_start(store, hook_input)
if recovered:
@@ -11,7 +11,6 @@ from __future__ import annotations
import functools
import hashlib
import json
import math
import os
import re
import sqlite3
@@ -27,9 +26,17 @@ from pathlib import Path
from typing import Any, Iterable
import telemetry
from message_utils import MAX_EXTRACTION_INPUT_TOKENS as MAX_EXTRACTION_INPUT_TOKENS
from message_utils import SECRET_PATTERNS as SECRET_PATTERNS
from message_utils import _estimated_tokens as _estimated_tokens
from message_utils import _is_agent_assignment as _is_agent_assignment
from message_utils import _is_agent_response as _is_agent_response
from message_utils import _message_tokens as _message_tokens
from message_utils import extraction_message_batches as extraction_message_batches
from message_utils import redact as redact
DEFAULT_API_URL = "https://api.mem0.ai"
PLUGIN_VERSION = "0.3.2"
PLUGIN_VERSION = "0.3.1"
_harness_name: str = "generic"
_harness_env_prefix: str = "MEM0_PLUGIN"
@@ -66,7 +73,6 @@ CHECKPOINT_EXCHANGES = 5
CHECKPOINT_MESSAGES = 10
CHECKPOINT_SOURCE_CHARS = 40000
DEFAULT_MAX_CONTEXT_CHARS = 4000
MAX_EXTRACTION_INPUT_TOKENS = 24000
MAX_FLUSH_ATTEMPTS = 5
FORGET_PAGE_SIZE = 100
FORGET_MAX_PAGES = 50
@@ -139,48 +145,11 @@ BUILD_COMMAND_RE = re.compile(
re.IGNORECASE,
)
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(
r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"
),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r'|(?:access|refresh|session)[_-]?token|token|authorization|credential'
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def redact(value: Any) -> str:
text = (
value
if isinstance(value, str)
else json.dumps(value, ensure_ascii=False, default=str)
)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def bounded(value: Any, limit: int) -> str:
text = redact(value).strip()
if len(text) <= limit:
@@ -1706,128 +1675,6 @@ def build_extraction_messages(structured: dict[str, Any]) -> list[dict[str, str]
return messages
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if (
_is_agent_assignment(message)
and index + 1 < len(exchange)
and _is_agent_response(exchange[index + 1])
):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
# Platform surface attribution. Read from the generated per-host module so a new
# entrypoint is correct without remembering to configure anything.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
except ImportError:
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
def platform_headers(key: str) -> dict[str, str]:
"""Auth plus the three surface-identity headers.
X-Mem0-Source and X-Application are set-once by contract: this is the
outermost layer, so it sets them, and nothing below may overwrite them.
X-Mem0-Client is append-only — anything downstream adds itself to the tail.
"""
headers = {
"Authorization": f"Token {key}",
"Content-Type": "application/json",
"X-Mem0-Source": _PLATFORM_SOURCE,
"X-Mem0-Client": f"mem0-plugin/{PLUGIN_VERSION}",
}
if _PLATFORM_APPLICATION:
headers["X-Application"] = _PLATFORM_APPLICATION
return headers
def _request_json(
url: str, key: str, payload: dict[str, Any], timeout: float
) -> tuple[dict[str, Any] | list[Any], int, int]:
@@ -1835,7 +1682,7 @@ def _request_json(
request = urllib.request.Request(
url,
data=raw,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -1862,7 +1709,7 @@ def _get_json(
) -> tuple[dict[str, Any] | list[Any], int]:
request = urllib.request.Request(
url,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="GET",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -2008,13 +1855,6 @@ def flush_session(
"user_id": write_user,
"app_id": repo.app_id,
"run_id": session_id,
# Top level, not metadata: the backend reads `source` from the body or
# the query string, never from metadata, which is where this used to
# sit. The X-Mem0-Source header is also read, but only from the
# platform release that ships alongside this change, so the body value
# is what makes attribution work on both. The harness tag stays in
# metadata as hook provenance.
"source": _PLATFORM_SOURCE,
"metadata": {**metadata, "author": write_user, "dirs": directory_chain(repo)},
"agent_custom_instructions": PROJECT_MEMORY_INSTRUCTIONS,
"custom_instructions": PERSONAL_MEMORY_INSTRUCTIONS,
@@ -2558,7 +2398,7 @@ def _collect_memory_ids(
def _delete_memory(api_url: str, key: str, memory_id: str) -> bool:
request = urllib.request.Request(
f"{api_url}/v1/memories/{urllib.parse.quote(memory_id)}/",
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="DELETE",
)
try:
@@ -0,0 +1,127 @@
"""Shared, host-independent redaction and lossless extraction batching."""
from __future__ import annotations
import json
import math
import re
from typing import Any
MAX_EXTRACTION_INPUT_TOKENS = 24000
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r"|(?:access|refresh|session)[_-]?token|token|authorization|credential"
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def redact(value: Any) -> str:
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, default=str)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if _is_agent_assignment(message) and index + 1 < len(exchange) and _is_agent_response(exchange[index + 1]):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
@@ -1,9 +1,5 @@
#!/usr/bin/env python3
"""Usage telemetry for Mem0 agent plugins.
Events are linked to your Mem0 account email when an API key is configured, and
to a random per-machine id otherwise. Not anonymous — the Python SDK and CLI
attribute the same way.
"""Anonymous usage telemetry for Mem0 agent plugins.
Hooks run on a 3-6 second budget and fire on every tool call, so recording never
touches the network: `record` appends one JSON line to a local spool and returns.
@@ -13,8 +9,7 @@ started once per session and again from the flush worker that is already detache
Pure stdlib, matching the rest of the plugin. Opt out with MEM0_TELEMETRY=false.
Never sends prompts, memory text, queries, file paths, repository names, or API
keys: only event names, durations, counts, coarse outcomes, and repo/session
identifiers hashed with a random per-install salt.
keys: only event names, durations, counts, coarse outcomes, and salted hashes.
"""
from __future__ import annotations
@@ -34,24 +29,8 @@ from typing import Any
import memory_core
# Seeded from the per-host module the build generates into core/. Two processes
# in this pipeline never call init() — mcp_server.py, and the detached
# `python3 telemetry.py` sender that spawn_flush() starts — so a module default
# was what every one of their events got labelled with.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import HARNESS_ID as _DEFAULT_HARNESS
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
from _harness_id import SOURCE_TAG as _DEFAULT_SOURCE_TAG
except ImportError:
_DEFAULT_HARNESS = "generic"
_DEFAULT_SOURCE_TAG = "MEM0_PLUGIN"
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
_salt_cache: str = ""
_harness: str = _DEFAULT_HARNESS
_source_tag: str = _DEFAULT_SOURCE_TAG
_harness: str = "generic"
_source_tag: str = "MEM0_PLUGIN"
_PRIVATE_KEYS = {
"apikey",
"authorization",
@@ -77,19 +56,10 @@ _PRIVATE_KEYS = {
}
def init(harness: str = "", source_tag: str = "") -> None:
"""Override the generated identity. Optional — core/_harness_id.py is the default.
The fallback shape matches memory_core.configure_harness's (``<HOST>_PLUGIN``).
It used to be ``MEM0_<HOST>_PLUGIN`` here and ``<host>_plugin`` there, which
meant one plugin could emit three different source values depending on which
process happened to send the batch.
"""
def init(harness: str = "generic", source_tag: str = "") -> None:
global _harness, _source_tag
_harness = harness or _DEFAULT_HARNESS
_source_tag = source_tag or (
f"{_harness.upper().replace('-', '_')}_PLUGIN" if harness else _DEFAULT_SOURCE_TAG
)
_harness = harness
_source_tag = source_tag or f"MEM0_{harness.upper().replace('-', '_')}_PLUGIN"
POSTHOG_API_KEY = "phc_hgJkUVJFYtmaJqrvf6CYN67TIQ8yhXAkWzUn9AMU4yX"
POSTHOG_CAPTURE_URL = "https://us.i.posthog.com/i/v0/e/"
@@ -100,16 +70,6 @@ BATCH_SIZE = 100
SEND_TIMEOUT = 5
CLAIM_STALE_SECONDS = 120
CLAIM_EXPIRY_SECONDS = 7 * 24 * 60 * 60
# A batch is only discarded once it has genuinely been retried this many times.
MAX_CLAIM_ATTEMPTS = 3
# Parked claims drained per run, after the live spool. Bounded so a long backlog
# cannot turn one flush into an unbounded send loop.
MAX_PARKED_PER_RUN = 3
# Added to the wait before a released claim becomes reclaimable, per attempt
# already spent. Releasing straight to "reclaimable now" let two senders burn the
# whole budget within seconds of one another on a single momentary failure, and
# discard a batch a retry a minute later would have delivered.
RETRY_COOLDOWN_SECONDS = 60
def is_enabled() -> bool:
@@ -123,126 +83,9 @@ def is_enabled() -> bool:
def _digest(value: str, length: int = 16) -> str:
"""Unsalted digest. Only for values that are already secrets (API keys)."""
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:length]
def _salt_path() -> Path:
return memory_core.data_dir() / "telemetry-salt"
def _install_salt() -> str:
"""Random per-install salt, created once and memoized for the process.
Deliberately its own file, claimed with O_CREAT|O_EXCL, rather than a key in
the identity file. Three reasons, all of which produced wrong data when this
lived in the identity dict:
- Hooks are short-lived separate processes firing on every tool call, and
people run more than one agent window. A read-modify-write would let each
process mint its own salt, so one repository would hash several ways in the
window before a writer won.
- resolve_distinct_id holds a copy of the identity dict across a network call
to /v1/ping/, so whichever write landed second erased the other's key —
losing either the salt (repo_hash changes mid-stream) or the email (a
second $identify, splitting the person).
- Touching the identity file from record() would create it, and is_first_run
keys off that file, so recording an event would silently suppress the
install event.
Published atomically, and there is deliberately no derived fallback. Creating
the file with O_CREAT|O_EXCL and then writing into it leaves a window where
the file exists and is empty, and a concurrent hook that reads it in that
window gets nothing. Falling back to a digest of the path would hand that
process a salt an attacker can compute, memoized for its whole run, which is
the privacy control this function exists to provide silently turning itself
off under load. The salt is written to a private temp file first and linked
into place, so the name either does not exist or already has the full value.
Returns "" when it genuinely cannot persist. Callers omit the hash entirely
rather than emit an unsalted one.
"""
global _salt_cache
if _salt_cache:
return _salt_cache
path = _salt_path()
# Read before writing. Hooks are separate processes firing on every tool
# call, so all but the first find the salt already published; going straight
# to create-fsync-link-unlink meant every one of them paid an fsync to
# discover that, on a path whose whole promise is appending a line and
# returning.
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
if _salt_cache:
return _salt_cache
except OSError:
pass
temporary = path.with_name(f"{path.name}.{os.getpid()}.tmp")
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(handle, "w", encoding="utf-8") as stream:
stream.write(uuid.uuid4().hex)
stream.flush()
os.fsync(stream.fileno())
try:
# Atomic claim: fails if another process already published one.
# os.link rather than replace, which would clobber theirs.
os.link(temporary, path)
except FileExistsError:
pass
except OSError:
# No hardlinks here (some network mounts, some container volumes).
# Claim the name directly instead. That reopens the empty-file
# window, but the window is now benign: a reader that lands in it
# gets "" and omits the hash for that process rather than caching a
# guessable one. Losing the hashes on every run of an entire
# filesystem is the worse failure.
try:
fallback = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(fallback, "w", encoding="utf-8") as stream:
stream.write(temporary.read_text(encoding="utf-8"))
except OSError:
pass
except OSError:
pass
finally:
try:
temporary.unlink()
except OSError:
pass
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
except OSError:
_salt_cache = ""
return _salt_cache
def _scoped_digest(value: str, length: int = 16) -> str:
"""Salted digest for values drawn from a guessable space.
repo.identity is a git remote URL, or ``local:<absolute path>`` when there is
no remote — which normally contains the account username. Sixteen unsalted
hex characters over that input space is enumerable, so this is not a
privacy control without the salt. Salting per install keeps every
within-account join the analytics actually use and gives up only
cross-machine joins on the same repository, which nothing computes.
Returns "" when there is no salt, so record() omits the property. An
unsalted digest over this input space is close to plaintext, and emitting one
under a name that implies it is hashed is worse than sending nothing.
"""
if not value:
return ""
salt = _install_salt()
if not salt:
return ""
return hashlib.sha256(f"{salt}:{value}".encode("utf-8")).hexdigest()[:length]
def _safe_value(value: Any) -> Any:
if isinstance(value, str):
return memory_core.redact(value)
@@ -302,176 +145,9 @@ def anonymous_id(identity: dict[str, str] | None = None) -> str:
return created
def _rotate_anonymous_id(identity: dict[str, str]) -> str:
"""Mint a fresh anonymous id because the account context is gone.
The previous id may already have been merged into a person profile by an
$identify, and that merge is permanent. Reusing it after a logout or a key
change attributes everything that follows to the account that just went
away, which is the same misattribution the key fingerprint exists to stop,
only arriving through the anonymous path instead.
`aliased` is cleared with it: the new id has never been merged, so it is
eligible to be aliased into whatever account comes next.
"""
created = f"code-anon-{uuid.uuid4().hex}"
identity["anonymous_id"] = created
identity.pop("aliased", None)
_write_identity(identity)
return created
def _install_state_path() -> Path:
return memory_core.data_dir() / "install-state.json"
def is_first_run() -> bool:
"""Whether install has never been recorded on this machine.
Deliberately NOT the identity file. That file is only written by a
successful flush, so an offline or firewalled user recorded code.install on
every single session, forever — and every 0.2.x user recorded one on their
first 0.3.x session because 0.2.x never wrote it at all.
"""
return not _install_state_path().exists()
def data_dir_was_empty() -> bool:
"""Whether the data directory is untouched. Call BEFORE anything writes to it.
hook_runner reaches claim_install() only after cache_plugin_api_key() has
written `api-key` and EvidenceStore() has created `evidence.sqlite3`, so
asking at claim time always saw content and every fresh install reported an
upgrade. The caller snapshots this at the top of the run instead.
"""
return not _data_dir_has_content()
def claim_install(was_empty: bool | None = None) -> str | None:
"""Claim the one install/upgrade record for this machine, atomically.
Returns the event to record ("install" or "upgrade"), or None if another
session already claimed it. O_CREAT|O_EXCL so two sessions starting together
cannot both win.
`was_empty` must come from data_dir_was_empty() called before this process
wrote anything. Omitting it falls back to checking now, which is only
correct for a caller that has touched nothing.
"""
if not is_enabled():
# Never consume the one-shot claim while the user is opted out, or they
# would silently lose their install event if they later opt in.
return None
path = _install_state_path()
upgrading = not (data_dir_was_empty() if was_empty is None else was_empty)
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
except FileExistsError:
return None
except OSError:
return None
try:
with os.fdopen(handle, "w", encoding="utf-8") as stream:
json.dump(
{
"plugin_version": memory_core.PLUGIN_VERSION,
"installed_at": memory_core.utc_now(),
"upgraded": upgrading,
},
stream,
)
# Durable before this returns. The O_EXCL open is what makes the
# claim exclusive, so it cannot be replaced by a temp-and-rename
# without losing that, which leaves the content as the thing to make
# safe. A kill between the open and this fsync used to leave a marker
# that exists but parses to nothing: is_first_run reads it as claimed
# and claim_version_change cannot read a version out of it.
stream.flush()
os.fsync(stream.fileno())
except OSError:
pass
return "upgrade" if upgrading else "install"
def _data_dir_has_content() -> bool:
"""Whether anything predates this session in the plugin data directory."""
try:
for entry in memory_core.data_dir().iterdir():
if entry.name != "install-state.json":
return True
except OSError:
pass
return False
def _repair_install_state(path: Path) -> None:
"""Rewrite an unparseable marker so version tracking can resume."""
try:
temporary = path.with_suffix(f".{os.getpid()}.tmp")
temporary.write_text(
json.dumps({"plugin_version": memory_core.PLUGIN_VERSION, "repaired_at": memory_core.utc_now()}),
encoding="utf-8",
)
temporary.replace(path)
except OSError:
pass
def claim_version_change() -> str | None:
"""Return the previously recorded version if it differs, updating the marker.
Only meaningful once the marker exists — the first transition into 0.3.x has
no recorded predecessor and reports "pre-0.3" instead. Claiming by rewriting
the marker means the next session sees no change and records nothing.
"""
path = _install_state_path()
try:
state = json.loads(path.read_text(encoding="utf-8"))
except OSError:
return None
except json.JSONDecodeError:
# A crash between O_EXCL and the write leaves an empty marker. Left
# alone it disables every future upgrade event on this machine, because
# claim_install sees the file and this function cannot parse it.
state = None
if not isinstance(state, dict):
_repair_install_state(path)
return None
previous = str(state.get("plugin_version") or "")
if not previous or previous == memory_core.PLUGIN_VERSION:
return None
# Claim the transition with an exclusive sentinel before rewriting the
# marker. A plain read-modify-write let every concurrently starting session
# observe the old version and each record its own upgrade — and the first
# session after a version bump is exactly when several agent windows restart
# together.
sentinel = path.with_name(f"upgraded-{memory_core.PLUGIN_VERSION}")
try:
os.close(os.open(sentinel, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600))
except FileExistsError:
return None
except OSError:
return None
state["plugin_version"] = memory_core.PLUGIN_VERSION
state["upgraded_at"] = memory_core.utc_now()
temporary = path.with_suffix(f".{os.getpid()}.tmp")
try:
temporary.write_text(json.dumps(state), encoding="utf-8")
temporary.replace(path)
except OSError:
# Release the claim. The marker still records the old version, so
# without this the sentinel makes claim_version_change return early on
# every later run and this version's upgrade is never recorded again.
for leftover in (sentinel, temporary):
try:
leftover.unlink()
except OSError:
pass
return None
return previous
"""Whether this machine has never recorded a plugin event before."""
return not _identity_path().exists()
def record(
@@ -492,32 +168,19 @@ def record(
except OSError:
pass
properties = _safe_value(properties)
# Stamped in the RECORDING process, beside harness. `source` used to be
# read in the sending process from a module global, so whichever process
# drained the spool named every event in it. flush() spreads per-event
# properties last, so this now wins over any sender's default.
properties.update(
harness=_harness,
source=_source_tag,
plugin_version=memory_core.PLUGIN_VERSION,
os=sys.platform,
python_version=platform.python_version(),
)
# Assigned only when the digest is real. _scoped_digest returns "" when
# the salt could not be persisted, and an empty property is worse than an
# absent one: it survives the None filter below and reads as a value.
if repo is not None:
repo_hash = _scoped_digest(getattr(repo, "identity", ""))
if repo_hash:
properties["repo_hash"] = repo_hash
properties["repo_hash"] = _digest(getattr(repo, "identity", ""))
if session_id:
session_hash = _scoped_digest(session_id)
if session_hash:
properties["session_hash"] = session_hash
properties["session_hash"] = _digest(session_id)
line = json.dumps(
{
"event": f"{EVENT_PREFIX}.{event}",
"uuid": str(uuid.uuid4()),
"timestamp": memory_core.utc_now(),
"properties": {
key: value for key, value in properties.items() if value is not None
@@ -576,201 +239,38 @@ def spawn_flush() -> bool:
return False
def _claim_name(attempt: int = 0) -> str:
"""Claim filename. The attempt count rides in the name so the 7-day expiry
only ever discards a batch that was actually retried and failed."""
return f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}-a{attempt}.sending"
def _claim_attempt(claim: Path) -> int:
"""Attempts recorded in a claim filename; 0 for the pre-attempt-count shape.
Anchored on field position, not on a leading "a": the legacy shape is
``telemetry-<pid>-<hex>.sending`` and a hex id such as ``a1234567`` would
otherwise parse as attempt 1234567 and be discarded unsent on the first
flush after an upgrade.
"""
stem = claim.name[: -len(".sending")] if claim.name.endswith(".sending") else claim.name
parts = stem.split("-")
if len(parts) != 4:
return 0
tail = parts[3]
if tail.startswith("a") and tail[1:].isdigit():
return int(tail[1:])
return 0
def _touch(path: Path) -> None:
"""Refresh mtime so a claim's age measures time since it was claimed.
``Path.replace`` is ``os.rename``, which preserves mtime — so a claim created
after a quiet minute inherited the spool's last-write time and looked
abandoned the instant it was made. A second sender would then take it over
while the first was still posting, and both would deliver the batch.
"""
try:
os.utime(path, None)
except OSError:
pass
def _claim_spool() -> Path | None:
"""Rename the spool aside so exactly one sender owns each batch."""
directory = memory_core.data_dir()
claim = directory / _claim_name()
claim = directory / f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}.sending"
spool = _spool_path()
try:
spool.replace(claim)
_touch(claim)
return claim
except OSError:
pass
return _claim_parked(directory)
def _sweep_debris(directory: Path) -> None:
"""Remove files nothing else will ever pick up again.
*.partial is a temp file orphaned by a crash between write and rename.
*.corrupt is a batch quarantined for undecodable content. No glob in this
module matches either, so without this they accumulate on disk for the life
of the install.
Quarantined batches are kept far longer than debris: they are the only
evidence left of events that could not be delivered, and someone diagnosing
a report of missing telemetry has to be able to find one.
"""
now = time.time()
for debris in directory.glob("telemetry-*.partial"):
try:
if now - debris.stat().st_mtime > CLAIM_STALE_SECONDS:
debris.unlink()
except OSError:
continue
for quarantined in directory.glob("telemetry-*.corrupt"):
try:
if now - quarantined.stat().st_mtime > CLAIM_EXPIRY_SECONDS:
quarantined.unlink()
except OSError:
continue
# The same reasoning covers *.tmp. _write_identity and _install_salt both
# create one and unlink it in a finally, which a SIGKILL skips, and no glob
# in this module matches the leftovers either.
for temporary in directory.glob("telemetry-*.tmp"):
try:
if now - temporary.stat().st_mtime > CLAIM_STALE_SECONDS:
temporary.unlink()
except OSError:
continue
def _claim_parked(directory: Path) -> Path | None:
"""Take the oldest abandoned claim, if any lease has actually expired.
Kept separate from the live spool so flush() can drain both in one run.
Previously parked batches were only reachable when no spool existed at all,
and because sessions keep recording there usually was one — so a batch
parked by a failed send waited until the 7-day expiry deleted it unsent,
even though its own presence is what started the sender.
"""
now = time.time()
for orphan in sorted(directory.glob("telemetry-*.sending"), key=_safe_mtime):
for orphan in sorted(directory.glob("telemetry-*.sending")):
try:
age = now - orphan.stat().st_mtime
except OSError:
continue
if age < CLAIM_STALE_SECONDS:
# Someone else holds a live lease on it. This check has to come
# first. Claiming a file bumps its attempt count and refreshes its
# mtime, so a sender that has just taken the final attempt looks
# exhausted to everyone else while it is actively draining. Judging
# exhaustion before liveness let a second sender unlink a batch out
# from under its owner, losing every event in it.
continue
# Attempts, not age. Every re-claim touches the mtime and every release
# backdates it by a fixed amount, so age is pinned near the stale
# threshold and never reaches the expiry. Age stays only as a backstop
# for files that never carried an attempt marker.
if _claim_attempt(orphan) >= MAX_CLAIM_ATTEMPTS or age > CLAIM_EXPIRY_SECONDS:
if age > CLAIM_EXPIRY_SECONDS:
try:
orphan.unlink()
except OSError:
pass
continue
claim = orphan.parent / _claim_name(_claim_attempt(orphan) + 1)
if age < CLAIM_STALE_SECONDS:
continue
try:
orphan.replace(claim)
_touch(claim)
return claim
except OSError:
continue
return None
def _safe_mtime(path: Path) -> float:
try:
return path.stat().st_mtime
except OSError:
return 0.0
def _rewrite_claim(claim: Path, remaining: list[dict[str, Any]]) -> bool:
"""Persist the unsent remainder, atomically, and refresh the lease.
Called after every successful batch. Two jobs: a retry resumes where the
send stopped instead of re-posting from the top, and the rewrite doubles as
the lease heartbeat, so a slow sender does not have its claim stolen
mid-flight. Interval is one batch, well inside CLAIM_STALE_SECONDS.
"""
if not remaining:
try:
claim.unlink()
except OSError:
pass
return True
temporary = claim.with_suffix(f".{os.getpid()}.partial")
try:
payload = "".join(json.dumps(event, separators=(",", ":"), default=str) + "\n" for event in remaining)
# fsync before the rename: without it the rename can land while the
# bytes have not, and the claim comes back empty or truncated after a
# crash. _drain then reads zero events and unlinks it.
with open(temporary, "w", encoding="utf-8") as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
temporary.replace(claim)
_touch(claim)
return True
except OSError:
try:
temporary.unlink()
except OSError:
pass
return False
def _release_claim(claim: Path, remaining: list[dict[str, Any]]) -> None:
"""Persist the remainder and drop the lease, because this sender has given up.
Distinct from the per-batch heartbeat: heartbeating on the way out would
make an abandoned batch look actively owned for a further
CLAIM_STALE_SECONDS, delaying the retry for no reason. Ageing it past the
threshold lets the next flush pick it up immediately, while the attempt
count in the filename still bounds how many times that can happen.
"""
if not _rewrite_claim(claim, remaining):
return
try:
# Backdate past the stale threshold so the next flush can pick it up,
# minus a cooldown that grows with the attempts already spent. Clamped so
# the mtime never lands in the future, which would read as a live lease.
cooldown = min(_claim_attempt(claim) * RETRY_COOLDOWN_SECONDS, CLAIM_STALE_SECONDS)
released = time.time() - CLAIM_STALE_SECONDS - 1 + cooldown
os.utime(claim, (released, released))
except OSError:
pass
def _resolve_email(key: str) -> str:
"""Trade the API key for the account email so events join other Mem0 surfaces."""
url = os.environ.get("MEM0_API_URL", memory_core.DEFAULT_API_URL).rstrip("/") + "/v1/ping/"
@@ -800,130 +300,34 @@ def _post(payload: dict[str, Any], url: str) -> bool:
def resolve_distinct_id() -> tuple[str, str]:
"""Return the PostHog distinct id and the anonymous id it replaced, if any.
The second value becomes a PostHog $identify alias. It is ONLY ever an
anonymous id: aliasing one account email to another merges two real person
profiles and cannot be undone, so a key that now belongs to a different
account re-resolves with no alias.
"""
"""Return the PostHog distinct id and the anonymous id it replaced, if any."""
identity = _read_identity()
key = memory_core.api_key()
fingerprint = _digest(key) if key else ""
email = identity.get("email", "")
if email and fingerprint:
recorded = identity.get("key_fingerprint", "")
if recorded == fingerprint:
return email, ""
if not recorded:
# Rows written before fingerprints existed. Verify rather than
# adopt: a key changed before the upgrade would otherwise bind the
# new key to the previous account's email, permanently, and the
# fingerprint would then agree with itself forever after.
verified = _resolve_email(key)
if not verified:
# Offline, firewalled, or the API is down. Keep the previous
# behaviour and retry on the next flush rather than dropping a
# real account attribution. Safe because the same network that
# failed /v1/ping/ is about to fail the PostHog POST, so nothing
# is delivered under the unverified identity in the meantime.
return email, ""
identity["email"] = verified
identity["key_fingerprint"] = fingerprint
_write_identity(identity)
return verified, ""
if email:
return email, ""
key = memory_core.api_key()
if not key:
# No key to verify the account with; do not keep attributing to it.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
return anonymous_id(identity), ""
resolved = _resolve_email(key)
if not resolved:
# The key changed and will not resolve (revoked, offline, API down).
# Reaching here with an email means the recorded fingerprint disagreed,
# so the key really did change. Drop the account and rotate: the stored
# anonymous id may already be merged into that account's person, and
# reusing it would keep the events on the profile we are trying to
# leave.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
email = _resolve_email(key)
if not email:
return anonymous_id(identity), ""
# Alias only when going anonymous -> email for the first time. Once an anon
# id has been merged into an account it must never be offered again: an
# alias naming an already-identified id is what could link two real people.
previous = "" if (email or identity.get("aliased")) else identity.get("anonymous_id", "")
if previous:
identity["aliased"] = True
identity["email"] = resolved
identity["key_fingerprint"] = fingerprint
previous = identity.get("anonymous_id", "")
identity["email"] = email
_write_identity(identity)
return resolved, previous
return email, previous
def flush() -> int:
"""Drain the live spool, then any parked claims, and return events sent."""
"""Drain claimed spools to PostHog and return the number of events sent."""
if not is_enabled():
return 0
sent, delivered = _drain(_claim_spool())
if not delivered:
# The network is failing. Retrying other batches now would only burn
# their attempt budget against the same broken connection.
return sent
# Parked batches used to starve behind the live spool indefinitely. Bounded
# per run so a long backlog cannot turn one flush into an unbounded loop.
directory = memory_core.data_dir()
_sweep_debris(directory)
for _ in range(MAX_PARKED_PER_RUN):
parked = _claim_parked(directory)
if parked is None:
break
count, delivered = _drain(parked)
sent += count
if not delivered:
break
return sent
def _drain(claim: Path | None) -> tuple[int, bool]:
"""Post one claimed batch file, recording progress after every batch.
Returns (events sent, whether everything was delivered).
"""
claim = _claim_spool()
if claim is None:
return 0, True
return 0
try:
lines = claim.read_text(encoding="utf-8").splitlines()
except ValueError:
# UnicodeDecodeError from a torn write: the content is unrecoverable, so
# quarantine rather than retry. flush() runs from a bare `finally:` in
# flush_worker, so raising here also skips the handoff cleanup, and an
# undecodable file would otherwise be re-read on every flush forever.
# Reported as delivered because there is nothing left to deliver and the
# rest of the run should continue.
try:
claim.replace(claim.with_suffix(".corrupt"))
except OSError:
try:
claim.unlink()
except OSError:
pass
return 0, True
except OSError:
# Could not read it, which is not the same as having nothing to send.
# The file is left exactly where it is: a vanished or briefly unreadable
# claim is retryable, and quarantining it here would discard events over
# a transient filesystem error. Reported as undelivered so the run stops
# instead of counting a batch nothing was posted from as delivered.
return 0, False
return 0
events = []
for line in lines:
try:
@@ -933,18 +337,11 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
if isinstance(value, dict) and value.get("event"):
events.append(value)
if not events:
# Only delete when the file really is empty. A non-empty file that
# parses to nothing is a torn write, and its contents are the unsent
# remainder — deleting it is the data loss this PR exists to prevent.
try:
empty = claim.stat().st_size == 0
except OSError:
empty = True
try:
claim.replace(claim.with_suffix(".corrupt")) if not empty else claim.unlink()
claim.unlink()
except OSError:
pass
return 0, True
return 0
distinct_id, aliased_anonymous_id = resolve_distinct_id()
if aliased_anonymous_id:
@@ -963,17 +360,12 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
sent = 0
for start in range(0, len(events), BATCH_SIZE):
chunk = events[start : start + BATCH_SIZE]
batch = [
{
"event": event["event"],
"distinct_id": distinct_id,
# Carried through from record() so a resend can be collapsed.
"uuid": event.get("uuid"),
"timestamp": event.get("timestamp"),
"properties": {
# Fallback only: events recorded by a build before source
# moved into record() have none of their own.
"source": _source_tag,
"language": "python",
"$process_person_profile": False,
@@ -981,24 +373,16 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
**(event.get("properties") or {}),
},
}
for event in chunk
for event in events[start : start + BATCH_SIZE]
]
if not _post({"api_key": POSTHOG_API_KEY, "batch": batch}, POSTHOG_BATCH_URL):
# Keep only what has not been delivered, and release the lease.
# Previously the whole file was kept and the retry re-posted every
# batch, including the ones that had already arrived.
_release_claim(claim, events[start:])
return sent, False
sent += len(chunk)
# Record progress and refresh the lease after each successful batch, so
# a crash repeats at most one batch instead of the entire file. If the
# rewrite fails the claim still holds delivered events, so stop rather
# than carry on as though progress were recorded — continuing is how the
# duplicate delivery this PR fixes would come back.
if not _rewrite_claim(claim, events[start + len(chunk) :]):
_release_claim(claim, events[start + len(chunk) :])
return sent, False
return sent, True
return sent
sent += len(batch)
try:
claim.unlink()
except OSError:
pass
return sent
def main() -> int:
@@ -1,3 +1,4 @@
jsonschema>=4.23,<5
pytest>=8,<10
skills-ref==0.1.1
httpx>=0.27,<1
@@ -7,8 +7,8 @@ disable-model-invocation: true
# Pause memory capture
To pause (hooks stop capturing and sending session content; a minimal
telemetry ping still fires at session start, under your Mem0 account email,
unless `MEM0_TELEMETRY=false`):
anonymous telemetry ping still fires at session start unless
`MEM0_TELEMETRY=false`):
```bash
python3 "{{PLUGIN_ROOT}}/core/memory_cli.py" --harness "{{HARNESS_ID}}" {{PLUGIN_DATA_ARG}} pause
@@ -15,6 +15,7 @@ from build.build import ( # noqa: E402
bundle_drift,
render_template,
replace_output,
sync_generated,
)
from build.validate import validate_bundle # noqa: E402
@@ -70,38 +71,6 @@ def test_portable_bundle_is_conformant_and_self_contained(tmp_path: Path) -> Non
assert not any(path.is_symlink() for path in root.rglob("*"))
def _harness_identity(root: Path) -> dict[str, str]:
"""Read the generated core/_harness_id.py without importing it."""
values: dict[str, str] = {}
for line in (root / "core" / "_harness_id.py").read_text(encoding="utf-8").splitlines():
if "=" in line and not line.lstrip().startswith("#"):
name, _, raw = line.partition("=")
values[name.strip()] = raw.strip().strip('"')
return values
def test_the_portable_bundle_declares_no_host_application(tmp_path: Path) -> None:
"""It runs in whatever editor a user drops it into, so it cannot know the host.
X-Application is allowlisted server-side. A guessed value is silently dropped
there, which is the worst outcome: the wire says we know the host and the
stored event says we do not.
"""
identity = _harness_identity(build("mem0-agent-plugin", "portable", tmp_path / "portable"))
assert identity["PLATFORM_APPLICATION"] == ""
# The PostHog-side label is still useful for grouping and stays populated.
assert identity["HARNESS_ID"] == "coding-agent"
assert identity["PLATFORM_SOURCE"] == "MEM0_PLUGIN"
@pytest.mark.parametrize("host", ["claude-code", "cursor", "codex", "kimi", "antigravity"])
def test_a_native_bundle_names_the_host_it_was_built_for(host: str, tmp_path: Path) -> None:
identity = _harness_identity(build(host, "native", tmp_path / host))
assert identity["PLATFORM_APPLICATION"] == host
@pytest.mark.parametrize("host", ["claude-code", "cursor", "codex", "kimi", "antigravity"])
def test_native_bundle_is_self_contained(host: str, tmp_path: Path) -> None:
root = build(host, "native", tmp_path / host)
@@ -145,6 +114,7 @@ def test_native_control_skills_select_the_host_store(host: str, tmp_path: Path)
("codex", "native"),
("kimi", "native"),
("antigravity", "native"),
("hermes", "native"),
],
)
def test_installable_plugin_directories_are_current(host: str, kind: str) -> None:
@@ -165,3 +135,91 @@ def test_marketplaces_keep_public_names_and_reference_real_plugins() -> None:
assert [plugin["name"] for plugin in codex_marketplace["plugins"]] == ["mem0"]
codex = codex_marketplace["plugins"][0]
assert codex["source"]["path"] == "./integrations/codex-plugin"
def test_native_bundle_can_select_runtime_without_skills(tmp_path: Path, monkeypatch) -> None:
from build import build as builder
source = tmp_path / "plugin"
source.mkdir()
(source / "__init__.py").write_text("# Native plugin adapter\n", encoding="utf-8")
(source / "plugin-build.json").write_text(
json.dumps(
{
"native": {
"pluginRoot": "${PLUGIN_ROOT}",
"pythonFiles": ["message_utils.py"],
"skills": False,
"files": {"__init__.py": "__init__.py"},
}
}
),
encoding="utf-8",
)
monkeypatch.setitem(builder.NATIVE_PLUGINS, "test-host", source)
stale_skill = source / "skills" / "remember" / "SKILL.md"
stale_skill.parent.mkdir(parents=True)
stale_skill.write_text("stale generated skill", encoding="utf-8")
root = build("test-host", "native", tmp_path / "output")
assert {path.name for path in (root / "core").iterdir()} == {"message_utils.py"}
assert not (root / "skills").exists()
assert (root / "__init__.py").read_text(encoding="utf-8") == "# Native plugin adapter\n"
sync_generated("test-host", "native")
assert bundle_drift("test-host", "native") == []
assert not (source / "skills").exists()
@pytest.mark.parametrize("files", ["message_utils.py", ["../README.md"], ["/tmp/source.py"], [7], ["missing.py"]])
def test_native_runtime_selection_rejects_invalid_sources(files, tmp_path: Path) -> None:
from build.build import _build_native
with pytest.raises(ValueError, match="pythonFiles|native source file"):
_build_native(
"test",
tmp_path,
tmp_path,
{
"native": {
"pluginRoot": "${PLUGIN_ROOT}",
"pythonFiles": files,
}
},
)
def test_native_runtime_selection_rejects_escaping_symlink(tmp_path: Path, monkeypatch) -> None:
from build import build as builder
source = tmp_path / "shared" / "python"
source.mkdir(parents=True)
secret = tmp_path / "outside.py"
secret.write_text("secret", encoding="utf-8")
(source / "message_utils.py").symlink_to(secret)
monkeypatch.setattr(builder, "CORE_ROOT", source.parent)
staged = tmp_path / "bundle"
staged.mkdir()
with pytest.raises(ValueError, match="inside their roots"):
builder._build_native(
"test",
tmp_path,
staged,
{
"native": {
"pluginRoot": "${PLUGIN_ROOT}",
"pythonFiles": ["message_utils.py"],
}
},
)
def test_hermes_bundle_uses_only_host_independent_runtime(tmp_path: Path) -> None:
root = build("hermes", "native", tmp_path / "hermes")
assert {path.name for path in (root / "core").iterdir()} == {"message_utils.py"}
assert (root / "__init__.py").is_file()
assert (root / "plugin.yaml").is_file()
assert not (root / "skills").exists()
assert not (root / "agents").exists()
@@ -10,10 +10,9 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from conformance import run as conformance_run # noqa: E402
from conformance.run import _command_check # noqa: E402
PLUGIN_ROOT = Path(__file__).resolve().parents[1]
RUNNER = PLUGIN_ROOT / "conformance" / "run.py"
PYTHON_HOSTS = {"claude-code", "cursor", "codex", "kimi", "antigravity"}
PYTHON_HOSTS = {"claude-code", "cursor", "codex", "kimi", "antigravity", "hermes"}
def test_python_bundle_conformance_builds_every_host(tmp_path: Path) -> None:
@@ -78,6 +77,12 @@ def test_conformance_plan_covers_every_runtime(tmp_path: Path) -> None:
"deepseek",
}
assert all(entry["status"] == "planned" for entry in payload["checks"])
python_commands = [entry["command"] for entry in payload["checks"] if entry["group"] == "python-tests"]
assert any(
"integrations/hermes-plugin/tests" in command
and "--confcutdir=integrations/hermes-plugin/tests" in command
for command in python_commands
)
assert {
entry["group"]
for entry in payload["checks"]
@@ -0,0 +1,39 @@
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
CORE = Path(__file__).resolve().parents[1] / "python"
def test_message_helpers_are_standalone_and_keep_legacy_exports() -> None:
result = subprocess.run(
[
sys.executable,
"-c",
"""
import json
import sys
import message_utils
assert "memory_core" not in sys.modules
assert "telemetry" not in sys.modules
text = "prefix " + "日本語🙂" * 250 + ' {"password": "secret value"}'
redacted = message_utils.redact(text)
assert "secret value" not in redacted
messages = [{"role": "user", "content": redacted}]
batches = message_utils.extraction_message_batches(messages, max_tokens=100)
assert len(batches) > 1
assert all(message_utils._message_tokens(batch) <= 100 for batch in batches)
assert "".join(m["content"] for batch in batches for m in batch) == redacted
import memory_core
for name in ("redact", "SECRET_PATTERNS", "extraction_message_batches", "_message_tokens"):
assert getattr(memory_core, name) is getattr(message_utils, name)
""",
],
cwd=CORE,
text=True,
capture_output=True,
check=False,
)
assert result.returncode == 0, result.stdout + result.stderr
@@ -1,446 +0,0 @@
"""Delivery semantics of the telemetry spool: no duplicates, no starvation.
These run against a built host's core in-process (not a subprocess) because they
need to inject failures into ``_post``. The identity tests next door cover the
uninitialised-process case that needs a real interpreter.
"""
from __future__ import annotations
import importlib
import json
import os
import sys
import time
from pathlib import Path
import pytest
CORE_ROOT = Path(__file__).resolve().parents[1]
REPOSITORY_ROOT = CORE_ROOT.parents[1]
HOST_CORE = REPOSITORY_ROOT / "integrations" / "claude-code-plugin" / "core"
pytestmark = pytest.mark.skipif(not HOST_CORE.exists(), reason="claude-code-plugin is not built")
@pytest.fixture()
def telemetry(tmp_path, monkeypatch):
# CI runs this directory and claude-code-plugin/tests in ONE pytest process,
# and that suite's conftest sets MEM0_TELEMETRY=false at import, process-wide.
# Without this the whole file silently no-ops: record() returns early and
# every assertion sees an empty spool. Do not rely on ambient env.
monkeypatch.setenv("MEM0_TELEMETRY", "true")
monkeypatch.setenv("MEM0_CODE_DATA_DIR", str(tmp_path / "data"))
monkeypatch.syspath_prepend(str(HOST_CORE))
# Save and RESTORE rather than delete. claude-code-plugin/tests/conftest.py
# imports memory_core once at collection and calls configure_harness() on it;
# dropping the module left a later re-import with default harness config, so
# tests in that suite failed depending on collection order.
names = ("telemetry", "memory_core", "_harness_id")
saved = {name: sys.modules.get(name) for name in names}
for name in names:
sys.modules.pop(name, None)
module = importlib.import_module("telemetry")
monkeypatch.setattr(module, "resolve_distinct_id", lambda: ("tester@example.com", ""))
try:
yield module
finally:
for name in names:
sys.modules.pop(name, None)
if saved[name] is not None:
sys.modules[name] = saved[name]
def _delivered(payloads):
return [event for payload in payloads if "batch" in payload for event in payload["batch"]]
def test_a_partial_failure_does_not_redeliver_what_already_arrived(telemetry):
"""Defect 2a: flush kept the whole claim on failure and retried from the top.
150 events across two batches, the second failing, previously delivered 250.
"""
for index in range(150):
telemetry.record("search", index=index)
sent: list[dict] = []
calls = {"n": 0}
def flaky(payload, url):
calls["n"] += 1
if calls["n"] == 2: # second batch fails
return False
sent.append(payload)
return True
telemetry._post = flaky
telemetry.flush()
telemetry._post = lambda payload, url: sent.append(payload) or True
telemetry.flush()
events = _delivered(sent)
assert len(events) == 150
assert len({event["uuid"] for event in events}) == 150
def test_a_fresh_claim_is_not_immediately_stealable(telemetry):
"""Defect 2b: rename preserves mtime, so a claim inherited the spool's age.
With the last write older than the stale threshold, a claim made now looked
abandoned the instant it existed and a second sender took it over.
"""
telemetry.record("search")
spool = telemetry._spool_path()
old = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(spool, (old, old))
first = telemetry._claim_spool()
assert first is not None
# A second sender starting right now must find nothing to take.
assert telemetry._claim_parked(first.parent) is None
def test_a_live_final_attempt_is_not_deleted_by_another_sender(telemetry):
"""Review finding: exhaustion was judged before liveness, so owners lost batches.
Claiming a parked file bumps its attempt count and refreshes its mtime. Once
the count reaches the budget, the owner draining it looked exhausted to every
other sender, which unlinked the file out from under it. Everything in that
batch was gone, which is precisely the loss this PR exists to stop.
"""
telemetry.record("search", reason="owned-by-the-first-sender")
spool = telemetry._spool_path()
stale = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(spool, (stale, stale))
claim = telemetry._claim_spool()
assert claim is not None
# Walk it to the final attempt, ageing it each round so it can be re-claimed.
# _claim_spool hands back a0 and _release_claim keeps the name, so it takes
# one full round per attempt to reach the budget.
for _ in range(telemetry.MAX_CLAIM_ATTEMPTS):
# Carry the marker through each rewrite so the final assertion proves the
# events survived, not merely that some file with the right name did.
telemetry._release_claim(claim, [{"event": "code.search", "uuid": "owned-by-the-first-sender"}])
parked = sorted(claim.parent.glob("telemetry-*.sending"))
assert parked, "the batch was dropped while still inside its budget"
os.utime(parked[0], (stale, stale))
claim = telemetry._claim_parked(claim.parent)
assert claim is not None
assert telemetry._claim_attempt(claim) >= telemetry.MAX_CLAIM_ATTEMPTS
assert claim.exists()
# The owner is draining it right now: fresh mtime, live lease.
second_sender = telemetry._claim_parked(claim.parent)
assert second_sender is None, "a second sender took a batch under a live lease"
assert claim.exists(), "a second sender deleted a batch its owner was draining"
assert "owned-by-the-first-sender" in claim.read_text(encoding="utf-8")
def test_an_exhausted_batch_is_still_discarded_once_its_lease_lapses(telemetry):
"""The liveness check must defer the cleanup, not cancel it.
Guards the obvious over-correction: skipping live claims is only safe if an
abandoned one at the same attempt count is still reaped on a later run.
"""
telemetry.record("search")
spool = telemetry._spool_path()
stale = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(spool, (stale, stale))
claim = telemetry._claim_spool()
assert claim is not None
exhausted = claim.parent / telemetry._claim_name(telemetry.MAX_CLAIM_ATTEMPTS)
claim.replace(exhausted)
os.utime(exhausted, (stale, stale))
assert telemetry._claim_parked(exhausted.parent) is None
assert not exhausted.exists(), "an abandoned exhausted batch was left behind forever"
def test_a_parked_batch_is_drained_behind_the_live_spool(telemetry):
"""Defect 6: parked claims were only reachable when no spool existed.
Because sessions keep recording there usually was one, so a batch parked by
a failed send waited until the 7-day expiry deleted it unsent — even though
its own presence is what starts the sender.
"""
telemetry.record("parked")
telemetry._post = lambda payload, url: False
telemetry.flush()
parked = list(telemetry.memory_core.data_dir().glob("telemetry-*.sending"))
assert len(parked) == 1
old = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(parked[0], (old, old))
telemetry.record("fresh")
sent: list[dict] = []
telemetry._post = lambda payload, url: sent.append(payload) or True
telemetry.flush()
names = {event["event"] for event in _delivered(sent)}
assert names == {"code.parked", "code.fresh"}
def test_a_batch_is_retried_until_the_budget_is_spent_not_discarded(telemetry):
"""Expiry discards what failed repeatedly, not what merely sat for a while.
The budget is the attempt count, because age cannot be one: every re-claim
touches the mtime and every release backdates it, so age never accumulates.
"""
telemetry.record("parked")
telemetry._post = lambda payload, url: False
telemetry.flush()
parked = list(telemetry.memory_core.data_dir().glob("telemetry-*.sending"))
assert len(parked) == 1
assert telemetry._claim_attempt(parked[0]) < telemetry.MAX_CLAIM_ATTEMPTS
sent: list[dict] = []
telemetry._post = lambda payload, url: sent.append(payload) or True
telemetry.flush()
assert [event["event"] for event in _delivered(sent)] == ["code.parked"]
def test_progress_is_recorded_after_every_batch(telemetry):
"""A crash repeats at most one batch, not the whole file."""
for index in range(250):
telemetry.record("search", index=index)
calls = {"n": 0}
def die_after_two(payload, url):
calls["n"] += 1
if calls["n"] > 2:
return False
return True
telemetry._post = die_after_two
telemetry.flush()
parked = list(telemetry.memory_core.data_dir().glob("telemetry-*.sending"))
assert len(parked) == 1
remaining = parked[0].read_text(encoding="utf-8").strip().splitlines()
# Two batches of 100 landed; only the last 50 should still be pending.
assert len(remaining) == 50
assert json.loads(remaining[0])["properties"]["index"] == 200
def test_the_heartbeat_actually_refreshes_the_lease(telemetry):
"""The claim rewrite doubles as the lease heartbeat.
Previously asserted `SEND_TIMEOUT * 4 < CLAIM_STALE_SECONDS`, which compares
two constants and executes none of the code under test. Drive the real
rewrite and watch the mtime move instead.
"""
for index in range(150):
telemetry.record("search", index=index)
claim = telemetry._claim_spool()
assert claim is not None
stale = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(claim, (stale, stale))
assert time.time() - claim.stat().st_mtime > telemetry.CLAIM_STALE_SECONDS
telemetry._rewrite_claim(claim, [{"event": "code.x", "properties": {}}])
assert time.time() - claim.stat().st_mtime < telemetry.CLAIM_STALE_SECONDS
def test_an_undeliverable_batch_is_eventually_given_up_on(telemetry):
"""Expiry has to be reachable from a state the state machine can produce.
It was not: every re-claim touched the mtime and every release backdated it
by a fixed amount, so age hovered near the stale threshold and the 7-day
expiry never fired. An undeliverable batch lived on disk forever, and
spawn_flush saw it and started a sender on every hook.
"""
telemetry.record("doomed")
telemetry._post = lambda payload, url: False
directory = telemetry.memory_core.data_dir()
for _ in range(telemetry.MAX_CLAIM_ATTEMPTS + 3):
telemetry.flush()
# Attempts now carry a cooldown, so a released claim is not instantly
# reclaimable. Age it to stand in for the wall time a real retry waits;
# without this the loop spins inside one cooldown and proves nothing.
for parked in directory.glob("telemetry-*.sending"):
stale = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(parked, (stale, stale))
leftover = list(directory.glob("telemetry-*.sending"))
assert leftover == [], f"batch never given up on: {[p.name for p in leftover]}"
def test_a_batch_that_cannot_be_read_is_not_counted_as_delivered(telemetry):
"""Review finding: a read failure reported 'everything delivered'.
Nothing was posted, so calling it delivered lets flush() carry on to other
claims as though this batch had arrived, and hides the failure from the one
signal that says the run went badly. It also must not quarantine: a briefly
unreadable file is retryable, and moving it to .corrupt discards the events
over a transient filesystem error, because nothing ever re-globs .corrupt.
"""
telemetry.record("search")
spool = telemetry._spool_path()
stale = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(spool, (stale, stale))
claim = telemetry._claim_spool()
assert claim is not None
original = Path.read_text
def unreadable(self, *args, **kwargs):
if self == claim:
raise OSError(5, "I/O error")
return original(self, *args, **kwargs)
Path.read_text = unreadable
try:
sent, delivered = telemetry._drain(claim)
finally:
Path.read_text = original
assert sent == 0
assert delivered is False, "an unread batch was reported as delivered"
assert claim.exists(), "a transient read error discarded the batch"
assert not list(claim.parent.glob("*.corrupt")), "quarantined over a transient error"
def test_undecodable_content_is_still_quarantined_and_the_run_continues(telemetry):
"""The other half: genuinely unrecoverable content must not block the run.
Guards the over-correction. If every read problem returned undelivered, one
torn file would stop every later claim on every flush, forever.
"""
telemetry.record("search")
spool = telemetry._spool_path()
stale = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(spool, (stale, stale))
claim = telemetry._claim_spool()
assert claim is not None
claim.write_bytes(b"\xff\xfe torn \x00 write")
sent, delivered = telemetry._drain(claim)
assert (sent, delivered) == (0, True)
assert not claim.exists()
assert list(claim.parent.glob("*.corrupt")), "unrecoverable content was not quarantined"
def test_retries_are_spread_over_real_time_not_burned_at_once(telemetry):
"""Review finding: releasing straight to reclaimable spent the budget instantly.
Two senders hitting one momentary failure could walk a batch from attempt 0
to the limit within seconds and discard it, when a retry a minute later would
have delivered. Each release now has to age past a cooldown that grows with
the attempts already spent.
"""
telemetry.record("doomed")
telemetry._post = lambda payload, url: False
directory = telemetry.memory_core.data_dir()
telemetry.flush()
parked = list(directory.glob("telemetry-*.sending"))
assert parked, "the batch was discarded on its first failure"
assert telemetry._claim_attempt(parked[0]) == 0
# Second sender, immediately: the cooldown has not elapsed, so it must not
# be able to spend another attempt.
telemetry.flush()
still = list(directory.glob("telemetry-*.sending"))
assert len(still) == 1
assert telemetry._claim_attempt(still[0]) <= 1, "burned attempts without waiting"
def test_a_legacy_claim_filename_is_not_mistaken_for_a_huge_attempt_count(telemetry):
"""The old shape is telemetry-<pid>-<hex>.sending, and hex can start with 'a'."""
assert telemetry._claim_attempt(Path("telemetry-999-deadbeef.sending")) == 0
assert telemetry._claim_attempt(Path("telemetry-999-a1234567.sending")) == 0
assert telemetry._claim_attempt(Path("telemetry-999-deadbeef-a2.sending")) == 2
def test_a_torn_claim_is_quarantined_not_deleted(telemetry):
"""A non-empty file that parses to nothing is the remainder, not garbage."""
telemetry.record("search")
claim = telemetry._claim_spool()
claim.write_bytes(b"\xff\xfe not utf-8 at all")
stale = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(claim, (stale, stale))
sent = telemetry.flush()
assert sent == 0
assert not claim.exists()
quarantined = list(telemetry.memory_core.data_dir().glob("*.corrupt"))
assert len(quarantined) == 1, "torn claim was destroyed instead of kept"
def test_a_failed_rewrite_stops_instead_of_redelivering(telemetry):
"""Ignoring the rewrite result reintroduced the duplicates this PR fixes."""
for index in range(250):
telemetry.record("search", index=index)
telemetry._rewrite_claim = lambda claim, remaining: False
delivered = []
telemetry._post = lambda payload, url: delivered.extend(payload.get("batch", [])) or True
telemetry.flush()
assert len(delivered) == 100, f"kept going after a failed rewrite: {len(delivered)}"
def test_partial_files_are_swept(telemetry):
"""Nothing else globs *.partial, so a crash mid-rename orphans one forever."""
data_dir = telemetry.memory_core.data_dir()
data_dir.mkdir(parents=True, exist_ok=True)
debris = data_dir / "telemetry-1-abc-a0.1.partial"
debris.write_text("x", encoding="utf-8")
old = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(debris, (old, old))
telemetry.flush()
assert not debris.exists()
def test_quarantined_batches_are_eventually_collected(telemetry):
"""Nothing re-globs .corrupt, so without a sweep they live on disk forever.
Kept much longer than .partial debris on purpose: a quarantined batch is the
only remaining evidence of events that could not be delivered.
"""
directory = telemetry.memory_core.data_dir()
directory.mkdir(parents=True, exist_ok=True)
fresh = directory / "telemetry-1-aaaaaaaa-a0.corrupt"
old = directory / "telemetry-2-bbbbbbbb-a0.corrupt"
for path in (fresh, old):
path.write_text("torn", encoding="utf-8")
expired = time.time() - (telemetry.CLAIM_EXPIRY_SECONDS + 60)
os.utime(old, (expired, expired))
telemetry._sweep_debris(directory)
assert fresh.exists(), "a recent quarantine was discarded before anyone could look at it"
assert not old.exists(), "an expired quarantine was left on disk forever"
def test_temp_files_orphaned_by_a_kill_are_collected(telemetry):
"""_write_identity and _install_salt unlink in a finally, which SIGKILL skips."""
directory = telemetry.memory_core.data_dir()
directory.mkdir(parents=True, exist_ok=True)
orphan = directory / "telemetry-salt.999.tmp"
orphan.write_text("abandoned", encoding="utf-8")
stale = time.time() - (telemetry.CLAIM_STALE_SECONDS + 60)
os.utime(orphan, (stale, stale))
telemetry._sweep_debris(directory)
assert not orphan.exists(), "a killed process left a temp file on disk forever"
@@ -1,274 +0,0 @@
"""Core telemetry behaviour with NO telemetry.init(), in a real subprocess.
Why this file exists
--------------------
``telemetry.py`` lives in ``agent-plugin-core/python/`` but its only tests lived
under ``claude-code-plugin/tests/``, behind a ``conftest.py`` that calls
``configure_harness()`` and ``telemetry.init()`` at import. Core behaviour was
therefore only ever exercised inside an already-configured module.
Two processes in the real pipeline never call ``init()``:
- ``mcp_server.py``, which records every manual search;
- the detached ``python3 telemetry.py`` sender that ``spawn_flush()`` starts at
session start, after every skill command, and when the MCP server exits.
Both fell back to module defaults, so MCP searches reported ``harness=generic``
and everything that sender delivered was labelled ``MEM0_PLUGIN`` regardless of
which of the six plugins produced it. The suite stayed green throughout.
These tests run in a fresh interpreter with no conftest, against a built host
bundle, which is the only arrangement that can catch that class of bug.
"""
from __future__ import annotations
import json
import subprocess
import sys
import tempfile
from pathlib import Path
import pytest
CORE_ROOT = Path(__file__).resolve().parents[1]
REPOSITORY_ROOT = CORE_ROOT.parents[1]
HOSTS = {
"claude-code": ("claude-code-plugin", "CLAUDE_CODE_PLUGIN"),
"cursor": ("cursor-plugin", "CURSOR_PLUGIN"),
"codex": ("codex-plugin", "CODEX_PLUGIN"),
"kimi": ("kimi-plugin", "KIMI_PLUGIN"),
"antigravity": ("antigravity-plugin", "ANTIGRAVITY_PLUGIN"),
# Portable: no flush_worker and no hook_runner, so its ONLY sender is the
# uninitialised telemetry.py. A native-only test passes here vacuously.
"coding-agent": ("mem0-agent-plugin", "CODING_AGENT_PLUGIN"),
}
def _core_dir(directory: str) -> Path:
return REPOSITORY_ROOT / "integrations" / directory / "core"
def _run(core: Path, data_dir: Path, body: str) -> str:
"""Execute `body` in a fresh interpreter with only the host's core on sys.path."""
script = f"import sys; sys.path.insert(0, {str(core)!r})\n{body}"
result = subprocess.run(
[sys.executable, "-c", script],
capture_output=True,
text=True,
env={
"MEM0_CODE_DATA_DIR": str(data_dir),
"PATH": "/usr/bin:/bin",
"HOME": str(data_dir),
},
)
assert result.returncode == 0, result.stderr
return result.stdout.strip()
@pytest.mark.parametrize("harness,spec", sorted(HOSTS.items()))
def test_identity_resolves_without_init(harness, spec):
"""Every built host knows what it is with no configuration call at all."""
directory, source_tag = spec
core = _core_dir(directory)
if not core.exists():
pytest.skip(f"{directory} is not built in this tree")
with tempfile.TemporaryDirectory() as tmp:
out = _run(
core,
Path(tmp),
"import telemetry; print(telemetry._harness, telemetry._source_tag)",
)
assert out == f"{harness} {source_tag}"
def test_mcp_server_records_the_real_harness():
"""mcp_server imports telemetry and never initialises it (server.py has no init).
Its recorded events used to carry harness=generic for every plugin.
"""
core = _core_dir("claude-code-plugin")
if not core.exists():
pytest.skip("claude-code-plugin is not built in this tree")
with tempfile.TemporaryDirectory() as tmp:
data_dir = Path(tmp)
_run(
core,
data_dir,
"import mcp_server, telemetry; telemetry.record('search', trigger='mcp-search')",
)
spooled = (data_dir / "telemetry.jsonl").read_text(encoding="utf-8").strip()
event = json.loads(spooled)
assert event["properties"]["harness"] == "claude-code"
assert event["properties"]["source"] == "CLAUDE_CODE_PLUGIN"
def test_the_detached_sender_does_not_relabel_events():
"""`python3 telemetry.py` is the sender spawn_flush() starts, and never inits.
source is stamped at record time now, so which process sends is irrelevant.
"""
core = _core_dir("claude-code-plugin")
if not core.exists():
pytest.skip("claude-code-plugin is not built in this tree")
with tempfile.TemporaryDirectory() as tmp:
data_dir = Path(tmp)
_run(core, data_dir, "import telemetry; telemetry.record('search')")
captured = data_dir / "captured.json"
# Drain with a fresh, unconfigured interpreter, capturing the payload
# instead of posting it.
_run(
core,
data_dir,
"import json, telemetry\n"
"sent = []\n"
"telemetry._post = lambda payload, url: sent.append(payload) or True\n"
"telemetry.flush()\n"
f"open({str(captured)!r}, 'w').write(json.dumps(sent))",
)
payloads = json.loads(captured.read_text(encoding="utf-8"))
batches = [p for p in payloads if "batch" in p]
assert batches, "nothing was sent"
properties = batches[0]["batch"][0]["properties"]
assert properties["source"] == "CLAUDE_CODE_PLUGIN"
assert properties["harness"] == "claude-code"
def test_every_event_carries_a_uuid_for_dedupe():
core = _core_dir("claude-code-plugin")
if not core.exists():
pytest.skip("claude-code-plugin is not built in this tree")
with tempfile.TemporaryDirectory() as tmp:
data_dir = Path(tmp)
_run(core, data_dir, "import telemetry; telemetry.record('search'); telemetry.record('flush')")
lines = (data_dir / "telemetry.jsonl").read_text(encoding="utf-8").strip().splitlines()
ids = [json.loads(line)["uuid"] for line in lines]
assert len(ids) == 2
assert len(set(ids)) == 2
def test_source_tag_defaults_agree_between_the_two_modules():
"""configure_harness and telemetry.init must derive the same tag.
They disagreed: `<host>_plugin` in one and `MEM0_<HOST>_PLUGIN` in the other,
so one plugin could emit three different source values depending on which
process sent the batch.
"""
core = _core_dir("claude-code-plugin")
if not core.exists():
pytest.skip("claude-code-plugin is not built in this tree")
with tempfile.TemporaryDirectory() as tmp:
out = _run(
core,
Path(tmp),
"import memory_core, telemetry\n"
"memory_core.configure_harness('kimi')\n"
"telemetry.init(harness='kimi')\n"
"print(memory_core.harness_config()['source_tag'].upper(), telemetry._source_tag)",
)
left, right = out.split()
assert left == right == "KIMI_PLUGIN"
def test_the_plugin_declares_its_surface_in_the_body_and_the_headers():
"""Body and headers both, because only the body works on every backend."""
core = _core_dir("claude-code-plugin")
if not core.exists():
pytest.skip("claude-code-plugin is not built in this tree")
with tempfile.TemporaryDirectory() as tmp:
out = _run(
core,
Path(tmp),
"import json, memory_core\n"
"h = memory_core.platform_headers('k')\n"
"print(json.dumps({'source': h.get('X-Mem0-Source'),"
" 'app': h.get('X-Application'),"
" 'client': h.get('X-Mem0-Client'),"
" 'auth': h.get('Authorization'),"
" 'ctype': h.get('Content-Type')}))",
)
headers = json.loads(out)
assert headers["source"] == "MEM0_PLUGIN"
assert headers["app"] == "claude-code"
assert headers["client"].startswith("mem0-plugin/")
# The transport headers the three call sites relied on must survive.
assert headers["auth"] == "Token k"
assert headers["ctype"] == "application/json"
def _session_start(core: Path, data_dir: Path) -> list[str]:
"""Drive the real hook_runner session-start path and return lifecycle events."""
recorded = "\n".join(
[
"import io, json, sys",
f"sys.path.insert(0, {str(core)!r})",
"import telemetry, hook_runner",
"seen = []",
"telemetry.record = lambda event, **kw: seen.append(event) or None",
"telemetry.spawn_flush = lambda: False",
# run() reads sys.argv through argparse; it takes no positional args.
"sys.argv = ['hook_runner', 'session-start']",
"sys.stdin = io.StringIO('{}')",
"hook_runner.run()",
"print(json.dumps([e for e in seen if e in ('install', 'upgrade')]))",
]
)
import json as _json
return _json.loads(_run(core, data_dir, recorded) or "[]")
def test_a_fresh_install_reports_install_not_upgrade():
"""The decision must survive the writes hook_runner does before asking.
claim_install() is reached only after cache_plugin_api_key() has written
`api-key` and EvidenceStore() has created `evidence.sqlite3`. Asking "is the
data dir empty" at that point always saw content, so code.install could
never fire and every new user was counted as an upgrade.
"""
core = _core_dir("claude-code-plugin")
if not core.exists():
pytest.skip("claude-code-plugin is not built in this tree")
with tempfile.TemporaryDirectory() as tmp:
data_dir = Path(tmp) / "data"
assert _session_start(core, data_dir) == ["install"]
def test_the_lifecycle_event_fires_exactly_once():
core = _core_dir("claude-code-plugin")
if not core.exists():
pytest.skip("claude-code-plugin is not built in this tree")
with tempfile.TemporaryDirectory() as tmp:
data_dir = Path(tmp) / "data"
first = _session_start(core, data_dir)
second = _session_start(core, data_dir)
third = _session_start(core, data_dir)
assert first == ["install"]
assert second == []
assert third == []
def test_an_existing_data_dir_reports_upgrade():
core = _core_dir("claude-code-plugin")
if not core.exists():
pytest.skip("claude-code-plugin is not built in this tree")
with tempfile.TemporaryDirectory() as tmp:
data_dir = Path(tmp) / "data"
data_dir.mkdir(parents=True)
# A 0.2.x leftover: the data dir survives the upgrade.
(data_dir / "requirements.txt").write_text("mem0ai\n", encoding="utf-8")
assert _session_start(core, data_dir) == ["upgrade"]
@@ -1,5 +1,3 @@
import { randomUUID } from "node:crypto";
import { redactSecrets } from "./lifecycle.ts";
const POSTHOG_API_KEY = "phc_hgJkUVJFYtmaJqrvf6CYN67TIQ8yhXAkWzUn9AMU4yX";
@@ -76,108 +74,34 @@ export function errorKind(error: unknown): string {
return error instanceof Error ? error.constructor.name : "other";
}
// Delivery is retried in memory, not spooled to disk, and that is a decision
// rather than an omission. The Python core spools because its hooks are separate
// processes that fire per tool call and exit immediately, so nothing survives
// without a file. These plugins are loaded into a host that lives for a whole
// session, so re-queueing covers the same transient failures without the claim
// and lease machinery a correct cross-process spool needs. What that leaves
// uncovered is narrow: a session that both starts and ends with no connectivity.
const RETRY_BACKOFF_CEILING_MS = 60_000;
// Consecutive failed flushes before the queue is dropped. Deliberately NOT the
// same thing as Python's budget, which rides in the claim filename and so
// follows one batch: this counter lives in the closure and counts the outage,
// not the payload. Events captured between attempts join the same queue and go
// with it. Per-batch accounting would need an attempt count on every event, and
// the queue is already bounded, so the simpler rule is the one in force here.
// Without any bound a payload the server will never accept is retried for the
// whole session and, now that the backlog is preferred over new events, holds
// the queue against everything behind it.
const MAX_DELIVERY_ATTEMPTS = 5;
export function createTelemetry(config: TelemetryConfig) {
let queue: Record<string, unknown>[] = [];
let timer: ReturnType<typeof setInterval> | undefined;
let consecutiveFailures = 0;
let retryNotBefore = 0;
let exitFlushAttempted = false;
let flushing = false;
const flushThreshold = config.flushThreshold ?? 10;
const maxQueueSize = config.maxQueueSize ?? 100;
const deliver = config.delivery ?? (async (batch: Record<string, unknown>[]) => {
const response = await fetch(POSTHOG_BATCH_URL, {
await fetch(POSTHOG_BATCH_URL, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ api_key: POSTHOG_API_KEY, batch }),
signal: AbortSignal.timeout(3_000),
});
// fetch only rejects on a network-level failure. Without this check a 500,
// a 503 or a 429 resolved normally and the batch was counted as delivered
// and dropped, which is the likelier outage than a refused connection.
// Any non-2xx is retried, matching the Python core: the backoff and the
// queue bound contain a payload that will never be accepted, because the
// re-queued batch sits at the front and is the first thing evicted.
if (!response.ok) throw new Error(`posthog responded ${response.status}`);
});
async function flush(force = false): Promise<void> {
// One at a time. Two overlapping flushes each detach the queue and each
// prepend their own batch back on failure, so the later batch lands in front
// of the earlier one and the truncation then drops the OLDER events first,
// inverting the priority the failure path exists to establish. A second
// caller returns immediately; the queue waits for the next flush.
if (flushing) return;
async function flush(): Promise<void> {
if (!queue.length) return;
// `force` skips the cooldown. beforeExit is the last chance this process
// gets, and gating it on the same backoff meant that after any failure the
// exit flush did nothing and the queue died with the process, which is the
// loss this whole mechanism exists to prevent.
if (!force && Date.now() < retryNotBefore) return;
const batch = queue;
queue = [];
flushing = true;
try {
await deliver(batch);
consecutiveFailures = 0;
retryNotBefore = 0;
} catch {
// Put it back. Detaching the batch and swallowing the error deleted the
// events outright, so any blip silently dropped telemetry with nothing
// recording that it had happened. Every event carries a uuid, so a retry
// that duplicates one PostHog already accepted is collapsed there.
//
consecutiveFailures += 1;
if (consecutiveFailures >= MAX_DELIVERY_ATTEMPTS) {
// Give up on the queue so a failing outage cannot hold it for the
// session. This drops whatever is queued now, which includes events
// captured during the outage, not only the batch that kept failing.
consecutiveFailures = 0;
retryNotBefore = 0;
return;
}
// Keep the FRONT on overflow, so the batch being retried survives and a
// new event is what gets dropped. Matches the Python core, where record()
// refuses new events once the spool is full rather than evicting the
// backlog. Keeping the newest would throw away exactly the events this
// retry exists to save.
queue = [...batch, ...queue].slice(0, maxQueueSize);
retryNotBefore = Date.now() + Math.min(2 ** consecutiveFailures * 1_000, RETRY_BACKOFF_CEILING_MS);
} finally {
flushing = false;
// Telemetry must never affect plugin behavior.
}
}
function beforeExit(): void {
// Once, and only once. Node re-emits beforeExit whenever the handler
// schedules more async work, so an unconditional forced flush looped until
// the attempt budget was spent: five attempts against a 3s delivery timeout
// is fifteen seconds added to the shutdown of whatever editor or CLI is
// hosting this. The backoff used to end that loop after one attempt, and
// removing it for the forced path removed the only thing bounding it.
if (exitFlushAttempted) return;
exitFlushAttempted = true;
void flush(true);
void flush();
}
function build(event: string, properties: Record<string, unknown> = {}): Record<string, unknown> | null {
@@ -188,15 +112,6 @@ export function createTelemetry(config: TelemetryConfig) {
return {
event: config.eventName?.(event) ?? event,
distinct_id: distinctId,
// Stamped once, at capture. This is what makes retrying safe: a batch
// re-sent after a failure carries the same ids, so PostHog collapses
// anything it already accepted instead of counting it twice.
uuid: randomUUID(),
// Capture time, not ingestion time. Events now sit through backoff and
// across a whole outage, so without this PostHog records them whenever
// delivery happened to succeed. It also matters for the uuid dedupe
// above, whose key includes the event date.
timestamp: new Date().toISOString(),
properties: {
...safeProperties(properties),
...safeProperties(config.commonProperties ?? {}),
@@ -219,10 +134,8 @@ export function createTelemetry(config: TelemetryConfig) {
try {
const payload = build(event, properties);
if (!payload) return;
// Full means drop this event, not evict the backlog. Same rule as the
// failure path above and as Python's record().
if (queue.length >= maxQueueSize) return;
queue.push(payload);
if (queue.length > maxQueueSize) queue = queue.slice(-maxQueueSize);
if (!timer) {
timer = setInterval(() => void flush(), config.flushIntervalMs ?? 5_000);
timer.unref?.();
@@ -236,8 +149,6 @@ export function createTelemetry(config: TelemetryConfig) {
function resetForTesting(): void {
queue = [];
consecutiveFailures = 0;
retryNotBefore = 0;
if (timer) clearInterval(timer);
timer = undefined;
process.off("beforeExit", beforeExit);
@@ -122,250 +122,3 @@ test("error classification does not expose messages", () => {
assert.equal(errorKind(new Error("request timeout")), "timeout");
assert.equal(errorKind(new Error("fetch failed")), "network");
});
test("a failed delivery keeps the batch instead of deleting it", async () => {
// The defect: the queue was detached before the await and the error swallowed,
// so one blip destroyed the events with nothing recording that it happened.
const attempts: Record<string, unknown>[][] = [];
let failNext = true;
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d",
flushThreshold: 1000,
delivery: async (batch) => {
attempts.push(batch);
if (failNext) throw new Error("network down");
},
});
telemetry.capture("one");
telemetry.capture("two");
await telemetry.flush();
assert.equal(attempts.length, 1);
assert.equal(telemetry.queueForTesting().length, 2, "events were dropped on failure");
failNext = false;
// Backoff is in force, so wait it out the way wall time would.
await new Promise((resolve) => setTimeout(resolve, 2_100));
await telemetry.flush();
assert.equal(attempts.length, 2, "never retried");
assert.equal(telemetry.queueForTesting().length, 0);
telemetry.resetForTesting();
});
test("a retried event carries the same uuid so PostHog can collapse it", async () => {
const attempts: Record<string, unknown>[][] = [];
let failNext = true;
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d",
flushThreshold: 1000,
delivery: async (batch) => {
attempts.push(batch);
if (failNext) throw new Error("network down");
},
});
telemetry.capture("once");
await telemetry.flush();
failNext = false;
await new Promise((resolve) => setTimeout(resolve, 2_100));
await telemetry.flush();
assert.equal(attempts.length, 2);
const first = attempts[0][0].uuid;
assert.ok(first, "events carry no uuid, so a retry would double count");
assert.equal(attempts[1][0].uuid, first, "retry minted a new uuid");
telemetry.resetForTesting();
});
test("repeated failures back off instead of retrying every flush", async () => {
let calls = 0;
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d",
flushThreshold: 1000,
delivery: async () => { calls += 1; throw new Error("blocked"); },
});
telemetry.capture("one");
await telemetry.flush();
await telemetry.flush();
await telemetry.flush();
assert.equal(calls, 1, "a blocked host was hammered on every flush");
assert.equal(telemetry.queueForTesting().length, 1, "the event was lost while backing off");
telemetry.resetForTesting();
});
test("a full queue drops the new event and keeps the batch being retried", async () => {
// Python's record() refuses new events once the spool is full rather than
// evicting the backlog. Keeping the newest here would throw away exactly the
// events the retry exists to save.
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d",
flushThreshold: 1000, maxQueueSize: 3,
delivery: async () => { throw new Error("down"); },
});
// Fill past the cap BEFORE the flush, so the re-queue actually has to truncate.
// Capturing only two left the queue empty at re-queue time and the slice on the
// failure path never ran, which is the half that decides the direction.
telemetry.capture("a");
telemetry.capture("b");
telemetry.capture("c");
await telemetry.flush();
telemetry.capture("d");
telemetry.capture("e");
const events = telemetry.queueForTesting().map((e) => (e as any).event);
assert.equal(events.length, 3, "queue grew past maxQueueSize");
assert.deepEqual(events, ["a", "b", "c"], "the retried batch was evicted instead of the new events");
telemetry.resetForTesting();
});
test("the exit-time flush ignores the backoff", async () => {
// beforeExit is the last chance the process gets. Gating it on the same
// cooldown meant that after any failure it did nothing and the queue died.
let attempts = 0;
let failing = true;
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d", flushThreshold: 1000,
delivery: async () => { attempts += 1; if (failing) throw new Error("down"); },
});
telemetry.capture("a");
await telemetry.flush();
assert.equal(attempts, 1);
failing = false;
await telemetry.flush();
assert.equal(attempts, 1, "the backoff should still hold for an ordinary flush");
await telemetry.flush(true);
assert.equal(attempts, 2, "the exit flush was suppressed by the backoff");
assert.equal(telemetry.queueForTesting().length, 0);
telemetry.resetForTesting();
});
test("every event carries a capture-time timestamp", async () => {
const sent: Record<string, unknown>[][] = [];
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d", flushThreshold: 1000,
delivery: async (batch) => { sent.push(batch); },
});
telemetry.capture("a");
const capturedAt = Date.now();
await new Promise((resolve) => setTimeout(resolve, 50));
await telemetry.flush();
const stamped = sent[0][0].timestamp as string;
assert.ok(stamped, "no timestamp, so PostHog would record delivery time");
assert.ok(Math.abs(Date.parse(stamped) - capturedAt) < 1_000, "not capture time");
telemetry.resetForTesting();
});
test("a batch the server will never accept is eventually given up on", async () => {
let attempts = 0;
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d", flushThreshold: 1000,
delivery: async () => { attempts += 1; throw new Error("permanently bad"); },
});
telemetry.capture("doomed");
for (let i = 0; i < 8; i += 1) await telemetry.flush(true);
assert.ok(attempts <= 6, `retried ${attempts} times with no cap`);
assert.equal(telemetry.queueForTesting().length, 0, "a doomed batch held the queue forever");
telemetry.resetForTesting();
});
test("an HTTP error response is a failure, not a delivery", async () => {
// fetch only rejects on a network-level failure, so a 500 used to resolve
// normally and the batch was dropped as delivered. Exercises the real default
// delivery path rather than an injected one, which is where this hid.
const realFetch = globalThis.fetch;
let calls = 0;
globalThis.fetch = (async () => {
calls += 1;
return new Response("upstream is unwell", { status: 503 });
}) as typeof fetch;
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d", flushThreshold: 1000,
});
try {
telemetry.capture("during.outage");
await telemetry.flush();
assert.equal(calls, 1, "never reached the network");
assert.equal(telemetry.queueForTesting().length, 1, "a 503 was counted as delivered");
} finally {
globalThis.fetch = realFetch;
telemetry.resetForTesting();
}
});
test("a 2xx is a delivery", async () => {
const realFetch = globalThis.fetch;
globalThis.fetch = (async () => new Response("ok", { status: 200 })) as typeof fetch;
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d", flushThreshold: 1000,
});
try {
telemetry.capture("fine");
await telemetry.flush();
assert.equal(telemetry.queueForTesting().length, 0, "a good response did not clear the queue");
} finally {
globalThis.fetch = realFetch;
telemetry.resetForTesting();
}
});
test("the exit flush is attempted once, not until the budget is spent", async () => {
// Node re-emits beforeExit whenever the handler schedules async work, so an
// unconditional forced flush looped until MAX_DELIVERY_ATTEMPTS. Against the
// real 3s delivery timeout that is fifteen seconds added to a host's shutdown.
let attempts = 0;
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d", flushThreshold: 1000,
delivery: async () => { attempts += 1; throw new Error("down"); },
});
telemetry.capture("a");
const handlers = process.listeners("beforeExit");
const ours = handlers[handlers.length - 1] as () => void;
ours();
ours();
ours();
await new Promise((resolve) => setTimeout(resolve, 20));
assert.equal(attempts, 1, `exit flush ran ${attempts} times`);
telemetry.resetForTesting();
});
test("overlapping flushes do not reorder the backlog behind newer events", async () => {
// Each flush detaches the queue and prepends its own batch back on failure, so
// two in flight at once put the LATER batch in front of the earlier one. The
// truncation then drops the older events first, inverting the priority the
// failure path exists to establish.
let release: (() => void)[] = [];
const telemetry = createTelemetry({
host: "h", source: "S", version: "1", distinctId: "d", flushThreshold: 1000,
delivery: () => new Promise((_resolve, reject) => { release.push(() => reject(new Error("down"))); }),
});
telemetry.capture("first");
const a = telemetry.flush();
telemetry.capture("second");
const b = telemetry.flush();
release.forEach((fn) => fn());
await Promise.all([a, b]);
const events = telemetry.queueForTesting().map((e) => (e as any).event);
assert.equal(release.length, 1, "a second delivery started while one was in flight");
assert.deepEqual(events, ["first", "second"], `backlog reordered: ${events.join(",")}`);
telemetry.resetForTesting();
});
@@ -1,11 +0,0 @@
"""Generated by integrations/agent-plugin-core/build/build.py. Do not edit."""
HARNESS_ID = "antigravity"
SOURCE_TAG = "ANTIGRAVITY_PLUGIN"
# Platform-side vocabulary (mem0_event.source + X-Application). The whole
# plugin family is one source; which editor it runs in is the application.
# An empty application means the host is unknown, and memory_core omits
# the header entirely rather than sending a placeholder.
PLATFORM_SOURCE = "MEM0_PLUGIN"
PLATFORM_APPLICATION = "antigravity"
@@ -290,11 +290,6 @@ def run(
if args.plugin_data_dir:
os.environ[data_dir_env] = args.plugin_data_dir
# Snapshot BEFORE anything writes to the data dir: cache_plugin_api_key
# writes `api-key` and EvidenceStore creates `evidence.sqlite3`, so asking
# after them always saw content and every fresh install reported an upgrade.
data_dir_was_empty = telemetry.data_dir_was_empty()
cache_plugin_api_key()
if args.action == "session-start":
clear_stale_api_key_cache()
@@ -310,19 +305,8 @@ def run(
return 0
if args.action == "session-start":
# Claims the marker atomically and says which event to record, so a
# second session starting alongside this one cannot record it too.
first_event = telemetry.claim_install(was_empty=data_dir_was_empty)
if first_event == "install":
if telemetry.is_first_run():
telemetry.record("install")
elif first_event == "upgrade":
# First run after a build that never wrote the marker; the
# predecessor version was never recorded anywhere.
telemetry.record("upgrade", from_version="pre-0.3")
else:
previous = telemetry.claim_version_change()
if previous:
telemetry.record("upgrade", from_version=previous)
recovered = recover_pending_handoffs()
record_session_start(store, hook_input)
if recovered:
@@ -11,7 +11,6 @@ from __future__ import annotations
import functools
import hashlib
import json
import math
import os
import re
import sqlite3
@@ -27,9 +26,17 @@ from pathlib import Path
from typing import Any, Iterable
import telemetry
from message_utils import MAX_EXTRACTION_INPUT_TOKENS as MAX_EXTRACTION_INPUT_TOKENS
from message_utils import SECRET_PATTERNS as SECRET_PATTERNS
from message_utils import _estimated_tokens as _estimated_tokens
from message_utils import _is_agent_assignment as _is_agent_assignment
from message_utils import _is_agent_response as _is_agent_response
from message_utils import _message_tokens as _message_tokens
from message_utils import extraction_message_batches as extraction_message_batches
from message_utils import redact as redact
DEFAULT_API_URL = "https://api.mem0.ai"
PLUGIN_VERSION = "0.3.2"
PLUGIN_VERSION = "0.3.1"
_harness_name: str = "generic"
_harness_env_prefix: str = "MEM0_PLUGIN"
@@ -66,7 +73,6 @@ CHECKPOINT_EXCHANGES = 5
CHECKPOINT_MESSAGES = 10
CHECKPOINT_SOURCE_CHARS = 40000
DEFAULT_MAX_CONTEXT_CHARS = 4000
MAX_EXTRACTION_INPUT_TOKENS = 24000
MAX_FLUSH_ATTEMPTS = 5
FORGET_PAGE_SIZE = 100
FORGET_MAX_PAGES = 50
@@ -139,48 +145,11 @@ BUILD_COMMAND_RE = re.compile(
re.IGNORECASE,
)
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(
r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"
),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r'|(?:access|refresh|session)[_-]?token|token|authorization|credential'
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def redact(value: Any) -> str:
text = (
value
if isinstance(value, str)
else json.dumps(value, ensure_ascii=False, default=str)
)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def bounded(value: Any, limit: int) -> str:
text = redact(value).strip()
if len(text) <= limit:
@@ -1706,128 +1675,6 @@ def build_extraction_messages(structured: dict[str, Any]) -> list[dict[str, str]
return messages
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if (
_is_agent_assignment(message)
and index + 1 < len(exchange)
and _is_agent_response(exchange[index + 1])
):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
# Platform surface attribution. Read from the generated per-host module so a new
# entrypoint is correct without remembering to configure anything.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
except ImportError:
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
def platform_headers(key: str) -> dict[str, str]:
"""Auth plus the three surface-identity headers.
X-Mem0-Source and X-Application are set-once by contract: this is the
outermost layer, so it sets them, and nothing below may overwrite them.
X-Mem0-Client is append-only — anything downstream adds itself to the tail.
"""
headers = {
"Authorization": f"Token {key}",
"Content-Type": "application/json",
"X-Mem0-Source": _PLATFORM_SOURCE,
"X-Mem0-Client": f"mem0-plugin/{PLUGIN_VERSION}",
}
if _PLATFORM_APPLICATION:
headers["X-Application"] = _PLATFORM_APPLICATION
return headers
def _request_json(
url: str, key: str, payload: dict[str, Any], timeout: float
) -> tuple[dict[str, Any] | list[Any], int, int]:
@@ -1835,7 +1682,7 @@ def _request_json(
request = urllib.request.Request(
url,
data=raw,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -1862,7 +1709,7 @@ def _get_json(
) -> tuple[dict[str, Any] | list[Any], int]:
request = urllib.request.Request(
url,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="GET",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -2008,13 +1855,6 @@ def flush_session(
"user_id": write_user,
"app_id": repo.app_id,
"run_id": session_id,
# Top level, not metadata: the backend reads `source` from the body or
# the query string, never from metadata, which is where this used to
# sit. The X-Mem0-Source header is also read, but only from the
# platform release that ships alongside this change, so the body value
# is what makes attribution work on both. The harness tag stays in
# metadata as hook provenance.
"source": _PLATFORM_SOURCE,
"metadata": {**metadata, "author": write_user, "dirs": directory_chain(repo)},
"agent_custom_instructions": PROJECT_MEMORY_INSTRUCTIONS,
"custom_instructions": PERSONAL_MEMORY_INSTRUCTIONS,
@@ -2558,7 +2398,7 @@ def _collect_memory_ids(
def _delete_memory(api_url: str, key: str, memory_id: str) -> bool:
request = urllib.request.Request(
f"{api_url}/v1/memories/{urllib.parse.quote(memory_id)}/",
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="DELETE",
)
try:
@@ -0,0 +1,127 @@
"""Shared, host-independent redaction and lossless extraction batching."""
from __future__ import annotations
import json
import math
import re
from typing import Any
MAX_EXTRACTION_INPUT_TOKENS = 24000
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r"|(?:access|refresh|session)[_-]?token|token|authorization|credential"
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def redact(value: Any) -> str:
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, default=str)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if _is_agent_assignment(message) and index + 1 < len(exchange) and _is_agent_response(exchange[index + 1]):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
+39 -655
View File
@@ -1,9 +1,5 @@
#!/usr/bin/env python3
"""Usage telemetry for Mem0 agent plugins.
Events are linked to your Mem0 account email when an API key is configured, and
to a random per-machine id otherwise. Not anonymous — the Python SDK and CLI
attribute the same way.
"""Anonymous usage telemetry for Mem0 agent plugins.
Hooks run on a 3-6 second budget and fire on every tool call, so recording never
touches the network: `record` appends one JSON line to a local spool and returns.
@@ -13,8 +9,7 @@ started once per session and again from the flush worker that is already detache
Pure stdlib, matching the rest of the plugin. Opt out with MEM0_TELEMETRY=false.
Never sends prompts, memory text, queries, file paths, repository names, or API
keys: only event names, durations, counts, coarse outcomes, and repo/session
identifiers hashed with a random per-install salt.
keys: only event names, durations, counts, coarse outcomes, and salted hashes.
"""
from __future__ import annotations
@@ -34,24 +29,8 @@ from typing import Any
import memory_core
# Seeded from the per-host module the build generates into core/. Two processes
# in this pipeline never call init() — mcp_server.py, and the detached
# `python3 telemetry.py` sender that spawn_flush() starts — so a module default
# was what every one of their events got labelled with.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import HARNESS_ID as _DEFAULT_HARNESS
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
from _harness_id import SOURCE_TAG as _DEFAULT_SOURCE_TAG
except ImportError:
_DEFAULT_HARNESS = "generic"
_DEFAULT_SOURCE_TAG = "MEM0_PLUGIN"
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
_salt_cache: str = ""
_harness: str = _DEFAULT_HARNESS
_source_tag: str = _DEFAULT_SOURCE_TAG
_harness: str = "generic"
_source_tag: str = "MEM0_PLUGIN"
_PRIVATE_KEYS = {
"apikey",
"authorization",
@@ -77,19 +56,10 @@ _PRIVATE_KEYS = {
}
def init(harness: str = "", source_tag: str = "") -> None:
"""Override the generated identity. Optional — core/_harness_id.py is the default.
The fallback shape matches memory_core.configure_harness's (``<HOST>_PLUGIN``).
It used to be ``MEM0_<HOST>_PLUGIN`` here and ``<host>_plugin`` there, which
meant one plugin could emit three different source values depending on which
process happened to send the batch.
"""
def init(harness: str = "generic", source_tag: str = "") -> None:
global _harness, _source_tag
_harness = harness or _DEFAULT_HARNESS
_source_tag = source_tag or (
f"{_harness.upper().replace('-', '_')}_PLUGIN" if harness else _DEFAULT_SOURCE_TAG
)
_harness = harness
_source_tag = source_tag or f"MEM0_{harness.upper().replace('-', '_')}_PLUGIN"
POSTHOG_API_KEY = "phc_hgJkUVJFYtmaJqrvf6CYN67TIQ8yhXAkWzUn9AMU4yX"
POSTHOG_CAPTURE_URL = "https://us.i.posthog.com/i/v0/e/"
@@ -100,16 +70,6 @@ BATCH_SIZE = 100
SEND_TIMEOUT = 5
CLAIM_STALE_SECONDS = 120
CLAIM_EXPIRY_SECONDS = 7 * 24 * 60 * 60
# A batch is only discarded once it has genuinely been retried this many times.
MAX_CLAIM_ATTEMPTS = 3
# Parked claims drained per run, after the live spool. Bounded so a long backlog
# cannot turn one flush into an unbounded send loop.
MAX_PARKED_PER_RUN = 3
# Added to the wait before a released claim becomes reclaimable, per attempt
# already spent. Releasing straight to "reclaimable now" let two senders burn the
# whole budget within seconds of one another on a single momentary failure, and
# discard a batch a retry a minute later would have delivered.
RETRY_COOLDOWN_SECONDS = 60
def is_enabled() -> bool:
@@ -123,126 +83,9 @@ def is_enabled() -> bool:
def _digest(value: str, length: int = 16) -> str:
"""Unsalted digest. Only for values that are already secrets (API keys)."""
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:length]
def _salt_path() -> Path:
return memory_core.data_dir() / "telemetry-salt"
def _install_salt() -> str:
"""Random per-install salt, created once and memoized for the process.
Deliberately its own file, claimed with O_CREAT|O_EXCL, rather than a key in
the identity file. Three reasons, all of which produced wrong data when this
lived in the identity dict:
- Hooks are short-lived separate processes firing on every tool call, and
people run more than one agent window. A read-modify-write would let each
process mint its own salt, so one repository would hash several ways in the
window before a writer won.
- resolve_distinct_id holds a copy of the identity dict across a network call
to /v1/ping/, so whichever write landed second erased the other's key —
losing either the salt (repo_hash changes mid-stream) or the email (a
second $identify, splitting the person).
- Touching the identity file from record() would create it, and is_first_run
keys off that file, so recording an event would silently suppress the
install event.
Published atomically, and there is deliberately no derived fallback. Creating
the file with O_CREAT|O_EXCL and then writing into it leaves a window where
the file exists and is empty, and a concurrent hook that reads it in that
window gets nothing. Falling back to a digest of the path would hand that
process a salt an attacker can compute, memoized for its whole run, which is
the privacy control this function exists to provide silently turning itself
off under load. The salt is written to a private temp file first and linked
into place, so the name either does not exist or already has the full value.
Returns "" when it genuinely cannot persist. Callers omit the hash entirely
rather than emit an unsalted one.
"""
global _salt_cache
if _salt_cache:
return _salt_cache
path = _salt_path()
# Read before writing. Hooks are separate processes firing on every tool
# call, so all but the first find the salt already published; going straight
# to create-fsync-link-unlink meant every one of them paid an fsync to
# discover that, on a path whose whole promise is appending a line and
# returning.
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
if _salt_cache:
return _salt_cache
except OSError:
pass
temporary = path.with_name(f"{path.name}.{os.getpid()}.tmp")
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(handle, "w", encoding="utf-8") as stream:
stream.write(uuid.uuid4().hex)
stream.flush()
os.fsync(stream.fileno())
try:
# Atomic claim: fails if another process already published one.
# os.link rather than replace, which would clobber theirs.
os.link(temporary, path)
except FileExistsError:
pass
except OSError:
# No hardlinks here (some network mounts, some container volumes).
# Claim the name directly instead. That reopens the empty-file
# window, but the window is now benign: a reader that lands in it
# gets "" and omits the hash for that process rather than caching a
# guessable one. Losing the hashes on every run of an entire
# filesystem is the worse failure.
try:
fallback = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(fallback, "w", encoding="utf-8") as stream:
stream.write(temporary.read_text(encoding="utf-8"))
except OSError:
pass
except OSError:
pass
finally:
try:
temporary.unlink()
except OSError:
pass
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
except OSError:
_salt_cache = ""
return _salt_cache
def _scoped_digest(value: str, length: int = 16) -> str:
"""Salted digest for values drawn from a guessable space.
repo.identity is a git remote URL, or ``local:<absolute path>`` when there is
no remote — which normally contains the account username. Sixteen unsalted
hex characters over that input space is enumerable, so this is not a
privacy control without the salt. Salting per install keeps every
within-account join the analytics actually use and gives up only
cross-machine joins on the same repository, which nothing computes.
Returns "" when there is no salt, so record() omits the property. An
unsalted digest over this input space is close to plaintext, and emitting one
under a name that implies it is hashed is worse than sending nothing.
"""
if not value:
return ""
salt = _install_salt()
if not salt:
return ""
return hashlib.sha256(f"{salt}:{value}".encode("utf-8")).hexdigest()[:length]
def _safe_value(value: Any) -> Any:
if isinstance(value, str):
return memory_core.redact(value)
@@ -302,176 +145,9 @@ def anonymous_id(identity: dict[str, str] | None = None) -> str:
return created
def _rotate_anonymous_id(identity: dict[str, str]) -> str:
"""Mint a fresh anonymous id because the account context is gone.
The previous id may already have been merged into a person profile by an
$identify, and that merge is permanent. Reusing it after a logout or a key
change attributes everything that follows to the account that just went
away, which is the same misattribution the key fingerprint exists to stop,
only arriving through the anonymous path instead.
`aliased` is cleared with it: the new id has never been merged, so it is
eligible to be aliased into whatever account comes next.
"""
created = f"code-anon-{uuid.uuid4().hex}"
identity["anonymous_id"] = created
identity.pop("aliased", None)
_write_identity(identity)
return created
def _install_state_path() -> Path:
return memory_core.data_dir() / "install-state.json"
def is_first_run() -> bool:
"""Whether install has never been recorded on this machine.
Deliberately NOT the identity file. That file is only written by a
successful flush, so an offline or firewalled user recorded code.install on
every single session, forever — and every 0.2.x user recorded one on their
first 0.3.x session because 0.2.x never wrote it at all.
"""
return not _install_state_path().exists()
def data_dir_was_empty() -> bool:
"""Whether the data directory is untouched. Call BEFORE anything writes to it.
hook_runner reaches claim_install() only after cache_plugin_api_key() has
written `api-key` and EvidenceStore() has created `evidence.sqlite3`, so
asking at claim time always saw content and every fresh install reported an
upgrade. The caller snapshots this at the top of the run instead.
"""
return not _data_dir_has_content()
def claim_install(was_empty: bool | None = None) -> str | None:
"""Claim the one install/upgrade record for this machine, atomically.
Returns the event to record ("install" or "upgrade"), or None if another
session already claimed it. O_CREAT|O_EXCL so two sessions starting together
cannot both win.
`was_empty` must come from data_dir_was_empty() called before this process
wrote anything. Omitting it falls back to checking now, which is only
correct for a caller that has touched nothing.
"""
if not is_enabled():
# Never consume the one-shot claim while the user is opted out, or they
# would silently lose their install event if they later opt in.
return None
path = _install_state_path()
upgrading = not (data_dir_was_empty() if was_empty is None else was_empty)
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
except FileExistsError:
return None
except OSError:
return None
try:
with os.fdopen(handle, "w", encoding="utf-8") as stream:
json.dump(
{
"plugin_version": memory_core.PLUGIN_VERSION,
"installed_at": memory_core.utc_now(),
"upgraded": upgrading,
},
stream,
)
# Durable before this returns. The O_EXCL open is what makes the
# claim exclusive, so it cannot be replaced by a temp-and-rename
# without losing that, which leaves the content as the thing to make
# safe. A kill between the open and this fsync used to leave a marker
# that exists but parses to nothing: is_first_run reads it as claimed
# and claim_version_change cannot read a version out of it.
stream.flush()
os.fsync(stream.fileno())
except OSError:
pass
return "upgrade" if upgrading else "install"
def _data_dir_has_content() -> bool:
"""Whether anything predates this session in the plugin data directory."""
try:
for entry in memory_core.data_dir().iterdir():
if entry.name != "install-state.json":
return True
except OSError:
pass
return False
def _repair_install_state(path: Path) -> None:
"""Rewrite an unparseable marker so version tracking can resume."""
try:
temporary = path.with_suffix(f".{os.getpid()}.tmp")
temporary.write_text(
json.dumps({"plugin_version": memory_core.PLUGIN_VERSION, "repaired_at": memory_core.utc_now()}),
encoding="utf-8",
)
temporary.replace(path)
except OSError:
pass
def claim_version_change() -> str | None:
"""Return the previously recorded version if it differs, updating the marker.
Only meaningful once the marker exists — the first transition into 0.3.x has
no recorded predecessor and reports "pre-0.3" instead. Claiming by rewriting
the marker means the next session sees no change and records nothing.
"""
path = _install_state_path()
try:
state = json.loads(path.read_text(encoding="utf-8"))
except OSError:
return None
except json.JSONDecodeError:
# A crash between O_EXCL and the write leaves an empty marker. Left
# alone it disables every future upgrade event on this machine, because
# claim_install sees the file and this function cannot parse it.
state = None
if not isinstance(state, dict):
_repair_install_state(path)
return None
previous = str(state.get("plugin_version") or "")
if not previous or previous == memory_core.PLUGIN_VERSION:
return None
# Claim the transition with an exclusive sentinel before rewriting the
# marker. A plain read-modify-write let every concurrently starting session
# observe the old version and each record its own upgrade — and the first
# session after a version bump is exactly when several agent windows restart
# together.
sentinel = path.with_name(f"upgraded-{memory_core.PLUGIN_VERSION}")
try:
os.close(os.open(sentinel, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600))
except FileExistsError:
return None
except OSError:
return None
state["plugin_version"] = memory_core.PLUGIN_VERSION
state["upgraded_at"] = memory_core.utc_now()
temporary = path.with_suffix(f".{os.getpid()}.tmp")
try:
temporary.write_text(json.dumps(state), encoding="utf-8")
temporary.replace(path)
except OSError:
# Release the claim. The marker still records the old version, so
# without this the sentinel makes claim_version_change return early on
# every later run and this version's upgrade is never recorded again.
for leftover in (sentinel, temporary):
try:
leftover.unlink()
except OSError:
pass
return None
return previous
"""Whether this machine has never recorded a plugin event before."""
return not _identity_path().exists()
def record(
@@ -492,32 +168,19 @@ def record(
except OSError:
pass
properties = _safe_value(properties)
# Stamped in the RECORDING process, beside harness. `source` used to be
# read in the sending process from a module global, so whichever process
# drained the spool named every event in it. flush() spreads per-event
# properties last, so this now wins over any sender's default.
properties.update(
harness=_harness,
source=_source_tag,
plugin_version=memory_core.PLUGIN_VERSION,
os=sys.platform,
python_version=platform.python_version(),
)
# Assigned only when the digest is real. _scoped_digest returns "" when
# the salt could not be persisted, and an empty property is worse than an
# absent one: it survives the None filter below and reads as a value.
if repo is not None:
repo_hash = _scoped_digest(getattr(repo, "identity", ""))
if repo_hash:
properties["repo_hash"] = repo_hash
properties["repo_hash"] = _digest(getattr(repo, "identity", ""))
if session_id:
session_hash = _scoped_digest(session_id)
if session_hash:
properties["session_hash"] = session_hash
properties["session_hash"] = _digest(session_id)
line = json.dumps(
{
"event": f"{EVENT_PREFIX}.{event}",
"uuid": str(uuid.uuid4()),
"timestamp": memory_core.utc_now(),
"properties": {
key: value for key, value in properties.items() if value is not None
@@ -576,201 +239,38 @@ def spawn_flush() -> bool:
return False
def _claim_name(attempt: int = 0) -> str:
"""Claim filename. The attempt count rides in the name so the 7-day expiry
only ever discards a batch that was actually retried and failed."""
return f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}-a{attempt}.sending"
def _claim_attempt(claim: Path) -> int:
"""Attempts recorded in a claim filename; 0 for the pre-attempt-count shape.
Anchored on field position, not on a leading "a": the legacy shape is
``telemetry-<pid>-<hex>.sending`` and a hex id such as ``a1234567`` would
otherwise parse as attempt 1234567 and be discarded unsent on the first
flush after an upgrade.
"""
stem = claim.name[: -len(".sending")] if claim.name.endswith(".sending") else claim.name
parts = stem.split("-")
if len(parts) != 4:
return 0
tail = parts[3]
if tail.startswith("a") and tail[1:].isdigit():
return int(tail[1:])
return 0
def _touch(path: Path) -> None:
"""Refresh mtime so a claim's age measures time since it was claimed.
``Path.replace`` is ``os.rename``, which preserves mtime — so a claim created
after a quiet minute inherited the spool's last-write time and looked
abandoned the instant it was made. A second sender would then take it over
while the first was still posting, and both would deliver the batch.
"""
try:
os.utime(path, None)
except OSError:
pass
def _claim_spool() -> Path | None:
"""Rename the spool aside so exactly one sender owns each batch."""
directory = memory_core.data_dir()
claim = directory / _claim_name()
claim = directory / f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}.sending"
spool = _spool_path()
try:
spool.replace(claim)
_touch(claim)
return claim
except OSError:
pass
return _claim_parked(directory)
def _sweep_debris(directory: Path) -> None:
"""Remove files nothing else will ever pick up again.
*.partial is a temp file orphaned by a crash between write and rename.
*.corrupt is a batch quarantined for undecodable content. No glob in this
module matches either, so without this they accumulate on disk for the life
of the install.
Quarantined batches are kept far longer than debris: they are the only
evidence left of events that could not be delivered, and someone diagnosing
a report of missing telemetry has to be able to find one.
"""
now = time.time()
for debris in directory.glob("telemetry-*.partial"):
try:
if now - debris.stat().st_mtime > CLAIM_STALE_SECONDS:
debris.unlink()
except OSError:
continue
for quarantined in directory.glob("telemetry-*.corrupt"):
try:
if now - quarantined.stat().st_mtime > CLAIM_EXPIRY_SECONDS:
quarantined.unlink()
except OSError:
continue
# The same reasoning covers *.tmp. _write_identity and _install_salt both
# create one and unlink it in a finally, which a SIGKILL skips, and no glob
# in this module matches the leftovers either.
for temporary in directory.glob("telemetry-*.tmp"):
try:
if now - temporary.stat().st_mtime > CLAIM_STALE_SECONDS:
temporary.unlink()
except OSError:
continue
def _claim_parked(directory: Path) -> Path | None:
"""Take the oldest abandoned claim, if any lease has actually expired.
Kept separate from the live spool so flush() can drain both in one run.
Previously parked batches were only reachable when no spool existed at all,
and because sessions keep recording there usually was one — so a batch
parked by a failed send waited until the 7-day expiry deleted it unsent,
even though its own presence is what started the sender.
"""
now = time.time()
for orphan in sorted(directory.glob("telemetry-*.sending"), key=_safe_mtime):
for orphan in sorted(directory.glob("telemetry-*.sending")):
try:
age = now - orphan.stat().st_mtime
except OSError:
continue
if age < CLAIM_STALE_SECONDS:
# Someone else holds a live lease on it. This check has to come
# first. Claiming a file bumps its attempt count and refreshes its
# mtime, so a sender that has just taken the final attempt looks
# exhausted to everyone else while it is actively draining. Judging
# exhaustion before liveness let a second sender unlink a batch out
# from under its owner, losing every event in it.
continue
# Attempts, not age. Every re-claim touches the mtime and every release
# backdates it by a fixed amount, so age is pinned near the stale
# threshold and never reaches the expiry. Age stays only as a backstop
# for files that never carried an attempt marker.
if _claim_attempt(orphan) >= MAX_CLAIM_ATTEMPTS or age > CLAIM_EXPIRY_SECONDS:
if age > CLAIM_EXPIRY_SECONDS:
try:
orphan.unlink()
except OSError:
pass
continue
claim = orphan.parent / _claim_name(_claim_attempt(orphan) + 1)
if age < CLAIM_STALE_SECONDS:
continue
try:
orphan.replace(claim)
_touch(claim)
return claim
except OSError:
continue
return None
def _safe_mtime(path: Path) -> float:
try:
return path.stat().st_mtime
except OSError:
return 0.0
def _rewrite_claim(claim: Path, remaining: list[dict[str, Any]]) -> bool:
"""Persist the unsent remainder, atomically, and refresh the lease.
Called after every successful batch. Two jobs: a retry resumes where the
send stopped instead of re-posting from the top, and the rewrite doubles as
the lease heartbeat, so a slow sender does not have its claim stolen
mid-flight. Interval is one batch, well inside CLAIM_STALE_SECONDS.
"""
if not remaining:
try:
claim.unlink()
except OSError:
pass
return True
temporary = claim.with_suffix(f".{os.getpid()}.partial")
try:
payload = "".join(json.dumps(event, separators=(",", ":"), default=str) + "\n" for event in remaining)
# fsync before the rename: without it the rename can land while the
# bytes have not, and the claim comes back empty or truncated after a
# crash. _drain then reads zero events and unlinks it.
with open(temporary, "w", encoding="utf-8") as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
temporary.replace(claim)
_touch(claim)
return True
except OSError:
try:
temporary.unlink()
except OSError:
pass
return False
def _release_claim(claim: Path, remaining: list[dict[str, Any]]) -> None:
"""Persist the remainder and drop the lease, because this sender has given up.
Distinct from the per-batch heartbeat: heartbeating on the way out would
make an abandoned batch look actively owned for a further
CLAIM_STALE_SECONDS, delaying the retry for no reason. Ageing it past the
threshold lets the next flush pick it up immediately, while the attempt
count in the filename still bounds how many times that can happen.
"""
if not _rewrite_claim(claim, remaining):
return
try:
# Backdate past the stale threshold so the next flush can pick it up,
# minus a cooldown that grows with the attempts already spent. Clamped so
# the mtime never lands in the future, which would read as a live lease.
cooldown = min(_claim_attempt(claim) * RETRY_COOLDOWN_SECONDS, CLAIM_STALE_SECONDS)
released = time.time() - CLAIM_STALE_SECONDS - 1 + cooldown
os.utime(claim, (released, released))
except OSError:
pass
def _resolve_email(key: str) -> str:
"""Trade the API key for the account email so events join other Mem0 surfaces."""
url = os.environ.get("MEM0_API_URL", memory_core.DEFAULT_API_URL).rstrip("/") + "/v1/ping/"
@@ -800,130 +300,34 @@ def _post(payload: dict[str, Any], url: str) -> bool:
def resolve_distinct_id() -> tuple[str, str]:
"""Return the PostHog distinct id and the anonymous id it replaced, if any.
The second value becomes a PostHog $identify alias. It is ONLY ever an
anonymous id: aliasing one account email to another merges two real person
profiles and cannot be undone, so a key that now belongs to a different
account re-resolves with no alias.
"""
"""Return the PostHog distinct id and the anonymous id it replaced, if any."""
identity = _read_identity()
key = memory_core.api_key()
fingerprint = _digest(key) if key else ""
email = identity.get("email", "")
if email and fingerprint:
recorded = identity.get("key_fingerprint", "")
if recorded == fingerprint:
return email, ""
if not recorded:
# Rows written before fingerprints existed. Verify rather than
# adopt: a key changed before the upgrade would otherwise bind the
# new key to the previous account's email, permanently, and the
# fingerprint would then agree with itself forever after.
verified = _resolve_email(key)
if not verified:
# Offline, firewalled, or the API is down. Keep the previous
# behaviour and retry on the next flush rather than dropping a
# real account attribution. Safe because the same network that
# failed /v1/ping/ is about to fail the PostHog POST, so nothing
# is delivered under the unverified identity in the meantime.
return email, ""
identity["email"] = verified
identity["key_fingerprint"] = fingerprint
_write_identity(identity)
return verified, ""
if email:
return email, ""
key = memory_core.api_key()
if not key:
# No key to verify the account with; do not keep attributing to it.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
return anonymous_id(identity), ""
resolved = _resolve_email(key)
if not resolved:
# The key changed and will not resolve (revoked, offline, API down).
# Reaching here with an email means the recorded fingerprint disagreed,
# so the key really did change. Drop the account and rotate: the stored
# anonymous id may already be merged into that account's person, and
# reusing it would keep the events on the profile we are trying to
# leave.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
email = _resolve_email(key)
if not email:
return anonymous_id(identity), ""
# Alias only when going anonymous -> email for the first time. Once an anon
# id has been merged into an account it must never be offered again: an
# alias naming an already-identified id is what could link two real people.
previous = "" if (email or identity.get("aliased")) else identity.get("anonymous_id", "")
if previous:
identity["aliased"] = True
identity["email"] = resolved
identity["key_fingerprint"] = fingerprint
previous = identity.get("anonymous_id", "")
identity["email"] = email
_write_identity(identity)
return resolved, previous
return email, previous
def flush() -> int:
"""Drain the live spool, then any parked claims, and return events sent."""
"""Drain claimed spools to PostHog and return the number of events sent."""
if not is_enabled():
return 0
sent, delivered = _drain(_claim_spool())
if not delivered:
# The network is failing. Retrying other batches now would only burn
# their attempt budget against the same broken connection.
return sent
# Parked batches used to starve behind the live spool indefinitely. Bounded
# per run so a long backlog cannot turn one flush into an unbounded loop.
directory = memory_core.data_dir()
_sweep_debris(directory)
for _ in range(MAX_PARKED_PER_RUN):
parked = _claim_parked(directory)
if parked is None:
break
count, delivered = _drain(parked)
sent += count
if not delivered:
break
return sent
def _drain(claim: Path | None) -> tuple[int, bool]:
"""Post one claimed batch file, recording progress after every batch.
Returns (events sent, whether everything was delivered).
"""
claim = _claim_spool()
if claim is None:
return 0, True
return 0
try:
lines = claim.read_text(encoding="utf-8").splitlines()
except ValueError:
# UnicodeDecodeError from a torn write: the content is unrecoverable, so
# quarantine rather than retry. flush() runs from a bare `finally:` in
# flush_worker, so raising here also skips the handoff cleanup, and an
# undecodable file would otherwise be re-read on every flush forever.
# Reported as delivered because there is nothing left to deliver and the
# rest of the run should continue.
try:
claim.replace(claim.with_suffix(".corrupt"))
except OSError:
try:
claim.unlink()
except OSError:
pass
return 0, True
except OSError:
# Could not read it, which is not the same as having nothing to send.
# The file is left exactly where it is: a vanished or briefly unreadable
# claim is retryable, and quarantining it here would discard events over
# a transient filesystem error. Reported as undelivered so the run stops
# instead of counting a batch nothing was posted from as delivered.
return 0, False
return 0
events = []
for line in lines:
try:
@@ -933,18 +337,11 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
if isinstance(value, dict) and value.get("event"):
events.append(value)
if not events:
# Only delete when the file really is empty. A non-empty file that
# parses to nothing is a torn write, and its contents are the unsent
# remainder — deleting it is the data loss this PR exists to prevent.
try:
empty = claim.stat().st_size == 0
except OSError:
empty = True
try:
claim.replace(claim.with_suffix(".corrupt")) if not empty else claim.unlink()
claim.unlink()
except OSError:
pass
return 0, True
return 0
distinct_id, aliased_anonymous_id = resolve_distinct_id()
if aliased_anonymous_id:
@@ -963,17 +360,12 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
sent = 0
for start in range(0, len(events), BATCH_SIZE):
chunk = events[start : start + BATCH_SIZE]
batch = [
{
"event": event["event"],
"distinct_id": distinct_id,
# Carried through from record() so a resend can be collapsed.
"uuid": event.get("uuid"),
"timestamp": event.get("timestamp"),
"properties": {
# Fallback only: events recorded by a build before source
# moved into record() have none of their own.
"source": _source_tag,
"language": "python",
"$process_person_profile": False,
@@ -981,24 +373,16 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
**(event.get("properties") or {}),
},
}
for event in chunk
for event in events[start : start + BATCH_SIZE]
]
if not _post({"api_key": POSTHOG_API_KEY, "batch": batch}, POSTHOG_BATCH_URL):
# Keep only what has not been delivered, and release the lease.
# Previously the whole file was kept and the retry re-posted every
# batch, including the ones that had already arrived.
_release_claim(claim, events[start:])
return sent, False
sent += len(chunk)
# Record progress and refresh the lease after each successful batch, so
# a crash repeats at most one batch instead of the entire file. If the
# rewrite fails the claim still holds delivered events, so stop rather
# than carry on as though progress were recorded — continuing is how the
# duplicate delivery this PR fixes would come back.
if not _rewrite_claim(claim, events[start + len(chunk) :]):
_release_claim(claim, events[start + len(chunk) :])
return sent, False
return sent, True
return sent
sent += len(batch)
try:
claim.unlink()
except OSError:
pass
return sent
def main() -> int:
@@ -1,6 +1,6 @@
{
"id": "mem0",
"version": "0.3.2",
"version": "0.3.1",
"homepage": "https://docs.mem0.ai/integrations/antigravity",
"native": {
"pluginRoot": "${ANTIGRAVITY_PLUGIN_ROOT}",
@@ -7,8 +7,8 @@ disable-model-invocation: true
# Pause memory capture
To pause (hooks stop capturing and sending session content; a minimal
telemetry ping still fires at session start, under your Mem0 account email,
unless `MEM0_TELEMETRY=false`):
anonymous telemetry ping still fires at session start unless
`MEM0_TELEMETRY=false`):
```bash
python3 "${ANTIGRAVITY_PLUGIN_ROOT}/core/memory_cli.py" --harness "antigravity" pause
@@ -1,6 +1,6 @@
{
"name": "mem0",
"version": "0.3.2",
"version": "0.3.1",
"description": "Cross-session memory and token savings for coding agents.",
"author": {
"name": "Mem0"
+2 -12
View File
@@ -136,23 +136,13 @@ Local data lives in `${CLAUDE_PLUGIN_DATA}`:
- `pending/`: sessions waiting to be sent to Mem0 (retried after interruption)
- `flush-worker.log`: whether memory creation succeeded
- `plugin-errors.log`: hook errors (no credentials)
- `telemetry.jsonl` / `telemetry-identity.json`: usage events and the id they are sent under
- `telemetry-salt`: random per-install salt for the repo and session hashes
- `install-state.json`: records that install has been counted once on this machine
- `telemetry.jsonl` / `telemetry-identity.json`: anonymous usage events
Mem0 receives captured user messages, Claude's answers, sidekick assignments and completed responses, and changed file paths. When a failed command is recorded, extraction can also include bounded command details and results. Complete files and general tool output stay on your machine. Values that look like credentials are redacted before anything is sent.
## Telemetry
Usage events (which hook ran, timing, result counts, failure types) so Mem0 can identify what's used and what's breaking.
**These events are not anonymous.** When an API key is configured — which installing the plugin requires — events are sent under your Mem0 account email, the same way the Python SDK and the CLI attribute theirs. Without a key they are sent under a random per-machine id.
What each event carries: the event name, the plugin version, the harness it ran in, your OS and Python version, and per-event properties describing what happened — timings, counts, coarse outcome and failure labels, and which model was configured. Repository and session identifiers are hashed with a random salt generated on your machine, so they cannot be linked back to a repository name or path.
Rather than restate a list that drifts, the exact set is enforced in code: `telemetry.record` filters every property through a denylist of sensitive keys and redacts credential-shaped values. See `_PRIVATE_KEYS` in `core/telemetry.py`.
Prompts, memory text, queries, file paths, repository names, and API keys are never sent.
Anonymous usage events (which hook ran, timing, result counts, failure types) so Mem0 can identify what's used and what's breaking. Repo and session IDs are hashed before leaving your machine. Prompts, memory text, file paths, tool output, and API keys are never sent.
Turn it off:
@@ -1,11 +0,0 @@
"""Generated by integrations/agent-plugin-core/build/build.py. Do not edit."""
HARNESS_ID = "claude-code"
SOURCE_TAG = "CLAUDE_CODE_PLUGIN"
# Platform-side vocabulary (mem0_event.source + X-Application). The whole
# plugin family is one source; which editor it runs in is the application.
# An empty application means the host is unknown, and memory_core omits
# the header entirely rather than sending a placeholder.
PLATFORM_SOURCE = "MEM0_PLUGIN"
PLATFORM_APPLICATION = "claude-code"
@@ -290,11 +290,6 @@ def run(
if args.plugin_data_dir:
os.environ[data_dir_env] = args.plugin_data_dir
# Snapshot BEFORE anything writes to the data dir: cache_plugin_api_key
# writes `api-key` and EvidenceStore creates `evidence.sqlite3`, so asking
# after them always saw content and every fresh install reported an upgrade.
data_dir_was_empty = telemetry.data_dir_was_empty()
cache_plugin_api_key()
if args.action == "session-start":
clear_stale_api_key_cache()
@@ -310,19 +305,8 @@ def run(
return 0
if args.action == "session-start":
# Claims the marker atomically and says which event to record, so a
# second session starting alongside this one cannot record it too.
first_event = telemetry.claim_install(was_empty=data_dir_was_empty)
if first_event == "install":
if telemetry.is_first_run():
telemetry.record("install")
elif first_event == "upgrade":
# First run after a build that never wrote the marker; the
# predecessor version was never recorded anywhere.
telemetry.record("upgrade", from_version="pre-0.3")
else:
previous = telemetry.claim_version_change()
if previous:
telemetry.record("upgrade", from_version=previous)
recovered = recover_pending_handoffs()
record_session_start(store, hook_input)
if recovered:
@@ -11,7 +11,6 @@ from __future__ import annotations
import functools
import hashlib
import json
import math
import os
import re
import sqlite3
@@ -27,9 +26,17 @@ from pathlib import Path
from typing import Any, Iterable
import telemetry
from message_utils import MAX_EXTRACTION_INPUT_TOKENS as MAX_EXTRACTION_INPUT_TOKENS
from message_utils import SECRET_PATTERNS as SECRET_PATTERNS
from message_utils import _estimated_tokens as _estimated_tokens
from message_utils import _is_agent_assignment as _is_agent_assignment
from message_utils import _is_agent_response as _is_agent_response
from message_utils import _message_tokens as _message_tokens
from message_utils import extraction_message_batches as extraction_message_batches
from message_utils import redact as redact
DEFAULT_API_URL = "https://api.mem0.ai"
PLUGIN_VERSION = "0.3.2"
PLUGIN_VERSION = "0.3.1"
_harness_name: str = "generic"
_harness_env_prefix: str = "MEM0_PLUGIN"
@@ -66,7 +73,6 @@ CHECKPOINT_EXCHANGES = 5
CHECKPOINT_MESSAGES = 10
CHECKPOINT_SOURCE_CHARS = 40000
DEFAULT_MAX_CONTEXT_CHARS = 4000
MAX_EXTRACTION_INPUT_TOKENS = 24000
MAX_FLUSH_ATTEMPTS = 5
FORGET_PAGE_SIZE = 100
FORGET_MAX_PAGES = 50
@@ -139,48 +145,11 @@ BUILD_COMMAND_RE = re.compile(
re.IGNORECASE,
)
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(
r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"
),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r'|(?:access|refresh|session)[_-]?token|token|authorization|credential'
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def redact(value: Any) -> str:
text = (
value
if isinstance(value, str)
else json.dumps(value, ensure_ascii=False, default=str)
)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def bounded(value: Any, limit: int) -> str:
text = redact(value).strip()
if len(text) <= limit:
@@ -1706,128 +1675,6 @@ def build_extraction_messages(structured: dict[str, Any]) -> list[dict[str, str]
return messages
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if (
_is_agent_assignment(message)
and index + 1 < len(exchange)
and _is_agent_response(exchange[index + 1])
):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
# Platform surface attribution. Read from the generated per-host module so a new
# entrypoint is correct without remembering to configure anything.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
except ImportError:
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
def platform_headers(key: str) -> dict[str, str]:
"""Auth plus the three surface-identity headers.
X-Mem0-Source and X-Application are set-once by contract: this is the
outermost layer, so it sets them, and nothing below may overwrite them.
X-Mem0-Client is append-only — anything downstream adds itself to the tail.
"""
headers = {
"Authorization": f"Token {key}",
"Content-Type": "application/json",
"X-Mem0-Source": _PLATFORM_SOURCE,
"X-Mem0-Client": f"mem0-plugin/{PLUGIN_VERSION}",
}
if _PLATFORM_APPLICATION:
headers["X-Application"] = _PLATFORM_APPLICATION
return headers
def _request_json(
url: str, key: str, payload: dict[str, Any], timeout: float
) -> tuple[dict[str, Any] | list[Any], int, int]:
@@ -1835,7 +1682,7 @@ def _request_json(
request = urllib.request.Request(
url,
data=raw,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -1862,7 +1709,7 @@ def _get_json(
) -> tuple[dict[str, Any] | list[Any], int]:
request = urllib.request.Request(
url,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="GET",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -2008,13 +1855,6 @@ def flush_session(
"user_id": write_user,
"app_id": repo.app_id,
"run_id": session_id,
# Top level, not metadata: the backend reads `source` from the body or
# the query string, never from metadata, which is where this used to
# sit. The X-Mem0-Source header is also read, but only from the
# platform release that ships alongside this change, so the body value
# is what makes attribution work on both. The harness tag stays in
# metadata as hook provenance.
"source": _PLATFORM_SOURCE,
"metadata": {**metadata, "author": write_user, "dirs": directory_chain(repo)},
"agent_custom_instructions": PROJECT_MEMORY_INSTRUCTIONS,
"custom_instructions": PERSONAL_MEMORY_INSTRUCTIONS,
@@ -2558,7 +2398,7 @@ def _collect_memory_ids(
def _delete_memory(api_url: str, key: str, memory_id: str) -> bool:
request = urllib.request.Request(
f"{api_url}/v1/memories/{urllib.parse.quote(memory_id)}/",
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="DELETE",
)
try:
@@ -0,0 +1,127 @@
"""Shared, host-independent redaction and lossless extraction batching."""
from __future__ import annotations
import json
import math
import re
from typing import Any
MAX_EXTRACTION_INPUT_TOKENS = 24000
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r"|(?:access|refresh|session)[_-]?token|token|authorization|credential"
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def redact(value: Any) -> str:
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, default=str)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if _is_agent_assignment(message) and index + 1 < len(exchange) and _is_agent_response(exchange[index + 1]):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
+39 -655
View File
@@ -1,9 +1,5 @@
#!/usr/bin/env python3
"""Usage telemetry for Mem0 agent plugins.
Events are linked to your Mem0 account email when an API key is configured, and
to a random per-machine id otherwise. Not anonymous — the Python SDK and CLI
attribute the same way.
"""Anonymous usage telemetry for Mem0 agent plugins.
Hooks run on a 3-6 second budget and fire on every tool call, so recording never
touches the network: `record` appends one JSON line to a local spool and returns.
@@ -13,8 +9,7 @@ started once per session and again from the flush worker that is already detache
Pure stdlib, matching the rest of the plugin. Opt out with MEM0_TELEMETRY=false.
Never sends prompts, memory text, queries, file paths, repository names, or API
keys: only event names, durations, counts, coarse outcomes, and repo/session
identifiers hashed with a random per-install salt.
keys: only event names, durations, counts, coarse outcomes, and salted hashes.
"""
from __future__ import annotations
@@ -34,24 +29,8 @@ from typing import Any
import memory_core
# Seeded from the per-host module the build generates into core/. Two processes
# in this pipeline never call init() — mcp_server.py, and the detached
# `python3 telemetry.py` sender that spawn_flush() starts — so a module default
# was what every one of their events got labelled with.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import HARNESS_ID as _DEFAULT_HARNESS
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
from _harness_id import SOURCE_TAG as _DEFAULT_SOURCE_TAG
except ImportError:
_DEFAULT_HARNESS = "generic"
_DEFAULT_SOURCE_TAG = "MEM0_PLUGIN"
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
_salt_cache: str = ""
_harness: str = _DEFAULT_HARNESS
_source_tag: str = _DEFAULT_SOURCE_TAG
_harness: str = "generic"
_source_tag: str = "MEM0_PLUGIN"
_PRIVATE_KEYS = {
"apikey",
"authorization",
@@ -77,19 +56,10 @@ _PRIVATE_KEYS = {
}
def init(harness: str = "", source_tag: str = "") -> None:
"""Override the generated identity. Optional — core/_harness_id.py is the default.
The fallback shape matches memory_core.configure_harness's (``<HOST>_PLUGIN``).
It used to be ``MEM0_<HOST>_PLUGIN`` here and ``<host>_plugin`` there, which
meant one plugin could emit three different source values depending on which
process happened to send the batch.
"""
def init(harness: str = "generic", source_tag: str = "") -> None:
global _harness, _source_tag
_harness = harness or _DEFAULT_HARNESS
_source_tag = source_tag or (
f"{_harness.upper().replace('-', '_')}_PLUGIN" if harness else _DEFAULT_SOURCE_TAG
)
_harness = harness
_source_tag = source_tag or f"MEM0_{harness.upper().replace('-', '_')}_PLUGIN"
POSTHOG_API_KEY = "phc_hgJkUVJFYtmaJqrvf6CYN67TIQ8yhXAkWzUn9AMU4yX"
POSTHOG_CAPTURE_URL = "https://us.i.posthog.com/i/v0/e/"
@@ -100,16 +70,6 @@ BATCH_SIZE = 100
SEND_TIMEOUT = 5
CLAIM_STALE_SECONDS = 120
CLAIM_EXPIRY_SECONDS = 7 * 24 * 60 * 60
# A batch is only discarded once it has genuinely been retried this many times.
MAX_CLAIM_ATTEMPTS = 3
# Parked claims drained per run, after the live spool. Bounded so a long backlog
# cannot turn one flush into an unbounded send loop.
MAX_PARKED_PER_RUN = 3
# Added to the wait before a released claim becomes reclaimable, per attempt
# already spent. Releasing straight to "reclaimable now" let two senders burn the
# whole budget within seconds of one another on a single momentary failure, and
# discard a batch a retry a minute later would have delivered.
RETRY_COOLDOWN_SECONDS = 60
def is_enabled() -> bool:
@@ -123,126 +83,9 @@ def is_enabled() -> bool:
def _digest(value: str, length: int = 16) -> str:
"""Unsalted digest. Only for values that are already secrets (API keys)."""
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:length]
def _salt_path() -> Path:
return memory_core.data_dir() / "telemetry-salt"
def _install_salt() -> str:
"""Random per-install salt, created once and memoized for the process.
Deliberately its own file, claimed with O_CREAT|O_EXCL, rather than a key in
the identity file. Three reasons, all of which produced wrong data when this
lived in the identity dict:
- Hooks are short-lived separate processes firing on every tool call, and
people run more than one agent window. A read-modify-write would let each
process mint its own salt, so one repository would hash several ways in the
window before a writer won.
- resolve_distinct_id holds a copy of the identity dict across a network call
to /v1/ping/, so whichever write landed second erased the other's key —
losing either the salt (repo_hash changes mid-stream) or the email (a
second $identify, splitting the person).
- Touching the identity file from record() would create it, and is_first_run
keys off that file, so recording an event would silently suppress the
install event.
Published atomically, and there is deliberately no derived fallback. Creating
the file with O_CREAT|O_EXCL and then writing into it leaves a window where
the file exists and is empty, and a concurrent hook that reads it in that
window gets nothing. Falling back to a digest of the path would hand that
process a salt an attacker can compute, memoized for its whole run, which is
the privacy control this function exists to provide silently turning itself
off under load. The salt is written to a private temp file first and linked
into place, so the name either does not exist or already has the full value.
Returns "" when it genuinely cannot persist. Callers omit the hash entirely
rather than emit an unsalted one.
"""
global _salt_cache
if _salt_cache:
return _salt_cache
path = _salt_path()
# Read before writing. Hooks are separate processes firing on every tool
# call, so all but the first find the salt already published; going straight
# to create-fsync-link-unlink meant every one of them paid an fsync to
# discover that, on a path whose whole promise is appending a line and
# returning.
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
if _salt_cache:
return _salt_cache
except OSError:
pass
temporary = path.with_name(f"{path.name}.{os.getpid()}.tmp")
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(handle, "w", encoding="utf-8") as stream:
stream.write(uuid.uuid4().hex)
stream.flush()
os.fsync(stream.fileno())
try:
# Atomic claim: fails if another process already published one.
# os.link rather than replace, which would clobber theirs.
os.link(temporary, path)
except FileExistsError:
pass
except OSError:
# No hardlinks here (some network mounts, some container volumes).
# Claim the name directly instead. That reopens the empty-file
# window, but the window is now benign: a reader that lands in it
# gets "" and omits the hash for that process rather than caching a
# guessable one. Losing the hashes on every run of an entire
# filesystem is the worse failure.
try:
fallback = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(fallback, "w", encoding="utf-8") as stream:
stream.write(temporary.read_text(encoding="utf-8"))
except OSError:
pass
except OSError:
pass
finally:
try:
temporary.unlink()
except OSError:
pass
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
except OSError:
_salt_cache = ""
return _salt_cache
def _scoped_digest(value: str, length: int = 16) -> str:
"""Salted digest for values drawn from a guessable space.
repo.identity is a git remote URL, or ``local:<absolute path>`` when there is
no remote — which normally contains the account username. Sixteen unsalted
hex characters over that input space is enumerable, so this is not a
privacy control without the salt. Salting per install keeps every
within-account join the analytics actually use and gives up only
cross-machine joins on the same repository, which nothing computes.
Returns "" when there is no salt, so record() omits the property. An
unsalted digest over this input space is close to plaintext, and emitting one
under a name that implies it is hashed is worse than sending nothing.
"""
if not value:
return ""
salt = _install_salt()
if not salt:
return ""
return hashlib.sha256(f"{salt}:{value}".encode("utf-8")).hexdigest()[:length]
def _safe_value(value: Any) -> Any:
if isinstance(value, str):
return memory_core.redact(value)
@@ -302,176 +145,9 @@ def anonymous_id(identity: dict[str, str] | None = None) -> str:
return created
def _rotate_anonymous_id(identity: dict[str, str]) -> str:
"""Mint a fresh anonymous id because the account context is gone.
The previous id may already have been merged into a person profile by an
$identify, and that merge is permanent. Reusing it after a logout or a key
change attributes everything that follows to the account that just went
away, which is the same misattribution the key fingerprint exists to stop,
only arriving through the anonymous path instead.
`aliased` is cleared with it: the new id has never been merged, so it is
eligible to be aliased into whatever account comes next.
"""
created = f"code-anon-{uuid.uuid4().hex}"
identity["anonymous_id"] = created
identity.pop("aliased", None)
_write_identity(identity)
return created
def _install_state_path() -> Path:
return memory_core.data_dir() / "install-state.json"
def is_first_run() -> bool:
"""Whether install has never been recorded on this machine.
Deliberately NOT the identity file. That file is only written by a
successful flush, so an offline or firewalled user recorded code.install on
every single session, forever — and every 0.2.x user recorded one on their
first 0.3.x session because 0.2.x never wrote it at all.
"""
return not _install_state_path().exists()
def data_dir_was_empty() -> bool:
"""Whether the data directory is untouched. Call BEFORE anything writes to it.
hook_runner reaches claim_install() only after cache_plugin_api_key() has
written `api-key` and EvidenceStore() has created `evidence.sqlite3`, so
asking at claim time always saw content and every fresh install reported an
upgrade. The caller snapshots this at the top of the run instead.
"""
return not _data_dir_has_content()
def claim_install(was_empty: bool | None = None) -> str | None:
"""Claim the one install/upgrade record for this machine, atomically.
Returns the event to record ("install" or "upgrade"), or None if another
session already claimed it. O_CREAT|O_EXCL so two sessions starting together
cannot both win.
`was_empty` must come from data_dir_was_empty() called before this process
wrote anything. Omitting it falls back to checking now, which is only
correct for a caller that has touched nothing.
"""
if not is_enabled():
# Never consume the one-shot claim while the user is opted out, or they
# would silently lose their install event if they later opt in.
return None
path = _install_state_path()
upgrading = not (data_dir_was_empty() if was_empty is None else was_empty)
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
except FileExistsError:
return None
except OSError:
return None
try:
with os.fdopen(handle, "w", encoding="utf-8") as stream:
json.dump(
{
"plugin_version": memory_core.PLUGIN_VERSION,
"installed_at": memory_core.utc_now(),
"upgraded": upgrading,
},
stream,
)
# Durable before this returns. The O_EXCL open is what makes the
# claim exclusive, so it cannot be replaced by a temp-and-rename
# without losing that, which leaves the content as the thing to make
# safe. A kill between the open and this fsync used to leave a marker
# that exists but parses to nothing: is_first_run reads it as claimed
# and claim_version_change cannot read a version out of it.
stream.flush()
os.fsync(stream.fileno())
except OSError:
pass
return "upgrade" if upgrading else "install"
def _data_dir_has_content() -> bool:
"""Whether anything predates this session in the plugin data directory."""
try:
for entry in memory_core.data_dir().iterdir():
if entry.name != "install-state.json":
return True
except OSError:
pass
return False
def _repair_install_state(path: Path) -> None:
"""Rewrite an unparseable marker so version tracking can resume."""
try:
temporary = path.with_suffix(f".{os.getpid()}.tmp")
temporary.write_text(
json.dumps({"plugin_version": memory_core.PLUGIN_VERSION, "repaired_at": memory_core.utc_now()}),
encoding="utf-8",
)
temporary.replace(path)
except OSError:
pass
def claim_version_change() -> str | None:
"""Return the previously recorded version if it differs, updating the marker.
Only meaningful once the marker exists — the first transition into 0.3.x has
no recorded predecessor and reports "pre-0.3" instead. Claiming by rewriting
the marker means the next session sees no change and records nothing.
"""
path = _install_state_path()
try:
state = json.loads(path.read_text(encoding="utf-8"))
except OSError:
return None
except json.JSONDecodeError:
# A crash between O_EXCL and the write leaves an empty marker. Left
# alone it disables every future upgrade event on this machine, because
# claim_install sees the file and this function cannot parse it.
state = None
if not isinstance(state, dict):
_repair_install_state(path)
return None
previous = str(state.get("plugin_version") or "")
if not previous or previous == memory_core.PLUGIN_VERSION:
return None
# Claim the transition with an exclusive sentinel before rewriting the
# marker. A plain read-modify-write let every concurrently starting session
# observe the old version and each record its own upgrade — and the first
# session after a version bump is exactly when several agent windows restart
# together.
sentinel = path.with_name(f"upgraded-{memory_core.PLUGIN_VERSION}")
try:
os.close(os.open(sentinel, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600))
except FileExistsError:
return None
except OSError:
return None
state["plugin_version"] = memory_core.PLUGIN_VERSION
state["upgraded_at"] = memory_core.utc_now()
temporary = path.with_suffix(f".{os.getpid()}.tmp")
try:
temporary.write_text(json.dumps(state), encoding="utf-8")
temporary.replace(path)
except OSError:
# Release the claim. The marker still records the old version, so
# without this the sentinel makes claim_version_change return early on
# every later run and this version's upgrade is never recorded again.
for leftover in (sentinel, temporary):
try:
leftover.unlink()
except OSError:
pass
return None
return previous
"""Whether this machine has never recorded a plugin event before."""
return not _identity_path().exists()
def record(
@@ -492,32 +168,19 @@ def record(
except OSError:
pass
properties = _safe_value(properties)
# Stamped in the RECORDING process, beside harness. `source` used to be
# read in the sending process from a module global, so whichever process
# drained the spool named every event in it. flush() spreads per-event
# properties last, so this now wins over any sender's default.
properties.update(
harness=_harness,
source=_source_tag,
plugin_version=memory_core.PLUGIN_VERSION,
os=sys.platform,
python_version=platform.python_version(),
)
# Assigned only when the digest is real. _scoped_digest returns "" when
# the salt could not be persisted, and an empty property is worse than an
# absent one: it survives the None filter below and reads as a value.
if repo is not None:
repo_hash = _scoped_digest(getattr(repo, "identity", ""))
if repo_hash:
properties["repo_hash"] = repo_hash
properties["repo_hash"] = _digest(getattr(repo, "identity", ""))
if session_id:
session_hash = _scoped_digest(session_id)
if session_hash:
properties["session_hash"] = session_hash
properties["session_hash"] = _digest(session_id)
line = json.dumps(
{
"event": f"{EVENT_PREFIX}.{event}",
"uuid": str(uuid.uuid4()),
"timestamp": memory_core.utc_now(),
"properties": {
key: value for key, value in properties.items() if value is not None
@@ -576,201 +239,38 @@ def spawn_flush() -> bool:
return False
def _claim_name(attempt: int = 0) -> str:
"""Claim filename. The attempt count rides in the name so the 7-day expiry
only ever discards a batch that was actually retried and failed."""
return f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}-a{attempt}.sending"
def _claim_attempt(claim: Path) -> int:
"""Attempts recorded in a claim filename; 0 for the pre-attempt-count shape.
Anchored on field position, not on a leading "a": the legacy shape is
``telemetry-<pid>-<hex>.sending`` and a hex id such as ``a1234567`` would
otherwise parse as attempt 1234567 and be discarded unsent on the first
flush after an upgrade.
"""
stem = claim.name[: -len(".sending")] if claim.name.endswith(".sending") else claim.name
parts = stem.split("-")
if len(parts) != 4:
return 0
tail = parts[3]
if tail.startswith("a") and tail[1:].isdigit():
return int(tail[1:])
return 0
def _touch(path: Path) -> None:
"""Refresh mtime so a claim's age measures time since it was claimed.
``Path.replace`` is ``os.rename``, which preserves mtime — so a claim created
after a quiet minute inherited the spool's last-write time and looked
abandoned the instant it was made. A second sender would then take it over
while the first was still posting, and both would deliver the batch.
"""
try:
os.utime(path, None)
except OSError:
pass
def _claim_spool() -> Path | None:
"""Rename the spool aside so exactly one sender owns each batch."""
directory = memory_core.data_dir()
claim = directory / _claim_name()
claim = directory / f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}.sending"
spool = _spool_path()
try:
spool.replace(claim)
_touch(claim)
return claim
except OSError:
pass
return _claim_parked(directory)
def _sweep_debris(directory: Path) -> None:
"""Remove files nothing else will ever pick up again.
*.partial is a temp file orphaned by a crash between write and rename.
*.corrupt is a batch quarantined for undecodable content. No glob in this
module matches either, so without this they accumulate on disk for the life
of the install.
Quarantined batches are kept far longer than debris: they are the only
evidence left of events that could not be delivered, and someone diagnosing
a report of missing telemetry has to be able to find one.
"""
now = time.time()
for debris in directory.glob("telemetry-*.partial"):
try:
if now - debris.stat().st_mtime > CLAIM_STALE_SECONDS:
debris.unlink()
except OSError:
continue
for quarantined in directory.glob("telemetry-*.corrupt"):
try:
if now - quarantined.stat().st_mtime > CLAIM_EXPIRY_SECONDS:
quarantined.unlink()
except OSError:
continue
# The same reasoning covers *.tmp. _write_identity and _install_salt both
# create one and unlink it in a finally, which a SIGKILL skips, and no glob
# in this module matches the leftovers either.
for temporary in directory.glob("telemetry-*.tmp"):
try:
if now - temporary.stat().st_mtime > CLAIM_STALE_SECONDS:
temporary.unlink()
except OSError:
continue
def _claim_parked(directory: Path) -> Path | None:
"""Take the oldest abandoned claim, if any lease has actually expired.
Kept separate from the live spool so flush() can drain both in one run.
Previously parked batches were only reachable when no spool existed at all,
and because sessions keep recording there usually was one — so a batch
parked by a failed send waited until the 7-day expiry deleted it unsent,
even though its own presence is what started the sender.
"""
now = time.time()
for orphan in sorted(directory.glob("telemetry-*.sending"), key=_safe_mtime):
for orphan in sorted(directory.glob("telemetry-*.sending")):
try:
age = now - orphan.stat().st_mtime
except OSError:
continue
if age < CLAIM_STALE_SECONDS:
# Someone else holds a live lease on it. This check has to come
# first. Claiming a file bumps its attempt count and refreshes its
# mtime, so a sender that has just taken the final attempt looks
# exhausted to everyone else while it is actively draining. Judging
# exhaustion before liveness let a second sender unlink a batch out
# from under its owner, losing every event in it.
continue
# Attempts, not age. Every re-claim touches the mtime and every release
# backdates it by a fixed amount, so age is pinned near the stale
# threshold and never reaches the expiry. Age stays only as a backstop
# for files that never carried an attempt marker.
if _claim_attempt(orphan) >= MAX_CLAIM_ATTEMPTS or age > CLAIM_EXPIRY_SECONDS:
if age > CLAIM_EXPIRY_SECONDS:
try:
orphan.unlink()
except OSError:
pass
continue
claim = orphan.parent / _claim_name(_claim_attempt(orphan) + 1)
if age < CLAIM_STALE_SECONDS:
continue
try:
orphan.replace(claim)
_touch(claim)
return claim
except OSError:
continue
return None
def _safe_mtime(path: Path) -> float:
try:
return path.stat().st_mtime
except OSError:
return 0.0
def _rewrite_claim(claim: Path, remaining: list[dict[str, Any]]) -> bool:
"""Persist the unsent remainder, atomically, and refresh the lease.
Called after every successful batch. Two jobs: a retry resumes where the
send stopped instead of re-posting from the top, and the rewrite doubles as
the lease heartbeat, so a slow sender does not have its claim stolen
mid-flight. Interval is one batch, well inside CLAIM_STALE_SECONDS.
"""
if not remaining:
try:
claim.unlink()
except OSError:
pass
return True
temporary = claim.with_suffix(f".{os.getpid()}.partial")
try:
payload = "".join(json.dumps(event, separators=(",", ":"), default=str) + "\n" for event in remaining)
# fsync before the rename: without it the rename can land while the
# bytes have not, and the claim comes back empty or truncated after a
# crash. _drain then reads zero events and unlinks it.
with open(temporary, "w", encoding="utf-8") as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
temporary.replace(claim)
_touch(claim)
return True
except OSError:
try:
temporary.unlink()
except OSError:
pass
return False
def _release_claim(claim: Path, remaining: list[dict[str, Any]]) -> None:
"""Persist the remainder and drop the lease, because this sender has given up.
Distinct from the per-batch heartbeat: heartbeating on the way out would
make an abandoned batch look actively owned for a further
CLAIM_STALE_SECONDS, delaying the retry for no reason. Ageing it past the
threshold lets the next flush pick it up immediately, while the attempt
count in the filename still bounds how many times that can happen.
"""
if not _rewrite_claim(claim, remaining):
return
try:
# Backdate past the stale threshold so the next flush can pick it up,
# minus a cooldown that grows with the attempts already spent. Clamped so
# the mtime never lands in the future, which would read as a live lease.
cooldown = min(_claim_attempt(claim) * RETRY_COOLDOWN_SECONDS, CLAIM_STALE_SECONDS)
released = time.time() - CLAIM_STALE_SECONDS - 1 + cooldown
os.utime(claim, (released, released))
except OSError:
pass
def _resolve_email(key: str) -> str:
"""Trade the API key for the account email so events join other Mem0 surfaces."""
url = os.environ.get("MEM0_API_URL", memory_core.DEFAULT_API_URL).rstrip("/") + "/v1/ping/"
@@ -800,130 +300,34 @@ def _post(payload: dict[str, Any], url: str) -> bool:
def resolve_distinct_id() -> tuple[str, str]:
"""Return the PostHog distinct id and the anonymous id it replaced, if any.
The second value becomes a PostHog $identify alias. It is ONLY ever an
anonymous id: aliasing one account email to another merges two real person
profiles and cannot be undone, so a key that now belongs to a different
account re-resolves with no alias.
"""
"""Return the PostHog distinct id and the anonymous id it replaced, if any."""
identity = _read_identity()
key = memory_core.api_key()
fingerprint = _digest(key) if key else ""
email = identity.get("email", "")
if email and fingerprint:
recorded = identity.get("key_fingerprint", "")
if recorded == fingerprint:
return email, ""
if not recorded:
# Rows written before fingerprints existed. Verify rather than
# adopt: a key changed before the upgrade would otherwise bind the
# new key to the previous account's email, permanently, and the
# fingerprint would then agree with itself forever after.
verified = _resolve_email(key)
if not verified:
# Offline, firewalled, or the API is down. Keep the previous
# behaviour and retry on the next flush rather than dropping a
# real account attribution. Safe because the same network that
# failed /v1/ping/ is about to fail the PostHog POST, so nothing
# is delivered under the unverified identity in the meantime.
return email, ""
identity["email"] = verified
identity["key_fingerprint"] = fingerprint
_write_identity(identity)
return verified, ""
if email:
return email, ""
key = memory_core.api_key()
if not key:
# No key to verify the account with; do not keep attributing to it.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
return anonymous_id(identity), ""
resolved = _resolve_email(key)
if not resolved:
# The key changed and will not resolve (revoked, offline, API down).
# Reaching here with an email means the recorded fingerprint disagreed,
# so the key really did change. Drop the account and rotate: the stored
# anonymous id may already be merged into that account's person, and
# reusing it would keep the events on the profile we are trying to
# leave.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
email = _resolve_email(key)
if not email:
return anonymous_id(identity), ""
# Alias only when going anonymous -> email for the first time. Once an anon
# id has been merged into an account it must never be offered again: an
# alias naming an already-identified id is what could link two real people.
previous = "" if (email or identity.get("aliased")) else identity.get("anonymous_id", "")
if previous:
identity["aliased"] = True
identity["email"] = resolved
identity["key_fingerprint"] = fingerprint
previous = identity.get("anonymous_id", "")
identity["email"] = email
_write_identity(identity)
return resolved, previous
return email, previous
def flush() -> int:
"""Drain the live spool, then any parked claims, and return events sent."""
"""Drain claimed spools to PostHog and return the number of events sent."""
if not is_enabled():
return 0
sent, delivered = _drain(_claim_spool())
if not delivered:
# The network is failing. Retrying other batches now would only burn
# their attempt budget against the same broken connection.
return sent
# Parked batches used to starve behind the live spool indefinitely. Bounded
# per run so a long backlog cannot turn one flush into an unbounded loop.
directory = memory_core.data_dir()
_sweep_debris(directory)
for _ in range(MAX_PARKED_PER_RUN):
parked = _claim_parked(directory)
if parked is None:
break
count, delivered = _drain(parked)
sent += count
if not delivered:
break
return sent
def _drain(claim: Path | None) -> tuple[int, bool]:
"""Post one claimed batch file, recording progress after every batch.
Returns (events sent, whether everything was delivered).
"""
claim = _claim_spool()
if claim is None:
return 0, True
return 0
try:
lines = claim.read_text(encoding="utf-8").splitlines()
except ValueError:
# UnicodeDecodeError from a torn write: the content is unrecoverable, so
# quarantine rather than retry. flush() runs from a bare `finally:` in
# flush_worker, so raising here also skips the handoff cleanup, and an
# undecodable file would otherwise be re-read on every flush forever.
# Reported as delivered because there is nothing left to deliver and the
# rest of the run should continue.
try:
claim.replace(claim.with_suffix(".corrupt"))
except OSError:
try:
claim.unlink()
except OSError:
pass
return 0, True
except OSError:
# Could not read it, which is not the same as having nothing to send.
# The file is left exactly where it is: a vanished or briefly unreadable
# claim is retryable, and quarantining it here would discard events over
# a transient filesystem error. Reported as undelivered so the run stops
# instead of counting a batch nothing was posted from as delivered.
return 0, False
return 0
events = []
for line in lines:
try:
@@ -933,18 +337,11 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
if isinstance(value, dict) and value.get("event"):
events.append(value)
if not events:
# Only delete when the file really is empty. A non-empty file that
# parses to nothing is a torn write, and its contents are the unsent
# remainder — deleting it is the data loss this PR exists to prevent.
try:
empty = claim.stat().st_size == 0
except OSError:
empty = True
try:
claim.replace(claim.with_suffix(".corrupt")) if not empty else claim.unlink()
claim.unlink()
except OSError:
pass
return 0, True
return 0
distinct_id, aliased_anonymous_id = resolve_distinct_id()
if aliased_anonymous_id:
@@ -963,17 +360,12 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
sent = 0
for start in range(0, len(events), BATCH_SIZE):
chunk = events[start : start + BATCH_SIZE]
batch = [
{
"event": event["event"],
"distinct_id": distinct_id,
# Carried through from record() so a resend can be collapsed.
"uuid": event.get("uuid"),
"timestamp": event.get("timestamp"),
"properties": {
# Fallback only: events recorded by a build before source
# moved into record() have none of their own.
"source": _source_tag,
"language": "python",
"$process_person_profile": False,
@@ -981,24 +373,16 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
**(event.get("properties") or {}),
},
}
for event in chunk
for event in events[start : start + BATCH_SIZE]
]
if not _post({"api_key": POSTHOG_API_KEY, "batch": batch}, POSTHOG_BATCH_URL):
# Keep only what has not been delivered, and release the lease.
# Previously the whole file was kept and the retry re-posted every
# batch, including the ones that had already arrived.
_release_claim(claim, events[start:])
return sent, False
sent += len(chunk)
# Record progress and refresh the lease after each successful batch, so
# a crash repeats at most one batch instead of the entire file. If the
# rewrite fails the claim still holds delivered events, so stop rather
# than carry on as though progress were recorded — continuing is how the
# duplicate delivery this PR fixes would come back.
if not _rewrite_claim(claim, events[start + len(chunk) :]):
_release_claim(claim, events[start + len(chunk) :])
return sent, False
return sent, True
return sent
sent += len(batch)
try:
claim.unlink()
except OSError:
pass
return sent
def main() -> int:
@@ -1,6 +1,6 @@
{
"id": "mem0",
"version": "0.3.2",
"version": "0.3.1",
"homepage": "https://docs.mem0.ai/integrations/claude-code",
"native": {
"pluginRoot": "${CLAUDE_PLUGIN_ROOT}",
@@ -7,8 +7,8 @@ disable-model-invocation: true
# Pause memory capture
To pause (hooks stop capturing and sending session content; a minimal
telemetry ping still fires at session start, under your Mem0 account email,
unless `MEM0_TELEMETRY=false`):
anonymous telemetry ping still fires at session start unless
`MEM0_TELEMETRY=false`):
```bash
python3 "${CLAUDE_PLUGIN_ROOT}/core/memory_cli.py" --harness "claude-code" --plugin-data-dir "${CLAUDE_PLUGIN_DATA}" pause
@@ -1,7 +1,6 @@
from __future__ import annotations
import json
import re
import os
import sqlite3
import subprocess
@@ -3461,12 +3460,7 @@ def test_automatic_flush_can_be_disabled_for_external_harnesses(isolated_env):
def test_version_is_single_sourced():
manifest = json.loads((PLUGIN_ROOT / ".claude-plugin" / "plugin.json").read_text())
assert manifest["name"] == "mem0"
# Compared against PLUGIN_VERSION, never a literal. A hardcoded version here
# was one more place to edit on every release, inside the test asserting the
# version is single-sourced, and it caught nothing that the agreement checks
# below do not: fifteen places set to the same wrong value would still pass.
assert re.fullmatch(r"\d+\.\d+\.\d+", memory_core.PLUGIN_VERSION), memory_core.PLUGIN_VERSION
assert manifest["version"] == memory_core.PLUGIN_VERSION
assert manifest["version"] == memory_core.PLUGIN_VERSION == "0.3.1"
root = REPOSITORY_ROOT
for mp in (root / "marketplace.json", root / ".claude-plugin" / "marketplace.json"):
entry = next(p for p in json.loads(mp.read_text())["plugins"] if p["name"] == "mem0")
@@ -1,7 +1,6 @@
from __future__ import annotations
import json
import os
import sys
from pathlib import Path
from unittest.mock import patch
@@ -199,11 +198,9 @@ def test_a_stale_claim_is_reclaimed(isolated_env, monkeypatch):
telemetry.record("search")
orphan = telemetry._claim_spool()
assert orphan is not None
# Frozen rather than re-stat'd per call: flush() drains the live spool and
# then looks for parked claims in the same run, so by the second look this
# file no longer exists.
stale_now = orphan.stat().st_mtime + telemetry.CLAIM_STALE_SECONDS + 1
monkeypatch.setattr(telemetry.time, "time", lambda: stale_now)
monkeypatch.setattr(
telemetry.time, "time", lambda: orphan.stat().st_mtime + telemetry.CLAIM_STALE_SECONDS + 1
)
with patch.object(telemetry, "_post", lambda payload, url: True):
assert telemetry.flush() == 1
@@ -213,13 +210,9 @@ def test_an_expired_claim_is_dropped(isolated_env, monkeypatch):
telemetry.record("search")
orphan = telemetry._claim_spool()
assert orphan is not None
expired_now = orphan.stat().st_mtime + telemetry.CLAIM_EXPIRY_SECONDS + 1
monkeypatch.setattr(telemetry.time, "time", lambda: expired_now)
# Expiry now only discards a batch that was genuinely retried and failed,
# so age alone is not enough — age it past the attempt budget too.
retried = orphan.parent / orphan.name.replace("-a0.", f"-a{telemetry.MAX_CLAIM_ATTEMPTS}.")
orphan.replace(retried)
os.utime(retried, (expired_now, expired_now - telemetry.CLAIM_EXPIRY_SECONDS - 1))
monkeypatch.setattr(
telemetry.time, "time", lambda: orphan.stat().st_mtime + telemetry.CLAIM_EXPIRY_SECONDS + 1
)
assert telemetry._claim_spool() is None
assert not list(memory_core.data_dir().glob("telemetry-*.sending"))
@@ -265,150 +258,12 @@ def test_an_unresolvable_key_falls_back_to_the_anonymous_id(isolated_env, monkey
assert telemetry.resolve_distinct_id()[0].startswith("code-anon-")
def test_logging_out_does_not_leave_events_on_the_previous_account(isolated_env, monkeypatch):
"""Review finding: clearing the email kept an id already merged into a person.
The anonymous id is offered to PostHog as $anon_distinct_id on first sign-in,
and that merge is permanent. Keeping it after the key goes away means every
later anonymous event lands on the account that just left.
"""
# Run anonymously first, which is the only way an id exists to be merged.
merged = telemetry.anonymous_id()
monkeypatch.setenv("MEM0_API_KEY", "key-for-account-a")
with patch.object(telemetry, "_resolve_email", lambda key: "a@example.com"):
identified, alias = telemetry.resolve_distinct_id()
assert identified == "a@example.com"
assert alias == merged, "the anonymous id was merged into this account"
monkeypatch.delenv("MEM0_API_KEY", raising=False)
after_logout, logout_alias = telemetry.resolve_distinct_id()
assert after_logout.startswith("code-anon-")
assert after_logout != merged, "reused an id already merged into the previous account"
assert logout_alias == ""
assert "aliased" not in telemetry._read_identity(), "rotated id must be aliasable again"
def test_a_changed_key_that_will_not_resolve_rotates_the_anonymous_id(isolated_env, monkeypatch):
"""Same leak by the other route: fingerprint disagrees and the lookup fails."""
merged = telemetry.anonymous_id()
monkeypatch.setenv("MEM0_API_KEY", "key-for-account-a")
with patch.object(telemetry, "_resolve_email", lambda key: "a@example.com"):
telemetry.resolve_distinct_id()
monkeypatch.setenv("MEM0_API_KEY", "key-for-account-b")
with patch.object(telemetry, "_resolve_email", lambda key: ""):
after, alias = telemetry.resolve_distinct_id()
assert after.startswith("code-anon-")
assert after != merged
assert alias == ""
assert "email" not in telemetry._read_identity()
def test_a_legacy_cached_email_is_verified_before_the_key_is_bound(isolated_env, monkeypatch):
"""Review finding: a key changed before upgrading bound the wrong account.
Rows written before fingerprints existed carry an email and no fingerprint.
Adopting the current key without checking pinned that key to the previous
account's email, and every run after that agreed with itself.
"""
telemetry._write_identity({"email": "old@example.com", "anonymous_id": "code-anon-seed"})
monkeypatch.setenv("MEM0_API_KEY", "key-for-account-b")
with patch.object(telemetry, "_resolve_email", lambda key: "new@example.com"):
resolved, alias = telemetry.resolve_distinct_id()
assert resolved == "new@example.com"
assert alias == "", "email to email must never alias; it merges two real people"
stored = telemetry._read_identity()
assert stored["email"] == "new@example.com"
assert stored["key_fingerprint"] == telemetry._digest("key-for-account-b")
def test_a_legacy_row_keeps_working_when_the_account_cannot_be_checked(isolated_env, monkeypatch):
"""Firewalled users must not lose attribution, and must not bind unverified.
The same network that fails /v1/ping/ fails the PostHog POST, so nothing is
delivered under the unverified identity while this holds.
"""
telemetry._write_identity({"email": "old@example.com"})
monkeypatch.setenv("MEM0_API_KEY", "key-for-account-b")
with patch.object(telemetry, "_resolve_email", lambda key: ""):
resolved, _ = telemetry.resolve_distinct_id()
assert resolved == "old@example.com"
assert "key_fingerprint" not in telemetry._read_identity(), "bound an unverified key"
def test_a_failed_upgrade_claim_can_be_retried(isolated_env, monkeypatch):
"""Review finding: a failed rewrite left the sentinel and suppressed forever.
claim_version_change returns early on FileExistsError, and the marker still
holds the old version, so the upgrade for that version was never recorded
again on that machine.
"""
telemetry.claim_install()
state_path = memory_core.data_dir() / "install-state.json"
state = json.loads(state_path.read_text())
state["plugin_version"] = "0.0.1-old"
state_path.write_text(json.dumps(state), encoding="utf-8")
real_replace = Path.replace
def failing_replace(self, target):
raise OSError("disk full")
monkeypatch.setattr(Path, "replace", failing_replace)
assert telemetry.claim_version_change() is None
monkeypatch.setattr(Path, "replace", real_replace)
assert telemetry.claim_version_change() == "0.0.1-old", "sentinel suppressed the retry"
def test_first_run_is_not_flipped_by_writing_the_identity_file(isolated_env):
"""The identity file is written by a successful flush, not by recording.
Keying first-run off it meant an offline user recorded code.install on every
session forever, and every 0.2.x user recorded one on their first 0.3.x run.
"""
def test_is_first_run_flips_after_the_first_identity_write(isolated_env):
assert telemetry.is_first_run()
telemetry.anonymous_id()
assert telemetry.is_first_run()
def test_claiming_install_ends_first_run(isolated_env):
assert telemetry.claim_install() == "install"
assert not telemetry.is_first_run()
def test_install_can_only_be_claimed_once(isolated_env):
"""Two sessions starting together must not both record an install."""
assert telemetry.claim_install() == "install"
assert telemetry.claim_install() is None
def test_a_populated_data_dir_reads_as_an_upgrade(isolated_env):
"""A fresh install has an empty data directory; anything else predates it."""
data_dir = memory_core.data_dir()
data_dir.mkdir(parents=True, exist_ok=True)
(data_dir / "requirements.txt").write_text("mem0ai\n", encoding="utf-8")
assert telemetry.claim_install() == "upgrade"
def test_a_version_change_is_claimed_once(isolated_env):
telemetry.claim_install()
state_path = memory_core.data_dir() / "install-state.json"
state = json.loads(state_path.read_text())
state["plugin_version"] = "0.0.1-old"
state_path.write_text(json.dumps(state), encoding="utf-8")
assert telemetry.claim_version_change() == "0.0.1-old"
assert telemetry.claim_version_change() is None
def test_spawn_flush_does_nothing_without_a_spool(isolated_env):
with patch.object(telemetry.subprocess, "Popen") as popen:
assert telemetry.spawn_flush() is False
@@ -418,114 +273,3 @@ def test_spawn_flush_does_nothing_without_a_spool(isolated_env):
with patch.object(telemetry.subprocess, "Popen") as popen:
assert telemetry.spawn_flush() is True
popen.assert_called_once()
def test_salt_is_stable_across_processes(isolated_env):
"""Hooks are separate short-lived processes; one repo must hash one way.
An unlocked read-modify-write let each process mint its own salt, so a
repository hashed several ways in the window before one writer won.
"""
import subprocess as sp
core = str(Path(__file__).resolve().parents[1] / "core")
script = (
f"import sys; sys.path.insert(0, {core!r})\n"
"import telemetry\n"
"print(telemetry._install_salt())"
)
env = {**os.environ, "MEM0_CODE_DATA_DIR": str(memory_core.data_dir())}
salts = {
sp.run([sys.executable, "-c", script], capture_output=True, text=True, env=env).stdout.strip()
for _ in range(4)
}
assert len(salts) == 1, f"one repo hashed {len(salts)} ways: {salts}"
def test_salt_does_not_touch_the_identity_file(isolated_env):
"""The identity file is is_first_run's marker and the sender's email store.
Writing the salt into it would create it from record(), suppressing the
install event, and would race resolve_distinct_id, which holds a stale copy
of that dict across a network call.
"""
telemetry._install_salt()
assert not telemetry._identity_path().exists()
def test_no_salt_means_no_hash_rather_than_an_unsalted_one(isolated_env, monkeypatch):
"""A read-only data dir drops the property; it must not emit a weak digest.
The previous fallback was a digest of the salt file's own path, which an
attacker can compute, memoized for the whole process. A property named
repo_hash carrying an effectively unsalted digest is worse than no property:
it reads as protected and is not.
"""
telemetry._salt_cache = ""
monkeypatch.setattr(telemetry.os, "open", lambda *a, **k: (_ for _ in ()).throw(OSError("read-only")))
assert telemetry._install_salt() == ""
assert telemetry._scoped_digest("git@github.com:acme/secret.git") == ""
def test_a_half_written_salt_is_never_visible_to_another_process(isolated_env, monkeypatch):
"""The window this closes: file created, value not yet written.
O_CREAT|O_EXCL then write leaves the name present and empty in between. A
hook reading it there used to get "", fall back to the path digest and cache
that for its whole run, so the same repo hashed two ways depending on timing.
Publishing by link means the name either does not exist or is complete.
"""
telemetry._salt_cache = ""
salt_path = telemetry._salt_path()
observed = []
real_link = telemetry.os.link
def observing_link(source, target):
# Stand where the racing reader stands: after the temp file is written,
# before the real name exists.
observed.append(salt_path.exists())
return real_link(source, target)
monkeypatch.setattr(telemetry.os, "link", observing_link)
salt = telemetry._install_salt()
assert observed == [False], "the salt name existed before it held a value"
assert len(salt) == 32
assert salt_path.read_text(encoding="utf-8").strip() == salt
def test_a_filesystem_without_hardlinks_still_gets_a_salt(isolated_env, monkeypatch):
"""Publishing by link must not become a silent loss of the hashes.
Some network mounts and container volumes reject os.link. Returning ""
there would drop repo_hash and session_hash on every run for that whole
cohort, which is a bigger loss than the narrow race the link closes.
"""
telemetry._salt_cache = ""
monkeypatch.setattr(
telemetry.os, "link", lambda src, dst: (_ for _ in ()).throw(OSError(38, "not implemented"))
)
salt = telemetry._install_salt()
assert len(salt) == 32, "no salt on a filesystem without hardlinks"
assert telemetry._salt_path().read_text(encoding="utf-8").strip() == salt
assert telemetry._scoped_digest("git@github.com:acme/x.git") != ""
assert not list(telemetry._salt_path().parent.glob("telemetry-salt.*.tmp"))
def test_a_concurrent_writer_does_not_clobber_the_published_salt(isolated_env):
"""Second process to finish must adopt the first one's salt, not replace it.
os.link rather than os.replace is what makes losing the race harmless.
"""
telemetry._salt_cache = ""
first = telemetry._install_salt()
telemetry._salt_cache = ""
second = telemetry._install_salt()
assert second == first
assert not list(telemetry._salt_path().parent.glob("telemetry-salt.*.tmp")), "temp file left behind"
@@ -1,6 +1,6 @@
{
"name": "mem0",
"version": "0.3.2",
"version": "0.3.1",
"description": "Cross-session memory and token savings for coding agents.",
"author": { "name": "Mem0", "email": "support@mem0.ai" },
"homepage": "https://docs.mem0.ai/integrations/codex",
@@ -1,11 +0,0 @@
"""Generated by integrations/agent-plugin-core/build/build.py. Do not edit."""
HARNESS_ID = "codex"
SOURCE_TAG = "CODEX_PLUGIN"
# Platform-side vocabulary (mem0_event.source + X-Application). The whole
# plugin family is one source; which editor it runs in is the application.
# An empty application means the host is unknown, and memory_core omits
# the header entirely rather than sending a placeholder.
PLATFORM_SOURCE = "MEM0_PLUGIN"
PLATFORM_APPLICATION = "codex"
+1 -17
View File
@@ -290,11 +290,6 @@ def run(
if args.plugin_data_dir:
os.environ[data_dir_env] = args.plugin_data_dir
# Snapshot BEFORE anything writes to the data dir: cache_plugin_api_key
# writes `api-key` and EvidenceStore creates `evidence.sqlite3`, so asking
# after them always saw content and every fresh install reported an upgrade.
data_dir_was_empty = telemetry.data_dir_was_empty()
cache_plugin_api_key()
if args.action == "session-start":
clear_stale_api_key_cache()
@@ -310,19 +305,8 @@ def run(
return 0
if args.action == "session-start":
# Claims the marker atomically and says which event to record, so a
# second session starting alongside this one cannot record it too.
first_event = telemetry.claim_install(was_empty=data_dir_was_empty)
if first_event == "install":
if telemetry.is_first_run():
telemetry.record("install")
elif first_event == "upgrade":
# First run after a build that never wrote the marker; the
# predecessor version was never recorded anywhere.
telemetry.record("upgrade", from_version="pre-0.3")
else:
previous = telemetry.claim_version_change()
if previous:
telemetry.record("upgrade", from_version=previous)
recovered = recover_pending_handoffs()
record_session_start(store, hook_input)
if recovered:
+12 -172
View File
@@ -11,7 +11,6 @@ from __future__ import annotations
import functools
import hashlib
import json
import math
import os
import re
import sqlite3
@@ -27,9 +26,17 @@ from pathlib import Path
from typing import Any, Iterable
import telemetry
from message_utils import MAX_EXTRACTION_INPUT_TOKENS as MAX_EXTRACTION_INPUT_TOKENS
from message_utils import SECRET_PATTERNS as SECRET_PATTERNS
from message_utils import _estimated_tokens as _estimated_tokens
from message_utils import _is_agent_assignment as _is_agent_assignment
from message_utils import _is_agent_response as _is_agent_response
from message_utils import _message_tokens as _message_tokens
from message_utils import extraction_message_batches as extraction_message_batches
from message_utils import redact as redact
DEFAULT_API_URL = "https://api.mem0.ai"
PLUGIN_VERSION = "0.3.2"
PLUGIN_VERSION = "0.3.1"
_harness_name: str = "generic"
_harness_env_prefix: str = "MEM0_PLUGIN"
@@ -66,7 +73,6 @@ CHECKPOINT_EXCHANGES = 5
CHECKPOINT_MESSAGES = 10
CHECKPOINT_SOURCE_CHARS = 40000
DEFAULT_MAX_CONTEXT_CHARS = 4000
MAX_EXTRACTION_INPUT_TOKENS = 24000
MAX_FLUSH_ATTEMPTS = 5
FORGET_PAGE_SIZE = 100
FORGET_MAX_PAGES = 50
@@ -139,48 +145,11 @@ BUILD_COMMAND_RE = re.compile(
re.IGNORECASE,
)
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(
r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"
),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r'|(?:access|refresh|session)[_-]?token|token|authorization|credential'
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def redact(value: Any) -> str:
text = (
value
if isinstance(value, str)
else json.dumps(value, ensure_ascii=False, default=str)
)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def bounded(value: Any, limit: int) -> str:
text = redact(value).strip()
if len(text) <= limit:
@@ -1706,128 +1675,6 @@ def build_extraction_messages(structured: dict[str, Any]) -> list[dict[str, str]
return messages
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if (
_is_agent_assignment(message)
and index + 1 < len(exchange)
and _is_agent_response(exchange[index + 1])
):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
# Platform surface attribution. Read from the generated per-host module so a new
# entrypoint is correct without remembering to configure anything.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
except ImportError:
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
def platform_headers(key: str) -> dict[str, str]:
"""Auth plus the three surface-identity headers.
X-Mem0-Source and X-Application are set-once by contract: this is the
outermost layer, so it sets them, and nothing below may overwrite them.
X-Mem0-Client is append-only — anything downstream adds itself to the tail.
"""
headers = {
"Authorization": f"Token {key}",
"Content-Type": "application/json",
"X-Mem0-Source": _PLATFORM_SOURCE,
"X-Mem0-Client": f"mem0-plugin/{PLUGIN_VERSION}",
}
if _PLATFORM_APPLICATION:
headers["X-Application"] = _PLATFORM_APPLICATION
return headers
def _request_json(
url: str, key: str, payload: dict[str, Any], timeout: float
) -> tuple[dict[str, Any] | list[Any], int, int]:
@@ -1835,7 +1682,7 @@ def _request_json(
request = urllib.request.Request(
url,
data=raw,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -1862,7 +1709,7 @@ def _get_json(
) -> tuple[dict[str, Any] | list[Any], int]:
request = urllib.request.Request(
url,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="GET",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -2008,13 +1855,6 @@ def flush_session(
"user_id": write_user,
"app_id": repo.app_id,
"run_id": session_id,
# Top level, not metadata: the backend reads `source` from the body or
# the query string, never from metadata, which is where this used to
# sit. The X-Mem0-Source header is also read, but only from the
# platform release that ships alongside this change, so the body value
# is what makes attribution work on both. The harness tag stays in
# metadata as hook provenance.
"source": _PLATFORM_SOURCE,
"metadata": {**metadata, "author": write_user, "dirs": directory_chain(repo)},
"agent_custom_instructions": PROJECT_MEMORY_INSTRUCTIONS,
"custom_instructions": PERSONAL_MEMORY_INSTRUCTIONS,
@@ -2558,7 +2398,7 @@ def _collect_memory_ids(
def _delete_memory(api_url: str, key: str, memory_id: str) -> bool:
request = urllib.request.Request(
f"{api_url}/v1/memories/{urllib.parse.quote(memory_id)}/",
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="DELETE",
)
try:
@@ -0,0 +1,127 @@
"""Shared, host-independent redaction and lossless extraction batching."""
from __future__ import annotations
import json
import math
import re
from typing import Any
MAX_EXTRACTION_INPUT_TOKENS = 24000
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r"|(?:access|refresh|session)[_-]?token|token|authorization|credential"
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def redact(value: Any) -> str:
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, default=str)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if _is_agent_assignment(message) and index + 1 < len(exchange) and _is_agent_response(exchange[index + 1]):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
+39 -655
View File
@@ -1,9 +1,5 @@
#!/usr/bin/env python3
"""Usage telemetry for Mem0 agent plugins.
Events are linked to your Mem0 account email when an API key is configured, and
to a random per-machine id otherwise. Not anonymous — the Python SDK and CLI
attribute the same way.
"""Anonymous usage telemetry for Mem0 agent plugins.
Hooks run on a 3-6 second budget and fire on every tool call, so recording never
touches the network: `record` appends one JSON line to a local spool and returns.
@@ -13,8 +9,7 @@ started once per session and again from the flush worker that is already detache
Pure stdlib, matching the rest of the plugin. Opt out with MEM0_TELEMETRY=false.
Never sends prompts, memory text, queries, file paths, repository names, or API
keys: only event names, durations, counts, coarse outcomes, and repo/session
identifiers hashed with a random per-install salt.
keys: only event names, durations, counts, coarse outcomes, and salted hashes.
"""
from __future__ import annotations
@@ -34,24 +29,8 @@ from typing import Any
import memory_core
# Seeded from the per-host module the build generates into core/. Two processes
# in this pipeline never call init() — mcp_server.py, and the detached
# `python3 telemetry.py` sender that spawn_flush() starts — so a module default
# was what every one of their events got labelled with.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import HARNESS_ID as _DEFAULT_HARNESS
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
from _harness_id import SOURCE_TAG as _DEFAULT_SOURCE_TAG
except ImportError:
_DEFAULT_HARNESS = "generic"
_DEFAULT_SOURCE_TAG = "MEM0_PLUGIN"
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
_salt_cache: str = ""
_harness: str = _DEFAULT_HARNESS
_source_tag: str = _DEFAULT_SOURCE_TAG
_harness: str = "generic"
_source_tag: str = "MEM0_PLUGIN"
_PRIVATE_KEYS = {
"apikey",
"authorization",
@@ -77,19 +56,10 @@ _PRIVATE_KEYS = {
}
def init(harness: str = "", source_tag: str = "") -> None:
"""Override the generated identity. Optional — core/_harness_id.py is the default.
The fallback shape matches memory_core.configure_harness's (``<HOST>_PLUGIN``).
It used to be ``MEM0_<HOST>_PLUGIN`` here and ``<host>_plugin`` there, which
meant one plugin could emit three different source values depending on which
process happened to send the batch.
"""
def init(harness: str = "generic", source_tag: str = "") -> None:
global _harness, _source_tag
_harness = harness or _DEFAULT_HARNESS
_source_tag = source_tag or (
f"{_harness.upper().replace('-', '_')}_PLUGIN" if harness else _DEFAULT_SOURCE_TAG
)
_harness = harness
_source_tag = source_tag or f"MEM0_{harness.upper().replace('-', '_')}_PLUGIN"
POSTHOG_API_KEY = "phc_hgJkUVJFYtmaJqrvf6CYN67TIQ8yhXAkWzUn9AMU4yX"
POSTHOG_CAPTURE_URL = "https://us.i.posthog.com/i/v0/e/"
@@ -100,16 +70,6 @@ BATCH_SIZE = 100
SEND_TIMEOUT = 5
CLAIM_STALE_SECONDS = 120
CLAIM_EXPIRY_SECONDS = 7 * 24 * 60 * 60
# A batch is only discarded once it has genuinely been retried this many times.
MAX_CLAIM_ATTEMPTS = 3
# Parked claims drained per run, after the live spool. Bounded so a long backlog
# cannot turn one flush into an unbounded send loop.
MAX_PARKED_PER_RUN = 3
# Added to the wait before a released claim becomes reclaimable, per attempt
# already spent. Releasing straight to "reclaimable now" let two senders burn the
# whole budget within seconds of one another on a single momentary failure, and
# discard a batch a retry a minute later would have delivered.
RETRY_COOLDOWN_SECONDS = 60
def is_enabled() -> bool:
@@ -123,126 +83,9 @@ def is_enabled() -> bool:
def _digest(value: str, length: int = 16) -> str:
"""Unsalted digest. Only for values that are already secrets (API keys)."""
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:length]
def _salt_path() -> Path:
return memory_core.data_dir() / "telemetry-salt"
def _install_salt() -> str:
"""Random per-install salt, created once and memoized for the process.
Deliberately its own file, claimed with O_CREAT|O_EXCL, rather than a key in
the identity file. Three reasons, all of which produced wrong data when this
lived in the identity dict:
- Hooks are short-lived separate processes firing on every tool call, and
people run more than one agent window. A read-modify-write would let each
process mint its own salt, so one repository would hash several ways in the
window before a writer won.
- resolve_distinct_id holds a copy of the identity dict across a network call
to /v1/ping/, so whichever write landed second erased the other's key —
losing either the salt (repo_hash changes mid-stream) or the email (a
second $identify, splitting the person).
- Touching the identity file from record() would create it, and is_first_run
keys off that file, so recording an event would silently suppress the
install event.
Published atomically, and there is deliberately no derived fallback. Creating
the file with O_CREAT|O_EXCL and then writing into it leaves a window where
the file exists and is empty, and a concurrent hook that reads it in that
window gets nothing. Falling back to a digest of the path would hand that
process a salt an attacker can compute, memoized for its whole run, which is
the privacy control this function exists to provide silently turning itself
off under load. The salt is written to a private temp file first and linked
into place, so the name either does not exist or already has the full value.
Returns "" when it genuinely cannot persist. Callers omit the hash entirely
rather than emit an unsalted one.
"""
global _salt_cache
if _salt_cache:
return _salt_cache
path = _salt_path()
# Read before writing. Hooks are separate processes firing on every tool
# call, so all but the first find the salt already published; going straight
# to create-fsync-link-unlink meant every one of them paid an fsync to
# discover that, on a path whose whole promise is appending a line and
# returning.
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
if _salt_cache:
return _salt_cache
except OSError:
pass
temporary = path.with_name(f"{path.name}.{os.getpid()}.tmp")
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(handle, "w", encoding="utf-8") as stream:
stream.write(uuid.uuid4().hex)
stream.flush()
os.fsync(stream.fileno())
try:
# Atomic claim: fails if another process already published one.
# os.link rather than replace, which would clobber theirs.
os.link(temporary, path)
except FileExistsError:
pass
except OSError:
# No hardlinks here (some network mounts, some container volumes).
# Claim the name directly instead. That reopens the empty-file
# window, but the window is now benign: a reader that lands in it
# gets "" and omits the hash for that process rather than caching a
# guessable one. Losing the hashes on every run of an entire
# filesystem is the worse failure.
try:
fallback = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(fallback, "w", encoding="utf-8") as stream:
stream.write(temporary.read_text(encoding="utf-8"))
except OSError:
pass
except OSError:
pass
finally:
try:
temporary.unlink()
except OSError:
pass
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
except OSError:
_salt_cache = ""
return _salt_cache
def _scoped_digest(value: str, length: int = 16) -> str:
"""Salted digest for values drawn from a guessable space.
repo.identity is a git remote URL, or ``local:<absolute path>`` when there is
no remote — which normally contains the account username. Sixteen unsalted
hex characters over that input space is enumerable, so this is not a
privacy control without the salt. Salting per install keeps every
within-account join the analytics actually use and gives up only
cross-machine joins on the same repository, which nothing computes.
Returns "" when there is no salt, so record() omits the property. An
unsalted digest over this input space is close to plaintext, and emitting one
under a name that implies it is hashed is worse than sending nothing.
"""
if not value:
return ""
salt = _install_salt()
if not salt:
return ""
return hashlib.sha256(f"{salt}:{value}".encode("utf-8")).hexdigest()[:length]
def _safe_value(value: Any) -> Any:
if isinstance(value, str):
return memory_core.redact(value)
@@ -302,176 +145,9 @@ def anonymous_id(identity: dict[str, str] | None = None) -> str:
return created
def _rotate_anonymous_id(identity: dict[str, str]) -> str:
"""Mint a fresh anonymous id because the account context is gone.
The previous id may already have been merged into a person profile by an
$identify, and that merge is permanent. Reusing it after a logout or a key
change attributes everything that follows to the account that just went
away, which is the same misattribution the key fingerprint exists to stop,
only arriving through the anonymous path instead.
`aliased` is cleared with it: the new id has never been merged, so it is
eligible to be aliased into whatever account comes next.
"""
created = f"code-anon-{uuid.uuid4().hex}"
identity["anonymous_id"] = created
identity.pop("aliased", None)
_write_identity(identity)
return created
def _install_state_path() -> Path:
return memory_core.data_dir() / "install-state.json"
def is_first_run() -> bool:
"""Whether install has never been recorded on this machine.
Deliberately NOT the identity file. That file is only written by a
successful flush, so an offline or firewalled user recorded code.install on
every single session, forever — and every 0.2.x user recorded one on their
first 0.3.x session because 0.2.x never wrote it at all.
"""
return not _install_state_path().exists()
def data_dir_was_empty() -> bool:
"""Whether the data directory is untouched. Call BEFORE anything writes to it.
hook_runner reaches claim_install() only after cache_plugin_api_key() has
written `api-key` and EvidenceStore() has created `evidence.sqlite3`, so
asking at claim time always saw content and every fresh install reported an
upgrade. The caller snapshots this at the top of the run instead.
"""
return not _data_dir_has_content()
def claim_install(was_empty: bool | None = None) -> str | None:
"""Claim the one install/upgrade record for this machine, atomically.
Returns the event to record ("install" or "upgrade"), or None if another
session already claimed it. O_CREAT|O_EXCL so two sessions starting together
cannot both win.
`was_empty` must come from data_dir_was_empty() called before this process
wrote anything. Omitting it falls back to checking now, which is only
correct for a caller that has touched nothing.
"""
if not is_enabled():
# Never consume the one-shot claim while the user is opted out, or they
# would silently lose their install event if they later opt in.
return None
path = _install_state_path()
upgrading = not (data_dir_was_empty() if was_empty is None else was_empty)
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
except FileExistsError:
return None
except OSError:
return None
try:
with os.fdopen(handle, "w", encoding="utf-8") as stream:
json.dump(
{
"plugin_version": memory_core.PLUGIN_VERSION,
"installed_at": memory_core.utc_now(),
"upgraded": upgrading,
},
stream,
)
# Durable before this returns. The O_EXCL open is what makes the
# claim exclusive, so it cannot be replaced by a temp-and-rename
# without losing that, which leaves the content as the thing to make
# safe. A kill between the open and this fsync used to leave a marker
# that exists but parses to nothing: is_first_run reads it as claimed
# and claim_version_change cannot read a version out of it.
stream.flush()
os.fsync(stream.fileno())
except OSError:
pass
return "upgrade" if upgrading else "install"
def _data_dir_has_content() -> bool:
"""Whether anything predates this session in the plugin data directory."""
try:
for entry in memory_core.data_dir().iterdir():
if entry.name != "install-state.json":
return True
except OSError:
pass
return False
def _repair_install_state(path: Path) -> None:
"""Rewrite an unparseable marker so version tracking can resume."""
try:
temporary = path.with_suffix(f".{os.getpid()}.tmp")
temporary.write_text(
json.dumps({"plugin_version": memory_core.PLUGIN_VERSION, "repaired_at": memory_core.utc_now()}),
encoding="utf-8",
)
temporary.replace(path)
except OSError:
pass
def claim_version_change() -> str | None:
"""Return the previously recorded version if it differs, updating the marker.
Only meaningful once the marker exists — the first transition into 0.3.x has
no recorded predecessor and reports "pre-0.3" instead. Claiming by rewriting
the marker means the next session sees no change and records nothing.
"""
path = _install_state_path()
try:
state = json.loads(path.read_text(encoding="utf-8"))
except OSError:
return None
except json.JSONDecodeError:
# A crash between O_EXCL and the write leaves an empty marker. Left
# alone it disables every future upgrade event on this machine, because
# claim_install sees the file and this function cannot parse it.
state = None
if not isinstance(state, dict):
_repair_install_state(path)
return None
previous = str(state.get("plugin_version") or "")
if not previous or previous == memory_core.PLUGIN_VERSION:
return None
# Claim the transition with an exclusive sentinel before rewriting the
# marker. A plain read-modify-write let every concurrently starting session
# observe the old version and each record its own upgrade — and the first
# session after a version bump is exactly when several agent windows restart
# together.
sentinel = path.with_name(f"upgraded-{memory_core.PLUGIN_VERSION}")
try:
os.close(os.open(sentinel, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600))
except FileExistsError:
return None
except OSError:
return None
state["plugin_version"] = memory_core.PLUGIN_VERSION
state["upgraded_at"] = memory_core.utc_now()
temporary = path.with_suffix(f".{os.getpid()}.tmp")
try:
temporary.write_text(json.dumps(state), encoding="utf-8")
temporary.replace(path)
except OSError:
# Release the claim. The marker still records the old version, so
# without this the sentinel makes claim_version_change return early on
# every later run and this version's upgrade is never recorded again.
for leftover in (sentinel, temporary):
try:
leftover.unlink()
except OSError:
pass
return None
return previous
"""Whether this machine has never recorded a plugin event before."""
return not _identity_path().exists()
def record(
@@ -492,32 +168,19 @@ def record(
except OSError:
pass
properties = _safe_value(properties)
# Stamped in the RECORDING process, beside harness. `source` used to be
# read in the sending process from a module global, so whichever process
# drained the spool named every event in it. flush() spreads per-event
# properties last, so this now wins over any sender's default.
properties.update(
harness=_harness,
source=_source_tag,
plugin_version=memory_core.PLUGIN_VERSION,
os=sys.platform,
python_version=platform.python_version(),
)
# Assigned only when the digest is real. _scoped_digest returns "" when
# the salt could not be persisted, and an empty property is worse than an
# absent one: it survives the None filter below and reads as a value.
if repo is not None:
repo_hash = _scoped_digest(getattr(repo, "identity", ""))
if repo_hash:
properties["repo_hash"] = repo_hash
properties["repo_hash"] = _digest(getattr(repo, "identity", ""))
if session_id:
session_hash = _scoped_digest(session_id)
if session_hash:
properties["session_hash"] = session_hash
properties["session_hash"] = _digest(session_id)
line = json.dumps(
{
"event": f"{EVENT_PREFIX}.{event}",
"uuid": str(uuid.uuid4()),
"timestamp": memory_core.utc_now(),
"properties": {
key: value for key, value in properties.items() if value is not None
@@ -576,201 +239,38 @@ def spawn_flush() -> bool:
return False
def _claim_name(attempt: int = 0) -> str:
"""Claim filename. The attempt count rides in the name so the 7-day expiry
only ever discards a batch that was actually retried and failed."""
return f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}-a{attempt}.sending"
def _claim_attempt(claim: Path) -> int:
"""Attempts recorded in a claim filename; 0 for the pre-attempt-count shape.
Anchored on field position, not on a leading "a": the legacy shape is
``telemetry-<pid>-<hex>.sending`` and a hex id such as ``a1234567`` would
otherwise parse as attempt 1234567 and be discarded unsent on the first
flush after an upgrade.
"""
stem = claim.name[: -len(".sending")] if claim.name.endswith(".sending") else claim.name
parts = stem.split("-")
if len(parts) != 4:
return 0
tail = parts[3]
if tail.startswith("a") and tail[1:].isdigit():
return int(tail[1:])
return 0
def _touch(path: Path) -> None:
"""Refresh mtime so a claim's age measures time since it was claimed.
``Path.replace`` is ``os.rename``, which preserves mtime — so a claim created
after a quiet minute inherited the spool's last-write time and looked
abandoned the instant it was made. A second sender would then take it over
while the first was still posting, and both would deliver the batch.
"""
try:
os.utime(path, None)
except OSError:
pass
def _claim_spool() -> Path | None:
"""Rename the spool aside so exactly one sender owns each batch."""
directory = memory_core.data_dir()
claim = directory / _claim_name()
claim = directory / f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}.sending"
spool = _spool_path()
try:
spool.replace(claim)
_touch(claim)
return claim
except OSError:
pass
return _claim_parked(directory)
def _sweep_debris(directory: Path) -> None:
"""Remove files nothing else will ever pick up again.
*.partial is a temp file orphaned by a crash between write and rename.
*.corrupt is a batch quarantined for undecodable content. No glob in this
module matches either, so without this they accumulate on disk for the life
of the install.
Quarantined batches are kept far longer than debris: they are the only
evidence left of events that could not be delivered, and someone diagnosing
a report of missing telemetry has to be able to find one.
"""
now = time.time()
for debris in directory.glob("telemetry-*.partial"):
try:
if now - debris.stat().st_mtime > CLAIM_STALE_SECONDS:
debris.unlink()
except OSError:
continue
for quarantined in directory.glob("telemetry-*.corrupt"):
try:
if now - quarantined.stat().st_mtime > CLAIM_EXPIRY_SECONDS:
quarantined.unlink()
except OSError:
continue
# The same reasoning covers *.tmp. _write_identity and _install_salt both
# create one and unlink it in a finally, which a SIGKILL skips, and no glob
# in this module matches the leftovers either.
for temporary in directory.glob("telemetry-*.tmp"):
try:
if now - temporary.stat().st_mtime > CLAIM_STALE_SECONDS:
temporary.unlink()
except OSError:
continue
def _claim_parked(directory: Path) -> Path | None:
"""Take the oldest abandoned claim, if any lease has actually expired.
Kept separate from the live spool so flush() can drain both in one run.
Previously parked batches were only reachable when no spool existed at all,
and because sessions keep recording there usually was one — so a batch
parked by a failed send waited until the 7-day expiry deleted it unsent,
even though its own presence is what started the sender.
"""
now = time.time()
for orphan in sorted(directory.glob("telemetry-*.sending"), key=_safe_mtime):
for orphan in sorted(directory.glob("telemetry-*.sending")):
try:
age = now - orphan.stat().st_mtime
except OSError:
continue
if age < CLAIM_STALE_SECONDS:
# Someone else holds a live lease on it. This check has to come
# first. Claiming a file bumps its attempt count and refreshes its
# mtime, so a sender that has just taken the final attempt looks
# exhausted to everyone else while it is actively draining. Judging
# exhaustion before liveness let a second sender unlink a batch out
# from under its owner, losing every event in it.
continue
# Attempts, not age. Every re-claim touches the mtime and every release
# backdates it by a fixed amount, so age is pinned near the stale
# threshold and never reaches the expiry. Age stays only as a backstop
# for files that never carried an attempt marker.
if _claim_attempt(orphan) >= MAX_CLAIM_ATTEMPTS or age > CLAIM_EXPIRY_SECONDS:
if age > CLAIM_EXPIRY_SECONDS:
try:
orphan.unlink()
except OSError:
pass
continue
claim = orphan.parent / _claim_name(_claim_attempt(orphan) + 1)
if age < CLAIM_STALE_SECONDS:
continue
try:
orphan.replace(claim)
_touch(claim)
return claim
except OSError:
continue
return None
def _safe_mtime(path: Path) -> float:
try:
return path.stat().st_mtime
except OSError:
return 0.0
def _rewrite_claim(claim: Path, remaining: list[dict[str, Any]]) -> bool:
"""Persist the unsent remainder, atomically, and refresh the lease.
Called after every successful batch. Two jobs: a retry resumes where the
send stopped instead of re-posting from the top, and the rewrite doubles as
the lease heartbeat, so a slow sender does not have its claim stolen
mid-flight. Interval is one batch, well inside CLAIM_STALE_SECONDS.
"""
if not remaining:
try:
claim.unlink()
except OSError:
pass
return True
temporary = claim.with_suffix(f".{os.getpid()}.partial")
try:
payload = "".join(json.dumps(event, separators=(",", ":"), default=str) + "\n" for event in remaining)
# fsync before the rename: without it the rename can land while the
# bytes have not, and the claim comes back empty or truncated after a
# crash. _drain then reads zero events and unlinks it.
with open(temporary, "w", encoding="utf-8") as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
temporary.replace(claim)
_touch(claim)
return True
except OSError:
try:
temporary.unlink()
except OSError:
pass
return False
def _release_claim(claim: Path, remaining: list[dict[str, Any]]) -> None:
"""Persist the remainder and drop the lease, because this sender has given up.
Distinct from the per-batch heartbeat: heartbeating on the way out would
make an abandoned batch look actively owned for a further
CLAIM_STALE_SECONDS, delaying the retry for no reason. Ageing it past the
threshold lets the next flush pick it up immediately, while the attempt
count in the filename still bounds how many times that can happen.
"""
if not _rewrite_claim(claim, remaining):
return
try:
# Backdate past the stale threshold so the next flush can pick it up,
# minus a cooldown that grows with the attempts already spent. Clamped so
# the mtime never lands in the future, which would read as a live lease.
cooldown = min(_claim_attempt(claim) * RETRY_COOLDOWN_SECONDS, CLAIM_STALE_SECONDS)
released = time.time() - CLAIM_STALE_SECONDS - 1 + cooldown
os.utime(claim, (released, released))
except OSError:
pass
def _resolve_email(key: str) -> str:
"""Trade the API key for the account email so events join other Mem0 surfaces."""
url = os.environ.get("MEM0_API_URL", memory_core.DEFAULT_API_URL).rstrip("/") + "/v1/ping/"
@@ -800,130 +300,34 @@ def _post(payload: dict[str, Any], url: str) -> bool:
def resolve_distinct_id() -> tuple[str, str]:
"""Return the PostHog distinct id and the anonymous id it replaced, if any.
The second value becomes a PostHog $identify alias. It is ONLY ever an
anonymous id: aliasing one account email to another merges two real person
profiles and cannot be undone, so a key that now belongs to a different
account re-resolves with no alias.
"""
"""Return the PostHog distinct id and the anonymous id it replaced, if any."""
identity = _read_identity()
key = memory_core.api_key()
fingerprint = _digest(key) if key else ""
email = identity.get("email", "")
if email and fingerprint:
recorded = identity.get("key_fingerprint", "")
if recorded == fingerprint:
return email, ""
if not recorded:
# Rows written before fingerprints existed. Verify rather than
# adopt: a key changed before the upgrade would otherwise bind the
# new key to the previous account's email, permanently, and the
# fingerprint would then agree with itself forever after.
verified = _resolve_email(key)
if not verified:
# Offline, firewalled, or the API is down. Keep the previous
# behaviour and retry on the next flush rather than dropping a
# real account attribution. Safe because the same network that
# failed /v1/ping/ is about to fail the PostHog POST, so nothing
# is delivered under the unverified identity in the meantime.
return email, ""
identity["email"] = verified
identity["key_fingerprint"] = fingerprint
_write_identity(identity)
return verified, ""
if email:
return email, ""
key = memory_core.api_key()
if not key:
# No key to verify the account with; do not keep attributing to it.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
return anonymous_id(identity), ""
resolved = _resolve_email(key)
if not resolved:
# The key changed and will not resolve (revoked, offline, API down).
# Reaching here with an email means the recorded fingerprint disagreed,
# so the key really did change. Drop the account and rotate: the stored
# anonymous id may already be merged into that account's person, and
# reusing it would keep the events on the profile we are trying to
# leave.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
email = _resolve_email(key)
if not email:
return anonymous_id(identity), ""
# Alias only when going anonymous -> email for the first time. Once an anon
# id has been merged into an account it must never be offered again: an
# alias naming an already-identified id is what could link two real people.
previous = "" if (email or identity.get("aliased")) else identity.get("anonymous_id", "")
if previous:
identity["aliased"] = True
identity["email"] = resolved
identity["key_fingerprint"] = fingerprint
previous = identity.get("anonymous_id", "")
identity["email"] = email
_write_identity(identity)
return resolved, previous
return email, previous
def flush() -> int:
"""Drain the live spool, then any parked claims, and return events sent."""
"""Drain claimed spools to PostHog and return the number of events sent."""
if not is_enabled():
return 0
sent, delivered = _drain(_claim_spool())
if not delivered:
# The network is failing. Retrying other batches now would only burn
# their attempt budget against the same broken connection.
return sent
# Parked batches used to starve behind the live spool indefinitely. Bounded
# per run so a long backlog cannot turn one flush into an unbounded loop.
directory = memory_core.data_dir()
_sweep_debris(directory)
for _ in range(MAX_PARKED_PER_RUN):
parked = _claim_parked(directory)
if parked is None:
break
count, delivered = _drain(parked)
sent += count
if not delivered:
break
return sent
def _drain(claim: Path | None) -> tuple[int, bool]:
"""Post one claimed batch file, recording progress after every batch.
Returns (events sent, whether everything was delivered).
"""
claim = _claim_spool()
if claim is None:
return 0, True
return 0
try:
lines = claim.read_text(encoding="utf-8").splitlines()
except ValueError:
# UnicodeDecodeError from a torn write: the content is unrecoverable, so
# quarantine rather than retry. flush() runs from a bare `finally:` in
# flush_worker, so raising here also skips the handoff cleanup, and an
# undecodable file would otherwise be re-read on every flush forever.
# Reported as delivered because there is nothing left to deliver and the
# rest of the run should continue.
try:
claim.replace(claim.with_suffix(".corrupt"))
except OSError:
try:
claim.unlink()
except OSError:
pass
return 0, True
except OSError:
# Could not read it, which is not the same as having nothing to send.
# The file is left exactly where it is: a vanished or briefly unreadable
# claim is retryable, and quarantining it here would discard events over
# a transient filesystem error. Reported as undelivered so the run stops
# instead of counting a batch nothing was posted from as delivered.
return 0, False
return 0
events = []
for line in lines:
try:
@@ -933,18 +337,11 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
if isinstance(value, dict) and value.get("event"):
events.append(value)
if not events:
# Only delete when the file really is empty. A non-empty file that
# parses to nothing is a torn write, and its contents are the unsent
# remainder — deleting it is the data loss this PR exists to prevent.
try:
empty = claim.stat().st_size == 0
except OSError:
empty = True
try:
claim.replace(claim.with_suffix(".corrupt")) if not empty else claim.unlink()
claim.unlink()
except OSError:
pass
return 0, True
return 0
distinct_id, aliased_anonymous_id = resolve_distinct_id()
if aliased_anonymous_id:
@@ -963,17 +360,12 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
sent = 0
for start in range(0, len(events), BATCH_SIZE):
chunk = events[start : start + BATCH_SIZE]
batch = [
{
"event": event["event"],
"distinct_id": distinct_id,
# Carried through from record() so a resend can be collapsed.
"uuid": event.get("uuid"),
"timestamp": event.get("timestamp"),
"properties": {
# Fallback only: events recorded by a build before source
# moved into record() have none of their own.
"source": _source_tag,
"language": "python",
"$process_person_profile": False,
@@ -981,24 +373,16 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
**(event.get("properties") or {}),
},
}
for event in chunk
for event in events[start : start + BATCH_SIZE]
]
if not _post({"api_key": POSTHOG_API_KEY, "batch": batch}, POSTHOG_BATCH_URL):
# Keep only what has not been delivered, and release the lease.
# Previously the whole file was kept and the retry re-posted every
# batch, including the ones that had already arrived.
_release_claim(claim, events[start:])
return sent, False
sent += len(chunk)
# Record progress and refresh the lease after each successful batch, so
# a crash repeats at most one batch instead of the entire file. If the
# rewrite fails the claim still holds delivered events, so stop rather
# than carry on as though progress were recorded — continuing is how the
# duplicate delivery this PR fixes would come back.
if not _rewrite_claim(claim, events[start + len(chunk) :]):
_release_claim(claim, events[start + len(chunk) :])
return sent, False
return sent, True
return sent
sent += len(batch)
try:
claim.unlink()
except OSError:
pass
return sent
def main() -> int:
+1 -1
View File
@@ -1,6 +1,6 @@
{
"id": "mem0",
"version": "0.3.2",
"version": "0.3.1",
"homepage": "https://docs.mem0.ai/integrations/codex",
"native": {
"pluginRoot": "${PLUGIN_ROOT}",
@@ -7,8 +7,8 @@ disable-model-invocation: true
# Pause memory capture
To pause (hooks stop capturing and sending session content; a minimal
telemetry ping still fires at session start, under your Mem0 account email,
unless `MEM0_TELEMETRY=false`):
anonymous telemetry ping still fires at session start unless
`MEM0_TELEMETRY=false`):
```bash
python3 "${PLUGIN_ROOT}/core/memory_cli.py" --harness "codex" --plugin-data-dir "${PLUGIN_DATA}" pause
@@ -1,6 +1,6 @@
{
"name": "mem0",
"version": "0.3.2",
"version": "0.3.1",
"description": "Cross-session memory and token savings for coding agents.",
"author": { "name": "Mem0", "email": "support@mem0.ai" },
"homepage": "https://docs.mem0.ai/integrations/cursor",
@@ -1,11 +0,0 @@
"""Generated by integrations/agent-plugin-core/build/build.py. Do not edit."""
HARNESS_ID = "cursor"
SOURCE_TAG = "CURSOR_PLUGIN"
# Platform-side vocabulary (mem0_event.source + X-Application). The whole
# plugin family is one source; which editor it runs in is the application.
# An empty application means the host is unknown, and memory_core omits
# the header entirely rather than sending a placeholder.
PLATFORM_SOURCE = "MEM0_PLUGIN"
PLATFORM_APPLICATION = "cursor"
+1 -17
View File
@@ -290,11 +290,6 @@ def run(
if args.plugin_data_dir:
os.environ[data_dir_env] = args.plugin_data_dir
# Snapshot BEFORE anything writes to the data dir: cache_plugin_api_key
# writes `api-key` and EvidenceStore creates `evidence.sqlite3`, so asking
# after them always saw content and every fresh install reported an upgrade.
data_dir_was_empty = telemetry.data_dir_was_empty()
cache_plugin_api_key()
if args.action == "session-start":
clear_stale_api_key_cache()
@@ -310,19 +305,8 @@ def run(
return 0
if args.action == "session-start":
# Claims the marker atomically and says which event to record, so a
# second session starting alongside this one cannot record it too.
first_event = telemetry.claim_install(was_empty=data_dir_was_empty)
if first_event == "install":
if telemetry.is_first_run():
telemetry.record("install")
elif first_event == "upgrade":
# First run after a build that never wrote the marker; the
# predecessor version was never recorded anywhere.
telemetry.record("upgrade", from_version="pre-0.3")
else:
previous = telemetry.claim_version_change()
if previous:
telemetry.record("upgrade", from_version=previous)
recovered = recover_pending_handoffs()
record_session_start(store, hook_input)
if recovered:
+12 -172
View File
@@ -11,7 +11,6 @@ from __future__ import annotations
import functools
import hashlib
import json
import math
import os
import re
import sqlite3
@@ -27,9 +26,17 @@ from pathlib import Path
from typing import Any, Iterable
import telemetry
from message_utils import MAX_EXTRACTION_INPUT_TOKENS as MAX_EXTRACTION_INPUT_TOKENS
from message_utils import SECRET_PATTERNS as SECRET_PATTERNS
from message_utils import _estimated_tokens as _estimated_tokens
from message_utils import _is_agent_assignment as _is_agent_assignment
from message_utils import _is_agent_response as _is_agent_response
from message_utils import _message_tokens as _message_tokens
from message_utils import extraction_message_batches as extraction_message_batches
from message_utils import redact as redact
DEFAULT_API_URL = "https://api.mem0.ai"
PLUGIN_VERSION = "0.3.2"
PLUGIN_VERSION = "0.3.1"
_harness_name: str = "generic"
_harness_env_prefix: str = "MEM0_PLUGIN"
@@ -66,7 +73,6 @@ CHECKPOINT_EXCHANGES = 5
CHECKPOINT_MESSAGES = 10
CHECKPOINT_SOURCE_CHARS = 40000
DEFAULT_MAX_CONTEXT_CHARS = 4000
MAX_EXTRACTION_INPUT_TOKENS = 24000
MAX_FLUSH_ATTEMPTS = 5
FORGET_PAGE_SIZE = 100
FORGET_MAX_PAGES = 50
@@ -139,48 +145,11 @@ BUILD_COMMAND_RE = re.compile(
re.IGNORECASE,
)
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(
r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"
),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r'|(?:access|refresh|session)[_-]?token|token|authorization|credential'
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def redact(value: Any) -> str:
text = (
value
if isinstance(value, str)
else json.dumps(value, ensure_ascii=False, default=str)
)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def bounded(value: Any, limit: int) -> str:
text = redact(value).strip()
if len(text) <= limit:
@@ -1706,128 +1675,6 @@ def build_extraction_messages(structured: dict[str, Any]) -> list[dict[str, str]
return messages
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if (
_is_agent_assignment(message)
and index + 1 < len(exchange)
and _is_agent_response(exchange[index + 1])
):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
# Platform surface attribution. Read from the generated per-host module so a new
# entrypoint is correct without remembering to configure anything.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
except ImportError:
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
def platform_headers(key: str) -> dict[str, str]:
"""Auth plus the three surface-identity headers.
X-Mem0-Source and X-Application are set-once by contract: this is the
outermost layer, so it sets them, and nothing below may overwrite them.
X-Mem0-Client is append-only — anything downstream adds itself to the tail.
"""
headers = {
"Authorization": f"Token {key}",
"Content-Type": "application/json",
"X-Mem0-Source": _PLATFORM_SOURCE,
"X-Mem0-Client": f"mem0-plugin/{PLUGIN_VERSION}",
}
if _PLATFORM_APPLICATION:
headers["X-Application"] = _PLATFORM_APPLICATION
return headers
def _request_json(
url: str, key: str, payload: dict[str, Any], timeout: float
) -> tuple[dict[str, Any] | list[Any], int, int]:
@@ -1835,7 +1682,7 @@ def _request_json(
request = urllib.request.Request(
url,
data=raw,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -1862,7 +1709,7 @@ def _get_json(
) -> tuple[dict[str, Any] | list[Any], int]:
request = urllib.request.Request(
url,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="GET",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -2008,13 +1855,6 @@ def flush_session(
"user_id": write_user,
"app_id": repo.app_id,
"run_id": session_id,
# Top level, not metadata: the backend reads `source` from the body or
# the query string, never from metadata, which is where this used to
# sit. The X-Mem0-Source header is also read, but only from the
# platform release that ships alongside this change, so the body value
# is what makes attribution work on both. The harness tag stays in
# metadata as hook provenance.
"source": _PLATFORM_SOURCE,
"metadata": {**metadata, "author": write_user, "dirs": directory_chain(repo)},
"agent_custom_instructions": PROJECT_MEMORY_INSTRUCTIONS,
"custom_instructions": PERSONAL_MEMORY_INSTRUCTIONS,
@@ -2558,7 +2398,7 @@ def _collect_memory_ids(
def _delete_memory(api_url: str, key: str, memory_id: str) -> bool:
request = urllib.request.Request(
f"{api_url}/v1/memories/{urllib.parse.quote(memory_id)}/",
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="DELETE",
)
try:
@@ -0,0 +1,127 @@
"""Shared, host-independent redaction and lossless extraction batching."""
from __future__ import annotations
import json
import math
import re
from typing import Any
MAX_EXTRACTION_INPUT_TOKENS = 24000
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r"|(?:access|refresh|session)[_-]?token|token|authorization|credential"
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def redact(value: Any) -> str:
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, default=str)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if _is_agent_assignment(message) and index + 1 < len(exchange) and _is_agent_response(exchange[index + 1]):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
+39 -655
View File
@@ -1,9 +1,5 @@
#!/usr/bin/env python3
"""Usage telemetry for Mem0 agent plugins.
Events are linked to your Mem0 account email when an API key is configured, and
to a random per-machine id otherwise. Not anonymous — the Python SDK and CLI
attribute the same way.
"""Anonymous usage telemetry for Mem0 agent plugins.
Hooks run on a 3-6 second budget and fire on every tool call, so recording never
touches the network: `record` appends one JSON line to a local spool and returns.
@@ -13,8 +9,7 @@ started once per session and again from the flush worker that is already detache
Pure stdlib, matching the rest of the plugin. Opt out with MEM0_TELEMETRY=false.
Never sends prompts, memory text, queries, file paths, repository names, or API
keys: only event names, durations, counts, coarse outcomes, and repo/session
identifiers hashed with a random per-install salt.
keys: only event names, durations, counts, coarse outcomes, and salted hashes.
"""
from __future__ import annotations
@@ -34,24 +29,8 @@ from typing import Any
import memory_core
# Seeded from the per-host module the build generates into core/. Two processes
# in this pipeline never call init() — mcp_server.py, and the detached
# `python3 telemetry.py` sender that spawn_flush() starts — so a module default
# was what every one of their events got labelled with.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import HARNESS_ID as _DEFAULT_HARNESS
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
from _harness_id import SOURCE_TAG as _DEFAULT_SOURCE_TAG
except ImportError:
_DEFAULT_HARNESS = "generic"
_DEFAULT_SOURCE_TAG = "MEM0_PLUGIN"
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
_salt_cache: str = ""
_harness: str = _DEFAULT_HARNESS
_source_tag: str = _DEFAULT_SOURCE_TAG
_harness: str = "generic"
_source_tag: str = "MEM0_PLUGIN"
_PRIVATE_KEYS = {
"apikey",
"authorization",
@@ -77,19 +56,10 @@ _PRIVATE_KEYS = {
}
def init(harness: str = "", source_tag: str = "") -> None:
"""Override the generated identity. Optional — core/_harness_id.py is the default.
The fallback shape matches memory_core.configure_harness's (``<HOST>_PLUGIN``).
It used to be ``MEM0_<HOST>_PLUGIN`` here and ``<host>_plugin`` there, which
meant one plugin could emit three different source values depending on which
process happened to send the batch.
"""
def init(harness: str = "generic", source_tag: str = "") -> None:
global _harness, _source_tag
_harness = harness or _DEFAULT_HARNESS
_source_tag = source_tag or (
f"{_harness.upper().replace('-', '_')}_PLUGIN" if harness else _DEFAULT_SOURCE_TAG
)
_harness = harness
_source_tag = source_tag or f"MEM0_{harness.upper().replace('-', '_')}_PLUGIN"
POSTHOG_API_KEY = "phc_hgJkUVJFYtmaJqrvf6CYN67TIQ8yhXAkWzUn9AMU4yX"
POSTHOG_CAPTURE_URL = "https://us.i.posthog.com/i/v0/e/"
@@ -100,16 +70,6 @@ BATCH_SIZE = 100
SEND_TIMEOUT = 5
CLAIM_STALE_SECONDS = 120
CLAIM_EXPIRY_SECONDS = 7 * 24 * 60 * 60
# A batch is only discarded once it has genuinely been retried this many times.
MAX_CLAIM_ATTEMPTS = 3
# Parked claims drained per run, after the live spool. Bounded so a long backlog
# cannot turn one flush into an unbounded send loop.
MAX_PARKED_PER_RUN = 3
# Added to the wait before a released claim becomes reclaimable, per attempt
# already spent. Releasing straight to "reclaimable now" let two senders burn the
# whole budget within seconds of one another on a single momentary failure, and
# discard a batch a retry a minute later would have delivered.
RETRY_COOLDOWN_SECONDS = 60
def is_enabled() -> bool:
@@ -123,126 +83,9 @@ def is_enabled() -> bool:
def _digest(value: str, length: int = 16) -> str:
"""Unsalted digest. Only for values that are already secrets (API keys)."""
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:length]
def _salt_path() -> Path:
return memory_core.data_dir() / "telemetry-salt"
def _install_salt() -> str:
"""Random per-install salt, created once and memoized for the process.
Deliberately its own file, claimed with O_CREAT|O_EXCL, rather than a key in
the identity file. Three reasons, all of which produced wrong data when this
lived in the identity dict:
- Hooks are short-lived separate processes firing on every tool call, and
people run more than one agent window. A read-modify-write would let each
process mint its own salt, so one repository would hash several ways in the
window before a writer won.
- resolve_distinct_id holds a copy of the identity dict across a network call
to /v1/ping/, so whichever write landed second erased the other's key —
losing either the salt (repo_hash changes mid-stream) or the email (a
second $identify, splitting the person).
- Touching the identity file from record() would create it, and is_first_run
keys off that file, so recording an event would silently suppress the
install event.
Published atomically, and there is deliberately no derived fallback. Creating
the file with O_CREAT|O_EXCL and then writing into it leaves a window where
the file exists and is empty, and a concurrent hook that reads it in that
window gets nothing. Falling back to a digest of the path would hand that
process a salt an attacker can compute, memoized for its whole run, which is
the privacy control this function exists to provide silently turning itself
off under load. The salt is written to a private temp file first and linked
into place, so the name either does not exist or already has the full value.
Returns "" when it genuinely cannot persist. Callers omit the hash entirely
rather than emit an unsalted one.
"""
global _salt_cache
if _salt_cache:
return _salt_cache
path = _salt_path()
# Read before writing. Hooks are separate processes firing on every tool
# call, so all but the first find the salt already published; going straight
# to create-fsync-link-unlink meant every one of them paid an fsync to
# discover that, on a path whose whole promise is appending a line and
# returning.
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
if _salt_cache:
return _salt_cache
except OSError:
pass
temporary = path.with_name(f"{path.name}.{os.getpid()}.tmp")
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(handle, "w", encoding="utf-8") as stream:
stream.write(uuid.uuid4().hex)
stream.flush()
os.fsync(stream.fileno())
try:
# Atomic claim: fails if another process already published one.
# os.link rather than replace, which would clobber theirs.
os.link(temporary, path)
except FileExistsError:
pass
except OSError:
# No hardlinks here (some network mounts, some container volumes).
# Claim the name directly instead. That reopens the empty-file
# window, but the window is now benign: a reader that lands in it
# gets "" and omits the hash for that process rather than caching a
# guessable one. Losing the hashes on every run of an entire
# filesystem is the worse failure.
try:
fallback = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(fallback, "w", encoding="utf-8") as stream:
stream.write(temporary.read_text(encoding="utf-8"))
except OSError:
pass
except OSError:
pass
finally:
try:
temporary.unlink()
except OSError:
pass
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
except OSError:
_salt_cache = ""
return _salt_cache
def _scoped_digest(value: str, length: int = 16) -> str:
"""Salted digest for values drawn from a guessable space.
repo.identity is a git remote URL, or ``local:<absolute path>`` when there is
no remote — which normally contains the account username. Sixteen unsalted
hex characters over that input space is enumerable, so this is not a
privacy control without the salt. Salting per install keeps every
within-account join the analytics actually use and gives up only
cross-machine joins on the same repository, which nothing computes.
Returns "" when there is no salt, so record() omits the property. An
unsalted digest over this input space is close to plaintext, and emitting one
under a name that implies it is hashed is worse than sending nothing.
"""
if not value:
return ""
salt = _install_salt()
if not salt:
return ""
return hashlib.sha256(f"{salt}:{value}".encode("utf-8")).hexdigest()[:length]
def _safe_value(value: Any) -> Any:
if isinstance(value, str):
return memory_core.redact(value)
@@ -302,176 +145,9 @@ def anonymous_id(identity: dict[str, str] | None = None) -> str:
return created
def _rotate_anonymous_id(identity: dict[str, str]) -> str:
"""Mint a fresh anonymous id because the account context is gone.
The previous id may already have been merged into a person profile by an
$identify, and that merge is permanent. Reusing it after a logout or a key
change attributes everything that follows to the account that just went
away, which is the same misattribution the key fingerprint exists to stop,
only arriving through the anonymous path instead.
`aliased` is cleared with it: the new id has never been merged, so it is
eligible to be aliased into whatever account comes next.
"""
created = f"code-anon-{uuid.uuid4().hex}"
identity["anonymous_id"] = created
identity.pop("aliased", None)
_write_identity(identity)
return created
def _install_state_path() -> Path:
return memory_core.data_dir() / "install-state.json"
def is_first_run() -> bool:
"""Whether install has never been recorded on this machine.
Deliberately NOT the identity file. That file is only written by a
successful flush, so an offline or firewalled user recorded code.install on
every single session, forever — and every 0.2.x user recorded one on their
first 0.3.x session because 0.2.x never wrote it at all.
"""
return not _install_state_path().exists()
def data_dir_was_empty() -> bool:
"""Whether the data directory is untouched. Call BEFORE anything writes to it.
hook_runner reaches claim_install() only after cache_plugin_api_key() has
written `api-key` and EvidenceStore() has created `evidence.sqlite3`, so
asking at claim time always saw content and every fresh install reported an
upgrade. The caller snapshots this at the top of the run instead.
"""
return not _data_dir_has_content()
def claim_install(was_empty: bool | None = None) -> str | None:
"""Claim the one install/upgrade record for this machine, atomically.
Returns the event to record ("install" or "upgrade"), or None if another
session already claimed it. O_CREAT|O_EXCL so two sessions starting together
cannot both win.
`was_empty` must come from data_dir_was_empty() called before this process
wrote anything. Omitting it falls back to checking now, which is only
correct for a caller that has touched nothing.
"""
if not is_enabled():
# Never consume the one-shot claim while the user is opted out, or they
# would silently lose their install event if they later opt in.
return None
path = _install_state_path()
upgrading = not (data_dir_was_empty() if was_empty is None else was_empty)
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
except FileExistsError:
return None
except OSError:
return None
try:
with os.fdopen(handle, "w", encoding="utf-8") as stream:
json.dump(
{
"plugin_version": memory_core.PLUGIN_VERSION,
"installed_at": memory_core.utc_now(),
"upgraded": upgrading,
},
stream,
)
# Durable before this returns. The O_EXCL open is what makes the
# claim exclusive, so it cannot be replaced by a temp-and-rename
# without losing that, which leaves the content as the thing to make
# safe. A kill between the open and this fsync used to leave a marker
# that exists but parses to nothing: is_first_run reads it as claimed
# and claim_version_change cannot read a version out of it.
stream.flush()
os.fsync(stream.fileno())
except OSError:
pass
return "upgrade" if upgrading else "install"
def _data_dir_has_content() -> bool:
"""Whether anything predates this session in the plugin data directory."""
try:
for entry in memory_core.data_dir().iterdir():
if entry.name != "install-state.json":
return True
except OSError:
pass
return False
def _repair_install_state(path: Path) -> None:
"""Rewrite an unparseable marker so version tracking can resume."""
try:
temporary = path.with_suffix(f".{os.getpid()}.tmp")
temporary.write_text(
json.dumps({"plugin_version": memory_core.PLUGIN_VERSION, "repaired_at": memory_core.utc_now()}),
encoding="utf-8",
)
temporary.replace(path)
except OSError:
pass
def claim_version_change() -> str | None:
"""Return the previously recorded version if it differs, updating the marker.
Only meaningful once the marker exists — the first transition into 0.3.x has
no recorded predecessor and reports "pre-0.3" instead. Claiming by rewriting
the marker means the next session sees no change and records nothing.
"""
path = _install_state_path()
try:
state = json.loads(path.read_text(encoding="utf-8"))
except OSError:
return None
except json.JSONDecodeError:
# A crash between O_EXCL and the write leaves an empty marker. Left
# alone it disables every future upgrade event on this machine, because
# claim_install sees the file and this function cannot parse it.
state = None
if not isinstance(state, dict):
_repair_install_state(path)
return None
previous = str(state.get("plugin_version") or "")
if not previous or previous == memory_core.PLUGIN_VERSION:
return None
# Claim the transition with an exclusive sentinel before rewriting the
# marker. A plain read-modify-write let every concurrently starting session
# observe the old version and each record its own upgrade — and the first
# session after a version bump is exactly when several agent windows restart
# together.
sentinel = path.with_name(f"upgraded-{memory_core.PLUGIN_VERSION}")
try:
os.close(os.open(sentinel, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600))
except FileExistsError:
return None
except OSError:
return None
state["plugin_version"] = memory_core.PLUGIN_VERSION
state["upgraded_at"] = memory_core.utc_now()
temporary = path.with_suffix(f".{os.getpid()}.tmp")
try:
temporary.write_text(json.dumps(state), encoding="utf-8")
temporary.replace(path)
except OSError:
# Release the claim. The marker still records the old version, so
# without this the sentinel makes claim_version_change return early on
# every later run and this version's upgrade is never recorded again.
for leftover in (sentinel, temporary):
try:
leftover.unlink()
except OSError:
pass
return None
return previous
"""Whether this machine has never recorded a plugin event before."""
return not _identity_path().exists()
def record(
@@ -492,32 +168,19 @@ def record(
except OSError:
pass
properties = _safe_value(properties)
# Stamped in the RECORDING process, beside harness. `source` used to be
# read in the sending process from a module global, so whichever process
# drained the spool named every event in it. flush() spreads per-event
# properties last, so this now wins over any sender's default.
properties.update(
harness=_harness,
source=_source_tag,
plugin_version=memory_core.PLUGIN_VERSION,
os=sys.platform,
python_version=platform.python_version(),
)
# Assigned only when the digest is real. _scoped_digest returns "" when
# the salt could not be persisted, and an empty property is worse than an
# absent one: it survives the None filter below and reads as a value.
if repo is not None:
repo_hash = _scoped_digest(getattr(repo, "identity", ""))
if repo_hash:
properties["repo_hash"] = repo_hash
properties["repo_hash"] = _digest(getattr(repo, "identity", ""))
if session_id:
session_hash = _scoped_digest(session_id)
if session_hash:
properties["session_hash"] = session_hash
properties["session_hash"] = _digest(session_id)
line = json.dumps(
{
"event": f"{EVENT_PREFIX}.{event}",
"uuid": str(uuid.uuid4()),
"timestamp": memory_core.utc_now(),
"properties": {
key: value for key, value in properties.items() if value is not None
@@ -576,201 +239,38 @@ def spawn_flush() -> bool:
return False
def _claim_name(attempt: int = 0) -> str:
"""Claim filename. The attempt count rides in the name so the 7-day expiry
only ever discards a batch that was actually retried and failed."""
return f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}-a{attempt}.sending"
def _claim_attempt(claim: Path) -> int:
"""Attempts recorded in a claim filename; 0 for the pre-attempt-count shape.
Anchored on field position, not on a leading "a": the legacy shape is
``telemetry-<pid>-<hex>.sending`` and a hex id such as ``a1234567`` would
otherwise parse as attempt 1234567 and be discarded unsent on the first
flush after an upgrade.
"""
stem = claim.name[: -len(".sending")] if claim.name.endswith(".sending") else claim.name
parts = stem.split("-")
if len(parts) != 4:
return 0
tail = parts[3]
if tail.startswith("a") and tail[1:].isdigit():
return int(tail[1:])
return 0
def _touch(path: Path) -> None:
"""Refresh mtime so a claim's age measures time since it was claimed.
``Path.replace`` is ``os.rename``, which preserves mtime — so a claim created
after a quiet minute inherited the spool's last-write time and looked
abandoned the instant it was made. A second sender would then take it over
while the first was still posting, and both would deliver the batch.
"""
try:
os.utime(path, None)
except OSError:
pass
def _claim_spool() -> Path | None:
"""Rename the spool aside so exactly one sender owns each batch."""
directory = memory_core.data_dir()
claim = directory / _claim_name()
claim = directory / f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}.sending"
spool = _spool_path()
try:
spool.replace(claim)
_touch(claim)
return claim
except OSError:
pass
return _claim_parked(directory)
def _sweep_debris(directory: Path) -> None:
"""Remove files nothing else will ever pick up again.
*.partial is a temp file orphaned by a crash between write and rename.
*.corrupt is a batch quarantined for undecodable content. No glob in this
module matches either, so without this they accumulate on disk for the life
of the install.
Quarantined batches are kept far longer than debris: they are the only
evidence left of events that could not be delivered, and someone diagnosing
a report of missing telemetry has to be able to find one.
"""
now = time.time()
for debris in directory.glob("telemetry-*.partial"):
try:
if now - debris.stat().st_mtime > CLAIM_STALE_SECONDS:
debris.unlink()
except OSError:
continue
for quarantined in directory.glob("telemetry-*.corrupt"):
try:
if now - quarantined.stat().st_mtime > CLAIM_EXPIRY_SECONDS:
quarantined.unlink()
except OSError:
continue
# The same reasoning covers *.tmp. _write_identity and _install_salt both
# create one and unlink it in a finally, which a SIGKILL skips, and no glob
# in this module matches the leftovers either.
for temporary in directory.glob("telemetry-*.tmp"):
try:
if now - temporary.stat().st_mtime > CLAIM_STALE_SECONDS:
temporary.unlink()
except OSError:
continue
def _claim_parked(directory: Path) -> Path | None:
"""Take the oldest abandoned claim, if any lease has actually expired.
Kept separate from the live spool so flush() can drain both in one run.
Previously parked batches were only reachable when no spool existed at all,
and because sessions keep recording there usually was one — so a batch
parked by a failed send waited until the 7-day expiry deleted it unsent,
even though its own presence is what started the sender.
"""
now = time.time()
for orphan in sorted(directory.glob("telemetry-*.sending"), key=_safe_mtime):
for orphan in sorted(directory.glob("telemetry-*.sending")):
try:
age = now - orphan.stat().st_mtime
except OSError:
continue
if age < CLAIM_STALE_SECONDS:
# Someone else holds a live lease on it. This check has to come
# first. Claiming a file bumps its attempt count and refreshes its
# mtime, so a sender that has just taken the final attempt looks
# exhausted to everyone else while it is actively draining. Judging
# exhaustion before liveness let a second sender unlink a batch out
# from under its owner, losing every event in it.
continue
# Attempts, not age. Every re-claim touches the mtime and every release
# backdates it by a fixed amount, so age is pinned near the stale
# threshold and never reaches the expiry. Age stays only as a backstop
# for files that never carried an attempt marker.
if _claim_attempt(orphan) >= MAX_CLAIM_ATTEMPTS or age > CLAIM_EXPIRY_SECONDS:
if age > CLAIM_EXPIRY_SECONDS:
try:
orphan.unlink()
except OSError:
pass
continue
claim = orphan.parent / _claim_name(_claim_attempt(orphan) + 1)
if age < CLAIM_STALE_SECONDS:
continue
try:
orphan.replace(claim)
_touch(claim)
return claim
except OSError:
continue
return None
def _safe_mtime(path: Path) -> float:
try:
return path.stat().st_mtime
except OSError:
return 0.0
def _rewrite_claim(claim: Path, remaining: list[dict[str, Any]]) -> bool:
"""Persist the unsent remainder, atomically, and refresh the lease.
Called after every successful batch. Two jobs: a retry resumes where the
send stopped instead of re-posting from the top, and the rewrite doubles as
the lease heartbeat, so a slow sender does not have its claim stolen
mid-flight. Interval is one batch, well inside CLAIM_STALE_SECONDS.
"""
if not remaining:
try:
claim.unlink()
except OSError:
pass
return True
temporary = claim.with_suffix(f".{os.getpid()}.partial")
try:
payload = "".join(json.dumps(event, separators=(",", ":"), default=str) + "\n" for event in remaining)
# fsync before the rename: without it the rename can land while the
# bytes have not, and the claim comes back empty or truncated after a
# crash. _drain then reads zero events and unlinks it.
with open(temporary, "w", encoding="utf-8") as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
temporary.replace(claim)
_touch(claim)
return True
except OSError:
try:
temporary.unlink()
except OSError:
pass
return False
def _release_claim(claim: Path, remaining: list[dict[str, Any]]) -> None:
"""Persist the remainder and drop the lease, because this sender has given up.
Distinct from the per-batch heartbeat: heartbeating on the way out would
make an abandoned batch look actively owned for a further
CLAIM_STALE_SECONDS, delaying the retry for no reason. Ageing it past the
threshold lets the next flush pick it up immediately, while the attempt
count in the filename still bounds how many times that can happen.
"""
if not _rewrite_claim(claim, remaining):
return
try:
# Backdate past the stale threshold so the next flush can pick it up,
# minus a cooldown that grows with the attempts already spent. Clamped so
# the mtime never lands in the future, which would read as a live lease.
cooldown = min(_claim_attempt(claim) * RETRY_COOLDOWN_SECONDS, CLAIM_STALE_SECONDS)
released = time.time() - CLAIM_STALE_SECONDS - 1 + cooldown
os.utime(claim, (released, released))
except OSError:
pass
def _resolve_email(key: str) -> str:
"""Trade the API key for the account email so events join other Mem0 surfaces."""
url = os.environ.get("MEM0_API_URL", memory_core.DEFAULT_API_URL).rstrip("/") + "/v1/ping/"
@@ -800,130 +300,34 @@ def _post(payload: dict[str, Any], url: str) -> bool:
def resolve_distinct_id() -> tuple[str, str]:
"""Return the PostHog distinct id and the anonymous id it replaced, if any.
The second value becomes a PostHog $identify alias. It is ONLY ever an
anonymous id: aliasing one account email to another merges two real person
profiles and cannot be undone, so a key that now belongs to a different
account re-resolves with no alias.
"""
"""Return the PostHog distinct id and the anonymous id it replaced, if any."""
identity = _read_identity()
key = memory_core.api_key()
fingerprint = _digest(key) if key else ""
email = identity.get("email", "")
if email and fingerprint:
recorded = identity.get("key_fingerprint", "")
if recorded == fingerprint:
return email, ""
if not recorded:
# Rows written before fingerprints existed. Verify rather than
# adopt: a key changed before the upgrade would otherwise bind the
# new key to the previous account's email, permanently, and the
# fingerprint would then agree with itself forever after.
verified = _resolve_email(key)
if not verified:
# Offline, firewalled, or the API is down. Keep the previous
# behaviour and retry on the next flush rather than dropping a
# real account attribution. Safe because the same network that
# failed /v1/ping/ is about to fail the PostHog POST, so nothing
# is delivered under the unverified identity in the meantime.
return email, ""
identity["email"] = verified
identity["key_fingerprint"] = fingerprint
_write_identity(identity)
return verified, ""
if email:
return email, ""
key = memory_core.api_key()
if not key:
# No key to verify the account with; do not keep attributing to it.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
return anonymous_id(identity), ""
resolved = _resolve_email(key)
if not resolved:
# The key changed and will not resolve (revoked, offline, API down).
# Reaching here with an email means the recorded fingerprint disagreed,
# so the key really did change. Drop the account and rotate: the stored
# anonymous id may already be merged into that account's person, and
# reusing it would keep the events on the profile we are trying to
# leave.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
email = _resolve_email(key)
if not email:
return anonymous_id(identity), ""
# Alias only when going anonymous -> email for the first time. Once an anon
# id has been merged into an account it must never be offered again: an
# alias naming an already-identified id is what could link two real people.
previous = "" if (email or identity.get("aliased")) else identity.get("anonymous_id", "")
if previous:
identity["aliased"] = True
identity["email"] = resolved
identity["key_fingerprint"] = fingerprint
previous = identity.get("anonymous_id", "")
identity["email"] = email
_write_identity(identity)
return resolved, previous
return email, previous
def flush() -> int:
"""Drain the live spool, then any parked claims, and return events sent."""
"""Drain claimed spools to PostHog and return the number of events sent."""
if not is_enabled():
return 0
sent, delivered = _drain(_claim_spool())
if not delivered:
# The network is failing. Retrying other batches now would only burn
# their attempt budget against the same broken connection.
return sent
# Parked batches used to starve behind the live spool indefinitely. Bounded
# per run so a long backlog cannot turn one flush into an unbounded loop.
directory = memory_core.data_dir()
_sweep_debris(directory)
for _ in range(MAX_PARKED_PER_RUN):
parked = _claim_parked(directory)
if parked is None:
break
count, delivered = _drain(parked)
sent += count
if not delivered:
break
return sent
def _drain(claim: Path | None) -> tuple[int, bool]:
"""Post one claimed batch file, recording progress after every batch.
Returns (events sent, whether everything was delivered).
"""
claim = _claim_spool()
if claim is None:
return 0, True
return 0
try:
lines = claim.read_text(encoding="utf-8").splitlines()
except ValueError:
# UnicodeDecodeError from a torn write: the content is unrecoverable, so
# quarantine rather than retry. flush() runs from a bare `finally:` in
# flush_worker, so raising here also skips the handoff cleanup, and an
# undecodable file would otherwise be re-read on every flush forever.
# Reported as delivered because there is nothing left to deliver and the
# rest of the run should continue.
try:
claim.replace(claim.with_suffix(".corrupt"))
except OSError:
try:
claim.unlink()
except OSError:
pass
return 0, True
except OSError:
# Could not read it, which is not the same as having nothing to send.
# The file is left exactly where it is: a vanished or briefly unreadable
# claim is retryable, and quarantining it here would discard events over
# a transient filesystem error. Reported as undelivered so the run stops
# instead of counting a batch nothing was posted from as delivered.
return 0, False
return 0
events = []
for line in lines:
try:
@@ -933,18 +337,11 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
if isinstance(value, dict) and value.get("event"):
events.append(value)
if not events:
# Only delete when the file really is empty. A non-empty file that
# parses to nothing is a torn write, and its contents are the unsent
# remainder — deleting it is the data loss this PR exists to prevent.
try:
empty = claim.stat().st_size == 0
except OSError:
empty = True
try:
claim.replace(claim.with_suffix(".corrupt")) if not empty else claim.unlink()
claim.unlink()
except OSError:
pass
return 0, True
return 0
distinct_id, aliased_anonymous_id = resolve_distinct_id()
if aliased_anonymous_id:
@@ -963,17 +360,12 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
sent = 0
for start in range(0, len(events), BATCH_SIZE):
chunk = events[start : start + BATCH_SIZE]
batch = [
{
"event": event["event"],
"distinct_id": distinct_id,
# Carried through from record() so a resend can be collapsed.
"uuid": event.get("uuid"),
"timestamp": event.get("timestamp"),
"properties": {
# Fallback only: events recorded by a build before source
# moved into record() have none of their own.
"source": _source_tag,
"language": "python",
"$process_person_profile": False,
@@ -981,24 +373,16 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
**(event.get("properties") or {}),
},
}
for event in chunk
for event in events[start : start + BATCH_SIZE]
]
if not _post({"api_key": POSTHOG_API_KEY, "batch": batch}, POSTHOG_BATCH_URL):
# Keep only what has not been delivered, and release the lease.
# Previously the whole file was kept and the retry re-posted every
# batch, including the ones that had already arrived.
_release_claim(claim, events[start:])
return sent, False
sent += len(chunk)
# Record progress and refresh the lease after each successful batch, so
# a crash repeats at most one batch instead of the entire file. If the
# rewrite fails the claim still holds delivered events, so stop rather
# than carry on as though progress were recorded — continuing is how the
# duplicate delivery this PR fixes would come back.
if not _rewrite_claim(claim, events[start + len(chunk) :]):
_release_claim(claim, events[start + len(chunk) :])
return sent, False
return sent, True
return sent
sent += len(batch)
try:
claim.unlink()
except OSError:
pass
return sent
def main() -> int:
+1 -1
View File
@@ -1,6 +1,6 @@
{
"id": "mem0",
"version": "0.3.2",
"version": "0.3.1",
"homepage": "https://docs.mem0.ai/integrations/cursor",
"native": {
"pluginRoot": "${CURSOR_PLUGIN_ROOT}",
@@ -7,8 +7,8 @@ disable-model-invocation: true
# Pause memory capture
To pause (hooks stop capturing and sending session content; a minimal
telemetry ping still fires at session start, under your Mem0 account email,
unless `MEM0_TELEMETRY=false`):
anonymous telemetry ping still fires at session start unless
`MEM0_TELEMETRY=false`):
```bash
python3 "${CURSOR_PLUGIN_ROOT}/core/memory_cli.py" --harness "cursor" pause
+2 -2
View File
@@ -86,9 +86,9 @@ Per-call `userId` overrides are rejected unless the operator enables `allowUserO
## Telemetry
Writes are tagged `source="DEEPSEEK_HARNESS"`. That value has to exist in the backend's `EventSource` enum for usage to surface by name; until it does, these writes read as `OTHERS`. It is added by [mem0ai/platform#3602](https://github.com/mem0ai/platform/pull/3602), which has to ship before this claim is true.
Writes are tagged `source="DEEPSEEK_HARNESS"` so Mem0's backend can attribute usage to this integration. For it to surface by name (rather than bucketing into `OTHERS`), `DEEPSEEK_HARNESS` must be present in the backend's `KNOWN_EVENT_SOURCES` allowlist, a one-line platform change matching the existing `ZAPIER` / `STRANDS` sources.
The plugin also sends usage events (which tool ran, duration, result counts, coarse failure kind) so Mem0 can tell how the plugin is used and where it breaks. These are **not anonymous**: when an API key is configured they are sent under your Mem0 account email, the same way the SDK attributes its own. Queries, memory text, and entity ids are never sent. Turn it off with `MEM0_TELEMETRY=false`.
The plugin also sends anonymous usage events (which tool ran, duration, result counts, coarse failure kind) so Mem0 can tell how the plugin is used and where it breaks. Queries, memory text, and entity ids are never sent. Turn it off with `MEM0_TELEMETRY=false`.
## Status
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@mem0/deepseek-plugin",
"version": "0.3.1",
"version": "0.3.0",
"description": "Mem0 long-term memory as a native DeepSeek Harness (Cordis) plugin.",
"type": "module",
"license": "Apache-2.0",
+4 -3
View File
@@ -26,9 +26,10 @@ export const name = "mem0";
export const inject = ["tools", "systemPrompt"];
// Tags writes so Mem0's backend attributes them to this integration in
// telemetry. Values outside the backend's KNOWN_EVENT_SOURCES allowlist bucket
// into "OTHERS"; this one is added by mem0ai/platform#3602 and reads as OTHERS
// until that ships.
// telemetry. The backend keeps recognized values via its KNOWN_EVENT_SOURCES
// allowlist; unknown values bucket into "OTHERS", so "DEEPSEEK_HARNESS" must be
// added to that allowlist for usage to surface by name (a one-line backend PR,
// same pattern as the ZAPIER / STRANDS sources).
const SOURCE = "DEEPSEEK_HARNESS";
const DEFAULT_SEARCH_LIMIT = 10;
+64
View File
@@ -0,0 +1,64 @@
# Source and compatibility notes
Imported from [NousResearch/hermes-plugin-mem0](https://github.com/NousResearch/hermes-plugin-mem0)
version 1.3.0 at commit `3fc36950b2b7c19cdd81c6de99f10d2cbed850af`.
The original native provider MIT license is retained in `LICENSE`. Generated shared core
is Apache-2.0, with its license in `LICENSE-APACHE-2.0`; package metadata records both.
This Mem0-owned port is version 1.4.0.
The source was a handoff of Hermes' bundled provider, with package-relative imports and
standalone dependency declarations; its historical authors remain in the upstream repositories.
## Changes in this port
- Generates only shared `core/message_utils.py` through `plugin-build.json`.
- Uses shared secret redaction and token-aware batching for non-empty completed-turn text.
- Preserves full message text after redaction; explicit `sync_max_chars` and the OSS default of 450
limit chunk size instead of discarding the tail. Platform/HTTP has no default character cap.
- Attaches Hermes session IDs as top-level Mem0 `run_id` on writes; existing user recall
remains unfiltered by session. The queue is in-memory, with bounded shutdown, logged failures,
and no durable retry.
- Keeps `mem0`, `memory.provider`, `mem0.json`, environment fallbacks, all four tool names,
identity precedence, SDK backends, and the three setup modes.
- Refuses OSS collection dimension mismatches without deleting existing vectors.
## Hermes contract review
Reviewed the [official memory-provider documentation](https://hermes-agent.nousresearch.com/docs/developer-guide/memory-provider-plugin),
release [v0.21.3 / v2026.9.14](https://github.com/NousResearch/hermes-agent/releases/tag/v2026.9.14)
(commit `345cd2b057a452236de401d3534b8502a7465e8d`), and main at
`c62bd9f2078a946108f1c9d9b24bf118963277ef` on September 18, 2026.
- These Hermes revisions still bundle Mem0. Bundled providers take precedence over user
directories. See README for an isolated preview; installation alone does not replace core.
- The installer supports `owner/repo/subdirectory` and copies only that subtree. All runtime
imports must stay within this directory or use Hermes/declared Python dependencies.
- Main installs dependencies from `[project].dependencies` and reapplies them after updates;
v0.21.3 does not. Install dependencies explicitly in the Hermes environment on that release.
- `register(ctx)` registers a `MemoryProvider` instance. Hermes dispatches tool schemas and
lifecycle hooks; no general-plugin hook registration is required. Background work retains
context variables through `spawn_context_thread`, including profile and secret scope.
- Hermes supports the existing `sync_turn(user, assistant, *, session_id="")` signature.
New optional `messages`/`turn_author` arguments are passed only to providers accepting them.
- `post_setup(hermes_home, config)` takes over setup and activation. The copied wizard already
owns its prompt helpers; it still uses Hermes config, curses and credential utilities.
- The handoff's original notes mentioned `config_schema.py`, but neither its actual directory
nor the checked Hermes Mem0 directory contains it. This port preserves CLI setup and does
not claim a provider-specific dashboard configuration panel.
Older Hermes releases and live cloud/OSS services require additional validation; preserving
configuration and tool names does not imply compatibility with every historical host API.
Actual external-loader and `MemoryManager` smoke checks passed on both pinned host
revisions: initialization, tool dispatch, optional sync arguments, full-turn tail capture,
session ID forwarding and shutdown. SDK backends were mocked; no live service was contacted.
Reproduce the host smoke from the repository root (use the Hermes environment or a test
venv with its imported dependencies installed):
```bash
HERMES_SOURCE=/path/to/hermes-agent python integrations/hermes-plugin/tests/smoke_hermes.py
```
The check confirms bundled precedence, then isolates external discovery by replacing the
bundled search root with an empty temporary directory in the test process only.
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 Nous Research
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [2023] [Taranjeet Singh]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+273
View File
@@ -0,0 +1,273 @@
# Mem0 for Hermes Agent
Native Hermes memory provider, version **1.4.0**, based on the Nous Research
[1.3.0 handoff](https://github.com/NousResearch/hermes-plugin-mem0/tree/3fc36950b2b7c19cdd81c6de99f10d2cbed850af).
Keeps platform, self-hosted HTTP, and in-process OSS modes and all four existing tools.
See [HANDOFF.md](HANDOFF.md) for provenance and compatibility details.
## Install and activate
Requires Python 3.11+ and the Hermes memory-provider API present in **v0.21.3
(v2026.9.14)**. Prefer the latest Hermes release. Earlier Hermes versions have not
been validated. Install dependencies into the Python environment that runs Hermes:
```bash
# Replace this with your Hermes checkout path.
HERMES_REPO=/path/to/hermes-agent
uv pip install --python "$HERMES_REPO/.venv/bin/python" 'mem0ai>=2.0.10,<3' 'httpx>=0.27,<1'
```
Recent Hermes development versions install `pyproject.toml` dependencies automatically;
v0.21.3 needs the explicit command above. Optional OSS providers can require additional
packages installed by the setup wizard.
### Local worktree preview
From the Mem0 worktree root, copy the complete directory to the active Hermes profile.
This command refuses to overwrite an existing user plugin:
```bash
python3 - <<'PYINSTALL'
import os
import shutil
from pathlib import Path
home = Path(os.environ.get("HERMES_HOME", "~/.hermes")).expanduser()
shutil.copytree("integrations/hermes-plugin", home / "plugins" / "mem0")
PYINSTALL
```
**Installing is not enough on Hermes versions that bundle Mem0.** Both v0.21.3 and
Hermes main checked on September 18, 2026 still contain `plugins/memory/mem0`.
Hermes always selects that bundled provider ahead of a same-named user plugin;
`hermes plugins enable mem0` does not change this precedence.
To preview this version without changing your normal Hermes checkout, make a separate
Hermes worktree and replace its bundled copy. From the Mem0 worktree root:
```bash
MEM0_WORKTREE="$PWD"
HERMES_PREVIEW="${TMPDIR:-/tmp}/hermes-mem0-preview"
git -C "$HERMES_REPO" worktree add --detach "$HERMES_PREVIEW" HEAD
mv "$HERMES_PREVIEW/plugins/memory/mem0" "$HERMES_PREVIEW/.mem0-bundled-backup"
cp -R "$MEM0_WORKTREE/integrations/hermes-plugin" "$HERMES_PREVIEW/plugins/memory/mem0"
cd "$HERMES_PREVIEW"
"$HERMES_REPO/.venv/bin/python" -m hermes_cli.main memory setup
"$HERMES_REPO/.venv/bin/python" -m hermes_cli.main chat
```
Use a fresh `HERMES_HOME` for a separate test profile, or your existing profile to retain
its configuration and memory identity. The preview uses your existing Hermes environment;
the original provider remains in the normal checkout and in the preview backup.
Updating Hermes can restore its bundled copy, so recheck which provider is selected.
Once Hermes no longer bundles Mem0, the user plugin copy loads directly.
### After this integration is published
Only after `integrations/hermes-plugin` is available on the remote branch:
```bash
hermes plugins install mem0ai/mem0/integrations/hermes-plugin
hermes plugins enable mem0
hermes memory setup
hermes memory status
```
The bundled-provider precedence above still applies. This remote command does not install
unpublished worktree changes. Keep the name `mem0`, existing `mem0.json`, `MEM0_*` variables,
and `memory.provider: mem0`; no memory migration or new user ID is required.
## Capture and recall
Automatic capture processes the completed user/assistant turn with the shared Mem0
message preparation: known-secret redaction and token-aware batching. Empty text is skipped.
It preserves full completed-turn text after redaction instead of dropping everything after 450 characters.
Raw tool results and the full historical transcript are not captured. Platform mode sends
prepared turn text to Mem0 Cloud for extraction; HTTP and OSS modes use their configured
server or model providers. Pattern-based redaction cannot recognize every possible secret.
An explicit positive `sync_max_chars` still limits each message chunk; long messages are
split across chunks without losing their remaining text. The default is no character cap
for platform/HTTP and 450 characters per chunk for OSS. Batches also obey the shared token
budget. Writes attach the Hermes session as Mem0’s top-level `run_id`. Recall does not filter by
`run_id`, so the existing user identity continues to recall across sessions. Automatic
capture uses an in-memory queue: process crashes can lose queued turns, and failed backend
batches are logged without durable retries. Shutdown drains the queue for a bounded time;
a stuck backend can leave queued turns unfinished.
Only `core/message_utils.py` is generated from `agent-plugin-core`; Hermes owns provider
lifecycle, setup, tools, and SDK backends. This package does not add MCP or coding-agent skills.
## Config
Behavioral settings live in `$HERMES_HOME/mem0.json` (set them via `hermes memory setup`). Store credentials in the active profile’s `$HERMES_HOME/.env`.
| Key | Default | Description |
|-----|---------|-------------|
| `mode` | `platform` | `platform` (Mem0 Cloud) or `oss` (self-managed, in-process) |
| `host` | — | Self-hosted Mem0 server URL (the Docker dashboard). When set, connects over HTTP with `X-API-Key`. Don't combine with `mode: oss` |
| `user_id` | Gateway user ID, else `hermes-user` | Explicit config or `MEM0_USER_ID` preserves the same identity across hosts. The legacy `hermes-user` placeholder permits gateway fallback. |
| `agent_id` | `hermes` | Agent identifier |
| `rerank` | `false` | Rerank search results for relevance (platform mode only) |
| `sync_max_chars` | Uncapped (platform/HTTP), `450` (OSS) | Positive character limit per chunk; longer text is split, not discarded. Increase for larger OSS embedding windows. |
The plugin has three connection modes:
- **Platform** — Mem0's hosted cloud (`api.mem0.ai`). Set `MEM0_API_KEY`. (default)
- **Self-hosted dashboard** — a Mem0 server you run yourself via Docker. Set `host`. See below.
- **OSS** — run Mem0 in-process with your own LLM + vector store. Set `mode: oss`. See below.
## Self-Hosted Dashboard (Server) Mode
Connect the plugin to a standalone Mem0 server you run yourself — the Docker-shipped Mem0 dashboard/server with its own REST API. Unlike OSS mode (which runs `mem0ai` in-process with your own vector store), here the plugin just talks HTTP to your server.
1. Run the Mem0 server (FastAPI + pgvector) from its Docker image and note its URL and `ADMIN_API_KEY`.
2. Point the plugin at it — via the setup wizard:
```bash
hermes memory setup # select "mem0" → "Self-hosted server"
# Or non-interactive:
hermes memory setup mem0 --mode selfhosted --host http://localhost:8888 --api-key your-admin-api-key
```
or via env vars:
```bash
echo "MEM0_HOST=http://localhost:8888" >> ~/.hermes/.env
echo "MEM0_API_KEY=your-admin-api-key" >> ~/.hermes/.env
```
or in `$HERMES_HOME/mem0.json`:
```json
{
"host": "http://localhost:8888",
"api_key": "your-admin-api-key"
}
```
3. Start a fresh Hermes session and call `mem0_search` — it connects to your server.
The plugin authenticates with `X-API-Key` and uses the server's `/search` and `/memories` routes. `api_key` is optional — omit it only for servers running with `AUTH_DISABLED`.
> Setting `host` routes to the self-hosted server automatically. Don't set `mode: oss` — OSS takes precedence and ignores `host`.
## OSS (Self-Hosted) Mode
Run Mem0 locally with your own LLM, embedder, and vector store. This is the in-process SDK mode. To instead connect to a Mem0 server you run via Docker, see [Self-Hosted Dashboard (Server) Mode](#self-hosted-dashboard-server-mode) above.
### Interactive Setup
```bash
hermes memory setup
# Select "mem0" → "Open Source (self-hosted)"
# Follow prompts for LLM, embedder, and vector store
```
### Agent-Driven Setup (Flags)
```bash
hermes memory setup mem0 --mode oss \
--oss-llm openai --oss-llm-key sk-... \
--oss-vector qdrant
```
### Supported Providers
| Component | Providers |
|-----------|-----------|
| LLM | openai, ollama |
| Embedder | openai, ollama |
| Vector Store | qdrant (local/server), pgvector |
### Flags Reference
| Flag | Description |
|------|-------------|
| `--mode` | `platform`, `selfhosted`, or `oss` |
| `--oss-llm` | LLM provider (default: openai) |
| `--oss-llm-key` | LLM API key |
| `--oss-embedder` | Embedder provider (default: openai) |
| `--oss-vector` | Vector store (default: qdrant) |
| `--oss-vector-path` | Qdrant local path |
| `--user-id` | User identifier |
## Switching Modes
### Platform to OSS
```bash
hermes memory setup mem0 --mode oss --oss-llm-key sk-...
```
Or edit `$HERMES_HOME/mem0.json` directly:
```json
{
"mode": "oss",
"oss": {
"llm": {"provider": "openai", "config": {"model": "gpt-5-mini", "is_reasoning_model": true}},
"embedder": {"provider": "openai", "config": {"model": "text-embedding-3-small"}},
"vector_store": {"provider": "qdrant", "config": {"path": "~/.hermes/mem0_qdrant"}}
}
}
```
### OSS to Platform
```bash
hermes memory setup mem0 --mode platform --api-key sk-...
```
### Dry Run (preview without writing)
```bash
hermes memory setup mem0 --mode oss --oss-llm-key sk-... --dry-run
```
## Tools
| Tool | Description |
|------|-------------|
| `mem0_search` | Semantic search by meaning |
| `mem0_add` | Store an explicit fact after known-secret redaction (no LLM extraction) |
| `mem0_update` | Update a memory's text by ID |
| `mem0_delete` | Delete a memory by ID |
## Troubleshooting
### "Mem0 temporarily unavailable"
Circuit breaker tripped after 5 consecutive failures. Resets after 2 minutes.
- **Platform mode**: Check API key and internet connectivity.
- **OSS mode**: Check that your vector store (qdrant/pgvector) is running.
### OSS: Qdrant connection refused
```bash
# If using local Qdrant, check the storage path is writable:
ls -la ~/.hermes/mem0_qdrant
# If using Qdrant server, check it's reachable:
curl http://localhost:6333/healthz
```
### OSS: PGVector connection refused
```bash
# Verify PostgreSQL is running and accepting connections:
pg_isready -h localhost -p 5432
```
### OSS: Ollama not reachable
```bash
# Check Ollama is running:
curl http://localhost:11434/api/tags
```
### Memories not appearing
- `mem0_add` stores a redacted fact without extraction. Completed turns are extracted automatically.
- Search uses semantic matching — try broader queries.
- Check `user_id` matches between sessions (`$HERMES_HOME/mem0.json`).
### Existing OSS collection has different embedding dimensions
Initialization fails with a configuration error instead of deleting or recreating the
collection. Restore the original embedding model/dimensions, or explicitly configure a new
collection name. Existing vectors remain untouched.
+663
View File
@@ -0,0 +1,663 @@
"""Mem0 memory plugin — MemoryProvider interface.
Server-side fact extraction and semantic search via the Mem0 Platform API (cloud), a
self-hosted Mem0 server (MEM0_HOST, HTTP), or OSS Memory. Secrets live in $HERMES_HOME/.env
(MEM0_API_KEY, MEM0_HOST); settings in $HERMES_HOME/mem0.json via `hermes memory setup`:
mode ("platform"|"oss"), host, user_id (canonical id across gateways; unset → gateway-native
id), agent_id. MEM0_* env vars remain a fallback.
"""
from __future__ import annotations
import atexit
import json
import logging
import threading
import time
from collections import deque
from contextlib import suppress
from contextvars import copy_context
from pathlib import Path
from typing import Any, Dict, List
from agent.memory_provider import MemoryProvider, spawn_context_thread
from agent.secret_scope import get_secret
from tools.registry import tool_error
from utils import atomic_json_write, read_json_or_empty
from .core.message_utils import extraction_message_batches, redact
logger = logging.getLogger(__name__)
# Circuit breaker: after _BREAKER_THRESHOLD consecutive failures, pause API
# calls for _BREAKER_COOLDOWN_SECS to avoid hammering a down server.
_BREAKER_THRESHOLD, _BREAKER_COOLDOWN_SECS, _PREFETCH_WAIT_SECS = 5, 120, 3
_SHUTDOWN_WAIT_SECS = 5.0
_CLIENT_ERROR_TYPES = ("MemoryNotFoundError", "ValidationError")
# Placeholder user_id. initialize() treats it as "no operator-configured user_id"
# so legacy mem0.json files written by the wizard don't override gateway-native ids.
_DEFAULT_USER_ID = "hermes-user"
# Legacy OSS embedding windows remain configurable; split instead of discarding the tail.
_SYNC_MSG_MAX_CHARS = 450
# Sentence ends recognized when trimming a synced message. Deliberately unordered:
# the LAST boundary of ANY kind wins, so one CJK stop early in a mixed-script turn
# cannot outrank a Latin stop near the end of the window. ``".\n"`` is not listed —
# its index can never exceed the bare ``"."`` it starts with.
_SYNC_SENTENCE_ENDS = ("。", "!", "?", ".", "!", "?")
def _truncate_for_sync(text: str, max_len: int = _SYNC_MSG_MAX_CHARS) -> str:
"""Cap a synced message at its last sentence boundary within ``max_len``.
Short messages pass through unchanged; long ones keep the last complete
sentence inside the window so fact extraction still sees coherent statements,
with a hard cut as fallback when no boundary exists (or one only appears in
the first third of the window, which usually means unsegmented input).
"""
if len(text) <= max_len:
return text
window = text[:max_len]
cut = max(window.rfind(sep) for sep in _SYNC_SENTENCE_ENDS)
if cut > max_len // 3:
return text[: cut + 1]
return text[:max_len]
def _is_client_error(exc: Exception) -> bool:
"""True for user-caused errors (bad ID, not found) that should NOT trip circuit breaker."""
err_str = str(exc).lower()
return type(exc).__name__ in _CLIENT_ERROR_TYPES or any(s in err_str for s in ("404", "not found", "valid uuid"))
def _load_config() -> dict:
"""Env vars provide defaults; $HERMES_HOME/mem0.json overrides individual keys.
Layering avoids a silent failure when the JSON file exists but lacks fields
like ``api_key`` that the user set in ``.env``."""
from hermes_constants import get_hermes_home
# Identity (user/agent id), host and mode are .env values like the key: read them through the
# profile scope too, or a secondary profile's memories land in the default profile's account.
# A scope-less multiplex caller raises here on purpose — that is a spawn-site bug, and
# swallowing it would silently route the turn's memories to the default profile.
config = {
"mode": get_secret("MEM0_MODE", "") or "platform",
"host": get_secret("MEM0_HOST", "") or "",
"agent_id": get_secret("MEM0_AGENT_ID", "") or "hermes",
"oss": {},
}
if user_id := get_secret(
"MEM0_USER_ID", ""
): # only when explicitly configured, so initialize() can fall back to the gateway-native id
config["user_id"] = user_id
file_cfg = read_json_or_empty(get_hermes_home() / "mem0.json")
config.update({k: v for k, v in file_cfg.items() if v is not None and v != ""})
# MEM0_API_KEY authenticates the Platform and self-hosted HTTP backends; pure OSS mode builds its
# backend from the local ``oss`` config and has no platform credential to resolve, so a profile
# scope WITHOUT the key must still load an OSS config (#99121 as it stands today: the caller is
# scoped, the scope is just empty). Decided after mem0.json overrode the env defaults because
# the file may be what selects ``oss``. Scope-less callers already raised above.
if config.get("mode", "platform") == "oss":
config.setdefault("api_key", "")
elif not config.get("api_key"):
config["api_key"] = get_secret("MEM0_API_KEY", "")
return config
def _schema(name: str, description: str, properties: dict[str, tuple[str, str]], required: list[str]) -> dict:
props = {k: {"type": t, "description": d} for k, (t, d) in properties.items()}
return {
"name": name,
"description": description,
"parameters": {"type": "object", "properties": props, "required": required},
}
TOOL_SCHEMAS = [
_schema(
"mem0_search",
"Search the user's memories by meaning; returns facts ranked by relevance. Use this before answering any question that may depend on what you know about the user (preferences, facts, history, people, projects, past decisions). For multi-part or multi-hop questions, call it several times — vary the wording and run follow-up searches on what earlier results reveal; one search is rarely enough.",
{
"query": ("string", "What to search for."),
"top_k": ("integer", "Max results (default: 10, max: 50)."),
"rerank": ("boolean", "Rerank results for relevance (default: false, platform mode only)."),
},
["query"],
),
_schema(
"mem0_add",
"Store a durable fact about the user, verbatim (no LLM extraction). Call this the moment the user states a lasting preference, correction, decision, or personal detail worth recalling on future turns — don't wait to be asked to remember. Skip transient chit-chat and facts you've already stored.",
{"content": ("string", "The fact to store.")},
["content"],
),
_schema(
"mem0_update",
"Replace the text of an existing memory by its ID (take the ID from a mem0_search result). Use when a stored fact has changed or was wrong — correct it in place instead of adding a duplicate.",
{"memory_id": ("string", "Memory UUID to update."), "text": ("string", "New text content.")},
["memory_id", "text"],
),
_schema(
"mem0_delete",
"Delete a memory by its ID (take the ID from a mem0_search result). Use when a stored fact is obsolete or the user asks you to forget it; prefer mem0_update if the fact merely changed.",
{"memory_id": ("string", "Memory UUID to delete.")},
["memory_id"],
),
]
_PROMPT_BODY = (
"You have persistent memory of this user from past conversations. You should call mem0_search before answering anything that could depend on prior context (the user's preferences, facts, history, people, projects, or earlier decisions) — do not rely on the chat window alone, and do not assume you have no memory.\n"
"For multi-part or multi-hop questions, run several searches with different wording/angles and follow-up searches on what the first results surface; one search is rarely enough. Keep searching until you have every fact the question needs before you answer.\n"
"Tools: mem0_search to find memories, mem0_add to store facts, mem0_update and mem0_delete to manage by ID."
)
class Mem0MemoryProvider(MemoryProvider):
"""Mem0 memory with server-side extraction and semantic search (platform, self-hosted or OSS)."""
def __init__(self):
self._config = self._backend = self._sync_thread = self._prefetch_thread = None
self._mode, self._api_key, self._host, self._user_id, self._agent_id = (
"platform",
"",
"",
_DEFAULT_USER_ID,
"hermes",
)
self._rerank_default, self._channel = False, "cli" # channel = gateway name (cli/telegram/discord/...)
self._sync_max_chars = 0
self._session_id = ""
self._sync_queue = deque()
self._closed = False
self._prefetch_query = self._prefetch_result = ""
self._prefetch_done = self._atexit_registered = False
self._prefetch_threads = set()
self._consecutive_failures, self._breaker_open_until = 0, 0.0 # circuit breaker state
self._breaker_lock, self._sync_lock, self._prefetch_lock = threading.Lock(), threading.Lock(), threading.Lock()
@property
def name(self) -> str:
return "mem0"
def is_available(self) -> bool:
cfg = _load_config()
if cfg.get("mode", "platform") == "oss":
return bool(cfg.get("oss", {}).get("vector_store"))
return bool(
cfg.get("api_key") or cfg.get("host")
) # platform needs a key; self-hosted a host (key optional with AUTH_DISABLED)
def save_config(self, values, hermes_home):
"""Merge-write config to $HERMES_HOME/mem0.json."""
config_path = Path(hermes_home) / "mem0.json"
atomic_json_write(config_path, {**read_json_or_empty(config_path), **values}, mode=0o600)
def get_config_schema(self):
cfg = _load_config()
api_key_required = cfg.get("mode", "platform") != "oss" and not cfg.get("host")
return [
{
"key": "api_key",
"description": "Mem0 Platform API key",
"secret": True,
"required": api_key_required,
"env_var": "MEM0_API_KEY",
"url": "https://app.mem0.ai",
},
{
"key": "host",
"description": "Self-hosted Mem0 server URL (leave blank for cloud)",
"required": False,
"env_var": "MEM0_HOST",
},
{"key": "user_id", "description": "User identifier", "default": "hermes-user"},
{"key": "agent_id", "description": "Agent identifier", "default": "hermes"},
{
"key": "rerank",
"description": "Enable reranking for recall",
"default": "false",
"choices": ["true", "false"],
},
]
def post_setup(self, hermes_home: str, config: dict) -> None:
from ._setup import post_setup
post_setup(hermes_home, config)
def _oss_hint(self, template: str, default: str = "vector store") -> str:
"""OSS-only hint; ``{vs}`` is the configured vector-store provider. "" in other modes."""
return (
template.format(vs=self._config.get("oss", {}).get("vector_store", {}).get("provider", default))
if self._mode == "oss"
else ""
)
def _create_backend(self):
try:
from . import _backend
if self._mode == "oss":
return _backend.OSSBackend(self._config.get("oss", {}))
return (
_backend.SelfHostedBackend(self._api_key, self._host)
if self._host
else _backend.PlatformBackend(self._api_key)
)
except Exception as e:
logger.error("Mem0 backend failed to initialize (%s mode): %s", self._mode, redact(str(e)))
self._init_error = redact(str(e))
return None
def _is_breaker_open(self) -> bool:
"""True while the breaker is tripped; an expired cooldown resets the failure count."""
with self._breaker_lock:
if self._consecutive_failures >= _BREAKER_THRESHOLD and time.monotonic() < self._breaker_open_until:
return True
if self._consecutive_failures >= _BREAKER_THRESHOLD:
self._consecutive_failures = 0
return False
def _format_error(self, prefix: str, exc: Exception) -> str:
msg = f"{prefix}: {redact(str(exc))}"
if any(s in str(exc).lower() for s in ("connection", "refused", "timeout")):
msg += self._oss_hint(" (check that {vs} is running)")
return msg
def _record_success(self):
with self._breaker_lock:
self._consecutive_failures = 0
def _record_failure(self):
with self._breaker_lock:
self._consecutive_failures = count = self._consecutive_failures + 1
if count >= _BREAKER_THRESHOLD:
self._breaker_open_until = time.monotonic() + _BREAKER_COOLDOWN_SECS
if count >= _BREAKER_THRESHOLD:
hint = self._oss_hint(" Check that your {vs} vector store is running and reachable.", "unknown")
logger.warning(
"Mem0 circuit breaker tripped after %d consecutive failures. Pausing API calls for %ds.%s",
count,
_BREAKER_COOLDOWN_SECS,
hint,
)
def _try(self, call, log, msg: str):
"""Background-path wrapper: run ``call`` under the breaker; on error log ``msg`` and return None."""
try:
result = call()
except Exception as e:
self._record_failure()
log(msg, redact(str(e)))
return None
self._record_success()
return result
def initialize(self, session_id: str, **kwargs) -> None:
self._session_id = session_id
self._config = cfg = _load_config()
self._mode, self._api_key, self._host, self._agent_id = (
cfg.get("mode", "platform"),
cfg.get("api_key", ""),
cfg.get("host", ""),
cfg.get("agent_id", "hermes"),
)
# user_id precedence: operator-configured (env/mem0.json) > gateway-native id (kwargs) > _DEFAULT_USER_ID.
# The literal placeholder counts as unset so wizard users still get gateway-native ids.
configured = cfg.get("user_id")
self._user_id = (
(None if configured == _DEFAULT_USER_ID else configured) or kwargs.get("user_id") or _DEFAULT_USER_ID
)
# Persisted rerank preference: default for mem0_search when the model omits ``rerank``. Platform-only.
_rr = cfg.get("rerank", False)
self._rerank_default = _rr.lower() in ("true", "1", "yes") if isinstance(_rr, str) else bool(_rr)
self._channel = kwargs.get("platform") or "cli"
default_cap = _SYNC_MSG_MAX_CHARS if self._mode == "oss" else 0
self._sync_max_chars = int(cfg.get("sync_max_chars", default_cap))
if self._sync_max_chars < 0:
raise ValueError("sync_max_chars must be zero (unlimited) or positive")
self._backend = self._create_backend()
if self._backend and not self._atexit_registered:
atexit.register(self.shutdown)
self._atexit_registered = True
def _search(self, query: str, top_k: int = 10, rerank: bool = False, backend=None) -> list:
# Scoped to user_id only — by design — so recall surfaces memories from any gateway/agent under this
# principal; writes attach agent_id and metadata.channel so narrower views remain possible at query time.
return (backend or self._backend).search(
redact(query), filters={"user_id": self._user_id}, top_k=top_k, rerank=rerank
)
def _add(self, messages: list, infer: bool, *, session_id: str = ""):
"""Send messages already redacted at the capture or explicit-tool boundary."""
metadata = {"channel": self._channel} if self._channel else {}
return self._backend.add(
messages,
user_id=self._user_id,
agent_id=self._agent_id,
infer=infer,
metadata=metadata,
run_id=session_id or self._session_id,
)
def system_prompt_block(self) -> str:
# Mirror _create_backend precedence (oss > host > platform). Rerank is a Mem0 Platform feature only.
mode_label = (
"OSS (self-hosted)"
if self._mode == "oss"
else "self-hosted (HTTP API)"
if self._host
else "platform (cloud API)"
)
rerank_note = " Rerank is available on search." if (self._mode == "platform" and not self._host) else ""
return f"# Mem0 Memory\nActive. Mode: {mode_label}. User: {self._user_id}.\n{_PROMPT_BODY}{rerank_note}"
def on_session_switch(self, new_session_id: str, **kwargs) -> None:
self._session_id = new_session_id
with self._prefetch_lock:
self._prefetch_query = self._prefetch_result = ""
self._prefetch_done = False
def on_turn_start(self, turn_number: int, message: str, **kwargs) -> None:
self._start_prefetch(message)
def _consume_prefetch_result(self, query: str) -> str | None:
"""Pop the finished prefetch body for ``query`` (None if absent or still running)."""
with self._prefetch_lock:
if self._prefetch_query != query or not self._prefetch_done:
return None
result, self._prefetch_result, self._prefetch_done = self._prefetch_result, "", False
return result
def _start_prefetch(self, query: str) -> None:
backend = self._backend
if not query or backend is None or self._is_breaker_open():
return
def _run():
try:
results = self._try(
lambda: self._search(query, rerank=self._rerank_default, backend=backend),
logger.debug,
"Mem0 prefetch failed: %s",
)
lines = [redact(r.get("memory", "")) for r in (results or []) if r.get("memory")]
body = "## Mem0 Memory\n" + "\n".join(f"- {line}" for line in lines) if lines else ""
with self._prefetch_lock:
if self._prefetch_query == query:
self._prefetch_result, self._prefetch_done = body, True
finally:
with self._prefetch_lock:
self._prefetch_threads.discard(threading.current_thread())
self._close_if_idle()
with self._prefetch_lock:
if self._closed:
return
# Same query already answered or still in flight: don't restart it.
if self._prefetch_query == query and (
self._prefetch_done or (self._prefetch_thread and self._prefetch_thread.is_alive())
):
return
self._prefetch_query, self._prefetch_result, self._prefetch_done = query, "", False
self._prefetch_thread = t = spawn_context_thread(_run, name="mem0-prefetch")
self._prefetch_threads.add(t)
t.start()
def prefetch(self, query: str, *, session_id: str = "") -> str:
"""Recall memories for the CURRENT question with a short hot-path wait."""
if (cached := self._consume_prefetch_result(query)) is not None:
return cached
self._start_prefetch(query)
with self._prefetch_lock:
thread = self._prefetch_thread if self._prefetch_query == query else None
if thread:
thread.join(timeout=_PREFETCH_WAIT_SECS)
return (
self._consume_prefetch_result(query) or ""
) # slow backend: skip injection; mem0_search remains the backstop
def _sync_batches(self, messages: list) -> list:
if self._sync_max_chars and any(len(m["content"]) > self._sync_max_chars for m in messages):
chunks = []
for message in messages:
remaining = message["content"]
while remaining:
part = _truncate_for_sync(remaining, self._sync_max_chars)
chunks.extend(extraction_message_batches([{**message, "content": part}]))
remaining = remaining[len(part) :]
return chunks
return extraction_message_batches(messages)
def _drain_sync(self) -> None:
while True:
with self._sync_lock:
if not self._sync_queue:
self._sync_thread = None
break
context, messages, session_id = self._sync_queue.popleft()
context.run(self._sync_messages, messages, session_id)
self._close_if_idle()
def _sync_messages(self, messages: list, session_id: str) -> None:
for batch in self._sync_batches(messages):
if self._is_breaker_open():
logger.warning(
"Mem0 turn not synced for session %s: circuit breaker open; capture is best effort", session_id
)
return
self._try(
lambda: self._add(batch, infer=True, session_id=session_id),
logger.warning,
"Mem0 sync failed: %s",
)
def sync_turn(self, user_content: str, assistant_content: str, *, session_id: str = "") -> None:
"""Queue full, redacted turns without blocking the agent or dropping busy turns."""
if self._backend is None or self._is_breaker_open():
return
messages = [
{"role": role, "content": redact(content)}
for role, content in (("user", user_content), ("assistant", assistant_content))
if content
]
if not messages:
return
with self._sync_lock:
if self._closed:
return
# ponytail: in-memory queue; use a durable spool if crash recovery is required.
self._sync_queue.append((copy_context(), messages, session_id or self._session_id))
if self._sync_thread is None:
self._sync_thread = spawn_context_thread(self._drain_sync, name="mem0-sync")
self._sync_thread.start()
def get_tool_schemas(self) -> List[Dict[str, Any]]:
return list(TOOL_SCHEMAS)
# -- tool handlers: (required params, error label, body, client-error policy) ---
# Client errors (bad ID / not found) never trip the breaker, except for mem0_add
# where they count as failures; update/delete answer them with "Memory not found".
def _tool_search(self, args: dict) -> str:
top_k = max(1, min(int(args.get("top_k", 10)), 50))
rerank_raw = args.get("rerank", self._rerank_default)
rerank = rerank_raw.lower() not in ("false", "0", "no") if isinstance(rerank_raw, str) else bool(rerank_raw)
results = self._search(args["query"], top_k, rerank)
if not results:
return json.dumps({"result": "No relevant memories found."})
items = [
{"id": r.get("id"), "memory": redact(r.get("memory", "")), "score": r.get("score", 0)} for r in results
]
return json.dumps({"results": items, "count": len(items)})
def _tool_add(self, args: dict) -> str:
result = self._add([{"role": "user", "content": redact(args["content"])}], infer=False)
event_id = result.get("event_id") if isinstance(result, dict) else None
# Cloud add is async (server-side extraction); OSS and self-hosted store synchronously.
msg = "Fact stored." if (self._mode == "oss" or self._host) else "Fact queued for storage."
return json.dumps({"result": msg, "event_id": event_id})
_TOOL_HANDLERS = {
"mem0_search": (("query",), "Search failed", _tool_search, "skip"),
"mem0_add": (("content",), "Failed to store", _tool_add, "count"),
"mem0_update": (
("memory_id", "text"),
"Update failed",
lambda self, a: json.dumps(self._backend.update(a["memory_id"], redact(a["text"]))),
"not_found",
),
"mem0_delete": (
("memory_id",),
"Delete failed",
lambda self, a: json.dumps(self._backend.delete(a["memory_id"])),
"not_found",
),
}
def handle_tool_call(self, tool_name: str, args: dict, **kwargs) -> str:
if self._backend is None:
err = getattr(self, "_init_error", "unknown error")
return json.dumps(
{
"error": f"Mem0 backend not initialized: {err}.{self._oss_hint(' Check that {vs} is running and reachable.')}"
}
)
if self._is_breaker_open():
return json.dumps(
{
"error": f"Mem0 temporarily unavailable (multiple consecutive failures). Will retry automatically.{self._oss_hint(' Check that your {vs} is running.')}"
}
)
if tool_name not in self._TOOL_HANDLERS:
return tool_error(f"Unknown tool: {tool_name}")
required, label, body, on_client_error = self._TOOL_HANDLERS[tool_name]
if not isinstance(args, dict):
return tool_error("Tool arguments must be an object")
if missing := next((k for k in required if not isinstance(args.get(k), str) or not args[k].strip()), None):
return tool_error(f"Missing or invalid required parameter: {missing}")
if tool_name == "mem0_search":
try:
int(args.get("top_k", 10))
except (TypeError, ValueError, OverflowError):
return tool_error("top_k must be an integer")
try:
result = body(self, args)
except Exception as e:
client = _is_client_error(e)
if client and on_client_error == "not_found":
return tool_error(f"Memory not found: {args['memory_id']}")
if not client or on_client_error == "count":
self._record_failure()
return tool_error(self._format_error(label, e))
self._record_success()
return result
def _shutdown_backend(self):
with suppress(Exception):
if self._backend:
self._backend.close()
self._backend = None
def _close_if_idle(self) -> None:
with self._sync_lock, self._prefetch_lock:
if self._closed and self._sync_thread is None and not self._prefetch_threads:
self._shutdown_backend()
def shutdown(self) -> None:
with self._sync_lock, self._prefetch_lock:
self._closed = True
threads = [*self._prefetch_threads, self._sync_thread]
deadline = time.monotonic() + _SHUTDOWN_WAIT_SECS
for thread in threads:
if thread and thread.is_alive():
thread.join(timeout=max(0, deadline - time.monotonic()))
if any(thread and thread.is_alive() for thread in threads):
logger.warning("Mem0 shutdown timed out; pending capture may be lost if the process exits")
# Active workers close their backend when finished, never underneath a request.
self._close_if_idle()
def register(ctx) -> None:
"""Register Mem0 as a memory provider plugin."""
ctx.register_memory_provider(Mem0MemoryProvider())
# Compatibility names retained for callers of the original bundled provider.
ADD_SCHEMA = {
"name": "mem0_add",
"description": (
"Store a durable fact about the user, verbatim (no LLM extraction). "
"Call this the moment the user states a lasting preference, correction, "
"decision, or personal detail worth recalling on future turns — don't "
"wait to be asked to remember. Skip transient chit-chat and facts you've "
"already stored."
),
"parameters": {
"type": "object",
"properties": {
"content": {"type": "string", "description": "The fact to store."},
},
"required": ["content"],
},
}
DELETE_SCHEMA = {
"name": "mem0_delete",
"description": (
"Delete a memory by its ID (take the ID from a mem0_search "
"result). Use when a stored fact is obsolete or the user asks you to "
"forget it; prefer mem0_update if the fact merely changed."
),
"parameters": {
"type": "object",
"properties": {
"memory_id": {"type": "string", "description": "Memory UUID to delete."},
},
"required": ["memory_id"],
},
}
SEARCH_SCHEMA = {
"name": "mem0_search",
"description": (
"Search the user's memories by meaning; returns facts ranked by "
"relevance. Use this before answering any question that may depend on "
"what you know about the user (preferences, facts, history, people, "
"projects, past decisions). For multi-part or multi-hop questions, "
"call it several times — vary the wording and run follow-up searches "
"on what earlier results reveal; one search is rarely enough."
),
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "What to search for."},
"top_k": {"type": "integer", "description": "Max results (default: 10, max: 50)."},
"rerank": {
"type": "boolean",
"description": "Rerank results for relevance (default: false, platform mode only).",
},
},
"required": ["query"],
},
}
UPDATE_SCHEMA = {
"name": "mem0_update",
"description": (
"Replace the text of an existing memory by its ID (take the ID from a "
"mem0_search result). Use when a stored fact has changed "
"or was wrong — correct it in place instead of adding a duplicate."
),
"parameters": {
"type": "object",
"properties": {
"memory_id": {"type": "string", "description": "Memory UUID to update."},
"text": {"type": "string", "description": "New text content."},
},
"required": ["memory_id", "text"],
},
}
# ---- END PLUGIN-COMPAT ----
+298
View File
@@ -0,0 +1,298 @@
"""Backend abstraction for Mem0 Platform and OSS modes."""
from __future__ import annotations
from abc import ABC, abstractmethod
from contextlib import closing, suppress
from typing import Any
def _add_kwargs(user_id: str, agent_id: str, infer: bool, metadata: dict | None, run_id: str = "") -> dict[str, Any]:
return {
"user_id": user_id,
"agent_id": agent_id,
"infer": infer,
**({"metadata": metadata} if metadata else {}),
**({"run_id": run_id} if run_id else {}),
}
def _unwrap_results(response: Any) -> list:
"""Normalize API response — extract results list from dict or pass through."""
return response.get("results", []) if isinstance(response, dict) else response if isinstance(response, list) else []
class Mem0Backend(ABC):
"""Unified interface over Platform (MemoryClient), self-hosted (HTTP) and OSS (Memory) backends.
update()/delete() are template methods: subclasses implement raw ``_update``/``_delete``."""
@abstractmethod
def search(self, query: str, *, filters: dict, top_k: int = 10, rerank: bool = False) -> list[dict]: ...
@abstractmethod
def add(
self,
messages: list,
*,
user_id: str,
agent_id: str,
infer: bool = False,
metadata: dict | None = None,
run_id: str = "",
) -> dict: ...
@abstractmethod
def _update(self, memory_id: str, text: str) -> None: ...
@abstractmethod
def _delete(self, memory_id: str) -> None: ...
def update(self, memory_id: str, text: str) -> dict:
self._update(memory_id, text)
return {"result": "Memory updated.", "memory_id": memory_id}
def delete(self, memory_id: str) -> dict:
self._delete(memory_id)
return {"result": "Memory deleted.", "memory_id": memory_id}
def close(self) -> None:
pass
class PlatformBackend(Mem0Backend):
"""Wraps mem0.MemoryClient for Mem0 Platform (cloud API)."""
def __init__(self, api_key: str):
from mem0 import MemoryClient
self._client = MemoryClient(api_key=api_key)
def search(self, query: str, *, filters: dict, top_k: int = 10, rerank: bool = False) -> list[dict]:
return _unwrap_results(self._client.search(query, filters=filters, top_k=top_k, rerank=rerank))
def add(
self,
messages: list,
*,
user_id: str,
agent_id: str,
infer: bool = False,
metadata: dict | None = None,
run_id: str = "",
) -> dict:
return self._client.add(messages, **_add_kwargs(user_id, agent_id, infer, metadata, run_id))
def _update(self, memory_id: str, text: str) -> None:
self._client.update(memory_id=memory_id, text=text)
def _delete(self, memory_id: str) -> None:
self._client.delete(memory_id=memory_id)
class SelfHostedBackend(Mem0Backend):
"""Direct HTTP backend for a self-hosted Mem0 server (the FastAPI ``server/``).
mem0.MemoryClient is hardwired to the cloud API (``Authorization: Token``, ``GET /v1/ping/`` in ``__init__``),
so this speaks the server's real contract: ``X-API-Key`` auth and the ``/memories`` / ``/search`` routes."""
def __init__(self, api_key: str, host: str, transport=None):
import httpx
headers = {
"Content-Type": "application/json",
**({"X-API-Key": api_key} if api_key else {}),
} # key omitted only for AUTH_DISABLED servers
# Connect-level retries keep one dropped SYN from counting toward the breaker. ``transport`` is injectable for tests.
self._client = httpx.Client(
base_url=host.rstrip("/"),
headers=headers,
timeout=30.0,
transport=transport or httpx.HTTPTransport(retries=2),
)
def _json(self, method: str, path: str, **kwargs) -> Any:
resp = self._client.request(method, path, **kwargs)
resp.raise_for_status()
return resp.json() if resp.content else {}
def search(self, query: str, *, filters: dict, top_k: int = 10, rerank: bool = False) -> list[dict]:
# rerank is platform-only; the self-hosted /search ignores it. user_id belongs in filters (top-level is deprecated).
return _unwrap_results(
self._json(
"POST", "/search", json={"query": query, "top_k": top_k, **({"filters": filters} if filters else {})}
)
)
def add(
self,
messages: list,
*,
user_id: str,
agent_id: str,
infer: bool = False,
metadata: dict | None = None,
run_id: str = "",
) -> dict:
return self._json(
"POST", "/memories", json={"messages": messages, **_add_kwargs(user_id, agent_id, infer, metadata, run_id)}
)
def _update(self, memory_id: str, text: str) -> None:
self._json("PUT", f"/memories/{memory_id}", json={"text": text})
def _delete(self, memory_id: str) -> None:
self._json("DELETE", f"/memories/{memory_id}")
def close(self) -> None:
with suppress(Exception):
self._client.close()
_DIRECT_OPENAI_PROVIDER = "hermes_openai"
_DIRECT_OPENAI_CLASS_PATH = f"{__package__}._openai_llm.DirectOpenAILLM"
def _register_direct_openai_provider() -> None:
"""Register Hermes' OpenAI-only Mem0 LLM provider once per factory."""
from mem0.configs.llms.openai import OpenAIConfig
from mem0.utils.factory import LlmFactory
provider_map = getattr(LlmFactory, "provider_to_class", None)
register_provider = getattr(LlmFactory, "register_provider", None)
if not isinstance(provider_map, dict) or not callable(register_provider):
raise RuntimeError(
"mem0 LlmFactory does not support the provider registration required for the Hermes OpenAI OSS backend"
)
if provider_map.get(_DIRECT_OPENAI_PROVIDER) != (_DIRECT_OPENAI_CLASS_PATH, OpenAIConfig):
register_provider(_DIRECT_OPENAI_PROVIDER, _DIRECT_OPENAI_CLASS_PATH, OpenAIConfig)
class OSSBackend(Mem0Backend):
"""Wraps mem0.Memory for self-hosted (OSS) mode."""
def __init__(self, oss_config: dict):
import os
from mem0 import Memory
from ._oss_providers import EMBEDDER_PROVIDERS, KNOWN_DIMS, LLM_PROVIDERS
def _provider_block(name: str, registry: dict) -> dict:
"""Copy of oss_config[name] with the legacy ``api_base`` key mapped to the provider's canonical base-URL key."""
block = dict(oss_config[name])
provider_config = dict(block.get("config", {}))
legacy_base = provider_config.pop("api_base", None)
canonical_key = registry.get(str(block.get("provider") or "").strip().lower(), {}).get("base_url_key")
if legacy_base and canonical_key:
provider_config.setdefault(canonical_key, legacy_base)
block["config"] = provider_config
return block
vector_store = dict(oss_config["vector_store"])
vs_config = dict(vector_store.get("config", {}))
if "path" in vs_config:
vs_config["path"] = os.path.expanduser(vs_config["path"])
embedder_config = oss_config.get("embedder", {}).get("config", {})
dims = embedder_config.get("embedding_dims") or KNOWN_DIMS.get(embedder_config.get("model", ""))
if dims:
vs_config["embedding_model_dims"] = dims
self._recreate_collection_if_dims_changed(vector_store.get("provider", "qdrant"), vs_config, dims)
vector_store["config"] = vs_config
config = {
"vector_store": vector_store,
"llm": _provider_block("llm", LLM_PROVIDERS),
"embedder": _provider_block("embedder", EMBEDDER_PROVIDERS),
"version": "v1.1",
}
if str(config["llm"].get("provider") or "").strip().lower() == "openai":
# mem0 validates LlmConfig.provider before its factory lookup: build the supported OpenAI config, then swap the provider.
_register_direct_openai_provider()
from mem0.configs.base import MemoryConfig
memory_config = MemoryConfig(**config)
try:
memory_config.llm.provider = _DIRECT_OPENAI_PROVIDER
except (AttributeError, TypeError) as exc:
raise RuntimeError(
"mem0 MemoryConfig does not expose a mutable llm.provider for the Hermes OpenAI OSS backend"
) from exc
self._memory = Memory(memory_config)
else:
self._memory = Memory.from_config(config)
@staticmethod
def _recreate_collection_if_dims_changed(provider: str, vs_config: dict, expected_dims: int) -> None:
"""Reject dimension changes without deleting existing memories (legacy method name)."""
collection_name = vs_config.get("collection_name", "mem0")
current_dims = None
with suppress(Exception):
if provider == "qdrant":
from qdrant_client import QdrantClient
path, url = vs_config.get("path"), vs_config.get("url")
if path:
client = QdrantClient(path=path)
elif url:
client = QdrantClient(url=url, api_key=vs_config.get("api_key"))
else:
return
with closing(client):
if not client.collection_exists(collection_name):
return
vectors = client.get_collection(collection_name).config.params.vectors
# Named-vector collections expose a dict; unnamed expose an object with .size.
if isinstance(vectors, dict):
vectors = next(iter(vectors.values()), None)
current_dims = getattr(vectors, "size", None)
elif provider == "pgvector":
import psycopg2
conn_params = {
k: vs_config[k]
for k in ("host", "port", "user", "password", "dbname", "sslmode")
if vs_config.get(k)
}
with closing(psycopg2.connect(**conn_params)) as conn:
conn.autocommit = True
with closing(conn.cursor()) as cur:
cur.execute(
"SELECT atttypmod FROM pg_attribute WHERE attrelid = %s::regclass AND attname = 'vector'",
(collection_name,),
)
row = cur.fetchone()
current_dims = row[0] if row and row[0] > 0 else None
if current_dims is not None and current_dims != expected_dims:
raise ValueError(
f"Collection {collection_name!r} has {current_dims} embedding dimensions, but {expected_dims} are configured. "
"Existing memories were preserved. Restore the previous embedder or choose a new collection_name "
"and migrate your memories explicitly."
)
def search(self, query: str, *, filters: dict, top_k: int = 10, rerank: bool = False) -> list[dict]:
return _unwrap_results(self._memory.search(query, filters=filters, top_k=top_k))
def add(
self,
messages: list,
*,
user_id: str,
agent_id: str,
infer: bool = False,
metadata: dict | None = None,
run_id: str = "",
) -> dict:
return self._memory.add(messages, **_add_kwargs(user_id, agent_id, infer, metadata, run_id))
def _update(self, memory_id: str, text: str) -> None:
self._memory.update(memory_id, data=text)
def _delete(self, memory_id: str) -> None:
self._memory.delete(memory_id)
def close(self):
with suppress(Exception):
telemetry = getattr(self._memory, "telemetry", None)
if telemetry and hasattr(telemetry, "posthog"):
with suppress(Exception):
telemetry.posthog.shutdown()
vs = getattr(self._memory, "vector_store", None)
# Memory, then its vector store, then the store's raw client; the first failure aborts the chain.
for obj in filter(None, (self._memory, vs, getattr(vs, "client", None))):
if hasattr(obj, "close"):
obj.close()
+88
View File
@@ -0,0 +1,88 @@
"""OpenAI-only LLM adapter for Mem0 OSS mode."""
from __future__ import annotations
import logging
from typing import Dict, List, Optional, Union
from mem0.configs.llms.base import BaseLlmConfig
from mem0.configs.llms.openai import OpenAIConfig
from mem0.llms.base import LLMBase
from mem0.llms.openai import OpenAILLM
# BaseLlmConfig fields copied into OpenAIConfig; the last two may be absent on older mem0.
_COPIED_FIELDS = (
"model",
"temperature",
"api_key",
"max_tokens",
"top_p",
"top_k",
"enable_vision",
"vision_details",
"http_client_proxies",
)
_OPTIONAL_FIELDS = ("reasoning_effort", "is_reasoning_model")
class DirectOpenAILLM(OpenAILLM):
"""Use OpenAI credentials and requests regardless of router environment."""
def __init__(self, config: Optional[Union[BaseLlmConfig, OpenAIConfig, Dict]] = None):
if config is None:
config = OpenAIConfig()
elif isinstance(config, dict):
config = OpenAIConfig(**config)
elif isinstance(config, BaseLlmConfig) and not isinstance(config, OpenAIConfig):
fields = {k: getattr(config, k) for k in _COPIED_FIELDS}
fields.update({k: getattr(config, k, None) for k in _OPTIONAL_FIELDS})
config = OpenAIConfig(**fields)
if not config.model:
config.model = "gpt-5-mini"
# Configs predating the setup marker: keep the default model reasoning-safe
# without overriding an explicit user choice.
if config.model == "gpt-5-mini" and config.is_reasoning_model is None:
config.is_reasoning_model = True
# Bypass OpenAILLM.__init__ (it picks OpenRouter when OPENROUTER_API_KEY is
# set); LLMBase still owns validation and supported-parameter filtering.
LLMBase.__init__(self, config)
# OPENAI_API_KEY / OPENAI_BASE_URL are profile credentials: read them through the secret
# scope, never raw os.environ, or a multiplexed secondary's memory extraction runs on the
# default profile's OpenAI account (and its proxy).
from agent.secret_scope import get_secret
api_key = self.config.api_key or get_secret("OPENAI_API_KEY", "")
if not api_key:
raise ValueError("OpenAI API key is required for the Hermes Mem0 OSS provider")
from openai import OpenAI
self.client = OpenAI(
api_key=api_key,
base_url=self.config.openai_base_url or get_secret("OPENAI_BASE_URL", "") or "https://api.openai.com/v1",
)
def generate_response(
self,
messages: List[Dict[str, str]],
response_format=None,
tools: Optional[List[Dict]] = None,
tool_choice: str = "auto",
**kwargs,
):
params = self._get_supported_params(messages=messages, **kwargs)
params.update({"model": self.config.model, "messages": messages})
# No OpenRouter-only fields; ``store`` is opt-in so OpenAI-compatible endpoints never receive unknown fields.
if self.config.store is not None:
params["store"] = self.config.store
if response_format:
params["response_format"] = response_format
if tools:
params["tools"], params["tool_choice"] = tools, tool_choice
response = self.client.chat.completions.create(**params)
parsed_response = self._parse_response(response, tools)
if self.config.response_callback:
try:
self.config.response_callback(self, response, params)
except Exception:
logging.error("Error running Mem0 OpenAI response callback")
return parsed_response
@@ -0,0 +1,98 @@
"""OSS provider definitions for LLM, embedder, and vector store."""
from __future__ import annotations
import os
from typing import Any
from hermes_constants import get_hermes_home
LLM_PROVIDERS: dict[str, dict[str, Any]] = {
"openai": {
"label": "OpenAI",
"needs_key": True,
"env_var": "OPENAI_API_KEY",
"default_model": "gpt-5-mini",
"base_url_key": "openai_base_url",
},
"ollama": {
"label": "Ollama (local)",
"needs_key": False,
"default_model": "llama3.1:8b",
"default_url": "http://localhost:11434",
"base_url_key": "ollama_base_url",
"pip_dep": "ollama",
},
}
EMBEDDER_PROVIDERS: dict[str, dict[str, Any]] = {
"openai": {
"label": "OpenAI",
"needs_key": True,
"env_var": "OPENAI_API_KEY",
"default_model": "text-embedding-3-small",
"base_url_key": "openai_base_url",
"dims": 1536,
},
"ollama": {
"label": "Ollama (local)",
"needs_key": False,
"default_model": "nomic-embed-text",
"default_url": "http://localhost:11434",
"base_url_key": "ollama_base_url",
"dims": 768,
"pip_dep": "ollama",
},
}
VECTOR_PROVIDERS: dict[str, dict[str, Any]] = {
# Resolved lazily (see ``vector_default_config``): the profile home is a ContextVar at call time,
# not an import-time constant, and ``~/.hermes`` is wrong on Windows and under profiles.
"qdrant": {
"label": "Qdrant",
"default_config": {"path": lambda: str(get_hermes_home() / "mem0_qdrant")},
"pip_dep": "qdrant-client",
},
"pgvector": {
"label": "PGVector",
"default_config": {
"host": "localhost",
"port": 5432,
"user": os.getenv("USER", "postgres"),
"dbname": "postgres",
},
"pip_dep": "psycopg2-binary",
},
}
KNOWN_DIMS: dict[str, int] = {
"text-embedding-3-small": 1536,
"text-embedding-3-large": 3072,
"text-embedding-ada-002": 1536,
"nomic-embed-text": 768,
}
def vector_default_config(provider_id: str) -> dict[str, Any]:
"""A vector store's ``default_config`` with callable defaults resolved for the active profile."""
return {k: (v() if callable(v) else v) for k, v in VECTOR_PROVIDERS[provider_id]["default_config"].items()}
SECTION_REGISTRIES = (("llm", LLM_PROVIDERS), ("embedder", EMBEDDER_PROVIDERS), ("vector_store", VECTOR_PROVIDERS))
def validate_oss_config(oss_config: dict) -> list[str]:
"""Validate an OSS config dict. Returns list of error strings (empty = valid)."""
errors: list[str] = []
for section, registry in SECTION_REGISTRIES:
block = oss_config.get(section)
if not block or not isinstance(block, dict):
errors.append(f"Missing required section: {section}")
elif block.get("provider", "") not in registry:
errors.append(
f"Unknown {section} provider '{block.get('provider', '')}'. Valid: {', '.join(registry.keys())}"
)
vs = oss_config.get("vector_store", {})
if vs.get("provider") == "pgvector" and not vs.get("config", {}).get("user"):
errors.append("PGVector requires 'user' in vector_store.config")
return errors
+741
View File
@@ -0,0 +1,741 @@
"""Setup wizard for Mem0 plugin — interactive and flag-based modes."""
from __future__ import annotations
import getpass
import json
import os
import shutil
import socket
import subprocess
import sys
import tempfile
import time
import urllib.error
import urllib.request
from contextlib import suppress
from pathlib import Path
from typing import Any
from hermes_constants import get_hermes_home # noqa: F401 — patched by tests
from ._oss_providers import (
EMBEDDER_PROVIDERS,
KNOWN_DIMS,
LLM_PROVIDERS,
SECTION_REGISTRIES,
VECTOR_PROVIDERS,
validate_oss_config,
vector_default_config,
)
_OLLAMA_URL = "http://localhost:11434"
_PGVECTOR_CONTAINER, _PGVECTOR_IMAGE, _PGVECTOR_PASSWORD = "hermes-pgvector", "pgvector/pgvector:pg17", "hermes"
def _curses_select(title: str, items: list[tuple[str, str]], default: int = 0) -> int:
from hermes_cli.curses_ui import curses_radiolist
return curses_radiolist(
title,
[f"{label} {desc}" if desc else label for label, desc in items],
selected=default,
cancel_returns=default,
)
def _prompt(label: str, default: str | None = None, secret: bool = False) -> str:
"""Prompt for a value with optional default and secret masking."""
sys.stdout.write(f" {label}{f' [{default}]' if default else ''}: ")
sys.stdout.flush()
val = getpass.getpass(prompt="") if secret and sys.stdin.isatty() else sys.stdin.readline().strip()
return val or (default or "")
def _input(label: str, default: str) -> str:
return input(f" {label} [{default}]: ").strip() or default
def _masked(secret: str) -> str:
return f"...{secret[-4:]}" if len(secret) > 4 else "set"
def _http_get(url: str, path: str, timeout: int):
return urllib.request.urlopen(urllib.request.Request(f"{url.rstrip('/')}{path}", method="GET"), timeout=timeout)
def _prompt_api_key(label: str, env_var: str, hermes_home: str) -> str:
"""Prompt for API key, showing masked existing value if found."""
existing = os.environ.get(env_var, "")
if not existing:
from agent.secret_scope import load_env_file
existing = load_env_file(Path(hermes_home) / ".env").get(env_var, "")
hint = f" (current: {_masked(existing)}, blank to keep)" if existing else ""
return getpass.getpass(f" {label} API key{hint}: ").strip()
def _api_key_writes(
flags: dict, label: str, *, url: str | None = None, fresh_label: str | None = None
) -> dict[str, str]:
"""MEM0_API_KEY for .env: from --api-key, else prompt (masking any key already in the environment)."""
if flags.get("api_key"):
return {"MEM0_API_KEY": flags["api_key"]}
existing = os.environ.get("MEM0_API_KEY", "")
if url and not existing:
print(f" Get yours at {url}")
val = _prompt(
f"{label} (current: {_masked(existing)}, blank to keep)" if existing else fresh_label or label, secret=True
)
return {"MEM0_API_KEY": val} if val else {}
def _print_dry_run(summary: str, env_writes: dict, check=None) -> None:
print(f"\n [dry-run] Would save config: {summary}")
if env_writes:
print(" [dry-run] Would write API key to .env")
if check:
check()
print(" [dry-run] No files written.\n")
# --oss-vector-<key> flags accepted per vector store (also the pgvector key order).
_VECTOR_FLAG_KEYS = {"qdrant": ("path", "url"), "pgvector": ("host", "port", "user", "password", "dbname")}
_FLAG_KEYS = (
"mode",
"api_key",
"host",
*(f"oss_{s}{k}" for s in ("llm", "embedder") for k in ("", "_key", "_model", "_url")),
"oss_vector",
*(f"oss_vector_{k}" for ks in _VECTOR_FLAG_KEYS.values() for k in ks),
"user_id",
)
_FLAG_DEFAULTS = {"oss_llm": "openai", "oss_embedder": "openai", "oss_vector": "qdrant"}
def parse_flags(argv: list[str] | None = None) -> dict[str, str]:
args = argv if argv is not None else sys.argv[1:]
flags: dict[str, Any] = {**{k: _FLAG_DEFAULTS.get(k, "") for k in _FLAG_KEYS}, "dry_run": False}
flag_map = {"--" + k.replace("_", "-"): k for k in _FLAG_KEYS}
i = 0
while i < len(args):
if args[i] == "--dry-run":
flags["dry_run"] = True
elif args[i] in flag_map and i + 1 < len(args):
flags[flag_map[args[i]]] = args[i + 1]
i += 1
i += 1
return flags
def _model_block(flags: dict, registry: dict, prefix: str) -> tuple[str, dict, dict[str, Any]]:
"""Resolve (provider_id, provider_def, config) for an LLM/embedder section from flags."""
pid = flags.get(prefix, "openai")
pdef = registry[pid]
cfg: dict[str, Any] = {"model": flags.get(f"{prefix}_model") or pdef["default_model"]}
url = flags.get(f"{prefix}_url") or pdef.get("default_url")
if url and pdef.get("base_url_key"):
cfg[pdef["base_url_key"]] = url
return pid, pdef, cfg
def build_oss_config(flags: dict[str, str]) -> tuple[dict, dict[str, str]]:
"""Build (oss_config for mem0.json, env_writes of secrets for .env) from parsed flags."""
llm_id, llm_def, llm_config = _model_block(flags, LLM_PROVIDERS, "oss_llm")
if llm_id == "openai" and llm_config["model"] == "gpt-5-mini":
llm_config["is_reasoning_model"] = True
embedder_id, embedder_def, embedder_config = _model_block(flags, EMBEDDER_PROVIDERS, "oss_embedder")
dims = KNOWN_DIMS.get(embedder_config["model"])
if dims:
embedder_config["embedding_dims"] = dims
vector_id = flags.get("oss_vector", "qdrant")
vector_config = vector_default_config(vector_id)
for key in _VECTOR_FLAG_KEYS.get(vector_id, ()):
if val := flags.get(f"oss_vector_{key}"):
vector_config[key] = int(val) if key == "port" else val
if "url" in vector_config:
vector_config.pop("path", None) # a remote Qdrant URL replaces local storage
oss_config = {
"llm": {"provider": llm_id, "config": llm_config},
"embedder": {"provider": embedder_id, "config": embedder_config},
"vector_store": {"provider": vector_id, "config": vector_config},
}
# An embedder sharing the LLM's provider reuses the LLM key when no embedder key was given.
llm_key = flags.get("oss_llm_key") if llm_def.get("needs_key") else ""
emb_key = (
(flags.get("oss_embedder_key") or (flags.get("oss_llm_key") if embedder_id == llm_id else ""))
if embedder_def.get("needs_key")
else ""
)
env_writes = {d["env_var"]: k for d, k in ((llm_def, llm_key), (embedder_def, emb_key)) if k}
return oss_config, env_writes
def _write_env(env_path: Path, env_writes: dict[str, str]) -> None:
env_path.parent.mkdir(parents=True, exist_ok=True)
# utf-8-sig like the canonical .env readers: a BOM'd first line would miss the key match and get duplicated.
existing_lines = env_path.read_text(encoding="utf-8-sig").splitlines() if env_path.exists() else []
keys = [
line.split("=", 1)[0].strip() if "=" in line and not line.startswith("#") else None for line in existing_lines
]
new_lines = [f"{k}={env_writes[k]}" if k in env_writes else line for k, line in zip(keys, existing_lines)]
new_lines += [f"{k}={v}" for k, v in env_writes.items() if k not in keys]
fd, temporary = tempfile.mkstemp(prefix=".mem0-env-", dir=env_path.parent)
try:
with os.fdopen(fd, "w", encoding="utf-8") as stream:
stream.write("\n".join(new_lines) + "\n")
os.replace(temporary, env_path)
finally:
Path(temporary).unlink(missing_ok=True)
def _activate_provider(config: dict) -> None:
"""Point config.yaml's memory.provider at mem0."""
from hermes_cli.config import save_config
config["memory"]["provider"] = "mem0"
save_config(config)
def _persist_provider_config(
hermes_home: str,
config: dict,
provider_config: dict,
env_writes: dict[str, str],
label: str,
key_line: str,
server: str | None = None,
) -> None:
"""Shared platform/self-hosted tail: activate, write mem0.json (0600), then .env, then a saved summary."""
_activate_provider(config)
from . import Mem0MemoryProvider
Mem0MemoryProvider().save_config(provider_config, hermes_home)
if env_writes:
_write_env(Path(hermes_home) / ".env", env_writes)
if server:
_check_selfhosted_server(server)
print(
"\n".join(
[
"",
f" Memory provider: {label}",
*([f" Server: {server}"] if server else []),
" Activation saved to config.yaml",
" Provider config saved",
*([f" {key_line}"] if env_writes else []),
"",
" Start a new session to activate.",
"",
]
)
)
def _setup_platform(hermes_home: str, config: dict, flags: dict[str, str]) -> None:
"""Platform mode setup — prompts for API key (secret -> .env), user/agent ids and rerank (-> mem0.json)."""
from utils import read_json_or_empty
provider_config = read_json_or_empty(Path(hermes_home) / "mem0.json")
print("\n Configuring mem0:\n")
env_writes = _api_key_writes(flags, "Mem0 Platform API key", url="https://app.mem0.ai")
for key, desc, default in (
("user_id", "User identifier", "hermes-user"),
("agent_id", "Agent identifier", "hermes"),
):
if val := _prompt(desc, default=str(provider_config.get(key) or default)):
provider_config[key] = val
choices = ["true", "false"]
current = str(provider_config.get("rerank", "false") or "").lower()
provider_config["rerank"] = choices[
_curses_select(
" Enable reranking for recall",
[(c, "") for c in choices],
default=choices.index(current) if current in choices else 0,
)
]
if flags.get("dry_run"):
_print_dry_run(str(provider_config), env_writes)
return
# Routing checks ``host`` before platform, so clear a stale self-hosted host. "" rather than
# pop(): save_config merges into the existing mem0.json, so a popped key would survive.
provider_config.update(mode="platform", host="")
# _load_config() also seeds ``host`` from MEM0_HOST (.env); the file clear can't help there, so warn.
if os.environ.get("MEM0_HOST", "").strip():
print(
f"\n ⚠ MEM0_HOST is set in your environment ({os.environ['MEM0_HOST']}). It overrides platform mode — remove it from ~/.hermes/.env (or unset it) or Hermes will keep routing to the self-hosted server."
)
_persist_provider_config(hermes_home, config, provider_config, env_writes, "mem0", "API keys saved to .env")
def _check_selfhosted_server(host: str) -> None:
"""Best-effort reachability check for a self-hosted Mem0 server (non-fatal)."""
try:
_http_get(host, "/docs", 5)
print(f" ✓ Mem0 server reachable at {host}")
except urllib.error.HTTPError:
# Any HTTP response (401/403/404) still means something is listening.
print(f" ✓ Mem0 server responding at {host}")
except Exception:
print(f" ⚠ Could not reach {host} — check the URL and that the server is running.")
def _setup_selfhosted(hermes_home: str, config: dict, flags: dict[str, str]) -> None:
"""Self-hosted mode — point at an existing Mem0 server: URL -> mem0.json, key -> .env (MEM0_API_KEY)."""
from utils import read_json_or_empty
provider_config = read_json_or_empty(Path(hermes_home) / "mem0.json")
print("\n Configuring mem0 (self-hosted server):\n")
host = flags.get("host") or _prompt(
"Mem0 server URL (e.g. http://localhost:8888)", default=provider_config.get("host") or None
)
if not host:
print(" Error: a server URL is required for self-hosted mode.", file=sys.stderr)
return
host = host.rstrip("/")
env_writes = _api_key_writes(flags, "Server API key", fresh_label="Server API key (blank if AUTH_DISABLED)")
user_id = flags.get("user_id") or _prompt(
"User identifier", default=provider_config.get("user_id") or "hermes-user"
)
agent_id = _prompt("Agent identifier", default=provider_config.get("agent_id") or "hermes")
if flags.get("dry_run"):
_print_dry_run(
f"host={host}, user_id={user_id}, agent_id={agent_id}", env_writes, lambda: _check_selfhosted_server(host)
)
return
provider_config.update(
mode="platform", host=host, user_id=user_id, agent_id=agent_id
) # routing: oss > host > platform
_persist_provider_config(
hermes_home, config, provider_config, env_writes, "mem0 (self-hosted)", "API key saved to .env", server=host
)
def _print_oss_summary(oss_config: dict, env_writes: dict, dry_run: bool = False) -> None:
llm, emb = oss_config["llm"], oss_config["embedder"]
w = 0 if dry_run else 9 # final summary column-aligns the labels
lines = [
"",
" [dry-run] OSS config would be:" if dry_run else " ✓ Mem0 configured (OSS mode)",
f" {'LLM:':<{w}} {llm['provider']} ({llm['config'].get('model', '')})",
f" {'Embedder:':<{w}} {emb['provider']} ({emb['config'].get('model', '')})",
f" {'Vector:':<{w}} {oss_config['vector_store']['provider']}",
]
if dry_run:
lines += [f" Env vars: {', '.join(env_writes.keys())}"] if env_writes else []
else:
lines += [
*([" API keys saved to .env"] if env_writes else []),
" Config saved to mem0.json",
" Provider set in config.yaml",
"",
" Start a new session to activate.",
"",
]
print("\n".join(lines))
def _finish_oss(
hermes_home: str,
config: dict,
oss_config: dict,
env_writes: dict[str, str],
user_id: str,
agent_id: str,
pgvector_config: dict | None = None,
) -> None:
"""Shared OSS tail: write secrets + mem0.json, install deps, activate, check, summarize."""
from . import Mem0MemoryProvider
if env_writes:
_write_env(Path(hermes_home) / ".env", env_writes)
Mem0MemoryProvider().save_config(
{"mode": "oss", "user_id": user_id, "agent_id": agent_id, "oss": oss_config}, hermes_home
)
_install_provider_deps(
oss_config["llm"]["provider"], oss_config["embedder"]["provider"], oss_config["vector_store"]["provider"]
)
if pgvector_config:
_ensure_pgvector_extension(pgvector_config)
_activate_provider(config)
_run_connectivity_checks(oss_config)
_print_oss_summary(oss_config, env_writes)
def _setup_oss(hermes_home: str, config: dict, flags: dict[str, str]) -> None:
"""OSS mode — non-interactive when --mode was given, otherwise curses pickers."""
if not flags.get("_mode_from_flag"):
_setup_oss_interactive(hermes_home, config)
return
oss_config, env_writes = build_oss_config(flags)
if errors := validate_oss_config(oss_config):
print("".join(f" Error: {e}\n" for e in errors), end="", file=sys.stderr)
sys.exit(1)
if flags.get("dry_run"):
_print_oss_summary(oss_config, env_writes, dry_run=True)
_run_connectivity_checks(oss_config)
print(" [dry-run] No files written.\n")
return
_finish_oss(
hermes_home, config, oss_config, env_writes, flags.get("user_id") or os.getenv("USER", "hermes-user"), "hermes"
)
def _docker(*args: str, timeout: int, **kwargs) -> subprocess.CompletedProcess:
return subprocess.run(["docker", *args], capture_output=True, timeout=timeout, stdin=subprocess.DEVNULL, **kwargs)
def _pg_ready(host: str, port: int, wait: int) -> bool:
"""Wait up to ``wait`` seconds for the port, then report whether PostgreSQL answers."""
_wait_for_port(host, port, timeout=wait)
return _check_pgvector(host, port)[0]
def _ensure_pgvector(host: str = "localhost", port: int = 5432) -> dict | None:
"""Ensure pgvector is reachable, offering Docker if not; returns the started container's vector_config, else None."""
if _check_pgvector(host, port)[0]:
print(f" ✓ PostgreSQL reachable at {host}:{port}")
return None
print(f" PostgreSQL not reachable at {host}:{port}")
if not shutil.which("docker"):
print(" Docker not found. Install Docker to auto-start pgvector,\n or run PostgreSQL with pgvector manually.")
return None
with suppress(Exception): # restart our own container if it exists but is stopped
result = _docker(
"inspect",
_PGVECTOR_CONTAINER,
"--format",
"{{.State.Status}}",
timeout=10,
text=True,
encoding="utf-8",
errors="replace",
)
if result.returncode == 0 and "exited" in result.stdout:
print(f" Found stopped container '{_PGVECTOR_CONTAINER}', restarting...")
_docker("start", _PGVECTOR_CONTAINER, timeout=15)
if _pg_ready(host, port, 15):
print(" ✓ PostgreSQL container restarted")
return None
if input(" Start pgvector via Docker? [Y/n]: ").strip().lower() not in ("", "y", "yes"):
print(" Skipping Docker setup. Make sure PostgreSQL with pgvector is running.")
return None
try:
print(f" Pulling {_PGVECTOR_IMAGE}...")
_docker("pull", _PGVECTOR_IMAGE, timeout=120)
_docker("rm", "-f", _PGVECTOR_CONTAINER, timeout=10) # remove existing container if present
print(f" Starting container '{_PGVECTOR_CONTAINER}' on port {port}...")
_docker(
"run",
"-d",
"--name",
_PGVECTOR_CONTAINER,
"-e",
f"POSTGRES_PASSWORD={_PGVECTOR_PASSWORD}",
"-p",
f"{port}:5432",
_PGVECTOR_IMAGE,
timeout=30,
check=True,
)
if _pg_ready(host, port, 20):
print(f" ✓ pgvector running on {host}:{port}")
else:
print(
" Warning: Container started but PostgreSQL not yet accepting connections.\n It may need a few more seconds. Config will be saved; retry later."
)
return {"host": host, "port": port, "user": "postgres", "password": _PGVECTOR_PASSWORD, "dbname": "postgres"}
except subprocess.CalledProcessError as e:
print(f" Failed to start Docker container: {e}")
except Exception as e:
print(f" Docker error: {e}")
return None
def _ensure_ollama(models: list[str]) -> bool:
"""Ensure Ollama is running and ``models`` are pulled; False when the user must handle it manually."""
ollama_bin = shutil.which("ollama")
if not (ok := _check_ollama(_OLLAMA_URL)[0]):
if not ollama_bin:
print(
" Ollama not found. Install it:\n curl -fsSL https://ollama.com/install.sh | sh\n Or on macOS: brew install ollama"
)
return False
print(" Ollama installed but not running. Starting...")
try:
subprocess.Popen(
[ollama_bin, "serve"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
)
_wait_for_port("localhost", 11434, timeout=10)
if ok := _check_ollama(_OLLAMA_URL)[0]:
print(" ✓ Ollama started")
except Exception as e:
print(f" Could not start Ollama: {e}")
if not ok:
print(" Warning: Ollama not reachable. Models cannot be pulled.")
return False
for model in models:
try:
names = [
m.get("name", "") for m in json.loads(_http_get(_OLLAMA_URL, "/api/tags", 5).read()).get("models", [])
]
except Exception:
names = []
if any(model in n or model.split(":")[0] in n for n in names):
print(f" ✓ Model '{model}' available")
continue
print(f" Pulling '{model}'... (this may take a few minutes)")
try:
subprocess.run([ollama_bin or "ollama", "pull", model], timeout=600, stdin=subprocess.DEVNULL)
print(f" ✓ Model '{model}' pulled")
except Exception as e:
print(f" Warning: Could not pull '{model}': {e}\n Run manually: ollama pull {model}")
return True
def _ensure_pgvector_extension(pg_config: dict) -> None:
try:
import psycopg2
except ImportError:
return
defaults = {"host": "localhost", "port": 5432, "user": "postgres", "dbname": "postgres"}
try:
conn = psycopg2.connect(
**(defaults | {k: v for k, v in pg_config.items() if k in defaults or (k == "password" and v)})
)
conn.autocommit = True
conn.cursor().execute("CREATE EXTENSION IF NOT EXISTS vector")
conn.close()
print(" ✓ pgvector extension enabled")
except Exception as e:
print(f" Warning: Could not enable pgvector extension: {e}")
def _wait_for_port(host: str, port: int, timeout: int = 15) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
socket.create_connection((host, port), timeout=1).close()
return
except OSError:
time.sleep(0.5)
# Picker descriptions: LLM/embedder show model (+ URL); vector stores by provider id (default: the id itself).
_VECTOR_DESCRIPTIONS = {
"qdrant": lambda cfg: cfg.get("path", "local storage"),
"pgvector": lambda cfg: f"{cfg.get('host', 'localhost')}:{cfg.get('port', 5432)}",
}
def _configure_model_provider(
kind: str, registry: dict, hermes_home: str, env_writes: dict[str, str], llm: tuple[str, dict] | None = None
) -> tuple[str, dict, str, str | None]:
"""Pick an LLM/embedder provider, collect its key, and (for Ollama) model + URL -> (id, definition, model, url).
For the embedder (``llm`` given), a provider shared with the LLM reuses the LLM key instead of prompting again."""
items = [
(
v["label"],
f"{v.get('default_model', '')} ({v['default_url']})"
if v.get("default_url")
else v.get("default_model", ""),
)
for v in registry.values()
]
pid = list(registry)[_curses_select(f"{kind} Provider", items, 0)]
pdef = registry[pid]
model, url = pdef["default_model"], pdef.get("default_url")
if pdef["needs_key"]:
if llm is None or pid != llm[0]:
if key := _prompt_api_key(
pdef["label"] if llm is None else f"{pdef['label']} embedder", pdef["env_var"], hermes_home
):
env_writes[pdef["env_var"]] = key
elif llm[1].get("env_var") in env_writes:
env_writes[pdef["env_var"]] = env_writes[llm[1]["env_var"]]
if pid == "ollama":
model = _input(f"{kind} model", pdef["default_model"])
url = _input("Ollama URL", pdef["default_url"])
return pid, pdef, model, url
def _setup_oss_interactive(hermes_home: str, config: dict) -> None:
env_writes: dict[str, str] = {}
llm_id, llm_def, llm_model, llm_url = _configure_model_provider("LLM", LLM_PROVIDERS, hermes_home, env_writes)
embedder_id, _, embedder_model, embedder_url = _configure_model_provider(
"Embedder", EMBEDDER_PROVIDERS, hermes_home, env_writes, llm=(llm_id, llm_def)
)
vector_items = [
(v["label"], _VECTOR_DESCRIPTIONS.get(pid, lambda cfg: pid)(vector_default_config(pid)))
for pid, v in VECTOR_PROVIDERS.items()
]
vector_id = list(VECTOR_PROVIDERS)[_curses_select("Vector Store", vector_items, 0)]
# Auto-setup: ensure Ollama is running and models are pulled; ensure pgvector is reachable (offer Docker if not).
ollama_models = [m for pid, m in ((llm_id, llm_model), (embedder_id, embedder_model)) if pid == "ollama"]
if ollama_models:
_ensure_ollama(ollama_models)
pgvector_config = _ensure_pgvector() if vector_id == "pgvector" else None
if (
vector_id == "pgvector" and not pgvector_config
): # native PostgreSQL: prompt for connection details (user first, historical order)
pg = {
k: _input(f"PostgreSQL {label}", d)
for k, label, d in (
("user", "user", os.getenv("USER", "postgres")),
("host", "host", "localhost"),
("port", "port", "5432"),
("dbname", "database", "postgres"),
)
}
pg_password = getpass.getpass(" PostgreSQL password (blank if none): ").strip()
pgvector_config = {**pg, "port": int(pg["port"]), **({"password": pg_password} if pg_password else {})}
user_id = _input("User ID", os.getenv("USER", "hermes-user"))
agent_id = _input("Agent ID", "hermes")
flags = {
"oss_llm": llm_id,
"oss_llm_model": llm_model,
"oss_llm_url": llm_url or "",
"oss_llm_key": env_writes.get(llm_def["env_var"], "") if llm_def.get("env_var") else "",
"oss_embedder": embedder_id,
"oss_embedder_model": embedder_model,
"oss_embedder_url": embedder_url or "",
"oss_vector": vector_id,
"user_id": user_id,
}
flags.update({f"oss_vector_{key}": str(val) for key, val in (pgvector_config or {}).items() if val})
oss_config, _ = build_oss_config(flags)
_finish_oss(hermes_home, config, oss_config, env_writes, user_id, agent_id, pgvector_config)
def _install_provider_deps(llm_id: str, embedder_id: str, vector_id: str) -> None:
deps = {
registry[pid]["pip_dep"]
for (_, registry), pid in zip(SECTION_REGISTRIES, (llm_id, embedder_id, vector_id))
if registry.get(pid, {}).get("pip_dep")
}
for dep in sorted(deps):
print(f" Installing {dep}...")
try:
# Environment-aware install: sealed hosted venvs redirect to the durable data-volume target instead of /opt/hermes.
from tools.lazy_deps import install_specs
outcome = install_specs([dep], timeout=60)
except Exception:
outcome = None
print(
f" ✓ Installed {dep}"
if outcome is not None and outcome.ok
else f" Warning: cannot install {dep}: {outcome.reason}"
if outcome is not None and outcome.blocked
else f" Warning: Could not install {dep}. Install manually: uv pip install {dep}"
)
if deps:
import importlib
importlib.invalidate_caches()
def _probe(fn, ok: str, fail: str, exc=Exception) -> tuple[bool, str]:
"""Run ``fn``; (True, ok) on success, (False, "fail: <error>") on ``exc``."""
try:
fn()
return True, ok
except exc as e:
return False, f"{fail}: {e}"
def _check_qdrant_path(path: str) -> tuple[bool, str]:
"""Check that qdrant local storage parent dir is writable."""
parent = Path(path).expanduser().parent
return _probe(
lambda: parent.mkdir(parents=True, exist_ok=True),
f"Directory writable: {parent}",
f"Cannot write to {parent}",
OSError,
)
def _check_ollama(url: str) -> tuple[bool, str]:
return _probe(lambda: _http_get(url, "/api/tags", 3), "Ollama reachable", f"Ollama not reachable at {url}")
def _check_pgvector(host: str, port: int) -> tuple[bool, str]:
return _probe(
lambda: socket.create_connection((host, port), timeout=3).close(),
f"PGVector reachable at {host}:{port}",
f"PGVector not reachable at {host}:{port}",
)
def _warn_unless(check: tuple[bool, str]) -> None:
ok, msg = check
if not ok:
print(f" Warning: {msg}")
def _run_connectivity_checks(oss_config: dict) -> None:
vs = oss_config.get("vector_store", {})
cfg = vs.get("config", {})
if vs.get("provider") == "qdrant":
path, url = cfg.get("path"), cfg.get("url")
if path:
_warn_unless(_check_qdrant_path(path))
elif url:
_warn_unless(
_probe(lambda: _http_get(url, "/healthz", 3), "Qdrant reachable", f"Qdrant not reachable at {url}")
)
elif vs.get("provider") == "pgvector":
_warn_unless(_check_pgvector(cfg.get("host", "localhost"), cfg.get("port", 5432)))
llm = oss_config.get("llm", {})
if llm.get("provider") == "ollama":
_warn_unless(_check_ollama(llm.get("config", {}).get("ollama_base_url", _OLLAMA_URL)))
_MODE_HANDLERS = {
"oss": _setup_oss,
"selfhosted": _setup_selfhosted,
"self-hosted": _setup_selfhosted,
"platform": _setup_platform,
}
# Interactive picker order: Platform, Self-hosted server, Open Source.
_MODE_ITEMS = [
("Platform", "Mem0 Cloud API (lightweight, just needs an API key)"),
("Self-hosted server", "Connect to an existing self-hosted Mem0 server (Docker/FastAPI)"),
("Open Source", "Run Mem0 locally (self-hosted LLM + vector store)"),
]
_MODE_PICKER = (_setup_platform, _setup_selfhosted, _setup_oss)
def post_setup(hermes_home: str, config: dict) -> None:
"""Entry point for `hermes memory setup`: routes on --mode (platform / selfhosted / oss), else shows a picker.
OSS is non-interactive only when the mode came from the flag."""
with suppress(ImportError): # mem0ai must meet the minimum version from plugin.yaml
import mem0
installed_ver = getattr(mem0, "__version__", None)
if installed_ver and tuple(int(x) for x in installed_ver.split(".")[:3]) < (2, 0, 10):
print(
f"\n ⚠ mem0ai {installed_ver} installed but >=2.0.10 required.\n Run: uv pip install --python {sys.executable} 'mem0ai>=2.0.10'"
)
flags = parse_flags(sys.argv[1:])
handler = _MODE_HANDLERS.get(flags["mode"])
flags["_mode_from_flag"] = handler is not None
if handler is None:
handler = _MODE_PICKER[_curses_select(" Select mode", _MODE_ITEMS, 0)]
handler(hermes_home, config, flags)
# Compatibility name retained for callers of the original bundled provider.
def has_oss_flags() -> bool:
"""Check if OSS-related flags are present in sys.argv."""
flags = parse_flags(sys.argv[1:])
if flags["mode"] == "oss":
return True
if any(flags.get(k) for k in ("oss_llm_key", "oss_vector_path", "oss_vector_url")):
return True
return False
# ---- END PLUGIN-COMPAT ----
@@ -0,0 +1,127 @@
"""Shared, host-independent redaction and lossless extraction batching."""
from __future__ import annotations
import json
import math
import re
from typing import Any
MAX_EXTRACTION_INPUT_TOKENS = 24000
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r"|(?:access|refresh|session)[_-]?token|token|authorization|credential"
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def redact(value: Any) -> str:
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, default=str)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if _is_agent_assignment(message) and index + 1 < len(exchange) and _is_agent_response(exchange[index + 1]):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
@@ -0,0 +1,25 @@
{
"id": "mem0",
"version": "1.4.0",
"homepage": "https://docs.mem0.ai/integrations/hermes",
"native": {
"pluginRoot": "",
"pythonFiles": [
"message_utils.py"
],
"skills": false,
"files": {
"plugin.yaml": "plugin.yaml",
"pyproject.toml": "pyproject.toml",
"__init__.py": "__init__.py",
"_backend.py": "_backend.py",
"_setup.py": "_setup.py",
"_openai_llm.py": "_openai_llm.py",
"_oss_providers.py": "_oss_providers.py",
"README.md": "README.md",
"HANDOFF.md": "HANDOFF.md",
"LICENSE": "LICENSE",
"LICENSE-APACHE-2.0": "LICENSE-APACHE-2.0"
}
}
}
+5
View File
@@ -0,0 +1,5 @@
name: mem0
version: 1.4.0
description: "Mem0 — server-side LLM fact extraction with semantic search, automatic deduplication, and opt-in reranking (platform mode)."
pip_dependencies:
- mem0ai>=2.0.10,<3
+17
View File
@@ -0,0 +1,17 @@
[project]
name = "hermes-plugin-mem0"
version = "1.4.0"
description = "Hermes Agent memory provider plugin: mem0"
requires-python = ">=3.11"
license = { text = "MIT AND Apache-2.0" }
# Installed into the Hermes venv by `hermes plugins install` / `enable` and re-applied after
# every `hermes update` (hermes-agent#113851). Keep upper bounds: Hermes pins its own deps exactly
# and refuses a plugin whose requirements cannot resolve against them.
dependencies = [
"mem0ai>=2.0.10,<3",
"httpx>=0.27,<1",
]
[project.optional-dependencies]
postgres = ['psycopg2-binary>=2.9,<3']
qdrant = ['qdrant-client>=1.9,<2']
@@ -0,0 +1,53 @@
"""Offline contract smoke against a real Hermes checkout.
HERMES_SOURCE=/path/to/hermes-agent python tests/smoke_hermes.py
Host modules are real; only the Mem0 backend is mocked. Uses a temporary profile.
"""
import json
import os
import pathlib
import shutil
import sys
import tempfile
from unittest.mock import Mock
hermes_source = pathlib.Path(os.environ["HERMES_SOURCE"]).resolve()
plugin_source = pathlib.Path(__file__).resolve().parents[1]
sys.path.insert(0, str(hermes_source))
with tempfile.TemporaryDirectory() as d:
os.environ["HERMES_HOME"] = d
os.environ["MEM0_API_KEY"] = "test-key"
import plugins.memory as pm
from agent.memory_manager import MemoryManager
destination = pathlib.Path(d) / "plugins" / "mem0"
shutil.copytree(plugin_source, destination)
assert pm.find_provider_dir("mem0") == hermes_source / "plugins" / "memory" / "mem0"
pm._MEMORY_PLUGINS_DIR = pathlib.Path(d) / "empty-bundled"
pm._MEMORY_PLUGINS_DIR.mkdir()
provider = pm.load_memory_provider("mem0", register_skills=False)
assert provider is not None
assert provider.__class__.__module__.startswith("_hermes_user_memory.")
backend = Mock()
backend.search.return_value = []
backend.add.return_value = {}
provider._create_backend = lambda: backend
manager = MemoryManager()
manager.add_provider(provider)
manager.initialize_all("s1", platform="cli", user_id="alice")
result = manager.handle_tool_call("mem0_search", {"query": "favorite programming language"})
assert json.loads(result)["result"] == "No relevant memories found."
manager.sync_all(
"I prefer Python for tooling. " * 100 + "UNIQUE_END_MARKER",
"I will remember that.",
session_id="s2",
messages=[{"role": "tool", "content": "NOT_FOR_CAPTURE"}],
turn_author={"id": "alice"},
)
manager.shutdown_all()
assert backend.add.call_count >= 1
sent = str(backend.add.call_args_list)
assert "UNIQUE_END_MARKER" in sent and "NOT_FOR_CAPTURE" not in sent
assert backend.add.call_args.kwargs["run_id"] == "s2"
print("PASS:", hermes_source, "real external loader + MemoryManager lifecycle/tools/full capture/session ID")
@@ -0,0 +1,86 @@
"""Offline backend contracts; no Hermes install, credentials, or database needed."""
import importlib.util
import json
import sys
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import Mock, patch
import httpx
spec = importlib.util.spec_from_file_location("hermes_backend_test._backend", Path(__file__).parents[1] / "_backend.py")
backend = importlib.util.module_from_spec(spec)
spec.loader.exec_module(backend)
class BackendTests(unittest.TestCase):
def test_cloud_and_oss_preserve_scope_and_accept_optional_session(self):
for backend_type, client_attr in ((backend.PlatformBackend, "_client"), (backend.OSSBackend, "_memory")):
with self.subTest(backend=backend_type.__name__):
instance = backend_type.__new__(backend_type)
client = Mock()
setattr(instance, client_attr, client)
client.search.return_value = {"results": [{"id": "old", "memory": "Legacy memory"}]}
instance.add([{"role": "user", "content": "fact"}], user_id="user", agent_id="hermes")
self.assertNotIn("run_id", client.add.call_args.kwargs)
instance.add([], user_id="user", agent_id="hermes", run_id="session", metadata={"channel": "cli"})
self.assertEqual(client.add.call_args.kwargs["run_id"], "session")
self.assertEqual(client.add.call_args.kwargs["metadata"], {"channel": "cli"})
self.assertEqual(instance.search("fact", filters={"user_id": "user"}, top_k=3)[0]["id"], "old")
self.assertEqual(client.search.call_args.kwargs["filters"], {"user_id": "user"})
self.assertEqual(client.search.call_args.kwargs["top_k"], 3)
def test_selfhosted_http_contract(self):
requests = []
def respond(request):
requests.append(request)
return httpx.Response(200, json={"results": [{"id": "legacy", "memory": "fact"}]})
instance = backend.SelfHostedBackend(
"test-key", "http://localhost:8888/", transport=httpx.MockTransport(respond)
)
try:
instance.add([], user_id="user", agent_id="hermes", run_id="session", infer=True)
self.assertEqual(json.loads(requests[-1].content)["run_id"], "session")
self.assertEqual(requests[-1].headers["X-API-Key"], "test-key")
instance.search("fact", filters={"user_id": "user"}, top_k=7, rerank=True)
self.assertEqual(str(requests[-1].url), "http://localhost:8888/search")
self.assertEqual(
json.loads(requests[-1].content), {"query": "fact", "filters": {"user_id": "user"}, "top_k": 7}
)
instance.update("legacy", "new fact")
self.assertEqual((requests[-1].method, json.loads(requests[-1].content)), ("PUT", {"text": "new fact"}))
instance.delete("legacy")
self.assertEqual(requests[-1].method, "DELETE")
finally:
instance.close()
def test_qdrant_dimension_change_never_deletes_memories(self):
client = Mock()
client.collection_exists.return_value = True
client.get_collection.return_value.config.params.vectors = SimpleNamespace(size=1536)
with patch.dict(sys.modules, {"qdrant_client": SimpleNamespace(QdrantClient=Mock(return_value=client))}):
with self.assertRaisesRegex(ValueError, "1536.*768"):
backend.OSSBackend._recreate_collection_if_dims_changed("qdrant", {"path": "/unused"}, 768)
client.delete_collection.assert_not_called()
client.close.assert_called_once()
def test_pgvector_dimension_change_never_drops_table(self):
cursor = Mock()
cursor.fetchone.return_value = (1536,)
connection = Mock()
connection.cursor.return_value = cursor
driver = SimpleNamespace(connect=Mock(return_value=connection), sql=Mock())
with patch.dict(sys.modules, {"psycopg2": driver}):
with self.assertRaisesRegex(ValueError, "1536.*768"):
backend.OSSBackend._recreate_collection_if_dims_changed("pgvector", {"user": "test"}, 768)
self.assertEqual(cursor.execute.call_count, 1)
self.assertTrue(cursor.execute.call_args.args[0].startswith("SELECT"))
connection.close.assert_called_once()
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,225 @@
"""Offline contracts for the native Hermes provider and legacy configuration."""
import contextvars
import importlib.util
import json
import sys
import threading
import types
from pathlib import Path
from unittest.mock import Mock
import pytest
ROOT = Path(__file__).resolve().parents[1]
@pytest.fixture
def plugin(monkeypatch, tmp_path):
def spawn(target, *, name):
context = contextvars.copy_context()
return threading.Thread(target=context.run, args=(target,), name=name)
modules = {
"agent": {},
"agent.memory_provider": {"MemoryProvider": object, "spawn_context_thread": spawn},
"agent.secret_scope": {"get_secret": lambda key, default="": default},
"tools": {},
"tools.registry": {"tool_error": lambda text: json.dumps({"error": text})},
"utils": {
"read_json_or_empty": lambda p: json.loads(p.read_text()) if p.exists() else {},
"atomic_json_write": lambda p, value, **kw: p.write_text(json.dumps(value)),
},
"hermes_constants": {"get_hermes_home": lambda: tmp_path},
}
for name, values in modules.items():
module = types.ModuleType(name)
module.__dict__.update(values)
monkeypatch.setitem(sys.modules, name, module)
name = "hermes_test_mem0"
spec = importlib.util.spec_from_file_location(name, ROOT / "__init__.py", submodule_search_locations=[str(ROOT)])
module = importlib.util.module_from_spec(spec)
monkeypatch.setitem(sys.modules, name, module)
spec.loader.exec_module(module)
monkeypatch.setattr(module.atexit, "register", lambda *args: None)
return module
def provider(plugin, monkeypatch, config=None, **identity):
backend = Mock()
backend.add.return_value = {"event_id": "event-1"}
backend.search.return_value = []
monkeypatch.setattr(plugin, "_load_config", lambda: config or {})
monkeypatch.setattr(plugin.Mem0MemoryProvider, "_create_backend", lambda self: backend)
instance = plugin.Mem0MemoryProvider()
instance.initialize("session-1", **identity)
return instance, backend
def test_long_turns_preserve_tail_and_redact_before_chunking(plugin, monkeypatch):
instance, backend = provider(plugin, monkeypatch, {"sync_max_chars": 450})
text = "Opening. " + "x" * 1100 + " api_key=secret-value lasting preference at the end."
instance.sync_turn(text, "Noted.", session_id="turn-session")
instance.shutdown()
messages = [m for call in backend.add.call_args_list for m in call.args[0]]
assert "".join(m["content"] for m in messages if m["role"] == "user") == plugin.redact(text)
assert all(len(m["content"]) <= 450 for m in messages)
assert "secret-value" not in repr(backend.add.call_args_list)
assert all(call.kwargs["run_id"] == "turn-session" for call in backend.add.call_args_list)
def test_busy_capture_queues_each_turn_with_its_profile_context(plugin, monkeypatch):
instance, backend = provider(plugin, monkeypatch)
started, release = threading.Event(), threading.Event()
profile = contextvars.ContextVar("profile", default="first")
seen = []
def add(messages, **kwargs):
if not seen:
started.set()
assert release.wait(3)
seen.append((messages[0]["content"], profile.get()))
return {}
backend.add.side_effect = add
instance.sync_turn("first turn", "reply")
assert started.wait(3)
token = profile.set("second")
try:
instance.sync_turn("second turn", "reply")
finally:
profile.reset(token)
release.set()
instance.shutdown()
assert seen == [("first turn", "first"), ("second turn", "second")]
backend.close.assert_called_once()
@pytest.mark.parametrize(
"configured,gateway,expected",
[
(None, "telegram-42", "telegram-42"),
("hermes-user", "telegram-42", "telegram-42"),
("existing-account", "telegram-42", "existing-account"),
(None, None, "hermes-user"),
],
)
def test_legacy_identity_and_tool_contract(plugin, monkeypatch, configured, gateway, expected):
instance, backend = provider(plugin, monkeypatch, {"user_id": configured}, user_id=gateway)
assert instance.name == "mem0"
assert {s["name"] for s in instance.get_tool_schemas()} == {"mem0_search", "mem0_add", "mem0_update", "mem0_delete"}
instance.handle_tool_call("mem0_search", {"query": "api_key=secret-value"})
assert backend.search.call_args.kwargs["filters"] == {"user_id": expected}
assert backend.search.call_args.args[0] == "api_key=[REDACTED]"
instance.handle_tool_call("mem0_add", {"content": "password=secret-value"})
assert backend.add.call_args.kwargs["infer"] is False
assert backend.add.call_args.args[0][0]["content"] == "password=[REDACTED]"
instance.handle_tool_call("mem0_update", {"memory_id": "old-id", "text": "password=secret-value"})
backend.update.assert_called_once_with("old-id", "password=[REDACTED]")
instance.shutdown()
def test_config_keeps_legacy_file_over_env_precedence(plugin, monkeypatch, tmp_path):
env = {"MEM0_API_KEY": "env-key", "MEM0_USER_ID": "env-user", "MEM0_HOST": "http://localhost:8888"}
monkeypatch.setattr(plugin, "get_secret", lambda key, default="": env.get(key, default))
(tmp_path / "mem0.json").write_text(json.dumps({"user_id": "existing-user", "rerank": True}))
cfg = plugin._load_config()
assert (cfg["api_key"], cfg["user_id"], cfg["host"], cfg["rerank"]) == (
"env-key",
"existing-user",
"http://localhost:8888",
True,
)
def test_prefetch_respects_rerank_and_redacts_recalled_context(plugin, monkeypatch):
instance, backend = provider(plugin, monkeypatch, {"rerank": True})
backend.search.return_value = [{"memory": "password=old-secret"}]
assert "old-secret" not in instance.prefetch("preference")
assert backend.search.call_args.kwargs["rerank"] is True
instance.shutdown()
def test_session_switch_updates_writes_without_narrowing_recall(plugin, monkeypatch):
instance, backend = provider(plugin, monkeypatch)
instance.on_session_switch("resumed-session")
instance.handle_tool_call("mem0_add", {"content": "fact"})
assert backend.add.call_args.kwargs["run_id"] == "resumed-session"
instance.handle_tool_call("mem0_search", {"query": "fact"})
assert backend.search.call_args.kwargs["filters"] == {"user_id": "hermes-user"}
instance.shutdown()
def test_invalid_tool_input_never_reaches_backend_or_trips_breaker(plugin, monkeypatch):
instance, backend = provider(plugin, monkeypatch)
for args in ({"query": []}, {"query": "fact", "top_k": "not-a-number"}, None):
assert "error" in json.loads(instance.handle_tool_call("mem0_search", args))
backend.search.assert_not_called()
assert instance._consecutive_failures == 0
instance.shutdown()
def test_setup_writes_private_env_and_preserves_existing_values(plugin, tmp_path):
import importlib
setup = importlib.import_module(f"{plugin.__name__}._setup")
path = tmp_path / ".env"
path.write_text("EXISTING=value\nMEM0_API_KEY=old\n")
setup._write_env(path, {"MEM0_API_KEY": "new"})
assert path.read_text() == "EXISTING=value\nMEM0_API_KEY=new\n"
assert path.stat().st_mode & 0o777 == 0o600
def test_redaction_marker_split_at_chunk_boundary_is_lossless(plugin, monkeypatch):
instance, backend = provider(plugin, monkeypatch, {"sync_max_chars": 450})
text = "x" * 438 + " password=secret-value tail"
instance.sync_turn(text, "")
instance.shutdown()
stored = "".join(m["content"] for call in backend.add.call_args_list for m in call.args[0])
assert stored == plugin.redact(text)
def test_sync_queue_stops_network_calls_when_breaker_opens(plugin, monkeypatch, caplog):
instance, backend = provider(plugin, monkeypatch)
started, release = threading.Event(), threading.Event()
def unavailable(messages, **kwargs):
started.set()
assert release.wait(3)
raise RuntimeError("server unavailable")
backend.add.side_effect = unavailable
instance.sync_turn("first turn", "reply")
assert started.wait(3)
try:
for number in range(9):
instance.sync_turn(f"queued turn {number}", "reply")
finally:
release.set()
instance.shutdown()
assert backend.add.call_count == plugin._BREAKER_THRESHOLD
assert any("not synced" in record.message.lower() for record in caplog.records)
def test_shutdown_is_bounded_and_defers_close_until_sync_finishes(plugin, monkeypatch, caplog):
instance, backend = provider(plugin, monkeypatch)
started, release = threading.Event(), threading.Event()
monkeypatch.setattr(plugin, "_SHUTDOWN_WAIT_SECS", 0.01)
def add(*args, **kwargs):
started.set()
assert release.wait(3)
return {}
backend.add.side_effect = add
instance.sync_turn("fact", "reply")
assert started.wait(3)
worker = instance._sync_thread
try:
instance.shutdown()
backend.close.assert_not_called()
assert "shutdown timed out" in caplog.text
finally:
release.set()
worker.join(timeout=3)
backend.close.assert_called_once()
@@ -1,11 +0,0 @@
"""Generated by integrations/agent-plugin-core/build/build.py. Do not edit."""
HARNESS_ID = "kimi"
SOURCE_TAG = "KIMI_PLUGIN"
# Platform-side vocabulary (mem0_event.source + X-Application). The whole
# plugin family is one source; which editor it runs in is the application.
# An empty application means the host is unknown, and memory_core omits
# the header entirely rather than sending a placeholder.
PLATFORM_SOURCE = "MEM0_PLUGIN"
PLATFORM_APPLICATION = "kimi"
+1 -17
View File
@@ -290,11 +290,6 @@ def run(
if args.plugin_data_dir:
os.environ[data_dir_env] = args.plugin_data_dir
# Snapshot BEFORE anything writes to the data dir: cache_plugin_api_key
# writes `api-key` and EvidenceStore creates `evidence.sqlite3`, so asking
# after them always saw content and every fresh install reported an upgrade.
data_dir_was_empty = telemetry.data_dir_was_empty()
cache_plugin_api_key()
if args.action == "session-start":
clear_stale_api_key_cache()
@@ -310,19 +305,8 @@ def run(
return 0
if args.action == "session-start":
# Claims the marker atomically and says which event to record, so a
# second session starting alongside this one cannot record it too.
first_event = telemetry.claim_install(was_empty=data_dir_was_empty)
if first_event == "install":
if telemetry.is_first_run():
telemetry.record("install")
elif first_event == "upgrade":
# First run after a build that never wrote the marker; the
# predecessor version was never recorded anywhere.
telemetry.record("upgrade", from_version="pre-0.3")
else:
previous = telemetry.claim_version_change()
if previous:
telemetry.record("upgrade", from_version=previous)
recovered = recover_pending_handoffs()
record_session_start(store, hook_input)
if recovered:
+12 -172
View File
@@ -11,7 +11,6 @@ from __future__ import annotations
import functools
import hashlib
import json
import math
import os
import re
import sqlite3
@@ -27,9 +26,17 @@ from pathlib import Path
from typing import Any, Iterable
import telemetry
from message_utils import MAX_EXTRACTION_INPUT_TOKENS as MAX_EXTRACTION_INPUT_TOKENS
from message_utils import SECRET_PATTERNS as SECRET_PATTERNS
from message_utils import _estimated_tokens as _estimated_tokens
from message_utils import _is_agent_assignment as _is_agent_assignment
from message_utils import _is_agent_response as _is_agent_response
from message_utils import _message_tokens as _message_tokens
from message_utils import extraction_message_batches as extraction_message_batches
from message_utils import redact as redact
DEFAULT_API_URL = "https://api.mem0.ai"
PLUGIN_VERSION = "0.3.2"
PLUGIN_VERSION = "0.3.1"
_harness_name: str = "generic"
_harness_env_prefix: str = "MEM0_PLUGIN"
@@ -66,7 +73,6 @@ CHECKPOINT_EXCHANGES = 5
CHECKPOINT_MESSAGES = 10
CHECKPOINT_SOURCE_CHARS = 40000
DEFAULT_MAX_CONTEXT_CHARS = 4000
MAX_EXTRACTION_INPUT_TOKENS = 24000
MAX_FLUSH_ATTEMPTS = 5
FORGET_PAGE_SIZE = 100
FORGET_MAX_PAGES = 50
@@ -139,48 +145,11 @@ BUILD_COMMAND_RE = re.compile(
re.IGNORECASE,
)
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(
r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"
),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r'|(?:access|refresh|session)[_-]?token|token|authorization|credential'
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def redact(value: Any) -> str:
text = (
value
if isinstance(value, str)
else json.dumps(value, ensure_ascii=False, default=str)
)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def bounded(value: Any, limit: int) -> str:
text = redact(value).strip()
if len(text) <= limit:
@@ -1706,128 +1675,6 @@ def build_extraction_messages(structured: dict[str, Any]) -> list[dict[str, str]
return messages
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if (
_is_agent_assignment(message)
and index + 1 < len(exchange)
and _is_agent_response(exchange[index + 1])
):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
# Platform surface attribution. Read from the generated per-host module so a new
# entrypoint is correct without remembering to configure anything.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
except ImportError:
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
def platform_headers(key: str) -> dict[str, str]:
"""Auth plus the three surface-identity headers.
X-Mem0-Source and X-Application are set-once by contract: this is the
outermost layer, so it sets them, and nothing below may overwrite them.
X-Mem0-Client is append-only — anything downstream adds itself to the tail.
"""
headers = {
"Authorization": f"Token {key}",
"Content-Type": "application/json",
"X-Mem0-Source": _PLATFORM_SOURCE,
"X-Mem0-Client": f"mem0-plugin/{PLUGIN_VERSION}",
}
if _PLATFORM_APPLICATION:
headers["X-Application"] = _PLATFORM_APPLICATION
return headers
def _request_json(
url: str, key: str, payload: dict[str, Any], timeout: float
) -> tuple[dict[str, Any] | list[Any], int, int]:
@@ -1835,7 +1682,7 @@ def _request_json(
request = urllib.request.Request(
url,
data=raw,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -1862,7 +1709,7 @@ def _get_json(
) -> tuple[dict[str, Any] | list[Any], int]:
request = urllib.request.Request(
url,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="GET",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -2008,13 +1855,6 @@ def flush_session(
"user_id": write_user,
"app_id": repo.app_id,
"run_id": session_id,
# Top level, not metadata: the backend reads `source` from the body or
# the query string, never from metadata, which is where this used to
# sit. The X-Mem0-Source header is also read, but only from the
# platform release that ships alongside this change, so the body value
# is what makes attribution work on both. The harness tag stays in
# metadata as hook provenance.
"source": _PLATFORM_SOURCE,
"metadata": {**metadata, "author": write_user, "dirs": directory_chain(repo)},
"agent_custom_instructions": PROJECT_MEMORY_INSTRUCTIONS,
"custom_instructions": PERSONAL_MEMORY_INSTRUCTIONS,
@@ -2558,7 +2398,7 @@ def _collect_memory_ids(
def _delete_memory(api_url: str, key: str, memory_id: str) -> bool:
request = urllib.request.Request(
f"{api_url}/v1/memories/{urllib.parse.quote(memory_id)}/",
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="DELETE",
)
try:
@@ -0,0 +1,127 @@
"""Shared, host-independent redaction and lossless extraction batching."""
from __future__ import annotations
import json
import math
import re
from typing import Any
MAX_EXTRACTION_INPUT_TOKENS = 24000
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r"|(?:access|refresh|session)[_-]?token|token|authorization|credential"
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def redact(value: Any) -> str:
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, default=str)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if _is_agent_assignment(message) and index + 1 < len(exchange) and _is_agent_response(exchange[index + 1]):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
+39 -655
View File
@@ -1,9 +1,5 @@
#!/usr/bin/env python3
"""Usage telemetry for Mem0 agent plugins.
Events are linked to your Mem0 account email when an API key is configured, and
to a random per-machine id otherwise. Not anonymous — the Python SDK and CLI
attribute the same way.
"""Anonymous usage telemetry for Mem0 agent plugins.
Hooks run on a 3-6 second budget and fire on every tool call, so recording never
touches the network: `record` appends one JSON line to a local spool and returns.
@@ -13,8 +9,7 @@ started once per session and again from the flush worker that is already detache
Pure stdlib, matching the rest of the plugin. Opt out with MEM0_TELEMETRY=false.
Never sends prompts, memory text, queries, file paths, repository names, or API
keys: only event names, durations, counts, coarse outcomes, and repo/session
identifiers hashed with a random per-install salt.
keys: only event names, durations, counts, coarse outcomes, and salted hashes.
"""
from __future__ import annotations
@@ -34,24 +29,8 @@ from typing import Any
import memory_core
# Seeded from the per-host module the build generates into core/. Two processes
# in this pipeline never call init() — mcp_server.py, and the detached
# `python3 telemetry.py` sender that spawn_flush() starts — so a module default
# was what every one of their events got labelled with.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import HARNESS_ID as _DEFAULT_HARNESS
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
from _harness_id import SOURCE_TAG as _DEFAULT_SOURCE_TAG
except ImportError:
_DEFAULT_HARNESS = "generic"
_DEFAULT_SOURCE_TAG = "MEM0_PLUGIN"
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
_salt_cache: str = ""
_harness: str = _DEFAULT_HARNESS
_source_tag: str = _DEFAULT_SOURCE_TAG
_harness: str = "generic"
_source_tag: str = "MEM0_PLUGIN"
_PRIVATE_KEYS = {
"apikey",
"authorization",
@@ -77,19 +56,10 @@ _PRIVATE_KEYS = {
}
def init(harness: str = "", source_tag: str = "") -> None:
"""Override the generated identity. Optional — core/_harness_id.py is the default.
The fallback shape matches memory_core.configure_harness's (``<HOST>_PLUGIN``).
It used to be ``MEM0_<HOST>_PLUGIN`` here and ``<host>_plugin`` there, which
meant one plugin could emit three different source values depending on which
process happened to send the batch.
"""
def init(harness: str = "generic", source_tag: str = "") -> None:
global _harness, _source_tag
_harness = harness or _DEFAULT_HARNESS
_source_tag = source_tag or (
f"{_harness.upper().replace('-', '_')}_PLUGIN" if harness else _DEFAULT_SOURCE_TAG
)
_harness = harness
_source_tag = source_tag or f"MEM0_{harness.upper().replace('-', '_')}_PLUGIN"
POSTHOG_API_KEY = "phc_hgJkUVJFYtmaJqrvf6CYN67TIQ8yhXAkWzUn9AMU4yX"
POSTHOG_CAPTURE_URL = "https://us.i.posthog.com/i/v0/e/"
@@ -100,16 +70,6 @@ BATCH_SIZE = 100
SEND_TIMEOUT = 5
CLAIM_STALE_SECONDS = 120
CLAIM_EXPIRY_SECONDS = 7 * 24 * 60 * 60
# A batch is only discarded once it has genuinely been retried this many times.
MAX_CLAIM_ATTEMPTS = 3
# Parked claims drained per run, after the live spool. Bounded so a long backlog
# cannot turn one flush into an unbounded send loop.
MAX_PARKED_PER_RUN = 3
# Added to the wait before a released claim becomes reclaimable, per attempt
# already spent. Releasing straight to "reclaimable now" let two senders burn the
# whole budget within seconds of one another on a single momentary failure, and
# discard a batch a retry a minute later would have delivered.
RETRY_COOLDOWN_SECONDS = 60
def is_enabled() -> bool:
@@ -123,126 +83,9 @@ def is_enabled() -> bool:
def _digest(value: str, length: int = 16) -> str:
"""Unsalted digest. Only for values that are already secrets (API keys)."""
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:length]
def _salt_path() -> Path:
return memory_core.data_dir() / "telemetry-salt"
def _install_salt() -> str:
"""Random per-install salt, created once and memoized for the process.
Deliberately its own file, claimed with O_CREAT|O_EXCL, rather than a key in
the identity file. Three reasons, all of which produced wrong data when this
lived in the identity dict:
- Hooks are short-lived separate processes firing on every tool call, and
people run more than one agent window. A read-modify-write would let each
process mint its own salt, so one repository would hash several ways in the
window before a writer won.
- resolve_distinct_id holds a copy of the identity dict across a network call
to /v1/ping/, so whichever write landed second erased the other's key —
losing either the salt (repo_hash changes mid-stream) or the email (a
second $identify, splitting the person).
- Touching the identity file from record() would create it, and is_first_run
keys off that file, so recording an event would silently suppress the
install event.
Published atomically, and there is deliberately no derived fallback. Creating
the file with O_CREAT|O_EXCL and then writing into it leaves a window where
the file exists and is empty, and a concurrent hook that reads it in that
window gets nothing. Falling back to a digest of the path would hand that
process a salt an attacker can compute, memoized for its whole run, which is
the privacy control this function exists to provide silently turning itself
off under load. The salt is written to a private temp file first and linked
into place, so the name either does not exist or already has the full value.
Returns "" when it genuinely cannot persist. Callers omit the hash entirely
rather than emit an unsalted one.
"""
global _salt_cache
if _salt_cache:
return _salt_cache
path = _salt_path()
# Read before writing. Hooks are separate processes firing on every tool
# call, so all but the first find the salt already published; going straight
# to create-fsync-link-unlink meant every one of them paid an fsync to
# discover that, on a path whose whole promise is appending a line and
# returning.
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
if _salt_cache:
return _salt_cache
except OSError:
pass
temporary = path.with_name(f"{path.name}.{os.getpid()}.tmp")
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(handle, "w", encoding="utf-8") as stream:
stream.write(uuid.uuid4().hex)
stream.flush()
os.fsync(stream.fileno())
try:
# Atomic claim: fails if another process already published one.
# os.link rather than replace, which would clobber theirs.
os.link(temporary, path)
except FileExistsError:
pass
except OSError:
# No hardlinks here (some network mounts, some container volumes).
# Claim the name directly instead. That reopens the empty-file
# window, but the window is now benign: a reader that lands in it
# gets "" and omits the hash for that process rather than caching a
# guessable one. Losing the hashes on every run of an entire
# filesystem is the worse failure.
try:
fallback = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(fallback, "w", encoding="utf-8") as stream:
stream.write(temporary.read_text(encoding="utf-8"))
except OSError:
pass
except OSError:
pass
finally:
try:
temporary.unlink()
except OSError:
pass
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
except OSError:
_salt_cache = ""
return _salt_cache
def _scoped_digest(value: str, length: int = 16) -> str:
"""Salted digest for values drawn from a guessable space.
repo.identity is a git remote URL, or ``local:<absolute path>`` when there is
no remote — which normally contains the account username. Sixteen unsalted
hex characters over that input space is enumerable, so this is not a
privacy control without the salt. Salting per install keeps every
within-account join the analytics actually use and gives up only
cross-machine joins on the same repository, which nothing computes.
Returns "" when there is no salt, so record() omits the property. An
unsalted digest over this input space is close to plaintext, and emitting one
under a name that implies it is hashed is worse than sending nothing.
"""
if not value:
return ""
salt = _install_salt()
if not salt:
return ""
return hashlib.sha256(f"{salt}:{value}".encode("utf-8")).hexdigest()[:length]
def _safe_value(value: Any) -> Any:
if isinstance(value, str):
return memory_core.redact(value)
@@ -302,176 +145,9 @@ def anonymous_id(identity: dict[str, str] | None = None) -> str:
return created
def _rotate_anonymous_id(identity: dict[str, str]) -> str:
"""Mint a fresh anonymous id because the account context is gone.
The previous id may already have been merged into a person profile by an
$identify, and that merge is permanent. Reusing it after a logout or a key
change attributes everything that follows to the account that just went
away, which is the same misattribution the key fingerprint exists to stop,
only arriving through the anonymous path instead.
`aliased` is cleared with it: the new id has never been merged, so it is
eligible to be aliased into whatever account comes next.
"""
created = f"code-anon-{uuid.uuid4().hex}"
identity["anonymous_id"] = created
identity.pop("aliased", None)
_write_identity(identity)
return created
def _install_state_path() -> Path:
return memory_core.data_dir() / "install-state.json"
def is_first_run() -> bool:
"""Whether install has never been recorded on this machine.
Deliberately NOT the identity file. That file is only written by a
successful flush, so an offline or firewalled user recorded code.install on
every single session, forever — and every 0.2.x user recorded one on their
first 0.3.x session because 0.2.x never wrote it at all.
"""
return not _install_state_path().exists()
def data_dir_was_empty() -> bool:
"""Whether the data directory is untouched. Call BEFORE anything writes to it.
hook_runner reaches claim_install() only after cache_plugin_api_key() has
written `api-key` and EvidenceStore() has created `evidence.sqlite3`, so
asking at claim time always saw content and every fresh install reported an
upgrade. The caller snapshots this at the top of the run instead.
"""
return not _data_dir_has_content()
def claim_install(was_empty: bool | None = None) -> str | None:
"""Claim the one install/upgrade record for this machine, atomically.
Returns the event to record ("install" or "upgrade"), or None if another
session already claimed it. O_CREAT|O_EXCL so two sessions starting together
cannot both win.
`was_empty` must come from data_dir_was_empty() called before this process
wrote anything. Omitting it falls back to checking now, which is only
correct for a caller that has touched nothing.
"""
if not is_enabled():
# Never consume the one-shot claim while the user is opted out, or they
# would silently lose their install event if they later opt in.
return None
path = _install_state_path()
upgrading = not (data_dir_was_empty() if was_empty is None else was_empty)
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
except FileExistsError:
return None
except OSError:
return None
try:
with os.fdopen(handle, "w", encoding="utf-8") as stream:
json.dump(
{
"plugin_version": memory_core.PLUGIN_VERSION,
"installed_at": memory_core.utc_now(),
"upgraded": upgrading,
},
stream,
)
# Durable before this returns. The O_EXCL open is what makes the
# claim exclusive, so it cannot be replaced by a temp-and-rename
# without losing that, which leaves the content as the thing to make
# safe. A kill between the open and this fsync used to leave a marker
# that exists but parses to nothing: is_first_run reads it as claimed
# and claim_version_change cannot read a version out of it.
stream.flush()
os.fsync(stream.fileno())
except OSError:
pass
return "upgrade" if upgrading else "install"
def _data_dir_has_content() -> bool:
"""Whether anything predates this session in the plugin data directory."""
try:
for entry in memory_core.data_dir().iterdir():
if entry.name != "install-state.json":
return True
except OSError:
pass
return False
def _repair_install_state(path: Path) -> None:
"""Rewrite an unparseable marker so version tracking can resume."""
try:
temporary = path.with_suffix(f".{os.getpid()}.tmp")
temporary.write_text(
json.dumps({"plugin_version": memory_core.PLUGIN_VERSION, "repaired_at": memory_core.utc_now()}),
encoding="utf-8",
)
temporary.replace(path)
except OSError:
pass
def claim_version_change() -> str | None:
"""Return the previously recorded version if it differs, updating the marker.
Only meaningful once the marker exists — the first transition into 0.3.x has
no recorded predecessor and reports "pre-0.3" instead. Claiming by rewriting
the marker means the next session sees no change and records nothing.
"""
path = _install_state_path()
try:
state = json.loads(path.read_text(encoding="utf-8"))
except OSError:
return None
except json.JSONDecodeError:
# A crash between O_EXCL and the write leaves an empty marker. Left
# alone it disables every future upgrade event on this machine, because
# claim_install sees the file and this function cannot parse it.
state = None
if not isinstance(state, dict):
_repair_install_state(path)
return None
previous = str(state.get("plugin_version") or "")
if not previous or previous == memory_core.PLUGIN_VERSION:
return None
# Claim the transition with an exclusive sentinel before rewriting the
# marker. A plain read-modify-write let every concurrently starting session
# observe the old version and each record its own upgrade — and the first
# session after a version bump is exactly when several agent windows restart
# together.
sentinel = path.with_name(f"upgraded-{memory_core.PLUGIN_VERSION}")
try:
os.close(os.open(sentinel, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600))
except FileExistsError:
return None
except OSError:
return None
state["plugin_version"] = memory_core.PLUGIN_VERSION
state["upgraded_at"] = memory_core.utc_now()
temporary = path.with_suffix(f".{os.getpid()}.tmp")
try:
temporary.write_text(json.dumps(state), encoding="utf-8")
temporary.replace(path)
except OSError:
# Release the claim. The marker still records the old version, so
# without this the sentinel makes claim_version_change return early on
# every later run and this version's upgrade is never recorded again.
for leftover in (sentinel, temporary):
try:
leftover.unlink()
except OSError:
pass
return None
return previous
"""Whether this machine has never recorded a plugin event before."""
return not _identity_path().exists()
def record(
@@ -492,32 +168,19 @@ def record(
except OSError:
pass
properties = _safe_value(properties)
# Stamped in the RECORDING process, beside harness. `source` used to be
# read in the sending process from a module global, so whichever process
# drained the spool named every event in it. flush() spreads per-event
# properties last, so this now wins over any sender's default.
properties.update(
harness=_harness,
source=_source_tag,
plugin_version=memory_core.PLUGIN_VERSION,
os=sys.platform,
python_version=platform.python_version(),
)
# Assigned only when the digest is real. _scoped_digest returns "" when
# the salt could not be persisted, and an empty property is worse than an
# absent one: it survives the None filter below and reads as a value.
if repo is not None:
repo_hash = _scoped_digest(getattr(repo, "identity", ""))
if repo_hash:
properties["repo_hash"] = repo_hash
properties["repo_hash"] = _digest(getattr(repo, "identity", ""))
if session_id:
session_hash = _scoped_digest(session_id)
if session_hash:
properties["session_hash"] = session_hash
properties["session_hash"] = _digest(session_id)
line = json.dumps(
{
"event": f"{EVENT_PREFIX}.{event}",
"uuid": str(uuid.uuid4()),
"timestamp": memory_core.utc_now(),
"properties": {
key: value for key, value in properties.items() if value is not None
@@ -576,201 +239,38 @@ def spawn_flush() -> bool:
return False
def _claim_name(attempt: int = 0) -> str:
"""Claim filename. The attempt count rides in the name so the 7-day expiry
only ever discards a batch that was actually retried and failed."""
return f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}-a{attempt}.sending"
def _claim_attempt(claim: Path) -> int:
"""Attempts recorded in a claim filename; 0 for the pre-attempt-count shape.
Anchored on field position, not on a leading "a": the legacy shape is
``telemetry-<pid>-<hex>.sending`` and a hex id such as ``a1234567`` would
otherwise parse as attempt 1234567 and be discarded unsent on the first
flush after an upgrade.
"""
stem = claim.name[: -len(".sending")] if claim.name.endswith(".sending") else claim.name
parts = stem.split("-")
if len(parts) != 4:
return 0
tail = parts[3]
if tail.startswith("a") and tail[1:].isdigit():
return int(tail[1:])
return 0
def _touch(path: Path) -> None:
"""Refresh mtime so a claim's age measures time since it was claimed.
``Path.replace`` is ``os.rename``, which preserves mtime — so a claim created
after a quiet minute inherited the spool's last-write time and looked
abandoned the instant it was made. A second sender would then take it over
while the first was still posting, and both would deliver the batch.
"""
try:
os.utime(path, None)
except OSError:
pass
def _claim_spool() -> Path | None:
"""Rename the spool aside so exactly one sender owns each batch."""
directory = memory_core.data_dir()
claim = directory / _claim_name()
claim = directory / f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}.sending"
spool = _spool_path()
try:
spool.replace(claim)
_touch(claim)
return claim
except OSError:
pass
return _claim_parked(directory)
def _sweep_debris(directory: Path) -> None:
"""Remove files nothing else will ever pick up again.
*.partial is a temp file orphaned by a crash between write and rename.
*.corrupt is a batch quarantined for undecodable content. No glob in this
module matches either, so without this they accumulate on disk for the life
of the install.
Quarantined batches are kept far longer than debris: they are the only
evidence left of events that could not be delivered, and someone diagnosing
a report of missing telemetry has to be able to find one.
"""
now = time.time()
for debris in directory.glob("telemetry-*.partial"):
try:
if now - debris.stat().st_mtime > CLAIM_STALE_SECONDS:
debris.unlink()
except OSError:
continue
for quarantined in directory.glob("telemetry-*.corrupt"):
try:
if now - quarantined.stat().st_mtime > CLAIM_EXPIRY_SECONDS:
quarantined.unlink()
except OSError:
continue
# The same reasoning covers *.tmp. _write_identity and _install_salt both
# create one and unlink it in a finally, which a SIGKILL skips, and no glob
# in this module matches the leftovers either.
for temporary in directory.glob("telemetry-*.tmp"):
try:
if now - temporary.stat().st_mtime > CLAIM_STALE_SECONDS:
temporary.unlink()
except OSError:
continue
def _claim_parked(directory: Path) -> Path | None:
"""Take the oldest abandoned claim, if any lease has actually expired.
Kept separate from the live spool so flush() can drain both in one run.
Previously parked batches were only reachable when no spool existed at all,
and because sessions keep recording there usually was one — so a batch
parked by a failed send waited until the 7-day expiry deleted it unsent,
even though its own presence is what started the sender.
"""
now = time.time()
for orphan in sorted(directory.glob("telemetry-*.sending"), key=_safe_mtime):
for orphan in sorted(directory.glob("telemetry-*.sending")):
try:
age = now - orphan.stat().st_mtime
except OSError:
continue
if age < CLAIM_STALE_SECONDS:
# Someone else holds a live lease on it. This check has to come
# first. Claiming a file bumps its attempt count and refreshes its
# mtime, so a sender that has just taken the final attempt looks
# exhausted to everyone else while it is actively draining. Judging
# exhaustion before liveness let a second sender unlink a batch out
# from under its owner, losing every event in it.
continue
# Attempts, not age. Every re-claim touches the mtime and every release
# backdates it by a fixed amount, so age is pinned near the stale
# threshold and never reaches the expiry. Age stays only as a backstop
# for files that never carried an attempt marker.
if _claim_attempt(orphan) >= MAX_CLAIM_ATTEMPTS or age > CLAIM_EXPIRY_SECONDS:
if age > CLAIM_EXPIRY_SECONDS:
try:
orphan.unlink()
except OSError:
pass
continue
claim = orphan.parent / _claim_name(_claim_attempt(orphan) + 1)
if age < CLAIM_STALE_SECONDS:
continue
try:
orphan.replace(claim)
_touch(claim)
return claim
except OSError:
continue
return None
def _safe_mtime(path: Path) -> float:
try:
return path.stat().st_mtime
except OSError:
return 0.0
def _rewrite_claim(claim: Path, remaining: list[dict[str, Any]]) -> bool:
"""Persist the unsent remainder, atomically, and refresh the lease.
Called after every successful batch. Two jobs: a retry resumes where the
send stopped instead of re-posting from the top, and the rewrite doubles as
the lease heartbeat, so a slow sender does not have its claim stolen
mid-flight. Interval is one batch, well inside CLAIM_STALE_SECONDS.
"""
if not remaining:
try:
claim.unlink()
except OSError:
pass
return True
temporary = claim.with_suffix(f".{os.getpid()}.partial")
try:
payload = "".join(json.dumps(event, separators=(",", ":"), default=str) + "\n" for event in remaining)
# fsync before the rename: without it the rename can land while the
# bytes have not, and the claim comes back empty or truncated after a
# crash. _drain then reads zero events and unlinks it.
with open(temporary, "w", encoding="utf-8") as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
temporary.replace(claim)
_touch(claim)
return True
except OSError:
try:
temporary.unlink()
except OSError:
pass
return False
def _release_claim(claim: Path, remaining: list[dict[str, Any]]) -> None:
"""Persist the remainder and drop the lease, because this sender has given up.
Distinct from the per-batch heartbeat: heartbeating on the way out would
make an abandoned batch look actively owned for a further
CLAIM_STALE_SECONDS, delaying the retry for no reason. Ageing it past the
threshold lets the next flush pick it up immediately, while the attempt
count in the filename still bounds how many times that can happen.
"""
if not _rewrite_claim(claim, remaining):
return
try:
# Backdate past the stale threshold so the next flush can pick it up,
# minus a cooldown that grows with the attempts already spent. Clamped so
# the mtime never lands in the future, which would read as a live lease.
cooldown = min(_claim_attempt(claim) * RETRY_COOLDOWN_SECONDS, CLAIM_STALE_SECONDS)
released = time.time() - CLAIM_STALE_SECONDS - 1 + cooldown
os.utime(claim, (released, released))
except OSError:
pass
def _resolve_email(key: str) -> str:
"""Trade the API key for the account email so events join other Mem0 surfaces."""
url = os.environ.get("MEM0_API_URL", memory_core.DEFAULT_API_URL).rstrip("/") + "/v1/ping/"
@@ -800,130 +300,34 @@ def _post(payload: dict[str, Any], url: str) -> bool:
def resolve_distinct_id() -> tuple[str, str]:
"""Return the PostHog distinct id and the anonymous id it replaced, if any.
The second value becomes a PostHog $identify alias. It is ONLY ever an
anonymous id: aliasing one account email to another merges two real person
profiles and cannot be undone, so a key that now belongs to a different
account re-resolves with no alias.
"""
"""Return the PostHog distinct id and the anonymous id it replaced, if any."""
identity = _read_identity()
key = memory_core.api_key()
fingerprint = _digest(key) if key else ""
email = identity.get("email", "")
if email and fingerprint:
recorded = identity.get("key_fingerprint", "")
if recorded == fingerprint:
return email, ""
if not recorded:
# Rows written before fingerprints existed. Verify rather than
# adopt: a key changed before the upgrade would otherwise bind the
# new key to the previous account's email, permanently, and the
# fingerprint would then agree with itself forever after.
verified = _resolve_email(key)
if not verified:
# Offline, firewalled, or the API is down. Keep the previous
# behaviour and retry on the next flush rather than dropping a
# real account attribution. Safe because the same network that
# failed /v1/ping/ is about to fail the PostHog POST, so nothing
# is delivered under the unverified identity in the meantime.
return email, ""
identity["email"] = verified
identity["key_fingerprint"] = fingerprint
_write_identity(identity)
return verified, ""
if email:
return email, ""
key = memory_core.api_key()
if not key:
# No key to verify the account with; do not keep attributing to it.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
return anonymous_id(identity), ""
resolved = _resolve_email(key)
if not resolved:
# The key changed and will not resolve (revoked, offline, API down).
# Reaching here with an email means the recorded fingerprint disagreed,
# so the key really did change. Drop the account and rotate: the stored
# anonymous id may already be merged into that account's person, and
# reusing it would keep the events on the profile we are trying to
# leave.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
email = _resolve_email(key)
if not email:
return anonymous_id(identity), ""
# Alias only when going anonymous -> email for the first time. Once an anon
# id has been merged into an account it must never be offered again: an
# alias naming an already-identified id is what could link two real people.
previous = "" if (email or identity.get("aliased")) else identity.get("anonymous_id", "")
if previous:
identity["aliased"] = True
identity["email"] = resolved
identity["key_fingerprint"] = fingerprint
previous = identity.get("anonymous_id", "")
identity["email"] = email
_write_identity(identity)
return resolved, previous
return email, previous
def flush() -> int:
"""Drain the live spool, then any parked claims, and return events sent."""
"""Drain claimed spools to PostHog and return the number of events sent."""
if not is_enabled():
return 0
sent, delivered = _drain(_claim_spool())
if not delivered:
# The network is failing. Retrying other batches now would only burn
# their attempt budget against the same broken connection.
return sent
# Parked batches used to starve behind the live spool indefinitely. Bounded
# per run so a long backlog cannot turn one flush into an unbounded loop.
directory = memory_core.data_dir()
_sweep_debris(directory)
for _ in range(MAX_PARKED_PER_RUN):
parked = _claim_parked(directory)
if parked is None:
break
count, delivered = _drain(parked)
sent += count
if not delivered:
break
return sent
def _drain(claim: Path | None) -> tuple[int, bool]:
"""Post one claimed batch file, recording progress after every batch.
Returns (events sent, whether everything was delivered).
"""
claim = _claim_spool()
if claim is None:
return 0, True
return 0
try:
lines = claim.read_text(encoding="utf-8").splitlines()
except ValueError:
# UnicodeDecodeError from a torn write: the content is unrecoverable, so
# quarantine rather than retry. flush() runs from a bare `finally:` in
# flush_worker, so raising here also skips the handoff cleanup, and an
# undecodable file would otherwise be re-read on every flush forever.
# Reported as delivered because there is nothing left to deliver and the
# rest of the run should continue.
try:
claim.replace(claim.with_suffix(".corrupt"))
except OSError:
try:
claim.unlink()
except OSError:
pass
return 0, True
except OSError:
# Could not read it, which is not the same as having nothing to send.
# The file is left exactly where it is: a vanished or briefly unreadable
# claim is retryable, and quarantining it here would discard events over
# a transient filesystem error. Reported as undelivered so the run stops
# instead of counting a batch nothing was posted from as delivered.
return 0, False
return 0
events = []
for line in lines:
try:
@@ -933,18 +337,11 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
if isinstance(value, dict) and value.get("event"):
events.append(value)
if not events:
# Only delete when the file really is empty. A non-empty file that
# parses to nothing is a torn write, and its contents are the unsent
# remainder — deleting it is the data loss this PR exists to prevent.
try:
empty = claim.stat().st_size == 0
except OSError:
empty = True
try:
claim.replace(claim.with_suffix(".corrupt")) if not empty else claim.unlink()
claim.unlink()
except OSError:
pass
return 0, True
return 0
distinct_id, aliased_anonymous_id = resolve_distinct_id()
if aliased_anonymous_id:
@@ -963,17 +360,12 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
sent = 0
for start in range(0, len(events), BATCH_SIZE):
chunk = events[start : start + BATCH_SIZE]
batch = [
{
"event": event["event"],
"distinct_id": distinct_id,
# Carried through from record() so a resend can be collapsed.
"uuid": event.get("uuid"),
"timestamp": event.get("timestamp"),
"properties": {
# Fallback only: events recorded by a build before source
# moved into record() have none of their own.
"source": _source_tag,
"language": "python",
"$process_person_profile": False,
@@ -981,24 +373,16 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
**(event.get("properties") or {}),
},
}
for event in chunk
for event in events[start : start + BATCH_SIZE]
]
if not _post({"api_key": POSTHOG_API_KEY, "batch": batch}, POSTHOG_BATCH_URL):
# Keep only what has not been delivered, and release the lease.
# Previously the whole file was kept and the retry re-posted every
# batch, including the ones that had already arrived.
_release_claim(claim, events[start:])
return sent, False
sent += len(chunk)
# Record progress and refresh the lease after each successful batch, so
# a crash repeats at most one batch instead of the entire file. If the
# rewrite fails the claim still holds delivered events, so stop rather
# than carry on as though progress were recorded — continuing is how the
# duplicate delivery this PR fixes would come back.
if not _rewrite_claim(claim, events[start + len(chunk) :]):
_release_claim(claim, events[start + len(chunk) :])
return sent, False
return sent, True
return sent
sent += len(batch)
try:
claim.unlink()
except OSError:
pass
return sent
def main() -> int:
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mem0",
"version": "0.3.2",
"version": "0.3.1",
"description": "Cross-session memory and token savings for coding agents.",
"keywords": ["memory", "coding-agents", "continual-learning", "token-efficiency"],
"author": { "name": "Mem0", "email": "support@mem0.ai" },
+1 -1
View File
@@ -1,6 +1,6 @@
{
"id": "mem0",
"version": "0.3.2",
"version": "0.3.1",
"homepage": "https://docs.mem0.ai/integrations/kimi",
"native": {
"pluginRoot": "${KIMI_PLUGIN_ROOT}",
@@ -7,8 +7,8 @@ disable-model-invocation: true
# Pause memory capture
To pause (hooks stop capturing and sending session content; a minimal
telemetry ping still fires at session start, under your Mem0 account email,
unless `MEM0_TELEMETRY=false`):
anonymous telemetry ping still fires at session start unless
`MEM0_TELEMETRY=false`):
```bash
python3 "${KIMI_PLUGIN_ROOT}/core/memory_cli.py" --harness "kimi" pause
@@ -1,11 +0,0 @@
"""Generated by integrations/agent-plugin-core/build/build.py. Do not edit."""
HARNESS_ID = "coding-agent"
SOURCE_TAG = "CODING_AGENT_PLUGIN"
# Platform-side vocabulary (mem0_event.source + X-Application). The whole
# plugin family is one source; which editor it runs in is the application.
# An empty application means the host is unknown, and memory_core omits
# the header entirely rather than sending a placeholder.
PLATFORM_SOURCE = "MEM0_PLUGIN"
PLATFORM_APPLICATION = ""
@@ -11,7 +11,6 @@ from __future__ import annotations
import functools
import hashlib
import json
import math
import os
import re
import sqlite3
@@ -27,9 +26,17 @@ from pathlib import Path
from typing import Any, Iterable
import telemetry
from message_utils import MAX_EXTRACTION_INPUT_TOKENS as MAX_EXTRACTION_INPUT_TOKENS
from message_utils import SECRET_PATTERNS as SECRET_PATTERNS
from message_utils import _estimated_tokens as _estimated_tokens
from message_utils import _is_agent_assignment as _is_agent_assignment
from message_utils import _is_agent_response as _is_agent_response
from message_utils import _message_tokens as _message_tokens
from message_utils import extraction_message_batches as extraction_message_batches
from message_utils import redact as redact
DEFAULT_API_URL = "https://api.mem0.ai"
PLUGIN_VERSION = "0.3.2"
PLUGIN_VERSION = "0.3.1"
_harness_name: str = "generic"
_harness_env_prefix: str = "MEM0_PLUGIN"
@@ -66,7 +73,6 @@ CHECKPOINT_EXCHANGES = 5
CHECKPOINT_MESSAGES = 10
CHECKPOINT_SOURCE_CHARS = 40000
DEFAULT_MAX_CONTEXT_CHARS = 4000
MAX_EXTRACTION_INPUT_TOKENS = 24000
MAX_FLUSH_ATTEMPTS = 5
FORGET_PAGE_SIZE = 100
FORGET_MAX_PAGES = 50
@@ -139,48 +145,11 @@ BUILD_COMMAND_RE = re.compile(
re.IGNORECASE,
)
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(
r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"
),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r'|(?:access|refresh|session)[_-]?token|token|authorization|credential'
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def redact(value: Any) -> str:
text = (
value
if isinstance(value, str)
else json.dumps(value, ensure_ascii=False, default=str)
)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def bounded(value: Any, limit: int) -> str:
text = redact(value).strip()
if len(text) <= limit:
@@ -1706,128 +1675,6 @@ def build_extraction_messages(structured: dict[str, Any]) -> list[dict[str, str]
return messages
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get(
"content", ""
).startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if (
_is_agent_assignment(message)
and index + 1 < len(exchange)
and _is_agent_response(exchange[index + 1])
):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
# Platform surface attribution. Read from the generated per-host module so a new
# entrypoint is correct without remembering to configure anything.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
except ImportError:
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
def platform_headers(key: str) -> dict[str, str]:
"""Auth plus the three surface-identity headers.
X-Mem0-Source and X-Application are set-once by contract: this is the
outermost layer, so it sets them, and nothing below may overwrite them.
X-Mem0-Client is append-only — anything downstream adds itself to the tail.
"""
headers = {
"Authorization": f"Token {key}",
"Content-Type": "application/json",
"X-Mem0-Source": _PLATFORM_SOURCE,
"X-Mem0-Client": f"mem0-plugin/{PLUGIN_VERSION}",
}
if _PLATFORM_APPLICATION:
headers["X-Application"] = _PLATFORM_APPLICATION
return headers
def _request_json(
url: str, key: str, payload: dict[str, Any], timeout: float
) -> tuple[dict[str, Any] | list[Any], int, int]:
@@ -1835,7 +1682,7 @@ def _request_json(
request = urllib.request.Request(
url,
data=raw,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -1862,7 +1709,7 @@ def _get_json(
) -> tuple[dict[str, Any] | list[Any], int]:
request = urllib.request.Request(
url,
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="GET",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
@@ -2008,13 +1855,6 @@ def flush_session(
"user_id": write_user,
"app_id": repo.app_id,
"run_id": session_id,
# Top level, not metadata: the backend reads `source` from the body or
# the query string, never from metadata, which is where this used to
# sit. The X-Mem0-Source header is also read, but only from the
# platform release that ships alongside this change, so the body value
# is what makes attribution work on both. The harness tag stays in
# metadata as hook provenance.
"source": _PLATFORM_SOURCE,
"metadata": {**metadata, "author": write_user, "dirs": directory_chain(repo)},
"agent_custom_instructions": PROJECT_MEMORY_INSTRUCTIONS,
"custom_instructions": PERSONAL_MEMORY_INSTRUCTIONS,
@@ -2558,7 +2398,7 @@ def _collect_memory_ids(
def _delete_memory(api_url: str, key: str, memory_id: str) -> bool:
request = urllib.request.Request(
f"{api_url}/v1/memories/{urllib.parse.quote(memory_id)}/",
headers=platform_headers(key),
headers={"Authorization": f"Token {key}", "Content-Type": "application/json"},
method="DELETE",
)
try:
@@ -0,0 +1,127 @@
"""Shared, host-independent redaction and lossless extraction batching."""
from __future__ import annotations
import json
import math
import re
from typing import Any
MAX_EXTRACTION_INPUT_TOKENS = 24000
SECRET_PATTERNS = [
re.compile(r"(?i)(authorization\s*[:=]\s*(?:bearer|token)\s+)[^\s\"']+"),
re.compile(r"(?i)((?:api[_-]?key|secret[_-]?access[_-]?key|session[_-]?token)\s*[:=]\s*)[^\s\"']+"),
re.compile(
r"(?i)((?:access[_-]?token|refresh[_-]?token|password|credential)"
r"\s*[:=]\s*)[^\s&\"']+"
),
re.compile(r"\b(?:sk|m0|mem0_sk|psk)-[A-Za-z0-9_\-]{12,}\b"),
re.compile(r"\b(?:ASIA|AKIA)[A-Z0-9]{12,}\b"),
re.compile(r"\b(?:ghp_|github_pat_|xox[baprs]-)[A-Za-z0-9_\-]{12,}\b"),
re.compile(
r"-----BEGIN [^-]*PRIVATE KEY-----.*?-----END [^-]*PRIVATE KEY-----",
re.DOTALL,
),
re.compile(
r'(?i)("(?:api[_-]?key|password|secret(?:[_-]?access[_-]?key)?'
r"|(?:access|refresh|session)[_-]?token|token|authorization|credential"
r')"\s*:\s*")(?:\\.|[^"\\])*'
),
]
def redact(value: Any) -> str:
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, default=str)
for pattern in SECRET_PATTERNS:
if pattern.groups:
text = pattern.sub(r"\1[REDACTED]", text)
else:
text = pattern.sub("[REDACTED]", text)
return text
def _estimated_tokens(value: str) -> int:
"""Conservatively estimate tokens without adding a tokenizer dependency."""
ascii_chars = sum(ord(char) < 128 for char in value)
return math.ceil((ascii_chars * 0.4) + (len(value) - ascii_chars))
def _message_tokens(messages: list[dict[str, str]]) -> int:
return _estimated_tokens(json.dumps(messages, ensure_ascii=False))
def _is_agent_assignment(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent assignment (")
def _is_agent_response(message: dict[str, str]) -> bool:
return message.get("role") == "assistant" and message.get("content", "").startswith("Subagent response (")
def extraction_message_batches(
messages: list[dict[str, str]],
*,
max_tokens: int = MAX_EXTRACTION_INPUT_TOKENS,
) -> list[list[dict[str, str]]]:
"""Keep exchanges together when possible; split oversized messages to enforce the request budget."""
if not messages or _message_tokens(messages) <= max_tokens:
return [messages]
exchanges: list[list[dict[str, str]]] = []
exchange: list[dict[str, str]] = []
for message in messages:
if message.get("role") == "user" and exchange:
exchanges.append(exchange)
exchange = []
exchange.append(message)
if exchange:
exchanges.append(exchange)
units: list[list[dict[str, str]]] = []
for exchange in exchanges:
if _message_tokens(exchange) <= max_tokens:
units.append(exchange)
continue
index = 0
while index < len(exchange):
message = exchange[index]
if _is_agent_assignment(message) and index + 1 < len(exchange) and _is_agent_response(exchange[index + 1]):
units.append(exchange[index : index + 2])
index += 2
else:
units.append([message])
index += 1
bounded_units: list[list[dict[str, str]]] = []
for unit in units:
if _message_tokens(unit) <= max_tokens:
bounded_units.append(unit)
continue
for message in unit:
remaining = message["content"]
while remaining:
low, high = 0, len(remaining)
while low < high:
middle = (low + high + 1) // 2
if _message_tokens([{**message, "content": remaining[:middle]}]) <= max_tokens:
low = middle
else:
high = middle - 1
if low == 0:
raise ValueError("Extraction token budget cannot fit a message")
bounded_units.append([{**message, "content": remaining[:low]}])
remaining = remaining[low:]
batches: list[list[dict[str, str]]] = []
batch: list[dict[str, str]] = []
for unit in bounded_units:
candidate = [*batch, *unit]
if batch and _message_tokens(candidate) > max_tokens:
batches.append(batch)
batch = list(unit)
else:
batch = candidate
if batch:
batches.append(batch)
return batches
+39 -655
View File
@@ -1,9 +1,5 @@
#!/usr/bin/env python3
"""Usage telemetry for Mem0 agent plugins.
Events are linked to your Mem0 account email when an API key is configured, and
to a random per-machine id otherwise. Not anonymous — the Python SDK and CLI
attribute the same way.
"""Anonymous usage telemetry for Mem0 agent plugins.
Hooks run on a 3-6 second budget and fire on every tool call, so recording never
touches the network: `record` appends one JSON line to a local spool and returns.
@@ -13,8 +9,7 @@ started once per session and again from the flush worker that is already detache
Pure stdlib, matching the rest of the plugin. Opt out with MEM0_TELEMETRY=false.
Never sends prompts, memory text, queries, file paths, repository names, or API
keys: only event names, durations, counts, coarse outcomes, and repo/session
identifiers hashed with a random per-install salt.
keys: only event names, durations, counts, coarse outcomes, and salted hashes.
"""
from __future__ import annotations
@@ -34,24 +29,8 @@ from typing import Any
import memory_core
# Seeded from the per-host module the build generates into core/. Two processes
# in this pipeline never call init() — mcp_server.py, and the detached
# `python3 telemetry.py` sender that spawn_flush() starts — so a module default
# was what every one of their events got labelled with.
try: # pragma: no cover - absent only in the un-built shared source tree
from _harness_id import HARNESS_ID as _DEFAULT_HARNESS
from _harness_id import PLATFORM_APPLICATION as _PLATFORM_APPLICATION
from _harness_id import PLATFORM_SOURCE as _PLATFORM_SOURCE
from _harness_id import SOURCE_TAG as _DEFAULT_SOURCE_TAG
except ImportError:
_DEFAULT_HARNESS = "generic"
_DEFAULT_SOURCE_TAG = "MEM0_PLUGIN"
_PLATFORM_SOURCE = "MEM0_PLUGIN"
_PLATFORM_APPLICATION = ""
_salt_cache: str = ""
_harness: str = _DEFAULT_HARNESS
_source_tag: str = _DEFAULT_SOURCE_TAG
_harness: str = "generic"
_source_tag: str = "MEM0_PLUGIN"
_PRIVATE_KEYS = {
"apikey",
"authorization",
@@ -77,19 +56,10 @@ _PRIVATE_KEYS = {
}
def init(harness: str = "", source_tag: str = "") -> None:
"""Override the generated identity. Optional — core/_harness_id.py is the default.
The fallback shape matches memory_core.configure_harness's (``<HOST>_PLUGIN``).
It used to be ``MEM0_<HOST>_PLUGIN`` here and ``<host>_plugin`` there, which
meant one plugin could emit three different source values depending on which
process happened to send the batch.
"""
def init(harness: str = "generic", source_tag: str = "") -> None:
global _harness, _source_tag
_harness = harness or _DEFAULT_HARNESS
_source_tag = source_tag or (
f"{_harness.upper().replace('-', '_')}_PLUGIN" if harness else _DEFAULT_SOURCE_TAG
)
_harness = harness
_source_tag = source_tag or f"MEM0_{harness.upper().replace('-', '_')}_PLUGIN"
POSTHOG_API_KEY = "phc_hgJkUVJFYtmaJqrvf6CYN67TIQ8yhXAkWzUn9AMU4yX"
POSTHOG_CAPTURE_URL = "https://us.i.posthog.com/i/v0/e/"
@@ -100,16 +70,6 @@ BATCH_SIZE = 100
SEND_TIMEOUT = 5
CLAIM_STALE_SECONDS = 120
CLAIM_EXPIRY_SECONDS = 7 * 24 * 60 * 60
# A batch is only discarded once it has genuinely been retried this many times.
MAX_CLAIM_ATTEMPTS = 3
# Parked claims drained per run, after the live spool. Bounded so a long backlog
# cannot turn one flush into an unbounded send loop.
MAX_PARKED_PER_RUN = 3
# Added to the wait before a released claim becomes reclaimable, per attempt
# already spent. Releasing straight to "reclaimable now" let two senders burn the
# whole budget within seconds of one another on a single momentary failure, and
# discard a batch a retry a minute later would have delivered.
RETRY_COOLDOWN_SECONDS = 60
def is_enabled() -> bool:
@@ -123,126 +83,9 @@ def is_enabled() -> bool:
def _digest(value: str, length: int = 16) -> str:
"""Unsalted digest. Only for values that are already secrets (API keys)."""
return hashlib.sha256(value.encode("utf-8")).hexdigest()[:length]
def _salt_path() -> Path:
return memory_core.data_dir() / "telemetry-salt"
def _install_salt() -> str:
"""Random per-install salt, created once and memoized for the process.
Deliberately its own file, claimed with O_CREAT|O_EXCL, rather than a key in
the identity file. Three reasons, all of which produced wrong data when this
lived in the identity dict:
- Hooks are short-lived separate processes firing on every tool call, and
people run more than one agent window. A read-modify-write would let each
process mint its own salt, so one repository would hash several ways in the
window before a writer won.
- resolve_distinct_id holds a copy of the identity dict across a network call
to /v1/ping/, so whichever write landed second erased the other's key —
losing either the salt (repo_hash changes mid-stream) or the email (a
second $identify, splitting the person).
- Touching the identity file from record() would create it, and is_first_run
keys off that file, so recording an event would silently suppress the
install event.
Published atomically, and there is deliberately no derived fallback. Creating
the file with O_CREAT|O_EXCL and then writing into it leaves a window where
the file exists and is empty, and a concurrent hook that reads it in that
window gets nothing. Falling back to a digest of the path would hand that
process a salt an attacker can compute, memoized for its whole run, which is
the privacy control this function exists to provide silently turning itself
off under load. The salt is written to a private temp file first and linked
into place, so the name either does not exist or already has the full value.
Returns "" when it genuinely cannot persist. Callers omit the hash entirely
rather than emit an unsalted one.
"""
global _salt_cache
if _salt_cache:
return _salt_cache
path = _salt_path()
# Read before writing. Hooks are separate processes firing on every tool
# call, so all but the first find the salt already published; going straight
# to create-fsync-link-unlink meant every one of them paid an fsync to
# discover that, on a path whose whole promise is appending a line and
# returning.
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
if _salt_cache:
return _salt_cache
except OSError:
pass
temporary = path.with_name(f"{path.name}.{os.getpid()}.tmp")
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(handle, "w", encoding="utf-8") as stream:
stream.write(uuid.uuid4().hex)
stream.flush()
os.fsync(stream.fileno())
try:
# Atomic claim: fails if another process already published one.
# os.link rather than replace, which would clobber theirs.
os.link(temporary, path)
except FileExistsError:
pass
except OSError:
# No hardlinks here (some network mounts, some container volumes).
# Claim the name directly instead. That reopens the empty-file
# window, but the window is now benign: a reader that lands in it
# gets "" and omits the hash for that process rather than caching a
# guessable one. Losing the hashes on every run of an entire
# filesystem is the worse failure.
try:
fallback = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
with os.fdopen(fallback, "w", encoding="utf-8") as stream:
stream.write(temporary.read_text(encoding="utf-8"))
except OSError:
pass
except OSError:
pass
finally:
try:
temporary.unlink()
except OSError:
pass
try:
_salt_cache = path.read_text(encoding="utf-8").strip()
except OSError:
_salt_cache = ""
return _salt_cache
def _scoped_digest(value: str, length: int = 16) -> str:
"""Salted digest for values drawn from a guessable space.
repo.identity is a git remote URL, or ``local:<absolute path>`` when there is
no remote — which normally contains the account username. Sixteen unsalted
hex characters over that input space is enumerable, so this is not a
privacy control without the salt. Salting per install keeps every
within-account join the analytics actually use and gives up only
cross-machine joins on the same repository, which nothing computes.
Returns "" when there is no salt, so record() omits the property. An
unsalted digest over this input space is close to plaintext, and emitting one
under a name that implies it is hashed is worse than sending nothing.
"""
if not value:
return ""
salt = _install_salt()
if not salt:
return ""
return hashlib.sha256(f"{salt}:{value}".encode("utf-8")).hexdigest()[:length]
def _safe_value(value: Any) -> Any:
if isinstance(value, str):
return memory_core.redact(value)
@@ -302,176 +145,9 @@ def anonymous_id(identity: dict[str, str] | None = None) -> str:
return created
def _rotate_anonymous_id(identity: dict[str, str]) -> str:
"""Mint a fresh anonymous id because the account context is gone.
The previous id may already have been merged into a person profile by an
$identify, and that merge is permanent. Reusing it after a logout or a key
change attributes everything that follows to the account that just went
away, which is the same misattribution the key fingerprint exists to stop,
only arriving through the anonymous path instead.
`aliased` is cleared with it: the new id has never been merged, so it is
eligible to be aliased into whatever account comes next.
"""
created = f"code-anon-{uuid.uuid4().hex}"
identity["anonymous_id"] = created
identity.pop("aliased", None)
_write_identity(identity)
return created
def _install_state_path() -> Path:
return memory_core.data_dir() / "install-state.json"
def is_first_run() -> bool:
"""Whether install has never been recorded on this machine.
Deliberately NOT the identity file. That file is only written by a
successful flush, so an offline or firewalled user recorded code.install on
every single session, forever — and every 0.2.x user recorded one on their
first 0.3.x session because 0.2.x never wrote it at all.
"""
return not _install_state_path().exists()
def data_dir_was_empty() -> bool:
"""Whether the data directory is untouched. Call BEFORE anything writes to it.
hook_runner reaches claim_install() only after cache_plugin_api_key() has
written `api-key` and EvidenceStore() has created `evidence.sqlite3`, so
asking at claim time always saw content and every fresh install reported an
upgrade. The caller snapshots this at the top of the run instead.
"""
return not _data_dir_has_content()
def claim_install(was_empty: bool | None = None) -> str | None:
"""Claim the one install/upgrade record for this machine, atomically.
Returns the event to record ("install" or "upgrade"), or None if another
session already claimed it. O_CREAT|O_EXCL so two sessions starting together
cannot both win.
`was_empty` must come from data_dir_was_empty() called before this process
wrote anything. Omitting it falls back to checking now, which is only
correct for a caller that has touched nothing.
"""
if not is_enabled():
# Never consume the one-shot claim while the user is opted out, or they
# would silently lose their install event if they later opt in.
return None
path = _install_state_path()
upgrading = not (data_dir_was_empty() if was_empty is None else was_empty)
try:
path.parent.mkdir(parents=True, exist_ok=True)
handle = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
except FileExistsError:
return None
except OSError:
return None
try:
with os.fdopen(handle, "w", encoding="utf-8") as stream:
json.dump(
{
"plugin_version": memory_core.PLUGIN_VERSION,
"installed_at": memory_core.utc_now(),
"upgraded": upgrading,
},
stream,
)
# Durable before this returns. The O_EXCL open is what makes the
# claim exclusive, so it cannot be replaced by a temp-and-rename
# without losing that, which leaves the content as the thing to make
# safe. A kill between the open and this fsync used to leave a marker
# that exists but parses to nothing: is_first_run reads it as claimed
# and claim_version_change cannot read a version out of it.
stream.flush()
os.fsync(stream.fileno())
except OSError:
pass
return "upgrade" if upgrading else "install"
def _data_dir_has_content() -> bool:
"""Whether anything predates this session in the plugin data directory."""
try:
for entry in memory_core.data_dir().iterdir():
if entry.name != "install-state.json":
return True
except OSError:
pass
return False
def _repair_install_state(path: Path) -> None:
"""Rewrite an unparseable marker so version tracking can resume."""
try:
temporary = path.with_suffix(f".{os.getpid()}.tmp")
temporary.write_text(
json.dumps({"plugin_version": memory_core.PLUGIN_VERSION, "repaired_at": memory_core.utc_now()}),
encoding="utf-8",
)
temporary.replace(path)
except OSError:
pass
def claim_version_change() -> str | None:
"""Return the previously recorded version if it differs, updating the marker.
Only meaningful once the marker exists — the first transition into 0.3.x has
no recorded predecessor and reports "pre-0.3" instead. Claiming by rewriting
the marker means the next session sees no change and records nothing.
"""
path = _install_state_path()
try:
state = json.loads(path.read_text(encoding="utf-8"))
except OSError:
return None
except json.JSONDecodeError:
# A crash between O_EXCL and the write leaves an empty marker. Left
# alone it disables every future upgrade event on this machine, because
# claim_install sees the file and this function cannot parse it.
state = None
if not isinstance(state, dict):
_repair_install_state(path)
return None
previous = str(state.get("plugin_version") or "")
if not previous or previous == memory_core.PLUGIN_VERSION:
return None
# Claim the transition with an exclusive sentinel before rewriting the
# marker. A plain read-modify-write let every concurrently starting session
# observe the old version and each record its own upgrade — and the first
# session after a version bump is exactly when several agent windows restart
# together.
sentinel = path.with_name(f"upgraded-{memory_core.PLUGIN_VERSION}")
try:
os.close(os.open(sentinel, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600))
except FileExistsError:
return None
except OSError:
return None
state["plugin_version"] = memory_core.PLUGIN_VERSION
state["upgraded_at"] = memory_core.utc_now()
temporary = path.with_suffix(f".{os.getpid()}.tmp")
try:
temporary.write_text(json.dumps(state), encoding="utf-8")
temporary.replace(path)
except OSError:
# Release the claim. The marker still records the old version, so
# without this the sentinel makes claim_version_change return early on
# every later run and this version's upgrade is never recorded again.
for leftover in (sentinel, temporary):
try:
leftover.unlink()
except OSError:
pass
return None
return previous
"""Whether this machine has never recorded a plugin event before."""
return not _identity_path().exists()
def record(
@@ -492,32 +168,19 @@ def record(
except OSError:
pass
properties = _safe_value(properties)
# Stamped in the RECORDING process, beside harness. `source` used to be
# read in the sending process from a module global, so whichever process
# drained the spool named every event in it. flush() spreads per-event
# properties last, so this now wins over any sender's default.
properties.update(
harness=_harness,
source=_source_tag,
plugin_version=memory_core.PLUGIN_VERSION,
os=sys.platform,
python_version=platform.python_version(),
)
# Assigned only when the digest is real. _scoped_digest returns "" when
# the salt could not be persisted, and an empty property is worse than an
# absent one: it survives the None filter below and reads as a value.
if repo is not None:
repo_hash = _scoped_digest(getattr(repo, "identity", ""))
if repo_hash:
properties["repo_hash"] = repo_hash
properties["repo_hash"] = _digest(getattr(repo, "identity", ""))
if session_id:
session_hash = _scoped_digest(session_id)
if session_hash:
properties["session_hash"] = session_hash
properties["session_hash"] = _digest(session_id)
line = json.dumps(
{
"event": f"{EVENT_PREFIX}.{event}",
"uuid": str(uuid.uuid4()),
"timestamp": memory_core.utc_now(),
"properties": {
key: value for key, value in properties.items() if value is not None
@@ -576,201 +239,38 @@ def spawn_flush() -> bool:
return False
def _claim_name(attempt: int = 0) -> str:
"""Claim filename. The attempt count rides in the name so the 7-day expiry
only ever discards a batch that was actually retried and failed."""
return f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}-a{attempt}.sending"
def _claim_attempt(claim: Path) -> int:
"""Attempts recorded in a claim filename; 0 for the pre-attempt-count shape.
Anchored on field position, not on a leading "a": the legacy shape is
``telemetry-<pid>-<hex>.sending`` and a hex id such as ``a1234567`` would
otherwise parse as attempt 1234567 and be discarded unsent on the first
flush after an upgrade.
"""
stem = claim.name[: -len(".sending")] if claim.name.endswith(".sending") else claim.name
parts = stem.split("-")
if len(parts) != 4:
return 0
tail = parts[3]
if tail.startswith("a") and tail[1:].isdigit():
return int(tail[1:])
return 0
def _touch(path: Path) -> None:
"""Refresh mtime so a claim's age measures time since it was claimed.
``Path.replace`` is ``os.rename``, which preserves mtime — so a claim created
after a quiet minute inherited the spool's last-write time and looked
abandoned the instant it was made. A second sender would then take it over
while the first was still posting, and both would deliver the batch.
"""
try:
os.utime(path, None)
except OSError:
pass
def _claim_spool() -> Path | None:
"""Rename the spool aside so exactly one sender owns each batch."""
directory = memory_core.data_dir()
claim = directory / _claim_name()
claim = directory / f"telemetry-{os.getpid()}-{uuid.uuid4().hex[:8]}.sending"
spool = _spool_path()
try:
spool.replace(claim)
_touch(claim)
return claim
except OSError:
pass
return _claim_parked(directory)
def _sweep_debris(directory: Path) -> None:
"""Remove files nothing else will ever pick up again.
*.partial is a temp file orphaned by a crash between write and rename.
*.corrupt is a batch quarantined for undecodable content. No glob in this
module matches either, so without this they accumulate on disk for the life
of the install.
Quarantined batches are kept far longer than debris: they are the only
evidence left of events that could not be delivered, and someone diagnosing
a report of missing telemetry has to be able to find one.
"""
now = time.time()
for debris in directory.glob("telemetry-*.partial"):
try:
if now - debris.stat().st_mtime > CLAIM_STALE_SECONDS:
debris.unlink()
except OSError:
continue
for quarantined in directory.glob("telemetry-*.corrupt"):
try:
if now - quarantined.stat().st_mtime > CLAIM_EXPIRY_SECONDS:
quarantined.unlink()
except OSError:
continue
# The same reasoning covers *.tmp. _write_identity and _install_salt both
# create one and unlink it in a finally, which a SIGKILL skips, and no glob
# in this module matches the leftovers either.
for temporary in directory.glob("telemetry-*.tmp"):
try:
if now - temporary.stat().st_mtime > CLAIM_STALE_SECONDS:
temporary.unlink()
except OSError:
continue
def _claim_parked(directory: Path) -> Path | None:
"""Take the oldest abandoned claim, if any lease has actually expired.
Kept separate from the live spool so flush() can drain both in one run.
Previously parked batches were only reachable when no spool existed at all,
and because sessions keep recording there usually was one — so a batch
parked by a failed send waited until the 7-day expiry deleted it unsent,
even though its own presence is what started the sender.
"""
now = time.time()
for orphan in sorted(directory.glob("telemetry-*.sending"), key=_safe_mtime):
for orphan in sorted(directory.glob("telemetry-*.sending")):
try:
age = now - orphan.stat().st_mtime
except OSError:
continue
if age < CLAIM_STALE_SECONDS:
# Someone else holds a live lease on it. This check has to come
# first. Claiming a file bumps its attempt count and refreshes its
# mtime, so a sender that has just taken the final attempt looks
# exhausted to everyone else while it is actively draining. Judging
# exhaustion before liveness let a second sender unlink a batch out
# from under its owner, losing every event in it.
continue
# Attempts, not age. Every re-claim touches the mtime and every release
# backdates it by a fixed amount, so age is pinned near the stale
# threshold and never reaches the expiry. Age stays only as a backstop
# for files that never carried an attempt marker.
if _claim_attempt(orphan) >= MAX_CLAIM_ATTEMPTS or age > CLAIM_EXPIRY_SECONDS:
if age > CLAIM_EXPIRY_SECONDS:
try:
orphan.unlink()
except OSError:
pass
continue
claim = orphan.parent / _claim_name(_claim_attempt(orphan) + 1)
if age < CLAIM_STALE_SECONDS:
continue
try:
orphan.replace(claim)
_touch(claim)
return claim
except OSError:
continue
return None
def _safe_mtime(path: Path) -> float:
try:
return path.stat().st_mtime
except OSError:
return 0.0
def _rewrite_claim(claim: Path, remaining: list[dict[str, Any]]) -> bool:
"""Persist the unsent remainder, atomically, and refresh the lease.
Called after every successful batch. Two jobs: a retry resumes where the
send stopped instead of re-posting from the top, and the rewrite doubles as
the lease heartbeat, so a slow sender does not have its claim stolen
mid-flight. Interval is one batch, well inside CLAIM_STALE_SECONDS.
"""
if not remaining:
try:
claim.unlink()
except OSError:
pass
return True
temporary = claim.with_suffix(f".{os.getpid()}.partial")
try:
payload = "".join(json.dumps(event, separators=(",", ":"), default=str) + "\n" for event in remaining)
# fsync before the rename: without it the rename can land while the
# bytes have not, and the claim comes back empty or truncated after a
# crash. _drain then reads zero events and unlinks it.
with open(temporary, "w", encoding="utf-8") as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
temporary.replace(claim)
_touch(claim)
return True
except OSError:
try:
temporary.unlink()
except OSError:
pass
return False
def _release_claim(claim: Path, remaining: list[dict[str, Any]]) -> None:
"""Persist the remainder and drop the lease, because this sender has given up.
Distinct from the per-batch heartbeat: heartbeating on the way out would
make an abandoned batch look actively owned for a further
CLAIM_STALE_SECONDS, delaying the retry for no reason. Ageing it past the
threshold lets the next flush pick it up immediately, while the attempt
count in the filename still bounds how many times that can happen.
"""
if not _rewrite_claim(claim, remaining):
return
try:
# Backdate past the stale threshold so the next flush can pick it up,
# minus a cooldown that grows with the attempts already spent. Clamped so
# the mtime never lands in the future, which would read as a live lease.
cooldown = min(_claim_attempt(claim) * RETRY_COOLDOWN_SECONDS, CLAIM_STALE_SECONDS)
released = time.time() - CLAIM_STALE_SECONDS - 1 + cooldown
os.utime(claim, (released, released))
except OSError:
pass
def _resolve_email(key: str) -> str:
"""Trade the API key for the account email so events join other Mem0 surfaces."""
url = os.environ.get("MEM0_API_URL", memory_core.DEFAULT_API_URL).rstrip("/") + "/v1/ping/"
@@ -800,130 +300,34 @@ def _post(payload: dict[str, Any], url: str) -> bool:
def resolve_distinct_id() -> tuple[str, str]:
"""Return the PostHog distinct id and the anonymous id it replaced, if any.
The second value becomes a PostHog $identify alias. It is ONLY ever an
anonymous id: aliasing one account email to another merges two real person
profiles and cannot be undone, so a key that now belongs to a different
account re-resolves with no alias.
"""
"""Return the PostHog distinct id and the anonymous id it replaced, if any."""
identity = _read_identity()
key = memory_core.api_key()
fingerprint = _digest(key) if key else ""
email = identity.get("email", "")
if email and fingerprint:
recorded = identity.get("key_fingerprint", "")
if recorded == fingerprint:
return email, ""
if not recorded:
# Rows written before fingerprints existed. Verify rather than
# adopt: a key changed before the upgrade would otherwise bind the
# new key to the previous account's email, permanently, and the
# fingerprint would then agree with itself forever after.
verified = _resolve_email(key)
if not verified:
# Offline, firewalled, or the API is down. Keep the previous
# behaviour and retry on the next flush rather than dropping a
# real account attribution. Safe because the same network that
# failed /v1/ping/ is about to fail the PostHog POST, so nothing
# is delivered under the unverified identity in the meantime.
return email, ""
identity["email"] = verified
identity["key_fingerprint"] = fingerprint
_write_identity(identity)
return verified, ""
if email:
return email, ""
key = memory_core.api_key()
if not key:
# No key to verify the account with; do not keep attributing to it.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
return anonymous_id(identity), ""
resolved = _resolve_email(key)
if not resolved:
# The key changed and will not resolve (revoked, offline, API down).
# Reaching here with an email means the recorded fingerprint disagreed,
# so the key really did change. Drop the account and rotate: the stored
# anonymous id may already be merged into that account's person, and
# reusing it would keep the events on the profile we are trying to
# leave.
if email:
identity.pop("email", None)
identity.pop("key_fingerprint", None)
return _rotate_anonymous_id(identity), ""
email = _resolve_email(key)
if not email:
return anonymous_id(identity), ""
# Alias only when going anonymous -> email for the first time. Once an anon
# id has been merged into an account it must never be offered again: an
# alias naming an already-identified id is what could link two real people.
previous = "" if (email or identity.get("aliased")) else identity.get("anonymous_id", "")
if previous:
identity["aliased"] = True
identity["email"] = resolved
identity["key_fingerprint"] = fingerprint
previous = identity.get("anonymous_id", "")
identity["email"] = email
_write_identity(identity)
return resolved, previous
return email, previous
def flush() -> int:
"""Drain the live spool, then any parked claims, and return events sent."""
"""Drain claimed spools to PostHog and return the number of events sent."""
if not is_enabled():
return 0
sent, delivered = _drain(_claim_spool())
if not delivered:
# The network is failing. Retrying other batches now would only burn
# their attempt budget against the same broken connection.
return sent
# Parked batches used to starve behind the live spool indefinitely. Bounded
# per run so a long backlog cannot turn one flush into an unbounded loop.
directory = memory_core.data_dir()
_sweep_debris(directory)
for _ in range(MAX_PARKED_PER_RUN):
parked = _claim_parked(directory)
if parked is None:
break
count, delivered = _drain(parked)
sent += count
if not delivered:
break
return sent
def _drain(claim: Path | None) -> tuple[int, bool]:
"""Post one claimed batch file, recording progress after every batch.
Returns (events sent, whether everything was delivered).
"""
claim = _claim_spool()
if claim is None:
return 0, True
return 0
try:
lines = claim.read_text(encoding="utf-8").splitlines()
except ValueError:
# UnicodeDecodeError from a torn write: the content is unrecoverable, so
# quarantine rather than retry. flush() runs from a bare `finally:` in
# flush_worker, so raising here also skips the handoff cleanup, and an
# undecodable file would otherwise be re-read on every flush forever.
# Reported as delivered because there is nothing left to deliver and the
# rest of the run should continue.
try:
claim.replace(claim.with_suffix(".corrupt"))
except OSError:
try:
claim.unlink()
except OSError:
pass
return 0, True
except OSError:
# Could not read it, which is not the same as having nothing to send.
# The file is left exactly where it is: a vanished or briefly unreadable
# claim is retryable, and quarantining it here would discard events over
# a transient filesystem error. Reported as undelivered so the run stops
# instead of counting a batch nothing was posted from as delivered.
return 0, False
return 0
events = []
for line in lines:
try:
@@ -933,18 +337,11 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
if isinstance(value, dict) and value.get("event"):
events.append(value)
if not events:
# Only delete when the file really is empty. A non-empty file that
# parses to nothing is a torn write, and its contents are the unsent
# remainder — deleting it is the data loss this PR exists to prevent.
try:
empty = claim.stat().st_size == 0
except OSError:
empty = True
try:
claim.replace(claim.with_suffix(".corrupt")) if not empty else claim.unlink()
claim.unlink()
except OSError:
pass
return 0, True
return 0
distinct_id, aliased_anonymous_id = resolve_distinct_id()
if aliased_anonymous_id:
@@ -963,17 +360,12 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
sent = 0
for start in range(0, len(events), BATCH_SIZE):
chunk = events[start : start + BATCH_SIZE]
batch = [
{
"event": event["event"],
"distinct_id": distinct_id,
# Carried through from record() so a resend can be collapsed.
"uuid": event.get("uuid"),
"timestamp": event.get("timestamp"),
"properties": {
# Fallback only: events recorded by a build before source
# moved into record() have none of their own.
"source": _source_tag,
"language": "python",
"$process_person_profile": False,
@@ -981,24 +373,16 @@ def _drain(claim: Path | None) -> tuple[int, bool]:
**(event.get("properties") or {}),
},
}
for event in chunk
for event in events[start : start + BATCH_SIZE]
]
if not _post({"api_key": POSTHOG_API_KEY, "batch": batch}, POSTHOG_BATCH_URL):
# Keep only what has not been delivered, and release the lease.
# Previously the whole file was kept and the retry re-posted every
# batch, including the ones that had already arrived.
_release_claim(claim, events[start:])
return sent, False
sent += len(chunk)
# Record progress and refresh the lease after each successful batch, so
# a crash repeats at most one batch instead of the entire file. If the
# rewrite fails the claim still holds delivered events, so stop rather
# than carry on as though progress were recorded — continuing is how the
# duplicate delivery this PR fixes would come back.
if not _rewrite_claim(claim, events[start + len(chunk) :]):
_release_claim(claim, events[start + len(chunk) :])
return sent, False
return sent, True
return sent
sent += len(batch)
try:
claim.unlink()
except OSError:
pass
return sent
def main() -> int:
+1 -1
View File
@@ -1,7 +1,7 @@
{
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
"name": "mem0",
"version": "0.3.2",
"version": "0.3.1",
"description": "Cross-session memory and token savings for coding agents.",
"author": {
"name": "Mem0",
@@ -6,8 +6,8 @@ description: Pause Mem0 memory capture on this machine. Use when the user wants
# Pause memory capture
To pause (hooks stop capturing and sending session content; a minimal
telemetry ping still fires at session start, under your Mem0 account email,
unless `MEM0_TELEMETRY=false`):
anonymous telemetry ping still fires at session start unless
`MEM0_TELEMETRY=false`):
```bash
python3 "${PLUGIN_ROOT}/core/memory_cli.py" --harness "coding-agent" pause
+3 -5
View File
@@ -79,11 +79,9 @@ the tool share one Mem0 backend and namespace.
## Telemetry
The store sends usage events (store configuration, operation, duration, result
counts, coarse failure kind) over the Mem0 SDK's existing telemetry client,
tagged `source="STRANDS"`. These are **not anonymous**: when an API key is
configured they are sent under your Mem0 account email, the same way the SDK
attributes its own. Queries, memory text, message content, entity ids, and
The store sends anonymous usage events (store configuration, operation, duration,
result counts, coarse failure kind) over the Mem0 SDK's existing telemetry client,
tagged `source="STRANDS"`. Queries, memory text, message content, entity ids, and
metadata are never sent. Turn it off with `MEM0_TELEMETRY=false`.
## Development

Some files were not shown because too many files have changed in this diff Show More