Compare commits

...

8 Commits

Author SHA1 Message Date
kartik-mem0 a4633aa065 docs(api-reference): address reviewer comments on search defaults table and retrieval_criteria shape
- search-memories.mdx: collapse V1/V2/V3 comparison table to a single "Default" column showing current (v3) values only (threshold 0.1, rerank false, top_k 10)
- organizations-projects.mdx: replace bogus {role, content} retrieval_criteria examples with the correct {name, description, weight} shape, mirroring the canonical criteria-retrieval.mdx examples; update prose to name the three required fields
2026-06-26 16:48:19 +05:30
kartik-mem0 711f866e59 docs(api-reference): use literal em-dash instead of \u2014 escape in openapi.json 2026-06-26 16:03:50 +05:30
kartik-mem0 46757cb1fb docs(api-reference): document project.update() retrieval_criteria and decay params 2026-06-25 14:02:51 +05:30
kartik-mem0 bbfe0f9bb2 docs(api-reference): align API docs and openapi schema with v3 spec
- create-memory-export.mdx: replace `session_id` filter with `app_id`
  (spec: /v1/exports/ filters = user_id, agent_id, app_id, run_id)
- get-memory-export.mdx: replace stale filter list (session_id removed)
  with correct fields: user_id, agent_id, app_id, run_id, created_at, updated_at
- search-memories.mdx: fix comparison table — rerank V1/V2 default was
  `true`, spec shows `false`; threshold V1 is absent (not "No default"),
  V2 default is 0.3 (not absent)
- organizations-projects.mdx: rewrite Key Capabilities bullet — org/project
  are NOT init params; they resolve automatically from the API key via /v1/ping/
- openapi.json DELETE /v1/batch/: fix request schema from {memory_ids:[uuid]}
  to {memories:[{memory_id:uuid}]} to match Python client.batch_delete()
- openapi.json PUT /v1/batch/: make `text` optional (remove from required),
  add optional `metadata` property — matches Python client.batch_update() docstring
2026-06-25 12:01:24 +05:30
Taranjeet Singh ac296f7534 docs: make example code fences copy-safe (#5833) 2026-06-24 22:19:40 -07:00
Kartik ac8f862ff7 fix(mem0-plugin): store files_touched as a list to stop double JSON-encoding (#5806) 2026-06-25 09:06:11 +05:30
soumil-rathi b33fa5427c fix(memory): align entity extraction precision (#5829)
Making the entity extraction function cleaner


Co-authored-by: Soumil Rathi <soumilrathi@gmail.com>
2026-06-24 15:48:37 -07:00
Ashutosh Kasudhan 5d573dd2ae docs: added typescript support to deepseek provider (#5723) 2026-06-24 20:55:14 +05:30
34 changed files with 1402 additions and 473 deletions
+1 -1
View File
@@ -12,7 +12,7 @@
"name": "mem0",
"source": "./integrations/mem0-plugin",
"description": "Mem0 memory layer for AI applications. Add persistent memory, personalization, and semantic search to Claude workflows.",
"version": "0.2.10"
"version": "0.2.11"
}
]
}
+1 -1
View File
@@ -12,7 +12,7 @@
"name": "mem0",
"source": "./integrations/mem0-plugin",
"description": "Mem0 memory layer for AI applications. Add persistent memory, personalization, and semantic search.",
"version": "0.2.10"
"version": "0.2.11"
}
]
}
@@ -4,4 +4,4 @@ description: "Submit an export job to create a structured memory export using a
openapi: post /v1/exports/
---
Submit a job to create a structured export of memories using a customizable Pydantic schema. This process may take some time to complete, especially if you're exporting a large number of memories. You can tailor the export by applying various filters (e.g., `user_id`, `agent_id`, `run_id`, or `session_id`) and by modifying the Pydantic schema to ensure the final data matches your exact needs.
Submit a job to create a structured export of memories using a customizable Pydantic schema. This process may take some time to complete, especially if you're exporting a large number of memories. You can tailor the export by applying various filters (e.g., `user_id`, `agent_id`, `app_id`, or `run_id`) and by modifying the Pydantic schema to ensure the final data matches your exact needs.
@@ -4,4 +4,4 @@ description: "Retrieve the latest structured memory export after submitting an e
openapi: post /v1/exports/get
---
Retrieve the latest structured memory export after submitting an export job. You can filter the export by `user_id`, `run_id`, `session_id`, or `app_id` to get the most recent export matching your filters.
Retrieve the latest structured memory export after submitting an export job. You can filter the export by `user_id`, `agent_id`, `app_id`, `run_id`, `created_at`, or `updated_at` to get the most recent export matching your filters.
@@ -20,11 +20,11 @@ The `filters` object supports complex logical operations (AND, OR, NOT) and comp
### Search parameter defaults
| Parameter | V1/V2 | V3 |
| --- | --- | --- |
| `top_k` | Supported (default 10) | Supported (1-1000, default 10) |
| `threshold` | No default | Default `0.1` (pass `0.0` to disable) |
| `rerank` | Default `true` | Default `false` (pass `true` to enable) |
| Parameter | Default |
| --- | --- |
| `top_k` | `10` (range 1–1000) |
| `threshold` | `0.1` (pass `0.0` to disable) |
| `rerank` | `false` (pass `true` to enable) |
<CodeGroup>
```python Platform API Example
+41 -2
View File
@@ -14,7 +14,7 @@ Organizations and projects are **optional** features. You can use Mem0 without t
## Key Capabilities
- **Multi-org/project Support**: Specify organization and project when initializing the Mem0 client to attribute API usage appropriately
- **Multi-org/project Support**: Organization and project are resolved automatically from your API key via `/v1/ping/` — no org or project params are accepted by `MemoryClient.__init__`. Use a project-specific API key to target a particular project.
- **Member Management**: Control access to data through organization and project membership
- **Access Control**: Only members can access memories and data within their organization/project scope
- **Team Isolation**: Maintain data separation between different teams and projects for secure collaboration
@@ -79,7 +79,7 @@ new_project = client.project.create(
### Update Project Settings
Modify project configuration including custom instructions, categories, and language preferences:
Modify project configuration including custom instructions, categories, language preferences, retrieval criteria, and memory decay:
```python
# Update project with custom categories
@@ -98,6 +98,17 @@ client.project.update(
# Use the input language for memory storage and retrieval
client.project.update(multilingual=True)
# Set retrieval criteria to control which memories are surfaced in search
client.project.update(
retrieval_criteria=[
{"name": "relevance", "description": "How directly relevant this memory is to the current topic or user query", "weight": 3},
{"name": "access_frequency", "description": "How often this memory has been accessed or surfaced recently", "weight": 1}
]
)
# Enable Memory Decay (boosts recently-accessed memories at search time)
client.project.update(decay=True)
# Update multiple settings at once
client.project.update(
custom_instructions="...",
@@ -109,6 +120,34 @@ client.project.update(
)
```
#### Set Retrieval Criteria
`retrieval_criteria` is a per-project list of dictionaries (`List[Dict]`) that shapes how memories are ranked and filtered during search. Each dictionary has three fields: `name` (identifier), `description` (interpreted by the LLM to score each memory), and `weight` (relative influence on the final score). Use this to focus retrieval on intent-aligned or signal-specific memories:
```python
client.project.update(
retrieval_criteria=[
{
"name": "joy",
"description": "Measure the intensity of positive emotions such as happiness, excitement, or amusement expressed in the memory. A higher score reflects greater joy.",
"weight": 3
},
{
"name": "curiosity",
"description": "Assess the extent to which the memory reflects inquisitiveness or interest in exploring new information. A higher score reflects stronger curiosity.",
"weight": 2
},
{
"name": "access_frequency",
"description": "How often this memory has been accessed or surfaced recently.",
"weight": 1
}
]
)
```
Pass an empty list to clear all criteria and restore default retrieval behaviour.
#### Toggle Memory Decay
`decay` is a per-project boolean that turns on [Memory Decay](/platform/features/memory-decay) — a search-time ranking bias that reinforces recently-accessed memories and gently dampens stale ones. The flag is `false` by default; set it via the same project-update endpoint:
+14
View File
@@ -7,6 +7,13 @@ mode: "wide"
<Tabs>
<Tab title="Python">
<Update label="2026-06-24" description="v2.0.9">
**Bug Fixes:**
- **Memory (OSS):** Improve entity extraction precision by avoiding sentence-start common noun noise, preserving useful topic phrases, and exact-deduplicating entity links before semantic matching ([#5829](https://github.com/mem0ai/mem0/pull/5829))
</Update>
<Update label="2026-06-24" description="v2.0.8">
**New Features:**
@@ -1063,6 +1070,13 @@ See the [OSS v1 to v2 migration guide](https://docs.mem0.ai/migration/oss-v1-to-
<Tab title="TypeScript">
<Update label="2026-06-24" description="v3.0.11">
**Bug Fixes:**
- **Memory (OSS):** Align entity extraction with Python by reducing generic entity noise, preserving useful topic phrases, and exact-deduplicating entity links before semantic matching ([#5829](https://github.com/mem0ai/mem0/pull/5829))
</Update>
<Update label="2026-06-24" description="v3.0.10">
**Bug Fixes:**
+28 -1
View File
@@ -7,7 +7,8 @@ To use DeepSeek LLM models, you have to set the `DEEPSEEK_API_KEY` environment v
## Usage
```python
<CodeGroup>
```python Python
import os
from mem0 import Memory
@@ -36,6 +37,32 @@ messages = [
m.add(messages, user_id="alice", metadata={"category": "movies"})
```
```typescript TypeScript
import { Memory } from 'mem0ai/oss';
const config = {
llm: {
provider: 'deepseek',
config: {
apiKey: process.env.DEEPSEEK_API_KEY || '',
model: 'deepseek-chat',
temperature: 0.2,
maxTokens: 2000,
top_p: 1.0,
},
},
};
const memory = new Memory(config);
const messages = [
{"role": "user", "content": "I'm planning to watch a movie tonight. Any recommendations?"},
{"role": "assistant", "content": "How about thriller movies? They can be quite engaging."},
{"role": "user", "content": "I’m not a big fan of thriller movies but I love sci-fi movies."},
{"role": "assistant", "content": "Got it! I'll avoid thriller recommendations and suggest sci-fi movies in the future."}
];
await memory.add(messages, { userId: 'alice', metadata: { category: 'movies' } });
```
</CodeGroup>
You can also configure the API base URL in the config:
```python
+1 -1
View File
@@ -345,7 +345,7 @@ When evaluating memory systems, keep these considerations in mind:
<Card title="Research" icon="flask" href="https://mem0.ai/research">
Published research papers and technical reports
</Card>
<Card title="Blog Post" icon="newspaper" href="https://mem0.ai/blog/new-algorithm">
<Card title="Blog Post" icon="newspaper" href="https://mem0.ai/blog/the-token-efficient-memory-algorithm-now-has-temporal-reasoning">
Detailed writeup of the new algorithm design and results
</Card>
<Card title="Platform Migration" icon="arrow-right" href="/migration/platform-v2-to-v3">
+1
View File
@@ -65,6 +65,7 @@ The plugin uses the same shell scripts as Claude Code, Cursor, and Codex — hoo
| **User prompt** | `UserPromptSubmit` | Searches relevant memories before each message |
| **Pre-tool** | `PreToolUse` | Blocks MEMORY.md writes, enforces `user_id`/`app_id` on mem0 tools |
| **Post-tool** | `PostToolUse` | Tracks stats, scans bash errors for related memories |
| **Stop** | `Stop` | Stores a session summary when the session ends |
## Troubleshooting
+3 -2
View File
@@ -64,7 +64,7 @@ Add the Mem0 MCP server directly with a single command:
npx mcp-add \
--name mem0-mcp \
--type http \
--url "https://mcp.mem0.ai/mcp" \
--url "https://mcp.mem0.ai/mcp/" \
--clients "claude code"
```
@@ -142,7 +142,8 @@ When installed via the plugin marketplace, Mem0 hooks into Claude Code's lifecyc
| **User prompt** | `UserPromptSubmit` | Searches relevant memories before each message; skips short prompts |
| **Pre-tool** | `PreToolUse` | Blocks MEMORY.md writes, enforces `user_id`/`app_id` on mem0 tool calls |
| **Post-tool** | `PostToolUse` | Tracks stats, scans bash errors for related memories |
| **Pre-compact** | `PreCompact` | Stores a session summary before context compaction |
| **Stop** | `Stop` | Stores a session summary when the session ends |
| **Pre-compact** | `PreCompact` | Stores a summary before the context is compacted |
## Example Workflow
+23 -9
View File
@@ -41,7 +41,17 @@ Install the full plugin including MCP server, lifecycle hooks, and SDK skill.
codex plugin marketplace add mem0ai/mem0
```
2. Restart Codex, open the Plugin Directory, browse the **Mem0 Plugins** marketplace, and install **Mem0**.
2. Install the plugin:
```bash
codex plugin add mem0@mem0-plugins
```
Or, in the app: restart Codex, open the Plugin Directory, browse the **Mem0 Plugins** marketplace, and install **Mem0**.
<Note>
Step 1 is required for the app UI. Mem0 isn't in OpenAI's curated directory yet, so **without `codex plugin marketplace add`, Mem0 won't appear in the Codex app's Plugin Directory** — searching for it returns nothing. Adding the marketplace surfaces it (under **Created by you**) and makes it installable.
</Note>
<Info>
Do not combine with Option B. The plugin manifest auto-registers the `mem0` MCP server, so adding both will create a duplicate registration.
@@ -49,27 +59,30 @@ Install the full plugin including MCP server, lifecycle hooks, and SDK skill.
### Option B — Direct MCP
The fastest way to connect Codex to Mem0 — no plugin, no marketplace. Add to `~/.codex/config.toml`:
The fastest way to connect Codex to Mem0 — no plugin, no marketplace. Add the MCP server with a single command:
```bash
codex mcp add mem0 --url https://mcp.mem0.ai/mcp/ --bearer-token-env-var MEM0_API_KEY
```
Or add it manually to `~/.codex/config.toml`:
```toml
[mcp_servers.mem0]
url = "https://mcp.mem0.ai/mcp"
url = "https://mcp.mem0.ai/mcp/"
bearer_token_env_var = "MEM0_API_KEY"
```
Make sure `MEM0_API_KEY` is exported in the shell you launch Codex from, then restart Codex.
<Info>
Codex's `codex mcp add` CLI only supports stdio MCP servers. Because Mem0's MCP is HTTP/streamable, you configure it by editing `config.toml` directly (or via the **Plugins → Connect to a custom MCP → Streamable HTTP** UI in the Codex app).
</Info>
This gives you the MCP tools but not the lifecycle hooks or SDK skill.
### Managing the Plugin
```bash
codex plugin marketplace upgrade # pull latest plugin versions
codex plugin marketplace remove mem0-plugins # unregister the marketplace
codex plugin remove mem0@mem0-plugins # uninstall the plugin (keeps the marketplace)
codex plugin marketplace remove mem0-plugins # unregister the marketplace entirely
```
To update, run `codex plugin marketplace upgrade` to pull the latest from the Mem0 repo.
@@ -112,7 +125,8 @@ When installed via the plugin marketplace, Mem0 hooks into Codex's lifecycle to
| **User prompt** | `UserPromptSubmit` | Searches relevant memories before each message |
| **Pre-tool** | `PreToolUse` | Blocks MEMORY.md writes, enforces `user_id`/`app_id` on mem0 tool calls |
| **Post-tool** | `PostToolUse` | Tracks stats, scans bash errors for related memories |
| **Pre-compact** | `PreCompact` | Stores a session summary before context compaction |
| **Stop** | `Stop` | Stores a session summary when the session ends |
| **Pre-compact** | `PreCompact` | Stores a summary before the context is compacted |
## Example Workflow
+3 -2
View File
@@ -47,7 +47,7 @@ The fastest way to get started. Click the link below to install the Mem0 MCP ser
npx mcp-add \
--name mem0-mcp \
--type http \
--url "https://mcp.mem0.ai/mcp" \
--url "https://mcp.mem0.ai/mcp/" \
--clients "cursor"
```
@@ -110,7 +110,8 @@ When installed via the Cursor Marketplace, Mem0 hooks into Cursor's lifecycle:
| **User prompt** | `beforeSubmitPrompt` | Searches relevant memories before each message; skips short prompts |
| **Pre-tool (2 handlers)** | `preToolUse` | Blocks MEMORY.md writes, enforces `user_id`/`app_id` on mem0 tool calls |
| **Post-tool (2 handlers)** | `postToolUse` | Tracks stats, scans bash errors for related memories |
| **Pre-compact** | `preCompact` | Stores a session summary before context compaction |
| **Stop** | `stop` | Stores a session summary when the session ends |
| **Pre-compact** | `preCompact` | Stores a summary before the context is compacted |
## Example Workflow
+54 -55
View File
@@ -100,12 +100,12 @@ This section:
Initialize both the ElevenLabs and Mem0 clients:
```python
# Initialize ElevenLabs client
client = ElevenLabs(api_key=API_KEY)
# Initialize ElevenLabs client
client = ElevenLabs(api_key=API_KEY)
# Initialize memory client and tools
client_tools = ClientTools()
mem0_client = AsyncMemoryClient()
# Initialize memory client and tools
client_tools = ClientTools()
mem0_client = AsyncMemoryClient()
```
Here we:
@@ -118,36 +118,36 @@ Here we:
Define the two key memory functions that will be registered as tools:
```python
# Define memory-related functions for the agent
async def add_memories(parameters):
"""Add a message to the memory store"""
message = parameters.get("message")
await mem0_client.add(
messages=message,
user_id=USER_ID
)
return "Memory added successfully"
# Define memory-related functions for the agent
async def add_memories(parameters):
"""Add a message to the memory store"""
message = parameters.get("message")
await mem0_client.add(
messages=message,
user_id=USER_ID
)
return "Memory added successfully"
async def retrieve_memories(parameters):
"""Retrieve relevant memories based on the input message"""
message = parameters.get("message")
async def retrieve_memories(parameters):
"""Retrieve relevant memories based on the input message"""
message = parameters.get("message")
# For Platform API, user_id goes in filters
filters = {"user_id": USER_ID}
# For Platform API, user_id goes in filters
filters = {"user_id": USER_ID}
# Search for relevant memories using the message as a query
results = await mem0_client.search(
query=message,
filters=filters
)
# Search for relevant memories using the message as a query
results = await mem0_client.search(
query=message,
filters=filters
)
# Extract and join the memory texts
memories = ' '.join([result["memory"] for result in results.get('results', [])])
print("[ Memories ]", memories)
# Extract and join the memory texts
memories = ' '.join([result["memory"] for result in results.get('results', [])])
print("[ Memories ]", memories)
if memories:
return memories
return "No memories found"
if memories:
return memories
return "No memories found"
```
These functions:
@@ -171,9 +171,9 @@ These functions:
Register the memory functions with the ElevenLabs ClientTools system:
```python
# Register the memory functions as tools for the agent
client_tools.register("addMemories", add_memories, is_async=True)
client_tools.register("retrieveMemories", retrieve_memories, is_async=True)
# Register the memory functions as tools for the agent
client_tools.register("addMemories", add_memories, is_async=True)
client_tools.register("retrieveMemories", retrieve_memories, is_async=True)
```
This allows the ElevenLabs agent to:
@@ -186,19 +186,19 @@ This allows the ElevenLabs agent to:
Configure the conversation with ElevenLabs:
```python
# Initialize the conversation
conversation = Conversation(
client,
AGENT_ID,
# Assume auth is required when API_KEY is set
requires_auth=bool(API_KEY),
audio_interface=DefaultAudioInterface(),
client_tools=client_tools,
callback_agent_response=lambda response: print(f"Agent: {response}"),
callback_agent_response_correction=lambda original, corrected: print(f"Agent: {original} -> {corrected}"),
callback_user_transcript=lambda transcript: print(f"User: {transcript}"),
# callback_latency_measurement=lambda latency: print(f"Latency: {latency}ms"),
)
# Initialize the conversation
conversation = Conversation(
client,
AGENT_ID,
# Assume auth is required when API_KEY is set
requires_auth=bool(API_KEY),
audio_interface=DefaultAudioInterface(),
client_tools=client_tools,
callback_agent_response=lambda response: print(f"Agent: {response}"),
callback_agent_response_correction=lambda original, corrected: print(f"Agent: {original} -> {corrected}"),
callback_user_transcript=lambda transcript: print(f"User: {transcript}"),
# callback_latency_measurement=lambda latency: print(f"Latency: {latency}ms"),
)
```
This sets up the conversation with:
@@ -217,16 +217,16 @@ This sets up the conversation with:
Start and manage the conversation:
```python
# Start the conversation
print(f"Starting conversation with user_id: {USER_ID}")
conversation.start_session()
# Start the conversation
print(f"Starting conversation with user_id: {USER_ID}")
conversation.start_session()
# Handle Ctrl+C to gracefully end the session
signal.signal(signal.SIGINT, lambda sig, frame: conversation.end_session())
# Handle Ctrl+C to gracefully end the session
signal.signal(signal.SIGINT, lambda sig, frame: conversation.end_session())
# Wait for the conversation to end and get the conversation ID
conversation_id = conversation.wait_for_session_end()
print(f"Conversation ID: {conversation_id}")
# Wait for the conversation to end and get the conversation ID
conversation_id = conversation.wait_for_session_end()
print(f"Conversation ID: {conversation_id}")
if __name__ == '__main__':
@@ -445,4 +445,3 @@ By integrating ElevenLabs Conversational AI with Mem0, you can create voice agen
Create voice-first AI applications
</Card>
</CardGroup>
+18 -10
View File
@@ -62,7 +62,8 @@ def add(
filters: dict = None,
output_format: str = None, # ❌ REMOVED
version: str = None # ❌ REMOVED
) -> Union[List[dict], dict]
) -> Union[List[dict], dict]:
...
```
#### v1.0.0 Signature
@@ -76,7 +77,8 @@ def add(
metadata: dict = None,
filters: dict = None,
infer: bool = True # ✅ NEW: Control memory inference
) -> dict # Always returns dict with "results" key
) -> dict: # Always returns dict with "results" key
...
```
#### Changes Summary
@@ -147,7 +149,8 @@ def search(
filters: dict = None, # Basic key-value only
output_format: str = None, # ❌ REMOVED
version: str = None # ❌ REMOVED
) -> Union[List[dict], dict]
) -> Union[List[dict], dict]:
...
```
#### v1.0.0 Signature
@@ -161,7 +164,8 @@ def search(
limit: int = 100,
filters: dict = None, # ✅ ENHANCED: Advanced operators
rerank: bool = True # ✅ NEW: Reranking support
) -> dict # Always returns dict with "results" key
) -> dict: # Always returns dict with "results" key
...
```
#### Enhanced Filtering
@@ -216,7 +220,8 @@ def get_all(
filters: dict = None,
output_format: str = None, # ❌ REMOVED
version: str = None # ❌ REMOVED
) -> Union[List[dict], dict]
) -> Union[List[dict], dict]:
...
```
#### v1.0.0 Signature
@@ -227,7 +232,8 @@ def get_all(
agent_id: str = None,
run_id: str = None,
filters: dict = None # ✅ ENHANCED: Advanced operators
) -> dict # Always returns dict with "results" key
) -> dict: # Always returns dict with "results" key
...
```
### update() Method
@@ -239,7 +245,8 @@ def update(
self,
memory_id: str,
data: str
) -> dict
) -> dict:
...
```
### delete() Method
@@ -250,7 +257,8 @@ def update(
def delete(
self,
memory_id: str
) -> dict
) -> dict:
...
```
### delete_all() Method
@@ -344,7 +352,7 @@ config = {
### New Configuration Options
#### Reranker Configuration
```python
```text
# Cohere reranker
"reranker": {
"provider": "cohere",
@@ -563,4 +571,4 @@ results = m.search(
<Info>
Use this reference to systematically update your codebase. Test each change thoroughly before deploying to production.
</Info>
</Info>
+21 -8
View File
@@ -4861,8 +4861,7 @@
"items": {
"type": "object",
"required": [
"memory_id",
"text"
"memory_id"
],
"properties": {
"memory_id": {
@@ -4873,6 +4872,11 @@
"text": {
"type": "string",
"description": "The new text content for the memory"
},
"metadata": {
"type": "object",
"additionalProperties": true,
"description": "Updated metadata to associate with the memory."
}
}
},
@@ -4948,18 +4952,27 @@
"schema": {
"type": "object",
"properties": {
"memory_ids": {
"memories": {
"type": "array",
"items": {
"type": "string",
"format": "uuid"
"type": "object",
"properties": {
"memory_id": {
"type": "string",
"format": "uuid",
"description": "The unique identifier of the memory to delete."
}
},
"required": [
"memory_id"
]
},
"maxItems": 1000,
"description": "Array of memory IDs to delete."
"description": "Array of memory objects to delete."
}
},
"required": [
"memory_ids"
"memories"
]
}
}
@@ -6256,4 +6269,4 @@
}
},
"x-original-swagger-version": "2.0"
}
}
@@ -117,7 +117,7 @@ results_without_criteria = client.search(
### Compare Results
### Search Results (with Criteria)
```python
```text
[
{"memory": "User feels refreshed and ready to take on anything on a beautiful sunny day", "score": 0.666, ...},
{"memory": "User finally has time to draw something after a long time", "score": 0.616, ...},
@@ -128,7 +128,7 @@ results_without_criteria = client.search(
```
### Search Results (without Criteria)
```python
```text
[
{"memory": "User is happy today", "score": 0.607, ...},
{"memory": "User feels refreshed and ready to take on anything on a beautiful sunny day", "score": 0.512, ...},
+1 -1
View File
@@ -190,7 +190,7 @@ messages = [
client.add(messages, user_id='alice')
```
```python Memories with categories
```text Memories with categories
# Following categories will be created for the memories added
Sometimes draws and sketches in free time (hobbies)
Is quite athletic (sports)
+5 -5
View File
@@ -108,10 +108,10 @@ print(response)
```javascript JavaScript
// Basic Export request
const filters = {"user_id": "alice"};
const basicFilters = {"user_id": "alice"};
const response = await client.createMemoryExport({
schema: json_schema,
filters: filters
filters: basicFilters
});
// Export with custom instructions and additional filters
@@ -124,16 +124,16 @@ const export_instructions = `
`;
// For create operation, using only user_id filter as requested
const filters = {
const exportFilters = {
"AND": [
{"user_id": "alex"},
{"created_at": {"gte": "2024-01-01"}}
]
}
};
const responseWithInstructions = await client.createMemoryExport({
schema: json_schema,
filters: filters,
filters: exportFilters,
exportInstructions: export_instructions
});
@@ -1,6 +1,6 @@
{
"name": "mem0",
"version": "0.2.10",
"version": "0.2.11",
"description": "Persistent memory for Claude Code. Remembers decisions, patterns, and preferences across sessions.",
"author": {
"name": "Mem0",
@@ -1,6 +1,6 @@
{
"name": "mem0",
"version": "0.2.10",
"version": "0.2.11",
"description": "Persistent memory for Codex. Remembers decisions, patterns, and preferences across sessions.",
"author": {
"name": "Mem0",
@@ -1,6 +1,6 @@
{
"name": "mem0",
"version": "0.2.10",
"version": "0.2.11",
"description": "Mem0 memory layer for AI applications. Add persistent memory, personalization, and semantic search using the Mem0 Platform MCP server.",
"author": {
"name": "Mem0",
+12
View File
@@ -2,6 +2,18 @@
All notable changes to the Mem0 plugin will be documented in this file.
## 0.2.11 — Session-summary metadata fix + rerank auto-injected context by default
> Versions: Claude Code / Cursor / Codex `0.2.11`; Antigravity `0.1.3`. All four editors share `scripts/`, so the fix below applies to every editor.
### Fixed
- **`files_touched` was double-JSON-encoded in session summaries (`scripts/capture_session_summary.py`):** the Stop-hook summary set `metadata["files_touched"] = json.dumps(files[:20])` — a pre-serialized JSON string — and then serialized the whole request body again with `json.dumps(body)`. The stored memory therefore carried an escaped string blob (`"[\"mem0/memory/main.py\", \"src/client/index.ts\"]"`) instead of a real array, so file paths surfaced as backslash- and slash-heavy escaped text when those memories were returned by `search_memories`/`get_memories` and shown in Claude Code, Cursor, Codex, and Antigravity. The fix stores the list directly (`metadata["files_touched"] = files[:20]`) so the body is encoded exactly once. New `tests/test_capture_session_summary.py` asserts the posted body contains a JSON array and no escaped-string artifact.
### Changed
- **Auto-injected memory context is now reranked by default (`scripts/_search.py`, `scripts/file_context.py`, `scripts/on_bash_output.sh`, `scripts/on_user_prompt.sh`):** the REST search endpoint does not rerank when `rerank` is omitted, so hook-injected context (file-context, bash-error lookup, session-resume prefetch) was ordered by raw vector similarity and the single most relevant memory could fall outside the injected `top_k` window. A new `should_rerank()` helper turns reranking on for every auto-injection path; the extra ~150–200 ms stays within the hook's curl budget. Opt out with `MEM0_RERANK=0` (also accepts `false`/`no`/`off`). (#5690)
## 0.2.10 — Accurate per-editor telemetry attribution
### Fixed
+1 -1
View File
@@ -1,7 +1,7 @@
{
"id": "mem0",
"name": "mem0",
"version": "0.1.2",
"version": "0.1.3",
"description": "Persistent semantic memory for Antigravity agents. Cross-session, user-level recall via the Mem0 Platform MCP server. 16 slash commands, lifecycle hooks for auto-capture and metadata enforcement.",
"author": { "name": "Mem0", "email": "support@mem0.ai" },
"publisher": "mem0ai",
@@ -173,7 +173,7 @@ def store_summary(
if branch:
metadata["branch"] = branch
if files:
metadata["files_touched"] = json.dumps(files[:20])
metadata["files_touched"] = files[:20]
body = {
"messages": [{"role": "user", "content": summary_prompt}],
@@ -0,0 +1,79 @@
"""Regression tests for capture_session_summary.py request body construction.
Guards against the double-JSON-encoding bug where ``files_touched`` was stored
as a pre-serialized JSON string and then encoded a second time with the rest of
the request body — surfacing as escaped, slash-heavy blobs in the memories shown
inside Claude Code / Cursor / Codex / Antigravity (all four editors share this
script).
"""
from __future__ import annotations
import json
class _FakeResp:
status = 200
def __enter__(self):
return self
def __exit__(self, *_):
return False
def _capture_request_body(monkeypatch):
"""Patch urlopen so store_summary posts nowhere; capture the request body."""
import capture_session_summary as css
captured: dict = {}
def fake_urlopen(req, timeout=0):
captured["raw"] = req.data.decode("utf-8")
captured["body"] = json.loads(captured["raw"])
return _FakeResp()
monkeypatch.setattr(css.urllib.request, "urlopen", fake_urlopen)
return captured, css
def test_files_touched_is_json_array_not_double_encoded(monkeypatch):
"""files_touched must be a real JSON array, encoded exactly once."""
captured, css = _capture_request_body(monkeypatch)
files = ["mem0/memory/main.py", "src/client/index.ts"]
css.store_summary(
api_key="test-key",
summary_prompt="did some work",
user_id="u1",
session_id="s1",
project_id="p1",
branch="main",
files=files,
)
files_touched = captured["body"]["metadata"]["files_touched"]
assert isinstance(files_touched, list), (
"files_touched must be a JSON array, not a double-encoded string; "
f"got {type(files_touched).__name__}: {files_touched!r}"
)
assert files_touched == files
# The file paths must not appear as an escaped JSON string inside the body.
assert '\\"' not in captured["raw"]
def test_files_touched_omitted_when_no_files(monkeypatch):
"""No files touched -> no files_touched key (unchanged behaviour)."""
captured, css = _capture_request_body(monkeypatch)
css.store_summary(
api_key="test-key",
summary_prompt="did some work",
user_id="u1",
session_id="s1",
project_id="p1",
branch="main",
files=[],
)
assert "files_touched" not in captured["body"]["metadata"]
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mem0ai",
"version": "3.0.10",
"version": "3.0.11",
"description": "The Memory Layer For Your AI Apps",
"main": "./dist/index.js",
"module": "./dist/index.mjs",
+72 -10
View File
@@ -296,6 +296,44 @@ export class Memory {
return filters;
}
private _normalizeEntityText(value: string): string {
return value.trim().toLowerCase().replace(/\s+/g, " ");
}
private async _existingEntitiesByText(
entityStore: VectorStore,
filters: Record<string, any>,
): Promise<Map<string, { id: string; payload: Record<string, any> }>> {
const rowsByText = new Map<
string,
{ id: string; payload: Record<string, any> }
>();
let rows: Array<{ id: string; payload: Record<string, any> }> = [];
try {
const listed = await entityStore.list(filters, 10000);
rows = (
Array.isArray(listed) && Array.isArray(listed[0])
? listed[0]
: (listed as any)
) as Array<{ id: string; payload: Record<string, any> }>;
} catch (e) {
console.debug(
`Exact entity lookup failed, falling back to semantic dedup: ${e}`,
);
return rowsByText;
}
for (const row of rows) {
const text = row.payload?.data;
if (typeof text !== "string") continue;
const key = this._normalizeEntityText(text);
if (key && !rowsByText.has(key)) {
rowsByText.set(key, row);
}
}
return rowsByText;
}
/**
* Remove `memoryId` from every entity record scoped to `filters`.
* If an entity's `linkedMemoryIds` becomes empty after removal, the
@@ -393,6 +431,10 @@ export class Memory {
if (entities.length === 0) return;
const entityStore = await this.getEntityStore();
const exactMatches = await this._existingEntitiesByText(
entityStore,
filters,
);
for (const entity of entities) {
try {
@@ -409,12 +451,21 @@ export class Memory {
score?: number;
payload: Record<string, any>;
}> = [];
try {
matches = await entityStore.search(entityVec, 1, filters);
} catch {}
const exactMatch = exactMatches.get(
this._normalizeEntityText(entity.text),
);
if (!exactMatch) {
try {
matches = await entityStore.search(entityVec, 1, filters);
} catch {}
}
if (matches.length > 0 && (matches[0].score ?? 0) >= 0.95) {
const match = matches[0];
const semanticMatch =
matches.length > 0 && (matches[0].score ?? 0) >= 0.95
? matches[0]
: undefined;
const match = exactMatch ?? semanticMatch;
if (match) {
const payload = match.payload || {};
const linked = new Set<string>(
Array.isArray(payload.linkedMemoryIds)
@@ -1062,6 +1113,10 @@ export class Memory {
if (valid.length > 0) {
const entityStore = await this.getEntityStore();
const exactMatches = await this._existingEntitiesByText(
entityStore,
filters,
);
// 7c: Search for existing entities one by one (no batch search)
const toInsertVectors: number[][] = [];
@@ -1077,13 +1132,20 @@ export class Memory {
score?: number;
payload: Record<string, any>;
}> = [];
try {
matches = await entityStore.search(entityVec, 1, filters);
} catch {}
const exactMatch = exactMatches.get(key);
if (!exactMatch) {
try {
matches = await entityStore.search(entityVec, 1, filters);
} catch {}
}
if (matches.length > 0 && (matches[0].score ?? 0) >= 0.95) {
const semanticMatch =
matches.length > 0 && (matches[0].score ?? 0) >= 0.95
? matches[0]
: undefined;
const match = exactMatch ?? semanticMatch;
if (match) {
// Update existing entity
const match = matches[0];
const payload = match.payload || {};
const linked = new Set<string>(payload.linkedMemoryIds ?? []);
for (const mid of memoryIds) linked.add(mid);
+227 -122
View File
@@ -4,8 +4,8 @@
* Extracts four types of entities from text:
* - PROPER: Capitalized multi-word sequences (person names, places, brands)
* - QUOTED: Text in single or double quotes (titles, specific terms)
* - COMPOUND: Multi-word noun phrases with specific modifiers (e.g., "machine learning")
* - NOUN: Single nouns from circumstantial compound patterns
* - TOPIC: Multi-word noun/topic phrases with specific modifiers
* - IDENTIFIER: Dotted technical identifiers such as person.properties.email
*
* Uses the `compromise` npm package for NLP-based extraction when available.
* Falls back to regex-only extraction if `compromise` is not installed.
@@ -196,6 +196,25 @@ const NON_SPECIFIC_ADJ: Set<string> = new Set([
"final",
"initial",
"side",
"top",
]);
/** Leading words that frame a topic but are not part of the topic itself. */
const TOPIC_PREFIX_WORDS: Set<string> = new Set([
"a",
"an",
"the",
"my",
"your",
"our",
"their",
"his",
"her",
"its",
"this",
"that",
"these",
"those",
]);
/** Generic tail words to strip from compound entities. */
@@ -267,6 +286,25 @@ const GENERIC_CAPS: Set<string> = new Set([
"disadvantages",
]);
/** Generic role/title words that should not become single-token entities. */
const GENERIC_SINGLE_ENTITY_TERMS: Set<string> = new Set([
"user",
"assistant",
"agent",
"customer",
"client",
"person",
"people",
"human",
"memory",
"message",
"conversation",
"chat",
"session",
"system",
"top",
]);
/** Markdown/formatting markers to skip during extraction. */
const FORMATTING_MARKERS: Set<string> = new Set([
"*",
@@ -287,7 +325,7 @@ const FORMATTING_MARKERS: Set<string> = new Set([
// ---------------------------------------------------------------------------
export interface ExtractedEntity {
type: "PROPER" | "QUOTED" | "COMPOUND" | "NOUN";
type: "PROPER" | "QUOTED" | "TOPIC" | "IDENTIFIER";
text: string;
}
@@ -338,32 +376,96 @@ function stripGenericEnding(words: string[]): string[] {
return words;
}
/**
* Determine if a token position is at the start of a sentence.
* Simple heuristic: index 0, or preceded by sentence-ending punctuation
* or formatting markers.
*/
function isSentenceStart(
tokens: string[],
idx: number,
rawText: string,
): boolean {
if (idx === 0) {
return true;
function stripTopicPrefix(words: string[]): string[] {
let start = 0;
while (
start < words.length &&
TOPIC_PREFIX_WORDS.has(words[start].toLowerCase())
) {
start++;
}
const prev = tokens[idx - 1];
if (/[.!?:]$/.test(prev)) {
return words.slice(start);
}
function cleanToken(token: string): string {
return token.replace(/^[^\w.]+|[^\w.]+$/g, "");
}
function tokenize(text: string): string[] {
return (
text.match(
/[A-Za-z_][\w-]*(?:\.[A-Za-z_][\w-]*)*|\d[\d,]*(?:\.\d+)?|[,:;.!?&]/g,
) ?? []
);
}
function isCapitalized(token: string): boolean {
return /^[A-Z]/.test(token) && /[A-Za-z]/.test(token);
}
function hasInternalCapOrDigit(token: string): boolean {
return (
/\d/.test(token) ||
/[A-Z]/.test(token.slice(1)) ||
/^[A-Z]{2,}$/.test(token)
);
}
function isBadSingleNameToken(token: string): boolean {
const lower = token.toLowerCase();
return GENERIC_SINGLE_ENTITY_TERMS.has(lower) || GENERIC_CAPS.has(lower);
}
function looksLikeMetricCount(token: string): boolean {
return /^\d[\d,]*(?:\.\d+)?$/.test(token);
}
function isMetricListContext(tokens: string[], idx: number): boolean {
const prev = idx > 0 ? tokens[idx - 1] : "";
const next = idx + 1 < tokens.length ? tokens[idx + 1] : "";
return [":", ",", ";"].includes(prev) || [",", ";"].includes(next);
}
function isSentenceStart(tokens: string[], idx: number): boolean {
if (idx === 0) return true;
return (
[".", "!", "?", ":"].includes(tokens[idx - 1]) ||
FORMATTING_MARKERS.has(tokens[idx - 1])
);
}
function isListItemNameToken(tokens: string[], idx: number): boolean {
const token = cleanToken(tokens[idx]);
if (!isCapitalized(token) || isBadSingleNameToken(token)) return false;
const next = idx + 1 < tokens.length ? cleanToken(tokens[idx + 1]) : "";
if (!looksLikeMetricCount(next)) return false;
return (
isMetricListContext(tokens, idx) || isMetricListContext(tokens, idx + 1)
);
}
function isNameToken(tokens: string[], idx: number): boolean {
const token = cleanToken(tokens[idx]);
if (!token || !isCapitalized(token) || isBadSingleNameToken(token))
return false;
if (hasInternalCapOrDigit(token) || isListItemNameToken(tokens, idx))
return true;
}
if (FORMATTING_MARKERS.has(prev)) {
return true;
}
// Check for newline before this token in the raw text
const tokenStart = rawText.indexOf(tokens[idx]);
if (tokenStart > 0 && rawText.charAt(tokenStart - 1) === "\n") {
return true;
}
return false;
return !isSentenceStart(tokens, idx);
}
function cleanEntityText(text: string): string {
return text
.replace(/^\*+\s*|\s*\*+$/g, "")
.replace(/\s*:+$/g, "")
.replace(/^\d+\s*\.\s*/, "")
.replace(/\s+\d[\d,]*(?:\.\d+)?$/g, "")
.replace(/[.,;!?]+$/, "")
.trim()
.replace(/\s+/g, " ");
}
function isCoordinatedNameTopic(text: string): boolean {
return /\b[A-Z][\w-]+\s+and\s+[A-Z][\w-]+\b/.test(text);
}
// ---------------------------------------------------------------------------
@@ -397,86 +499,79 @@ function extractQuoted(text: string): ExtractedEntity[] {
}
/**
* Extract proper noun sequences using capitalization heuristics.
* Finds sequences of capitalized words that are not at sentence starts.
* Extract dotted technical identifiers such as person.properties.email.
*/
function extractIdentifiers(text: string): ExtractedEntity[] {
const entities: ExtractedEntity[] = [];
const identifierRe = /\b[A-Za-z_][\w-]*(?:\.[A-Za-z_][\w-]*)+\b/g;
let match: RegExpExecArray | null;
while ((match = identifierRe.exec(text)) !== null) {
entities.push({ type: "IDENTIFIER", text: match[0] });
}
return entities;
}
/**
* Extract proper names using capitalization and list-context heuristics.
*/
function extractProper(text: string): ExtractedEntity[] {
const entities: ExtractedEntity[] = [];
// Tokenize on whitespace, preserving order
const tokens = text.split(/\s+/).filter(Boolean);
const functionWords = new Set([
"'s",
"of",
"the",
"in",
"and",
"for",
"at",
"is",
]);
const tokens = tokenize(text);
const innerConnectors = new Set(["of", "the", "in", "for", "at"]);
let i = 0;
while (i < tokens.length) {
const tok = tokens[i];
// Skip formatting markers
if (FORMATTING_MARKERS.has(tok)) {
const token = cleanToken(tokens[i]);
const next = i + 1 < tokens.length ? tokens[i + 1] : "";
const afterNext = i + 2 < tokens.length ? cleanToken(tokens[i + 2]) : "";
if (
token &&
next === "&" &&
afterNext &&
isCapitalized(token) &&
isCapitalized(afterNext) &&
!isBadSingleNameToken(token) &&
!isBadSingleNameToken(afterNext)
) {
entities.push({
type: "PROPER",
text: cleanEntityText(`${token} & ${afterNext}`),
});
i += 3;
continue;
}
if (!isNameToken(tokens, i)) {
i++;
continue;
}
const isLabel = i + 1 < tokens.length && tokens[i + 1] === ":";
const isCap =
tok.length > 0 &&
tok.charAt(0) === tok.charAt(0).toUpperCase() &&
/[A-Z]/.test(tok.charAt(0));
if (isCap && !isLabel) {
const seq: Array<{ token: string; idx: number }> = [
{ token: tok, idx: i },
];
let j = i + 1;
while (j < tokens.length) {
const t = tokens[j];
const tIsCap =
t.length > 0 &&
t.charAt(0) === t.charAt(0).toUpperCase() &&
/[A-Z]/.test(t.charAt(0));
if (tIsCap || functionWords.has(t.toLowerCase())) {
seq.push({ token: t, idx: j });
j++;
} else {
break;
}
const span = [cleanToken(tokens[i])];
let j = i + 1;
while (j < tokens.length) {
const current = cleanToken(tokens[j]);
if (isNameToken(tokens, j)) {
span.push(current);
j++;
continue;
}
// Strip trailing function words
while (
seq.length > 0 &&
functionWords.has(seq[seq.length - 1].token.toLowerCase())
if (
innerConnectors.has(current.toLowerCase()) &&
j + 1 < tokens.length &&
isNameToken(tokens, j + 1)
) {
seq.pop();
span.push(current, cleanToken(tokens[j + 1]));
j += 2;
continue;
}
if (seq.length > 0) {
// Check for at least one mid-sentence capitalized word
const hasMidCap = seq.some(({ token, idx: tokenIdx }) => {
const isCapWord =
/[A-Z]/.test(token.charAt(0)) &&
!functionWords.has(token.toLowerCase());
return isCapWord && !isSentenceStart(tokens, tokenIdx, text);
});
if (hasMidCap) {
const phrase = seq.map((s) => s.token).join(" ");
if (phrase.length > 2) {
entities.push({ type: "PROPER", text: phrase });
}
}
}
i = j;
} else {
i++;
break;
}
const phrase = cleanEntityText(span.join(" "));
if (phrase.length > 2) {
entities.push({ type: "PROPER", text: phrase });
}
i = Math.max(j, i + 1);
}
return entities;
@@ -484,7 +579,7 @@ function extractProper(text: string): ExtractedEntity[] {
/**
* Extract compound noun phrases using the `compromise` NLP library.
* Returns COMPOUND and NOUN entities derived from noun chunks.
* Returns TOPIC entities derived from noun chunks.
*/
function extractCompoundsWithNlp(text: string): ExtractedEntity[] {
if (!nlp) {
@@ -524,12 +619,12 @@ function extractCompoundsWithNlp(text: string): ExtractedEntity[] {
const filtered = words.filter(
(w) => !NON_SPECIFIC_ADJ.has(w.toLowerCase()),
);
const cleaned = stripGenericEnding(filtered);
const cleaned = stripGenericEnding(stripTopicPrefix(filtered));
if (cleaned.length >= 2) {
const phrase = cleaned.join(" ");
const phrase = cleanEntityText(cleaned.join(" "));
if (phrase.length > 3) {
entities.push({ type: "COMPOUND", text: phrase });
entities.push({ type: "TOPIC", text: phrase });
}
}
}
@@ -547,7 +642,7 @@ function extractCompoundsRegex(text: string): ExtractedEntity[] {
// Multi-word sequences with at least one non-trivial word
// Match sequences like "machine learning", "New York", "data science"
const compoundRe =
/\b([A-Z][a-z]+(?:\s+(?:of|and|the|for|in)\s+)?[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)\b/g;
/\b([A-Z][a-z]+(?:\s+(?:of|the|for|in)\s+)?[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)\b/g;
let match: RegExpExecArray | null;
while ((match = compoundRe.exec(text)) !== null) {
const phrase = match[1].trim();
@@ -558,9 +653,12 @@ function extractCompoundsRegex(text: string): ExtractedEntity[] {
const filtered = words.filter(
(w) => !NON_SPECIFIC_ADJ.has(w.toLowerCase()),
);
const cleaned = stripGenericEnding(filtered);
const cleaned = stripGenericEnding(stripTopicPrefix(filtered));
if (cleaned.length >= 2) {
entities.push({ type: "COMPOUND", text: cleaned.join(" ") });
entities.push({
type: "TOPIC",
text: cleanEntityText(cleaned.join(" ")),
});
}
}
}
@@ -590,9 +688,12 @@ function extractCompoundsRegex(text: string): ExtractedEntity[] {
const filtered = words.filter(
(w) => !NON_SPECIFIC_ADJ.has(w.toLowerCase()),
);
const cleaned = stripGenericEnding(filtered);
const cleaned = stripGenericEnding(stripTopicPrefix(filtered));
if (cleaned.length >= 2) {
entities.push({ type: "COMPOUND", text: cleaned.join(" ") });
entities.push({
type: "TOPIC",
text: cleanEntityText(cleaned.join(" ")),
});
}
}
}
@@ -614,9 +715,9 @@ function extractCompoundsRegex(text: string): ExtractedEntity[] {
*
* Entity types (in priority order for deduplication):
* PROPER - Capitalized multi-word sequences not at sentence start
* COMPOUND - Multi-word noun phrases with specific modifiers
* IDENTIFIER - Dotted technical identifiers
* QUOTED - Text in single or double quotes (min 3 chars)
* NOUN - Single nouns from circumstantial patterns
* TOPIC - Multi-word noun/topic phrases with specific modifiers
*
* @param text - Input text to extract entities from.
* @returns Deduplicated list of extracted entities.
@@ -630,7 +731,10 @@ export function extractEntities(text: string): ExtractedEntity[] {
// 2. PROPER entities (capitalization heuristics)
raw.push(...extractProper(text));
// 3. COMPOUND entities (NLP or regex fallback)
// 3. IDENTIFIER entities
raw.push(...extractIdentifiers(text));
// 4. TOPIC entities (NLP or regex fallback)
if (nlp) {
raw.push(...extractCompoundsWithNlp(text));
} else {
@@ -654,19 +758,17 @@ export function extractEntities(text: string): ExtractedEntity[] {
const cleaned: ExtractedEntity[] = [];
for (const entity of deduped) {
let txt = entity.text.trim();
// Strip leading/trailing asterisks
txt = txt.replace(/^\*+\s*|\s*\*+$/g, "");
// Strip trailing colons
txt = txt.replace(/\s*:+$/, "");
// Strip leading numbered list markers
txt = txt.replace(/^\d+\s*\.\s*/, "");
// Strip trailing sentence punctuation (".", ",", ";", "!", "?") — otherwise
// "Paris." and "Paris" produce different embeddings and break entity dedup.
txt = txt.replace(/[.,;!?]+$/, "").trim();
txt = cleanEntityText(txt);
if (!txt || txt.length <= 2 || hasArtifacts(txt)) {
continue;
}
if (
entity.type === "TOPIC" &&
(/^\d/.test(txt) || isCoordinatedNameTopic(txt))
) {
continue;
}
// Filter generic single-word PROPER nouns
if (
@@ -680,12 +782,12 @@ export function extractEntities(text: string): ExtractedEntity[] {
cleaned.push({ type: entity.type, text: txt });
}
// Keep best type per entity (PROPER > COMPOUND > QUOTED > NOUN)
// Keep best type per entity (PROPER > IDENTIFIER > QUOTED > TOPIC)
const typePriority: Record<string, number> = {
PROPER: 0,
COMPOUND: 1,
IDENTIFIER: 1,
QUOTED: 2,
NOUN: 3,
TOPIC: 3,
};
const best = new Map<string, ExtractedEntity>();
for (const entity of cleaned) {
@@ -700,14 +802,17 @@ export function extractEntities(text: string): ExtractedEntity[] {
}
const bestEntities = Array.from(best.values());
// Remove entities that are substrings of longer entities
const allLower = bestEntities.map((e) => e.text.toLowerCase());
// Remove entities that are token substrings of longer entities.
return bestEntities.filter(
(entity) =>
!allLower.some(
!bestEntities.some(
(other) =>
entity.text.toLowerCase() !== other &&
other.includes(entity.text.toLowerCase()),
entity.text.toLowerCase() !== other.text.toLowerCase() &&
(typePriority[entity.type] ?? 99) >=
(typePriority[other.type] ?? 99) &&
new RegExp(
`(^|\\s)${entity.text.toLowerCase().replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}(\\s|$)`,
).test(other.text.toLowerCase()),
),
);
}
@@ -0,0 +1,45 @@
import { extractEntities } from "../src/utils/entity_extraction";
describe("extractEntities", () => {
it("handles product lists, coordinated names, and identifiers", () => {
const text =
"User reported top inbound integration pages: OpenClaw 25,443, " +
"Claude Code 8,916, Codex 2,573, Dify 656. " +
"User compared Cartesia and Deepgram. " +
"The email field for Mem0 lives at person.properties.email. " +
"The qwen endpoint uses person.properties.email. " +
"Johnson & Johnson was mentioned. " +
"Glasses around my window. " +
"On 2026-05-27 there were 90 days of stats.";
const entityTexts = new Set(
extractEntities(text).map((entity) => entity.text),
);
const normalized = new Set(
[...entityTexts].map((entityText) => entityText.toLowerCase()),
);
for (const expected of [
"OpenClaw",
"Claude Code",
"Codex",
"Dify",
"Cartesia",
"Deepgram",
"Mem0",
]) {
expect(entityTexts.has(expected)).toBe(true);
}
expect(entityTexts.has("person.properties.email")).toBe(true);
expect(entityTexts.has("qwen endpoint")).toBe(true);
expect(entityTexts.has("Johnson & Johnson")).toBe(true);
expect(entityTexts.has("Johnson")).toBe(false);
expect(normalized.has("top")).toBe(false);
expect(normalized.has("glasses")).toBe(false);
expect(entityTexts.has("Cartesia and Deepgram")).toBe(false);
expect(entityTexts.has("Claude Code 8,916")).toBe(false);
for (const rejected of ["8,916", "2,573", "656", "2026-05-27", "90"]) {
expect(entityTexts.has(rejected)).toBe(false);
}
});
});
+97 -27
View File
@@ -81,6 +81,14 @@ warnings.filterwarnings("ignore", category=DeprecationWarning, message=".*swigva
logger = logging.getLogger(__name__)
def _vector_store_list_rows(listed):
if isinstance(listed, (list, tuple)) and listed and isinstance(listed[0], list):
return listed[0]
if isinstance(listed, (list, tuple)):
return listed
return []
# Fields that hold runtime auth/connection objects and must be preserved.
# These are non-serializable objects (e.g. AWSV4SignerAuth, RequestsHttpConnection)
# needed by clients like OpenSearch — not sensitive strings to redact.
@@ -499,22 +507,49 @@ class Memory(MemoryBase):
)
return self._entity_store
@staticmethod
def _normalize_entity_text(value: str) -> str:
return " ".join(value.strip().lower().split())
def _existing_entities_by_text(self, filters):
"""Return existing entity rows keyed by normalized payload data."""
try:
listed = self.entity_store.list(filters=filters, top_k=10000)
except Exception as e:
logger.debug(f"Exact entity lookup failed, falling back to semantic dedup: {e}")
return {}
rows_by_text = {}
for row in _vector_store_list_rows(listed):
payload = getattr(row, "payload", None) or {}
text = payload.get("data")
if not isinstance(text, str):
continue
normalized = self._normalize_entity_text(text)
if normalized and normalized not in rows_by_text:
rows_by_text[normalized] = row
return rows_by_text
def _upsert_entity(self, entity_text, entity_type, memory_id, filters):
"""Upsert an entity into the entity store, linking it to a memory."""
try:
entity_embedding = self.embedding_model.embed(entity_text, "add")
search_filters = {k: v for k, v in filters.items() if k in ("user_id", "agent_id", "run_id") and v}
exact_match = self._existing_entities_by_text(search_filters).get(self._normalize_entity_text(entity_text))
existing = self.entity_store.search(
query=entity_text,
vectors=entity_embedding,
top_k=1,
filters=search_filters,
)
existing = []
if exact_match is None:
existing = self.entity_store.search(
query=entity_text,
vectors=entity_embedding,
top_k=1,
filters=search_filters,
)
if existing and existing[0].score >= 0.95:
semantic_match = existing[0] if existing and existing[0].score >= 0.95 else None
match = exact_match or semantic_match
if match:
# Update existing entity's linked_memory_ids
match = existing[0]
payload = match.payload or {}
linked_ids = payload.get("linked_memory_ids", [])
if memory_id not in linked_ids:
@@ -609,7 +644,7 @@ class Memory(MemoryBase):
return
seen = set()
for entity_type, entity_text in entities:
key = entity_text.strip().lower()
key = self._normalize_entity_text(entity_text)
if not key or key in seen:
continue
seen.add(key)
@@ -971,7 +1006,7 @@ class Memory(MemoryBase):
for idx, (memory_id, text, embedding, payload) in enumerate(records):
entities = all_entities[idx] if idx < len(all_entities) else []
for entity_type, entity_text in entities:
key = entity_text.strip().lower()
key = self._normalize_entity_text(entity_text)
if key in global_entities:
global_entities[key][2].add(memory_id)
else:
@@ -1009,6 +1044,7 @@ class Memory(MemoryBase):
if valid:
valid_indices, valid_keys = zip(*valid)
valid_vectors = [entity_embeddings[i] for i in valid_indices]
exact_matches = self._existing_entities_by_text(search_filters)
# 7c: Batch search for existing entities
valid_texts = [global_entities[k][1] for k in valid_keys]
@@ -1024,10 +1060,12 @@ class Memory(MemoryBase):
for j, key in enumerate(valid_keys):
entity_type, entity_text, memory_ids = global_entities[key]
matches = existing_matches[j] if j < len(existing_matches) else []
exact_match = exact_matches.get(key)
if matches and matches[0].score >= 0.95:
semantic_match = matches[0] if matches and matches[0].score >= 0.95 else None
match = exact_match or semantic_match
if match:
# Update existing entity
match = matches[0]
payload = match.payload or {}
linked = set(payload.get("linked_memory_ids", []))
linked |= memory_ids
@@ -1603,7 +1641,7 @@ class Memory(MemoryBase):
seen = set()
deduped = []
for entity_type, entity_text in query_entities[:8]:
key = entity_text.strip().lower()
key = self._normalize_entity_text(entity_text)
if key and key not in seen:
seen.add(key)
deduped.append((entity_type, entity_text))
@@ -2045,22 +2083,51 @@ class AsyncMemory(MemoryBase):
)
return self._entity_store
@staticmethod
def _normalize_entity_text(value: str) -> str:
return " ".join(value.strip().lower().split())
def _existing_entities_by_text(self, filters):
"""Return existing entity rows keyed by normalized payload data."""
try:
listed = self.entity_store.list(filters=filters, top_k=10000)
except Exception as e:
logger.debug(f"Exact entity lookup failed, falling back to semantic dedup: {e}")
return {}
rows_by_text = {}
for row in _vector_store_list_rows(listed):
payload = getattr(row, "payload", None) or {}
text = payload.get("data")
if not isinstance(text, str):
continue
normalized = self._normalize_entity_text(text)
if normalized and normalized not in rows_by_text:
rows_by_text[normalized] = row
return rows_by_text
async def _upsert_entity_async(self, entity_text, entity_type, memory_id, filters):
"""Async variant of `_upsert_entity` — per-entity search-then-update-or-insert."""
try:
entity_embedding = await asyncio.to_thread(self.embedding_model.embed, entity_text, "add")
search_filters = {k: v for k, v in filters.items() if k in ("user_id", "agent_id", "run_id") and v}
exact_match = (
await asyncio.to_thread(self._existing_entities_by_text, search_filters)
).get(self._normalize_entity_text(entity_text))
existing = await asyncio.to_thread(
self.entity_store.search,
query=entity_text,
vectors=entity_embedding,
top_k=1,
filters=search_filters,
)
existing = []
if exact_match is None:
existing = await asyncio.to_thread(
self.entity_store.search,
query=entity_text,
vectors=entity_embedding,
top_k=1,
filters=search_filters,
)
if existing and existing[0].score >= 0.95:
match = existing[0]
semantic_match = existing[0] if existing and existing[0].score >= 0.95 else None
match = exact_match or semantic_match
if match:
payload = match.payload or {}
linked_ids = payload.get("linked_memory_ids", [])
if memory_id not in linked_ids:
@@ -2163,7 +2230,7 @@ class AsyncMemory(MemoryBase):
return
seen = set()
for entity_type, entity_text in entities:
key = entity_text.strip().lower()
key = self._normalize_entity_text(entity_text)
if not key or key in seen:
continue
seen.add(key)
@@ -2512,7 +2579,7 @@ class AsyncMemory(MemoryBase):
for idx, (memory_id, text, embedding, payload) in enumerate(records):
entities = all_entities[idx] if idx < len(all_entities) else []
for entity_type, entity_text in entities:
key = entity_text.strip().lower()
key = self._normalize_entity_text(entity_text)
if key in global_entities:
global_entities[key][2].add(memory_id)
else:
@@ -2547,6 +2614,7 @@ class AsyncMemory(MemoryBase):
if valid:
valid_indices, valid_keys = zip(*valid)
valid_vectors = [entity_embeddings[i] for i in valid_indices]
exact_matches = await asyncio.to_thread(self._existing_entities_by_text, search_filters)
# 7c: Batch search for existing entities
valid_texts = [global_entities[k][1] for k in valid_keys]
@@ -2563,9 +2631,11 @@ class AsyncMemory(MemoryBase):
for j, key in enumerate(valid_keys):
entity_type, entity_text, memory_ids = global_entities[key]
matches = existing_matches[j] if j < len(existing_matches) else []
exact_match = exact_matches.get(key)
if matches and matches[0].score >= 0.95:
match = matches[0]
semantic_match = matches[0] if matches and matches[0].score >= 0.95 else None
match = exact_match or semantic_match
if match:
payload = match.payload or {}
linked = set(payload.get("linked_memory_ids", []))
linked |= memory_ids
@@ -3137,7 +3207,7 @@ class AsyncMemory(MemoryBase):
seen = set()
deduped = []
for entity_type, entity_text in query_entities[:8]:
key = entity_text.strip().lower()
key = self._normalize_entity_text(entity_text)
if key and key not in seen:
seen.add(key)
deduped.append((entity_type, entity_text))
+606 -197
View File
@@ -1,82 +1,352 @@
"""
Entity extraction from text using spaCy NLP.
Extracts four types of entities from text:
Extracts three types of entities from a spaCy-processed document:
- **Proper nouns**: Capitalized multi-word sequences (person names, places, brands)
- **Quoted text**: Text in single or double quotes (titles, specific terms)
- **Noun compounds**: Multi-word noun phrases with specific modifiers (e.g., "machine learning")
- **Noun fallback**: Single nouns from circumstantial compound patterns
Public API:
extract_entities(text: str) -> List[Tuple[str, str]]
``extract_entities(text)`` accepts a string and owns spaCy model loading.
``extract_entities_batch(texts)`` uses ``nlp.pipe`` for batched extraction.
Internal:
_extract_entities_from_doc(doc) -> List[Tuple[str, str]]
Returns:
List of ``(entity_type, entity_text)`` tuples where entity_type is one of
PROPER, QUOTED, TOPIC, or IDENTIFIER. Returns ``[]`` if spaCy is unavailable.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
import re
from typing import List, Tuple
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class _EntityCandidate:
entity_type: str
text: str
source: str
start: int
end: int
confidence: float
priority: int
# Words that are too generic to be useful as entity heads
_GENERIC_HEADS = {
"thing", "stuff", "way", "time", "experience", "situation", "case",
"fact", "matter", "issue", "idea", "thought", "feeling", "place",
"area", "part", "kind", "type", "sort", "lot", "bit", "day", "year",
"week", "month", "moment", "instance", "example", "technique",
"method", "approach", "process", "step", "tool", "result", "outcome",
"goal", "task", "item", "topic", "scale", "size", "level", "degree",
"amount", "number", "style", "look", "color", "colour", "shape",
"form", "piece", "section", "side", "end", "edge", "surface", "point",
"thing",
"stuff",
"way",
"time",
"experience",
"situation",
"case",
"fact",
"matter",
"issue",
"idea",
"thought",
"feeling",
"place",
"area",
"part",
"kind",
"type",
"sort",
"lot",
"bit",
"day",
"year",
"week",
"month",
"moment",
"instance",
"example",
"technique",
"method",
"approach",
"process",
"step",
"tool",
"result",
"outcome",
"goal",
"task",
"item",
"topic",
"scale",
"size",
"level",
"degree",
"amount",
"number",
"style",
"look",
"color",
"colour",
"shape",
"form",
"piece",
"section",
"side",
"end",
"edge",
"surface",
"point",
}
# Entity labels emitted by spaCy that are usually safe to treat as named
# entities. Numeric and temporal labels are intentionally excluded.
_ACCEPTED_NER_LABELS = {
"PERSON",
"ORG",
"GPE",
"LOC",
"FAC",
"PRODUCT",
"WORK_OF_ART",
"EVENT",
"NORP",
"LAW",
"LANGUAGE",
}
_REJECTED_NER_LABELS = {
"DATE",
"TIME",
"CARDINAL",
"ORDINAL",
"QUANTITY",
"MONEY",
"PERCENT",
}
# Generic role words and title-cased English words that should not become
# single-token named entities just because spaCy tagged them as PROPN.
_GENERIC_SINGLE_ENTITY_TERMS = {
"user",
"assistant",
"agent",
"customer",
"client",
"person",
"people",
"human",
"memory",
"message",
"conversation",
"chat",
"session",
"system",
"top",
}
# Modifiers that describe circumstance, not content
_CIRCUMSTANTIAL_MODS = {
"solo", "individual", "team", "group", "joint", "collaborative",
"first", "last", "next", "previous", "final", "initial", "main", "side",
"solo",
"individual",
"team",
"group",
"joint",
"collaborative",
"first",
"last",
"next",
"previous",
"final",
"initial",
"main",
"side",
"top",
}
# Adjectives too vague to make a compound entity specific
_NON_SPECIFIC_ADJ = {
"many", "few", "several", "some", "any", "all", "most", "more",
"less", "much", "little", "enough", "various", "numerous", "multiple",
"countless", "great", "good", "bad", "nice", "terrible", "awful",
"awesome", "amazing", "wonderful", "horrible", "excellent", "poor",
"best", "worst", "fine", "okay", "new", "old", "recent", "past",
"future", "current", "previous", "next", "last", "first", "latest",
"early", "late", "former", "modern", "ancient", "big", "small",
"large", "tiny", "huge", "enormous", "long", "short", "tall", "high",
"low", "wide", "narrow", "thick", "thin", "deep", "shallow",
"similar", "different", "same", "other", "another", "such", "certain",
"important", "main", "major", "minor", "key", "primary", "real",
"actual", "true", "whole", "entire", "full", "complete", "total",
"basic", "simple", "interesting", "boring", "exciting", "special",
"particular", "general", "common", "unique", "rare", "typical",
"usual", "normal", "regular", "possible", "likely", "potential",
"available", "necessary", "only", "solo", "individual", "team",
"group", "joint", "collaborative", "final", "initial", "side",
"many",
"few",
"several",
"some",
"any",
"all",
"most",
"more",
"less",
"much",
"little",
"enough",
"various",
"numerous",
"multiple",
"countless",
"great",
"good",
"bad",
"nice",
"terrible",
"awful",
"awesome",
"amazing",
"wonderful",
"horrible",
"excellent",
"poor",
"best",
"worst",
"fine",
"okay",
"new",
"old",
"recent",
"past",
"future",
"current",
"previous",
"next",
"last",
"first",
"latest",
"early",
"late",
"former",
"modern",
"ancient",
"big",
"small",
"large",
"tiny",
"huge",
"enormous",
"long",
"short",
"tall",
"high",
"low",
"wide",
"narrow",
"thick",
"thin",
"deep",
"shallow",
"similar",
"different",
"same",
"other",
"another",
"such",
"certain",
"important",
"main",
"major",
"minor",
"key",
"primary",
"real",
"actual",
"true",
"whole",
"entire",
"full",
"complete",
"total",
"basic",
"simple",
"interesting",
"boring",
"exciting",
"special",
"particular",
"general",
"common",
"unique",
"rare",
"typical",
"usual",
"normal",
"regular",
"possible",
"likely",
"potential",
"available",
"necessary",
"only",
"solo",
"individual",
"team",
"group",
"joint",
"collaborative",
"final",
"initial",
"side",
}
# Generic tail words to strip from compound entities
_GENERIC_ENDINGS = {
"work", "works", "job", "jobs", "task", "tasks", "stuff", "things",
"thing", "info", "information", "details", "data", "content",
"material", "materials", "activities", "activity", "efforts", "effort",
"options", "option", "choices", "choice", "results", "result",
"output", "outputs", "products", "product", "items", "item",
"work",
"works",
"job",
"jobs",
"task",
"tasks",
"stuff",
"things",
"thing",
"info",
"information",
"details",
"data",
"content",
"material",
"materials",
"activities",
"activity",
"efforts",
"effort",
"options",
"option",
"choices",
"choice",
"results",
"result",
"output",
"outputs",
"products",
"product",
"items",
"item",
}
# Capitalized single words that are too generic to be proper nouns
_GENERIC_CAPS = {
"works", "items", "things", "stuff", "resources", "options", "tips",
"ideas", "steps", "ways", "methods", "tools", "features", "benefits",
"examples", "details", "notes", "instructions", "guidelines",
"recommendations", "suggestions", "overview", "summary", "conclusion",
"introduction", "pros", "cons", "advantages", "disadvantages",
"works",
"items",
"things",
"stuff",
"resources",
"options",
"tips",
"ideas",
"steps",
"ways",
"methods",
"tools",
"features",
"benefits",
"examples",
"details",
"notes",
"instructions",
"guidelines",
"recommendations",
"suggestions",
"overview",
"summary",
"conclusion",
"introduction",
"pros",
"cons",
"advantages",
"disadvantages",
}
# Markdown/formatting markers to skip during extraction
@@ -120,121 +390,208 @@ def _has_artifacts(txt: str) -> bool:
)
def extract_entities(text: str) -> List[Tuple[str, str]]:
"""Extract named entities, quoted text, and noun compounds from text.
This is the public API that accepts a string. It loads the spaCy model
internally and delegates to _extract_entities_from_doc().
Args:
text: Input text to extract entities from.
Returns:
Deduplicated list of (entity_type, entity_text) tuples.
Entity types: PROPER, QUOTED, COMPOUND, NOUN.
Returns empty list if spaCy is unavailable.
"""
from mem0.utils.spacy_models import get_nlp_full
nlp = get_nlp_full()
if nlp is None:
return []
doc = nlp(text)
return _extract_entities_from_doc(doc)
def _clean_text(txt: str) -> str:
txt = re.sub(r"^\*+\s*|\s*\*+$", "", txt.strip())
txt = re.sub(r"\s*:+$", "", txt)
txt = re.sub(r"^\d+\s*\.\s*", "", txt)
return " ".join(txt.split())
def extract_entities_batch(texts: List[str], batch_size: int = 32) -> List[List[Tuple[str, str]]]:
"""Extract entities from multiple texts using spaCy's nlp.pipe() for batched NER.
Uses spaCy's efficient batch processing pipeline instead of calling
nlp() individually per text. Significantly faster for multiple texts.
Args:
texts: List of input texts to extract entities from.
batch_size: Number of texts to process in each spaCy batch.
Returns:
List of entity lists, one per input text. Each entity list contains
(entity_type, entity_text) tuples. Returns list of empty lists if
spaCy is unavailable.
"""
if not texts:
return []
from mem0.utils.spacy_models import get_nlp_full
nlp = get_nlp_full()
if nlp is None:
return [[] for _ in texts]
results = []
for doc in nlp.pipe(texts, batch_size=batch_size):
results.append(_extract_entities_from_doc(doc))
return results
def _norm_text(txt: str) -> str:
return " ".join(txt.lower().split())
def _extract_entities_from_doc(doc) -> List[Tuple[str, str]]:
"""Extract entities from a spaCy Doc object.
def _looks_like_technical_identifier(text: str) -> bool:
return bool(re.fullmatch(r"[A-Za-z_][\w-]*(?:\.[A-Za-z_][\w-]*)+", text))
Ported from platform's shared.core.utils.entity_extraction.extract_entities().
"""
entities: List[Tuple[str, str]] = []
text = doc.text
def _has_internal_cap_or_digit(text: str) -> bool:
return any(ch.isdigit() for ch in text) or any(ch.isupper() for ch in text[1:])
def _looks_like_metric_count_token(tok) -> bool:
return tok.pos_ == "NUM" and bool(re.fullmatch(r"\d[\d,]*(?:\.\d+)?", tok.text))
def _is_metric_list_context(tokens: list, idx: int) -> bool:
prev_text = tokens[idx - 1].text if idx > 0 else ""
next_text = tokens[idx + 1].text if idx + 1 < len(tokens) else ""
return prev_text in {":", ",", ";"} or next_text in {",", ";"}
def _strip_trailing_metric_counts(span_tokens: list, all_tokens: list) -> list:
while len(span_tokens) > 1 and _looks_like_metric_count_token(span_tokens[-1]):
tok = span_tokens[-1]
if "," not in tok.text and not _is_metric_list_context(all_tokens, tok.i):
break
span_tokens = span_tokens[:-1]
return span_tokens
def _is_list_item_name_token(tokens: list, idx: int) -> bool:
tok = tokens[idx]
if not tok.text or tok.text in _FORMATTING_MARKERS or not tok.text[0].isupper():
return False
if not any(ch.isalpha() for ch in tok.text) or _is_bad_single_name_token(tok):
return False
next_tok = tokens[idx + 1] if idx + 1 < len(tokens) else None
if not next_tok or not _looks_like_metric_count_token(next_tok):
return False
return _is_metric_list_context(tokens, idx) or _is_metric_list_context(tokens, idx + 1)
def _is_name_like_token(tok, tokens: list | None = None, idx: int | None = None) -> bool:
if not tok.text or tok.text in _FORMATTING_MARKERS:
return False
if not tok.text[0].isupper():
return False
if not any(ch.isalpha() for ch in tok.text):
return False
if _is_bad_single_name_token(tok):
return False
if tok.pos_ == "PROPN" or tok.tag_ in {"NNP", "NNPS"}:
return True
if tokens is not None and idx is not None and _is_list_item_name_token(tokens, idx):
return True
if _has_internal_cap_or_digit(tok.text):
return True
return (
tokens is not None
and idx is not None
and tok.pos_ == "NOUN"
and tok.dep_ not in {"compound", "amod"}
and not _is_sentence_start(tokens, idx)
)
def _is_bad_single_name_token(tok) -> bool:
lower = tok.text.lower()
return lower in _GENERIC_SINGLE_ENTITY_TERMS or lower in _GENERIC_CAPS or tok.is_stop
def _add_candidate(
candidates: list[_EntityCandidate],
entity_type: str,
text: str,
source: str,
start: int,
end: int,
confidence: float,
priority: int,
) -> None:
cleaned = _clean_text(text)
if not cleaned or len(cleaned) <= 2 or _has_artifacts(cleaned):
return
candidates.append(
_EntityCandidate(
entity_type=entity_type,
text=cleaned,
source=source,
start=start,
end=end,
confidence=confidence,
priority=priority,
)
)
def _add_ner_candidates(doc, candidates: list[_EntityCandidate]) -> None:
tokens = list(doc)
for ent in doc.ents:
if ent.label_ in _REJECTED_NER_LABELS or ent.label_ not in _ACCEPTED_NER_LABELS:
continue
ent_tokens = _strip_trailing_metric_counts(list(ent), tokens)
if not ent_tokens:
continue
if any(tok.pos_ == "CCONJ" and tok.text.lower() == "and" for tok in ent_tokens):
continue
if len(ent_tokens) == 1 and _is_bad_single_name_token(ent_tokens[0]):
continue
if (
len(ent_tokens) == 1
and ent_tokens[0].dep_ in {"compound", "amod"}
and ent_tokens[0].head.pos_ in {"NOUN", "PROPN"}
):
continue
_add_candidate(
candidates,
"PROPER",
"".join(tok.text_with_ws for tok in ent_tokens).strip(),
"spacy_ner",
ent_tokens[0].i,
ent_tokens[-1].i + 1,
0.95,
0,
)
# === PROPER NOUN SEQUENCES ===
def _add_technical_identifier_candidates(tokens: list, candidates: list[_EntityCandidate]) -> None:
for tok in tokens:
if _looks_like_technical_identifier(tok.text):
_add_candidate(
candidates,
"IDENTIFIER",
tok.text,
"technical_identifier",
tok.i,
tok.i + 1,
0.9,
1,
)
def _add_proper_name_candidates(tokens: list, candidates: list[_EntityCandidate]) -> None:
allowed_inner_connectors = {"of", "the", "for", "at", "in"}
i = 0
while i < len(tokens):
tok = tokens[i]
if tok.text in _FORMATTING_MARKERS:
if not _is_name_like_token(tok, tokens, i):
i += 1
continue
is_cap = tok.text and tok.text[0].isupper()
is_label = i + 1 < len(tokens) and tokens[i + 1].text == ":"
if is_cap and not is_label and tok.pos_ in {"PROPN", "NOUN", "ADJ"}:
seq = [(tok, i)]
j = i + 1
while j < len(tokens):
t = tokens[j]
if (t.text and t.text[0].isupper()) or t.text.lower() in {
"'s", "of", "the", "in", "and", "for", "at", "is",
}:
seq.append((t, j))
j += 1
else:
break
# Strip trailing function words
while seq and seq[-1][0].text.lower() in {"of", "the", "in", "and", "for", "at", "is", "'s"}:
seq.pop()
if seq:
has_mid_cap = any(
not _is_sentence_start(tokens, idx)
for (t, idx) in seq
if t.text[0].isupper() and t.text.lower() not in {"'s", "of", "the", "in", "and", "for", "at", "is"}
)
if has_mid_cap:
phrase = "".join(t.text_with_ws for (t, idx) in seq).strip()
if len(phrase) > 2:
entities.append(("PROPER", phrase))
i = j
else:
i += 1
span_tokens = [tok]
j = i + 1
while j < len(tokens):
current = tokens[j]
if _is_name_like_token(current, tokens, j):
span_tokens.append(current)
j += 1
continue
if (
current.text.lower() in allowed_inner_connectors
and j + 1 < len(tokens)
and _is_name_like_token(tokens[j + 1], tokens, j + 1)
):
span_tokens.extend([current, tokens[j + 1]])
j += 2
continue
break
# === QUOTED TEXT ===
name_tokens = [
t
for t in span_tokens
if _is_name_like_token(t, tokens, t.i) or (0 <= t.i < len(tokens) and _is_list_item_name_token(tokens, t.i))
]
if len(name_tokens) > 1 or not _is_bad_single_name_token(name_tokens[0]):
text = "".join(t.text_with_ws for t in span_tokens).strip()
_add_candidate(candidates, "PROPER", text, "proper_name_span", i, j, 0.8, 2)
i = max(j, i + 1)
def _add_quoted_candidates(text: str, candidates: list[_EntityCandidate]) -> None:
for m in re.finditer(r'"([^"]+)"', text):
if len(m.group(1).strip()) > 2:
entities.append(("QUOTED", m.group(1).strip()))
_add_candidate(candidates, "QUOTED", m.group(1).strip(), "quoted", -1, -1, 0.75, 3)
for m in re.finditer(r"(?:^|[\s\(\[{,;])'([^']+)'(?=[\s\.,;:!?\)\]]|$)", text):
if len(m.group(1).strip()) > 2:
entities.append(("QUOTED", m.group(1).strip()))
_add_candidate(candidates, "QUOTED", m.group(1).strip(), "quoted", -1, -1, 0.75, 3)
# === NOUN-NOUN COMPOUNDS ===
def _add_topic_phrase_candidates(doc, candidates: list[_EntityCandidate]) -> None:
for chunk in doc.noun_chunks:
chunk_tokens = list(chunk)
split_indices: list = []
poss_splits: list = []
split_indices: list[int] = []
poss_splits: list[int] = []
for idx, tok in enumerate(chunk_tokens):
if tok.dep_ == "case" and tok.text in {"'s", "\u2019s", "'"}:
split_indices.append(idx)
@@ -243,14 +600,14 @@ def _extract_entities_from_doc(doc) -> List[Tuple[str, str]]:
split_indices.append(idx)
if split_indices:
groups: list = []
groups: list[list] = []
prev = 0
for split_idx in split_indices:
if split_idx > prev:
groups.append(chunk_tokens[prev:split_idx])
if split_idx in poss_splits:
next_split = next((s for s in split_indices if s > split_idx), None)
owned = chunk_tokens[split_idx + 1: next_split if next_split else len(chunk_tokens)]
owned = chunk_tokens[split_idx + 1 : next_split if next_split else len(chunk_tokens)]
if owned:
first_content = next((t for t in owned if t.pos_ not in {"PUNCT", "PART"}), None)
if not (first_content and first_content.text and first_content.text[0].isupper()):
@@ -272,7 +629,8 @@ def _extract_entities_from_doc(doc) -> List[Tuple[str, str]]:
content = [
t
for t in group
if t.pos_ not in {"DET", "PRON", "PUNCT", "PART", "ADP", "SCONJ", "NUM"} and (t.pos_ == "ADJ" or not t.is_stop)
if t.pos_ not in {"DET", "PRON", "PUNCT", "PART", "ADP", "SCONJ", "NUM"}
and (t.pos_ == "ADJ" or not t.is_stop)
]
if not content:
continue
@@ -286,78 +644,129 @@ def _extract_entities_from_doc(doc) -> List[Tuple[str, str]]:
if compound_toks:
is_circ = any(t.lemma_.lower() in _CIRCUMSTANTIAL_MODS for t in compound_toks)
if is_circ:
val = head.lemma_ if head.pos_ == "NOUN" else head.text
val = head.text
if len(val) > 2:
entities.append(("NOUN", val))
_add_candidate(
candidates,
"TOPIC",
val,
"topic_phrase",
head.i,
head.i + 1,
0.45,
4,
)
else:
filtered = _strip_generic_ending(
[t for t in content if not (t.pos_ == "ADJ" and t.lemma_.lower() in _NON_SPECIFIC_ADJ)]
)
if filtered:
phrase = _lemmatize_compound(filtered)
phrase = " ".join(t.text for t in filtered)
if len(phrase) > 3 and " " in phrase:
entities.append(("COMPOUND", phrase))
_add_candidate(
candidates,
"TOPIC",
phrase,
"topic_phrase",
filtered[0].i,
filtered[-1].i + 1,
0.45,
4,
)
elif len(content) > 1 and has_spec_adj:
filtered = _strip_generic_ending(
[t for t in content if not ((t.pos_ == "ADJ" or t.dep_ == "amod") and t.lemma_.lower() in _NON_SPECIFIC_ADJ)]
[
t
for t in content
if not ((t.pos_ == "ADJ" or t.dep_ == "amod") and t.lemma_.lower() in _NON_SPECIFIC_ADJ)
]
)
if filtered:
phrase = _lemmatize_compound(filtered)
phrase = " ".join(t.text for t in filtered)
if len(phrase) > 3 and " " in phrase:
entities.append(("COMPOUND", phrase))
_add_candidate(
candidates,
"TOPIC",
phrase,
"topic_phrase",
filtered[0].i,
filtered[-1].i + 1,
0.45,
4,
)
# === FALLBACK: Mis-tagged VERB heads ===
processed = {e[1].lower() for e in entities if e[0] == "COMPOUND"}
generic_verb_heads = _GENERIC_HEADS | {"find", "buy", "purchase", "sale", "deal", "trip", "visit"}
def collect_compounds(head):
return [t for t in doc if t.head == head and t.dep_ == "compound"]
def _spans_overlap(a: _EntityCandidate, b: _EntityCandidate) -> bool:
if a.start < 0 or b.start < 0:
return False
return a.start < b.end and b.start < a.end
for tok in doc:
if tok.pos_ == "VERB" and tok.dep_ in {"pobj", "dobj", "nsubj"}:
comps = sorted(collect_compounds(tok), key=lambda t: t.i)
if comps:
phrase_toks = comps if tok.lemma_.lower() in generic_verb_heads else comps + [tok]
phrase = " ".join(t.text for t in phrase_toks)
if phrase.lower() not in processed and len(phrase) > 3 and " " in phrase:
entities.append(("COMPOUND", phrase))
processed.add(phrase.lower())
# === DEDUPLICATION & CLEANUP ===
seen: set = set()
deduped = []
for t, e in entities:
k = e.lower().strip()
if k not in seen and len(k) > 2:
seen.add(k)
deduped.append((t, e))
def _resolve_candidates(candidates: list[_EntityCandidate]) -> list[tuple[str, str]]:
deduped_by_text: dict[str, _EntityCandidate] = {}
for candidate in candidates:
key = _norm_text(candidate.text)
current = deduped_by_text.get(key)
if current is None or (candidate.priority, -candidate.confidence) < (current.priority, -current.confidence):
deduped_by_text[key] = candidate
cleaned: List[Tuple[str, str]] = []
for etype, etext in deduped:
txt = re.sub(r"^\*+\s*|\s*\*+$", "", etext.strip())
txt = re.sub(r"\s*:+$", "", txt)
txt = re.sub(r"^\d+\s*\.\s*", "", txt)
if not txt or len(txt) <= 2 or _has_artifacts(txt):
ordered = sorted(
deduped_by_text.values(),
key=lambda c: (c.priority, -c.confidence, -(c.end - c.start), c.start),
)
accepted: list[_EntityCandidate] = []
for candidate in ordered:
if any(
_spans_overlap(candidate, existing)
and not (candidate.entity_type == "TOPIC" and " " in candidate.text and existing.entity_type == "PROPER")
for existing in accepted
):
continue
if etype == "PROPER" and " " not in txt and txt.lower() in _GENERIC_CAPS:
continue
cleaned.append((etype, txt))
accepted.append(candidate)
# Keep best type per entity (PROPER > COMPOUND > QUOTED > NOUN)
type_pri = {"PROPER": 0, "COMPOUND": 1, "QUOTED": 2, "NOUN": 3, "VERB": 4}
best: dict = {}
for t, e in cleaned:
k = e.lower()
if k not in best or type_pri.get(t, 99) < type_pri.get(best[k][0], 99):
best[k] = (t, e)
deduped = list(best.values())
accepted.sort(key=lambda c: (c.start if c.start >= 0 else 10**9, c.end, c.priority))
return [(candidate.entity_type, candidate.text) for candidate in accepted]
# Remove entities that are whole-word substrings of longer entities.
# Word-boundary anchoring avoids dropping distinct entities that only share a
# leading substring (e.g. "Sam" must survive alongside "Samsung").
all_lower = [e[1].lower() for e in deduped]
return [
(t, e)
for t, e in deduped
if not any(e.lower() != o and re.search(rf"\b{re.escape(e.lower())}\b", o) for o in all_lower)
]
def _extract_entities_from_doc(doc) -> list[tuple[str, str]]:
"""Extract typed entity candidates from a spaCy Doc.
Args:
doc: A spaCy ``Doc`` object (from ``nlp(text)``).
Returns:
Deduplicated list of ``(entity_type, entity_text)`` tuples.
Entity types include PROPER, QUOTED, TOPIC, and IDENTIFIER.
"""
tokens = list(doc)
candidates: list[_EntityCandidate] = []
_add_ner_candidates(doc, candidates)
_add_technical_identifier_candidates(tokens, candidates)
_add_proper_name_candidates(tokens, candidates)
_add_quoted_candidates(doc.text, candidates)
_add_topic_phrase_candidates(doc, candidates)
return _resolve_candidates(candidates)
def extract_entities(text: str) -> list[tuple[str, str]]:
"""Extract typed entity candidates from text."""
from mem0.utils.spacy_models import get_nlp_full
nlp = get_nlp_full()
if nlp is None:
return []
return _extract_entities_from_doc(nlp(text))
def extract_entities_batch(texts: list[str], batch_size: int = 32) -> list[list[tuple[str, str]]]:
"""Extract typed entity candidates from multiple texts."""
if not texts:
return []
from mem0.utils.spacy_models import get_nlp_full
nlp = get_nlp_full()
if nlp is None:
return [[] for _ in texts]
return [_extract_entities_from_doc(doc) for doc in nlp.pipe(texts, batch_size=batch_size)]
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "mem0ai"
version = "2.0.8"
version = "2.0.9"
description = "Long-term memory for AI Agents"
authors = [
{ name = "Mem0", email = "support@mem0.ai" }
+33 -3
View File
@@ -6,6 +6,7 @@ def _ensure_spacy():
"""Skip tests if spaCy model is not available."""
try:
import spacy
spacy.load("en_core_web_sm")
except Exception:
pytest.skip("spaCy en_core_web_sm model not available")
@@ -33,8 +34,9 @@ class TestExtractEntities:
entities = extract_entities("The machine learning engineer built a neural network")
entity_texts = [e[1].lower() for e in entities]
has_compound = any("machine" in t and "learning" in t for t in entity_texts) or \
any("neural" in t and "network" in t for t in entity_texts)
has_compound = any("machine" in t and "learning" in t for t in entity_texts) or any(
"neural" in t and "network" in t for t in entity_texts
)
assert has_compound, f"Expected compound nouns, got {entities}"
def test_empty_string(self):
@@ -76,9 +78,37 @@ class TestExtractEntities:
for entity in entities:
assert isinstance(entity, tuple)
assert len(entity) == 2
assert entity[0] in ("PROPER", "QUOTED", "COMPOUND", "NOUN")
assert entity[0] in ("PROPER", "QUOTED", "TOPIC", "IDENTIFIER")
assert isinstance(entity[1], str)
def test_handles_names_lists_and_identifiers(self):
from mem0.utils.entity_extraction import extract_entities
text = (
"User reported top inbound integration pages: OpenClaw 25,443, "
"Claude Code 8,916, Codex 2,573, Dify 656. "
"User compared Cartesia and Deepgram. "
"The email field for Mem0 lives at person.properties.email. "
"The qwen endpoint uses person.properties.email. "
"Johnson & Johnson was mentioned. "
"Glasses around my window. "
"On 2026-05-27 there were 90 days of stats."
)
entities = extract_entities(text)
entity_texts = {entity_text for _, entity_text in entities}
normalized = {entity_text.lower() for entity_text in entity_texts}
assert {"OpenClaw", "Claude Code", "Codex", "Dify", "Cartesia", "Deepgram", "Mem0"}.issubset(entity_texts)
assert "person.properties.email" in entity_texts
assert "qwen endpoint" in entity_texts
assert "Johnson & Johnson" in entity_texts
assert "top" not in normalized
assert "glasses" not in normalized
assert "Cartesia and Deepgram" not in entity_texts
assert "Claude Code 8,916" not in entity_texts
assert not {"8,916", "2,573", "656", "2026-05-27", "90"}.intersection(entity_texts)
class TestExtractEntitiesBatch:
def test_batch_processing(self):