Compare commits

..
Author SHA1 Message Date
mintlify[bot] f6936a0e07 Fix broken link to expiration-date page
Generated-By: mintlify-agent
2026-04-15 00:09:21 +00:00
KartikandGitHub a5a688295e fix: prevent arbitrary code execution via pickle in FAISS vector store (#4833) 2026-04-15 00:02:07 +05:30
Saket AryanandGitHub 5d40592e42 chore: version bump to beta1 (#4827) 2026-04-14 18:05:22 +05:30
a488e19044 feat(oss): port v3 pipeline with hybrid search, entity extraction, and additive scoring (#4805)
Co-authored-by: Soumil Rathi <soumilrathi@gmail.com>
Co-authored-by: Saket Aryan <saketaryan2002@gmail.com>
Co-authored-by: chaithanyak42 <chaithanya.kumar42a@gmail.com>
Co-authored-by: kartik-mem0 <kartik.labhshetwar@mem0.ai>
2026-04-14 18:00:58 +05:30
57f944e18a fix: allow anonymousTelemetryId in openclaw.json config (#4826)
Co-authored-by: parteeksachdeva-123 <parteek.sachdeva@aerchain.io>
2026-04-14 17:31:28 +05:30
Gabriel SteinandGitHub fe3f7ae618 fix(plugin): remove invalid keys from Claude plugin config (#4821) 2026-04-14 01:13:03 +05:30
shafdevandGitHub 4a7e166f9a fix(tests): use top_k instead of limit in test_server_params (#4820) 2026-04-14 01:11:31 +05:30
KartikandGitHub 85768e78e7 fix(docs): remove chrome extension cookbooks (#4813) 2026-04-13 22:14:00 +05:30
YunsuandGitHub 7b395f3bf7 fix(openai): make store opt-in so it stops leaking to non-OpenAI backends (#4757) 2026-04-13 21:53:22 +05:30
HUANG XIAOandGitHub 4180409b09 fix(s3vectors): handle vector=None in update() to prevent boto3 validation error (#4594) 2026-04-13 21:12:49 +05:30
Joe WuandGitHub 649e719ce6 fix: LLM config manager falls back to userConf.url for baseURL (#4715) (#4761) 2026-04-13 20:46:18 +05:30
KartikandGitHub 1a53852d93 test: update valkey cluster search test to use top_k parameter (#4815) 2026-04-13 20:44:18 +05:30
Chinnu AbeyandGitHub ac9cdd4840 Fix incorrect use of SentenceTransformer for cross-encoder reranker models (#4806) 2026-04-13 20:14:22 +05:30
Swarnaprakash UdayakumarandGitHub cf530c4bec feat(valkey): add cluster mode enabled (CME) support (#4759) 2026-04-13 20:07:11 +05:30
Saket AryanandGitHub 92b958c1cc chore: bump Python SDK to v2.0.0b0 and Node SDK to v3.0.0-beta.0 (#4810) 2026-04-13 15:51:33 +05:30
Asish KumarandGitHub c239d8a483 fix(client): prevent feedback telemetry TypeError (#4795) 2026-04-12 03:06:05 +05:30
32 changed files with 913 additions and 168 deletions
+21
View File
@@ -50,4 +50,25 @@ Here are the parameters available for configuring Valkey:
| `hnsw_m` | Number of bi-directional links for HNSW | `16` |
| `hnsw_ef_construction` | Size of dynamic candidate list for HNSW | `200` |
| `hnsw_ef_runtime` | Size of dynamic candidate list for search | `10` |
| `cluster_mode` | Enable cluster mode for Valkey cluster (CME) deployments | `false` |
| `distance_metric` | Distance metric for vector similarity | `cosine` |
## Cluster Mode
To use Valkey with cluster mode enabled (CME), set `cluster_mode` to `true`:
```python
config = {
"vector_store": {
"provider": "valkey",
"config": {
"collection_name": "memories",
"valkey_url": "valkey://cluster-endpoint:6379",
"embedding_model_dims": 1536,
"cluster_mode": True
}
}
}
```
When cluster mode is enabled, the connector uses `ValkeyCluster` instead of the standalone client, which handles `MOVED`/`ASK` redirections automatically. Search queries are coordinated across all shards by the valkey-search module's built-in coordinator. See the [valkey-search documentation](https://github.com/valkey-io/valkey-search) for details on cluster mode behavior.
@@ -1,70 +0,0 @@
---
title: Browser Extension Memory
description: "Add Mem0's universal memory layer to Chrome chat surfaces."
---
Enhance your AI interactions with Mem0, a Chrome extension that introduces a universal memory layer across platforms like ChatGPT, Claude, and Perplexity. Mem0 ensures seamless context sharing, making your AI experiences more personalized and efficient.
<Note>
We now support Grok! The Mem0 Chrome Extension has been updated to work with Grok, bringing the same powerful memory capabilities to your Grok conversations.
</Note>
## Features
- **Universal Memory Layer**: Share context seamlessly across ChatGPT, Claude, Perplexity, and Grok.
- **Smart Context Detection**: Automatically captures relevant information from your conversations.
- **Intelligent Memory Retrieval**: Surfaces pertinent memories at the right time.
- **One-Click Sync**: Easily synchronize with existing ChatGPT memories.
- **Memory Dashboard**: Manage all your memories in one centralized location.
## Installation
You can install the Mem0 Chrome Extension using one of the following methods:
### Method 1: Chrome Web Store Installation
1. **Download the Extension**: Open Google Chrome and navigate to the [Mem0 Chrome Extension page](https://chromewebstore.google.com/detail/mem0/onihkkbipkfeijkadecaafbgagkhglop?hl=en).
2. **Add to Chrome**: Click on the "Add to Chrome" button.
3. **Confirm Installation**: In the pop-up dialog, click "Add extension" to confirm. The Mem0 icon should now appear in your Chrome toolbar.
### Method 2: Manual Installation
1. **Download the Extension**: Clone or download the extension files from the [Mem0 Chrome Extension GitHub repository](https://github.com/mem0ai/mem0-chrome-extension).
2. **Access Chrome Extensions**: Open Google Chrome and navigate to `chrome://extensions`.
3. **Enable Developer Mode**: Toggle the "Developer mode" switch in the top right corner.
4. **Load Unpacked Extension**: Click "Load unpacked" and select the directory containing the extension files.
5. **Confirm Installation**: The Mem0 Chrome Extension should now appear in your Chrome toolbar.
## Usage
1. **Locate the Mem0 Icon**: After installation, find the Mem0 icon in your Chrome toolbar.
2. **Sign In**: Click the icon and sign in with your Google account.
3. **Interact with AI Assistants**:
- **ChatGPT and Perplexity**: Continue your conversations as usual; Mem0 operates seamlessly in the background.
- **Claude**: Click the Mem0 button or use the shortcut `Ctrl + M` to activate memory functions.
## Configuration
- **API Key**: Obtain your API key from the Mem0 Dashboard to connect the extension to the Mem0 API.
- **User ID**: This is your unique identifier in the Mem0 system. If not provided, it defaults to `chrome-extension-user`.
## Demo Video
<iframe width="700" height="400" src="https://www.youtube.com/embed/dqenCMMlfwQ?si=zhGVrkq6IS_0Jwyj" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
## Privacy and Data Security
Your messages are sent to the Mem0 API for extracting and retrieving memories. Mem0 is committed to ensuring your data's privacy and security.
---
<CardGroup cols={2}>
<Card title="Build a Mem0 Companion" icon="users" href="/cookbooks/essentials/building-ai-companion">
Learn the foundations of memory-powered assistants that work across platforms.
</Card>
<Card title="Multimodal Support" icon="image" href="/platform/features/multimodal-support">
Extend your browser interactions with vision and audio memory.
</Card>
</CardGroup>
+1 -4
View File
@@ -201,10 +201,7 @@ Here are some examples of how Mem0 can be integrated into various applications:
>
Persistent personality for Eliza agents.
</Card>
<Card title="Browser Extension Memory" icon="globe" href="/cookbooks/frameworks/chrome-extension">
Universal memory layer for Chrome.
</Card>
</CardGroup>
</CardGroup>
---
@@ -216,7 +216,7 @@ memory.delete_all(user_id="alice")
## Put it into practice
- Review the <Link href="/api-reference/memory/delete-memory">Delete Memory API reference</Link>, plus <Link href="/api-reference/memory/batch-delete">Batch Delete</Link> and <Link href="/api-reference/memory/delete-memories">Filtered Delete</Link>.
- Pair deletes with <Link href="/platform/features/expiration-date">Expiration Policies</Link> to automate retention.
- Pair deletes with <Link href="/cookbooks/essentials/memory-expiration-short-and-long-term">Expiration Policies</Link> to automate retention.
## See it live
@@ -236,6 +236,6 @@ memory.delete_all(user_id="alice")
title="Enable Expiration Policies"
description="Automate retention with the platform’s expiration feature."
icon="clock"
href="/platform/features/expiration-date"
href="/cookbooks/essentials/memory-expiration-short-and-long-term"
/>
</CardGroup>
+7 -4
View File
@@ -373,7 +373,6 @@
"cookbooks/frameworks/llamaindex-multiagent",
"cookbooks/frameworks/multimodal-retrieval",
"cookbooks/frameworks/eliza-os-character",
"cookbooks/frameworks/chrome-extension",
"cookbooks/frameworks/gemini-3-with-mem0-mcp",
"cookbooks/frameworks/mirofish-swarm-memory"
]
@@ -622,7 +621,7 @@
},
{
"source": "/platform/features/expiration-date",
"destination": "/"
"destination": "/cookbooks/essentials/memory-expiration-short-and-long-term"
},
{
"source": "/platform/features/async-mode-default-change",
@@ -798,7 +797,11 @@
},
{
"source": "/examples/chrome-extension",
"destination": "/cookbooks/frameworks/chrome-extension"
"destination": "/cookbooks/overview"
},
{
"source": "/cookbooks/frameworks/chrome-extension",
"destination": "/cookbooks/overview"
},
{
"source": "/examples",
@@ -838,7 +841,7 @@
},
{
"source": "/v0x/examples/chrome-extension",
"destination": "/cookbooks/frameworks/chrome-extension"
"destination": "/cookbooks/overview"
},
{
"source": "/v0x/examples/youtube-assistant",
-1
View File
@@ -245,7 +245,6 @@ Key differentiators:
- [LlamaIndex Multiagent](https://docs.mem0.ai/cookbooks/frameworks/llamaindex-multiagent): Multi-agent systems with shared memory
- [Multimodal Retrieval](https://docs.mem0.ai/cookbooks/frameworks/multimodal-retrieval): Memory systems handling text, images, and documents
- [Eliza OS Character](https://docs.mem0.ai/cookbooks/frameworks/eliza-os-character): Character-based AI with persistent personality
- [Chrome Extension](https://docs.mem0.ai/cookbooks/frameworks/chrome-extension): Browser extensions that remember user interactions
- [Gemini with Mem0 MCP](https://docs.mem0.ai/cookbooks/frameworks/gemini-3-with-mem0-mcp): Google Gemini integration using MCP server
- [Mirofish Swarm Memory](https://docs.mem0.ai/cookbooks/frameworks/mirofish-swarm-memory): Swarm-based multi-agent memory patterns
+2 -2
View File
@@ -8,6 +8,6 @@
},
"homepage": "https://mem0.ai",
"repository": "https://github.com/mem0ai/mem0",
"logo": "logo.svg",
"license": "Apache-2.0"
"license": "Apache-2.0",
"keywords": ["memory", "personalization", "mcp", "semantic-search"]
}
-1
View File
@@ -1,5 +1,4 @@
{
"description": "Mem0 memory capture hooks — automatic memory extraction at key lifecycle points",
"hooks": {
"SessionStart": [
{
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mem0ai",
"version": "2.4.6",
"version": "3.0.0-beta.1",
"description": "The Memory Layer For Your AI Apps",
"main": "./dist/index.js",
"module": "./dist/index.mjs",
+1
View File
@@ -110,6 +110,7 @@ export class ConfigManager {
((userConf as Record<string, unknown>)?.lmstudio_base_url as
| string
| undefined) ??
userConf?.url ??
defaultConf.baseURL;
return {
+1 -4
View File
@@ -195,10 +195,7 @@ export class Memory {
...this.config.vectorStore.config,
collectionName: entityCollectionName,
};
// For file-based stores (memory/SQLite), use a separate DB path for entities.
// If dbPath was set explicitly, derive the entity path from it. If it's unset,
// leave it unset — the vector store's default-path logic now scopes by
// collectionName, so the entity store will land in its own file automatically.
// For file-based stores (memory/SQLite), use a separate DB path for entities
if (entityConfig.dbPath) {
entityConfig.dbPath = entityConfig.dbPath.replace(
/\.db$/,
+2 -10
View File
@@ -2,16 +2,8 @@ import fs from "fs";
import os from "os";
import path from "path";
export function getDefaultVectorStoreDbPath(collectionName?: string): string {
// Scope the default DB file by collection name so that parallel stores
// (e.g. "memories" vs "memories_entities") don't collide in the same
// SQLite table. Without this, both collections write to the same
// `vectors` table and search results leak across them.
const filename =
collectionName && collectionName.length > 0
? `vector_store_${collectionName.replace(/[^a-zA-Z0-9_-]/g, "_")}.db`
: "vector_store.db";
return path.join(os.homedir(), ".mem0", filename);
export function getDefaultVectorStoreDbPath(): string {
return path.join(os.homedir(), ".mem0", "vector_store.db");
}
export function ensureSQLiteDirectory(dbPath: string): void {
+1 -4
View File
@@ -21,10 +21,7 @@ export class MemoryVectorStore implements VectorStore {
constructor(config: VectorStoreConfig) {
this.dimension = config.dimension || 1536; // Default OpenAI dimension
// Scope the default path by collectionName so that parallel stores
// (e.g. memory vs entity) don't share the same `vectors` table.
this.dbPath =
config.dbPath || getDefaultVectorStoreDbPath(config.collectionName);
this.dbPath = config.dbPath || getDefaultVectorStoreDbPath();
if (!config.dbPath) {
const oldDefault = path.join(process.cwd(), "vector_store.db");
@@ -127,6 +127,20 @@ describe("ConfigManager", () => {
expect(config.llm.config.url).toBe("http://fallback:11434");
});
it("should use url as baseURL fallback when no baseURL provided (issue #4715)", () => {
const config = ConfigManager.mergeConfig({
embedder: baseEmbedder,
vectorStore: baseVectorStore,
llm: {
provider: "ollama",
config: { model: "llama3.1:8b", url: "http://my-ollama-host:11434" },
},
});
expect(config.llm.config.baseURL).toBe("http://my-ollama-host:11434");
expect(config.llm.config.url).toBe("http://my-ollama-host:11434");
});
it("should use default baseURL when no url or baseURL provided", () => {
const config = ConfigManager.mergeConfig({
embedder: baseEmbedder,
+2 -2
View File
@@ -883,7 +883,7 @@ class MemoryClient:
response = self.client.post("/v1/feedback/", json=data)
response.raise_for_status()
capture_client_event("client.feedback", self, data, {"sync_type": "sync"})
capture_client_event("client.feedback", self, {**data, "sync_type": "sync"})
return response.json()
def _prepare_payload(self, messages: List[Dict[str, str]], kwargs: Dict[str, Any]) -> Dict[str, Any]:
@@ -1761,5 +1761,5 @@ class AsyncMemoryClient:
response = await self.async_client.post("/v1/feedback/", json=data)
response.raise_for_status()
capture_client_event("client.feedback", self, data, {"sync_type": "async"})
capture_client_event("client.feedback", self, {**data, "sync_type": "async"})
return response.json()
+6 -1
View File
@@ -29,7 +29,7 @@ class OpenAIConfig(BaseLlmConfig):
openrouter_base_url: Optional[str] = None,
site_url: Optional[str] = None,
app_name: Optional[str] = None,
store: bool = False,
store: Optional[bool] = None,
# Response monitoring callback
response_callback: Optional[Callable[[Any, dict, dict], None]] = None,
):
@@ -53,6 +53,11 @@ class OpenAIConfig(BaseLlmConfig):
openrouter_base_url: OpenRouter base URL, defaults to None
site_url: Site URL for OpenRouter, defaults to None
app_name: Application name for OpenRouter, defaults to None
store: Whether to store the conversation on OpenAI's server. Opt-in;
defaults to None (not sent). Set to True or False only if you
want the value forwarded to the OpenAI API. Leaving it None
avoids leaking the field into OpenAI-compatible backends that
reject unknown fields (Gemini, Groq, vLLM, etc.).
response_callback: Optional callback for monitoring LLM responses.
"""
# Initialize base parameters
+1
View File
@@ -14,6 +14,7 @@ class ValkeyConfig(BaseModel):
hnsw_m: int = Field(16, description="HNSW: number of connections per layer")
hnsw_ef_construction: int = Field(200, description="HNSW: search width during index construction")
hnsw_ef_runtime: int = Field(10, description="HNSW: search width during queries")
cluster_mode: bool = Field(False, description="Enable cluster mode for Valkey cluster (CME) deployments")
@model_validator(mode="before")
@classmethod
+6 -5
View File
@@ -126,11 +126,12 @@ class OpenAILLM(LLMBase):
params.update(**openrouter_params)
else:
openai_specific_generation_params = ["store"]
for param in openai_specific_generation_params:
if hasattr(self.config, param):
params[param] = getattr(self.config, param)
# Only send OpenAI-specific parameters when the user has explicitly
# configured them. OpenAI-compatible backends (Gemini, Groq, vLLM, etc.)
# reject unknown fields, so `store` must be opt-in, not opt-out.
if self.config.store is not None:
params["store"] = self.config.store
if response_format:
params["response_format"] = response_format
if tools: # TODO: Remove tools if no issues found with new memory addition logic
@@ -6,7 +6,7 @@ from mem0.configs.rerankers.base import BaseRerankerConfig
from mem0.configs.rerankers.sentence_transformer import SentenceTransformerRerankerConfig
try:
from sentence_transformers import SentenceTransformer
from sentence_transformers import CrossEncoder
SENTENCE_TRANSFORMERS_AVAILABLE = True
except ImportError:
SENTENCE_TRANSFORMERS_AVAILABLE = False
@@ -41,7 +41,7 @@ class SentenceTransformerReranker(BaseReranker):
)
self.config = config
self.model = SentenceTransformer(self.config.model, device=self.config.device)
self.model = CrossEncoder(self.config.model, device=self.config.device)
def rerank(self, query: str, documents: List[Dict[str, Any]], top_k: int = None) -> List[Dict[str, Any]]:
"""
@@ -74,8 +74,11 @@ class SentenceTransformerReranker(BaseReranker):
# Create query-document pairs
pairs = [[query, doc_text] for doc_text in doc_texts]
# Get similarity scores
scores = self.model.predict(pairs)
scores = self.model.predict(
pairs,
batch_size=self.config.batch_size,
show_progress_bar=self.config.show_progress_bar,
)
if isinstance(scores, np.ndarray):
scores = scores.tolist()
+155 -17
View File
@@ -1,10 +1,11 @@
import json
import logging
import os
import pickle
import uuid
import warnings
from pathlib import Path
from typing import Dict, List, Optional
from typing import Any, Dict, List, Optional
import numpy as np
from pydantic import BaseModel
@@ -13,7 +14,7 @@ try:
# Suppress SWIG deprecation warnings from FAISS
warnings.filterwarnings("ignore", category=DeprecationWarning, message=".*SwigPy.*")
warnings.filterwarnings("ignore", category=DeprecationWarning, message=".*swigvarlink.*")
logging.getLogger("faiss").setLevel(logging.WARNING)
logging.getLogger("faiss.loader").setLevel(logging.WARNING)
@@ -30,6 +31,93 @@ from mem0.vector_stores.base import VectorStoreBase
logger = logging.getLogger(__name__)
class SafeUnpickler(pickle.Unpickler):
"""
Restricted unpickler that only allows safe built-in types.
This prevents arbitrary code execution via pickle deserialization by only
allowing a whitelist of safe types (dict, list, str, int, float, bool, tuple, None).
"""
# Only allow builtins module
SAFE_MODULES = frozenset({"builtins", "__builtin__"})
# Only allow safe basic types
SAFE_NAMES = frozenset({"dict", "list", "str", "int", "float", "bool", "tuple", "set", "frozenset", "NoneType"})
def find_class(self, module: str, name: str) -> Any:
"""Override find_class to only allow safe types."""
if module in self.SAFE_MODULES and name in self.SAFE_NAMES:
import builtins
if hasattr(builtins, name):
return getattr(builtins, name)
# NoneType special case
if name == "NoneType":
return type(None)
raise pickle.UnpicklingError(
f"Unsafe pickle: attempted to load '{module}.{name}'. "
f"Only basic Python types are allowed for security reasons."
)
def _safe_pickle_load(file_path: str) -> Any:
"""
Safely load a pickle file using restricted unpickler.
Args:
file_path: Path to the pickle file.
Returns:
The deserialized object (only basic Python types allowed).
Raises:
pickle.UnpicklingError: If the pickle contains unsafe types.
"""
with open(file_path, "rb") as f:
return SafeUnpickler(f).load()
def _validate_docstore_structure(data: Any) -> tuple:
"""
Validate that loaded data has the expected structure.
Args:
data: The loaded data to validate.
Returns:
Tuple of (docstore, index_to_id) if valid.
Raises:
ValueError: If the data structure is invalid.
"""
if not isinstance(data, tuple) or len(data) != 2:
raise ValueError("Invalid docstore format: expected tuple of (docstore, index_to_id)")
docstore, index_to_id = data
if not isinstance(docstore, dict):
raise ValueError("Invalid docstore format: docstore must be a dict")
if not isinstance(index_to_id, dict):
raise ValueError("Invalid docstore format: index_to_id must be a dict")
# Validate docstore entries
for key, value in docstore.items():
if not isinstance(key, str):
raise ValueError(f"Invalid docstore key type: {type(key)}, expected str")
if not isinstance(value, dict):
raise ValueError(f"Invalid docstore value type: {type(value)}, expected dict")
# Validate index_to_id entries
for key, value in index_to_id.items():
if not isinstance(key, int):
raise ValueError(f"Invalid index_to_id key type: {type(key)}, expected int")
if not isinstance(value, str):
raise ValueError(f"Invalid index_to_id value type: {type(value)}, expected str")
return docstore, index_to_id
class OutputData(BaseModel):
id: Optional[str] # memory id
score: Optional[float] # distance
@@ -73,9 +161,13 @@ class FAISS(VectorStoreBase):
# Try to load existing index if available
index_path = f"{self.path}/{collection_name}.faiss"
docstore_path = f"{self.path}/{collection_name}.pkl"
if os.path.exists(index_path) and os.path.exists(docstore_path):
self._load(index_path, docstore_path)
json_docstore_path = f"{self.path}/{collection_name}.json"
pkl_docstore_path = f"{self.path}/{collection_name}.pkl"
# Check for index file and either JSON (preferred) or legacy pickle docstore
if os.path.exists(index_path) and (os.path.exists(json_docstore_path) or os.path.exists(pkl_docstore_path)):
# _load will prefer JSON over pickle and auto-migrate
self._load(index_path, pkl_docstore_path)
else:
self.create_col(collection_name)
@@ -83,34 +175,76 @@ class FAISS(VectorStoreBase):
"""
Load FAISS index and docstore from disk.
Supports both JSON (preferred) and legacy pickle formats. Pickle files are loaded
using a restricted unpickler that only allows basic Python types to prevent
arbitrary code execution (CVE mitigation).
Args:
index_path (str): Path to FAISS index file.
docstore_path (str): Path to docstore pickle file.
docstore_path (str): Path to docstore file (.json or legacy .pkl).
"""
try:
self.index = faiss.read_index(index_path)
with open(docstore_path, "rb") as f:
self.docstore, self.index_to_id = pickle.load(f)
logger.info(f"Loaded FAISS index from {index_path} with {self.index.ntotal} vectors")
# Determine docstore format - prefer JSON over pickle
json_docstore_path = docstore_path.replace(".pkl", ".json")
if os.path.exists(json_docstore_path):
# Load from JSON (safe, preferred format)
with open(json_docstore_path, "r", encoding="utf-8") as f:
data = json.load(f)
self.docstore = data.get("docstore", {})
# JSON keys are always strings, convert back to int
self.index_to_id = {int(k): v for k, v in data.get("index_to_id", {}).items()}
logger.info(f"Loaded FAISS index from {index_path} with {self.index.ntotal} vectors (JSON format)")
elif os.path.exists(docstore_path):
# Load from legacy pickle using safe unpickler
# This prevents arbitrary code execution from malicious pickle files
logger.warning(
f"Loading legacy pickle docstore from {docstore_path}. "
f"Consider migrating to JSON format for better security."
)
data = _safe_pickle_load(docstore_path)
self.docstore, self.index_to_id = _validate_docstore_structure(data)
logger.info(f"Loaded FAISS index from {index_path} with {self.index.ntotal} vectors (pickle format)")
# Auto-migrate to JSON format
self._save()
logger.info(f"Migrated docstore to JSON format: {json_docstore_path}")
else:
raise FileNotFoundError(f"No docstore found at {docstore_path} or {json_docstore_path}")
except pickle.UnpicklingError as e:
logger.error(f"Security error loading FAISS docstore: {e}")
raise ValueError(f"Failed to load FAISS docstore: potentially malicious pickle file. {e}") from e
except Exception as e:
logger.warning(f"Failed to load FAISS index: {e}")
self.docstore = {}
self.index_to_id = {}
def _save(self):
"""Save FAISS index and docstore to disk."""
"""Save FAISS index and docstore to disk using JSON format (secure)."""
if not self.path or not self.index:
return
try:
os.makedirs(self.path, exist_ok=True)
index_path = f"{self.path}/{self.collection_name}.faiss"
docstore_path = f"{self.path}/{self.collection_name}.pkl"
json_docstore_path = f"{self.path}/{self.collection_name}.json"
faiss.write_index(self.index, index_path)
with open(docstore_path, "wb") as f:
pickle.dump((self.docstore, self.index_to_id), f)
# Save docstore as JSON (safe format, no code execution risk)
# JSON keys must be strings, so convert int keys to str
data = {
"docstore": self.docstore,
"index_to_id": {str(k): v for k, v in self.index_to_id.items()},
}
with open(json_docstore_path, "w", encoding="utf-8") as f:
json.dump(data, f, indent=2)
except Exception as e:
logger.warning(f"Failed to save FAISS index: {e}")
@@ -417,12 +551,16 @@ class FAISS(VectorStoreBase):
if self.path:
try:
index_path = f"{self.path}/{self.collection_name}.faiss"
docstore_path = f"{self.path}/{self.collection_name}.pkl"
json_docstore_path = f"{self.path}/{self.collection_name}.json"
pkl_docstore_path = f"{self.path}/{self.collection_name}.pkl"
if os.path.exists(index_path):
os.remove(index_path)
if os.path.exists(docstore_path):
os.remove(docstore_path)
if os.path.exists(json_docstore_path):
os.remove(json_docstore_path)
# Also clean up legacy pickle files if they exist
if os.path.exists(pkl_docstore_path):
os.remove(pkl_docstore_path)
logger.info(f"Deleted collection {self.collection_name}")
except Exception as e:
+31 -1
View File
@@ -122,7 +122,37 @@ class S3Vectors(VectorStoreBase):
)
def update(self, vector_id, vector=None, payload=None):
# S3 Vectors uses put_vectors for updates (overwrite)
# S3 Vectors uses put_vectors for updates (overwrite).
# When vector=None (e.g. metadata-only update triggered by event=NONE),
# fetch the existing vector data first to avoid passing None to boto3
# which causes a parameter validation error:
# "Invalid type for parameter vectors[0].data.float32, value: None"
if vector is None:
existing = self.get(vector_id)
if existing is None:
logger.warning(f"update called with vector=None but {vector_id} not found; skipping")
return
try:
response = self.client.get_vectors(
vectorBucketName=self.vector_bucket_name,
indexName=self.collection_name,
keys=[vector_id],
returnData=True,
returnMetadata=True,
)
vectors = response.get("vectors", [])
if not vectors:
logger.warning(f"update: no vector data found for {vector_id}; skipping")
return
vector = vectors[0].get("data", {}).get("float32")
if vector is None:
logger.warning(f"update: float32 data is None for {vector_id}; skipping")
return
if payload is None:
payload = existing.payload
except Exception as e:
logger.error(f"update: failed to fetch existing vector for {vector_id}: {e}")
return
self.insert(vectors=[vector], payloads=[payload], ids=[vector_id])
def get(self, vector_id) -> Optional[OutputData]:
+13 -3
View File
@@ -52,6 +52,7 @@ class ValkeyDB(VectorStoreBase):
hnsw_m: int = 16,
hnsw_ef_construction: int = 200,
hnsw_ef_runtime: int = 10,
cluster_mode: bool = False,
):
"""
Initialize the Valkey vector store.
@@ -65,6 +66,7 @@ class ValkeyDB(VectorStoreBase):
hnsw_m (int, optional): HNSW M parameter (connections per node). Defaults to 16.
hnsw_ef_construction (int, optional): HNSW ef_construction parameter. Defaults to 200.
hnsw_ef_runtime (int, optional): HNSW ef_runtime parameter. Defaults to 10.
cluster_mode (bool, optional): Enable cluster mode for Valkey cluster (CME) deployments. Defaults to False.
"""
self.embedding_model_dims = embedding_model_dims
self.collection_name = collection_name
@@ -74,6 +76,7 @@ class ValkeyDB(VectorStoreBase):
self.hnsw_m = hnsw_m
self.hnsw_ef_construction = hnsw_ef_construction
self.hnsw_ef_runtime = hnsw_ef_runtime
self.cluster_mode = cluster_mode
# Validate index type
if self.index_type not in ["hnsw", "flat"]:
@@ -81,8 +84,13 @@ class ValkeyDB(VectorStoreBase):
# Connect to Valkey
try:
self.client = valkey.from_url(valkey_url)
logger.debug(f"Successfully connected to Valkey at {valkey_url}")
if self.cluster_mode:
from valkey.cluster import ValkeyCluster
self.client = ValkeyCluster.from_url(valkey_url)
else:
self.client = valkey.from_url(valkey_url)
logger.debug(f"Successfully connected to Valkey at {valkey_url} (cluster_mode={cluster_mode})")
except Exception as e:
logger.exception(f"Failed to connect to Valkey at {valkey_url}: {e}")
raise
@@ -185,7 +193,6 @@ class ValkeyDB(VectorStoreBase):
"""
# Check if the search module is available
try:
# Try to execute a search command
self.client.execute_command("FT._LIST")
except ResponseError as e:
if "unknown command" in str(e).lower():
@@ -353,6 +360,9 @@ class ValkeyDB(VectorStoreBase):
"""
Execute a search query.
In cluster mode, the valkey-search module's built-in coordinator handles
fan-out across all shards and aggregates results server-side.
Args:
query (str): The search query to execute.
params (dict): The query parameters.
+5
View File
@@ -147,6 +147,7 @@ export const DEFAULT_CUSTOM_CATEGORIES: Record<string, string> = {
const ALLOWED_KEYS = [
"mode",
"apiKey",
"anonymousTelemetryId",
"baseUrl",
"userId",
"userEmail",
@@ -215,6 +216,10 @@ export const mem0ConfigSchema = {
return {
mode,
apiKey: resolvedApiKey,
anonymousTelemetryId:
typeof cfg.anonymousTelemetryId === "string"
? cfg.anonymousTelemetryId
: undefined,
baseUrl: resolvedBaseUrl,
userId:
typeof cfg.userId === "string" && cfg.userId
+4
View File
@@ -134,6 +134,10 @@
"topK": {
"type": "number"
},
"anonymousTelemetryId": {
"type": "string",
"description": "Persistent anonymous telemetry identifier"
},
"oss": {
"type": "object",
"properties": {
+5
View File
@@ -88,6 +88,11 @@ describe("mem0ConfigSchema.parse() — defaults", () => {
const cfg = mem0ConfigSchema.parse({ apiKey: "test-key" });
expect(cfg.skills).toBeUndefined();
});
it("allows anonymousTelemetryId", () => {
const cfg = mem0ConfigSchema.parse({ apiKey: "test-key", anonymousTelemetryId: "123" });
expect(cfg.anonymousTelemetryId).toBe("123");
});
});
// ---------------------------------------------------------------------------
+1
View File
@@ -8,6 +8,7 @@ export type Mem0Config = {
mode: Mem0Mode;
// Platform-specific
apiKey?: string;
anonymousTelemetryId?: string;
baseUrl?: string;
customInstructions: string;
customCategories: Record<string, string>;
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "mem0ai"
version = "1.0.11"
version = "2.0.0b1"
description = "Long-term memory for AI Agents"
authors = [
{ name = "Mem0", email = "support@mem0.ai" }
+56 -2
View File
@@ -55,7 +55,7 @@ def test_generate_response_without_tools(mock_openai_client):
response = llm.generate_response(messages)
mock_openai_client.chat.completions.create.assert_called_once_with(
model="gpt-4.1-nano-2025-04-14", messages=messages, temperature=0.7, max_tokens=100, top_p=1.0, store=False
model="gpt-4.1-nano-2025-04-14", messages=messages, temperature=0.7, max_tokens=100, top_p=1.0
)
assert response == "I'm doing well, thank you for asking!"
@@ -97,7 +97,7 @@ def test_generate_response_with_tools(mock_openai_client):
response = llm.generate_response(messages, tools=tools)
mock_openai_client.chat.completions.create.assert_called_once_with(
model="gpt-4.1-nano-2025-04-14", messages=messages, temperature=0.7, max_tokens=100, top_p=1.0, tools=tools, tool_choice="auto", store=False
model="gpt-4.1-nano-2025-04-14", messages=messages, temperature=0.7, max_tokens=100, top_p=1.0, tools=tools, tool_choice="auto"
)
assert response["content"] == "I've added the memory for you."
@@ -231,6 +231,60 @@ def test_reasoning_effort_config_values():
assert config.reasoning_effort is None
def test_store_not_sent_by_default(mock_openai_client):
"""`store` must NOT be injected into requests when the user has not
explicitly configured it. Regression test for issue #4709, where
`store=False` was unconditionally sent and rejected by OpenAI-compatible
backends such as Google Gemini."""
config = OpenAIConfig(model="gpt-4.1-nano-2025-04-14", temperature=0.1)
assert config.store is None # new opt-in default
llm = OpenAILLM(config)
messages = [{"role": "user", "content": "Hello"}]
mock_response = Mock()
mock_response.choices = [Mock(message=Mock(content="Response"))]
mock_openai_client.chat.completions.create.return_value = mock_response
llm.generate_response(messages)
call_kwargs = mock_openai_client.chat.completions.create.call_args.kwargs
assert "store" not in call_kwargs
def test_store_sent_when_explicitly_true(mock_openai_client):
"""When the user explicitly sets `store=True`, the field must be forwarded."""
config = OpenAIConfig(model="gpt-4.1-nano-2025-04-14", store=True)
llm = OpenAILLM(config)
messages = [{"role": "user", "content": "Hello"}]
mock_response = Mock()
mock_response.choices = [Mock(message=Mock(content="Response"))]
mock_openai_client.chat.completions.create.return_value = mock_response
llm.generate_response(messages)
call_kwargs = mock_openai_client.chat.completions.create.call_args.kwargs
assert call_kwargs["store"] is True
def test_store_sent_when_explicitly_false(mock_openai_client):
"""When the user explicitly sets `store=False`, the field must still be
forwarded — explicit opt-out is a valid configuration for users who rely on
it for OpenAI's zero-data-retention behavior."""
config = OpenAIConfig(model="gpt-4.1-nano-2025-04-14", store=False)
llm = OpenAILLM(config)
messages = [{"role": "user", "content": "Hello"}]
mock_response = Mock()
mock_response.choices = [Mock(message=Mock(content="Response"))]
mock_openai_client.chat.completions.create.return_value = mock_response
llm.generate_response(messages)
call_kwargs = mock_openai_client.chat.completions.create.call_args.kwargs
assert call_kwargs["store"] is False
def test_callback_with_tools(mock_openai_client):
mock_callback = Mock()
config = OpenAIConfig(model="gpt-4.1-nano-2025-04-14", response_callback=mock_callback)
+59
View File
@@ -0,0 +1,59 @@
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from mem0.client.main import AsyncMemoryClient, MemoryClient
def _build_memory_client(response_payload=None):
client = MemoryClient.__new__(MemoryClient)
client.user_email = "user@example.com"
client.client = MagicMock()
response = client.client.post.return_value
response.json.return_value = response_payload or {"message": "Feedback recorded"}
response.raise_for_status.return_value = None
return client
def test_memory_client_feedback_uses_single_telemetry_payload():
client = _build_memory_client()
with patch("mem0.client.main.capture_client_event") as mock_capture:
result = client.feedback("mem_1", feedback="positive", feedback_reason="accurate")
assert result == {"message": "Feedback recorded"}
client.client.post.assert_called_once_with(
"/v1/feedback/",
json={"memory_id": "mem_1", "feedback": "POSITIVE", "feedback_reason": "accurate"},
)
mock_capture.assert_called_once_with(
"client.feedback",
client,
{"memory_id": "mem_1", "feedback": "POSITIVE", "feedback_reason": "accurate", "sync_type": "sync"},
)
@pytest.mark.asyncio
async def test_async_memory_client_feedback_uses_single_telemetry_payload():
client = AsyncMemoryClient.__new__(AsyncMemoryClient)
client.user_email = "user@example.com"
client.async_client = MagicMock()
response = MagicMock()
response.json.return_value = {"message": "Feedback recorded"}
response.raise_for_status.return_value = None
client.async_client.post = AsyncMock(return_value=response)
with patch("mem0.client.main.capture_client_event") as mock_capture:
result = await client.feedback("mem_1", feedback="negative", feedback_reason="outdated")
assert result == {"message": "Feedback recorded"}
client.async_client.post.assert_awaited_once_with(
"/v1/feedback/",
json={"memory_id": "mem_1", "feedback": "NEGATIVE", "feedback_reason": "outdated"},
)
mock_capture.assert_called_once_with(
"client.feedback",
client,
{"memory_id": "mem_1", "feedback": "NEGATIVE", "feedback_reason": "outdated", "sync_type": "async"},
)
+20 -20
View File
@@ -2,7 +2,7 @@
Verifies that the Pydantic request models in server/main.py correctly accept
and forward all parameters supported by the underlying Memory class methods,
including limit, threshold, infer, memory_type, and prompt — which were
including top_k, threshold, infer, memory_type, and prompt — which were
previously silently dropped by Pydantic v2's default extra='ignore' behavior.
"""
@@ -55,31 +55,31 @@ def mock_memory(_mock_memory):
# ===========================================================================
# SearchRequest: limit parameter
# SearchRequest: top_k parameter
# ===========================================================================
class TestSearchLimit:
"""Verify that the limit parameter is accepted and forwarded to Memory.search()."""
"""Verify that the top_k parameter is accepted and forwarded to Memory.search()."""
def test_limit_forwarded(self, client, mock_memory):
resp = client.post("/search", json={"query": "food", "user_id": "u1", "limit": 5})
resp = client.post("/search", json={"query": "food", "user_id": "u1", "top_k": 5})
assert resp.status_code == 200
_, kwargs = mock_memory.search.call_args
assert kwargs["limit"] == 5
assert kwargs["top_k"] == 5
def test_limit_one(self, client, mock_memory):
resp = client.post("/search", json={"query": "food", "user_id": "u1", "limit": 1})
resp = client.post("/search", json={"query": "food", "user_id": "u1", "top_k": 1})
assert resp.status_code == 200
_, kwargs = mock_memory.search.call_args
assert kwargs["limit"] == 1
assert kwargs["top_k"] == 1
def test_limit_omitted_uses_memory_default(self, client, mock_memory):
"""When limit is not sent, it should not appear in the kwargs,
"""When top_k is not sent, it should not appear in the kwargs,
allowing Memory.search() to use its own default (100)."""
resp = client.post("/search", json={"query": "food", "user_id": "u1"})
assert resp.status_code == 200
_, kwargs = mock_memory.search.call_args
assert "limit" not in kwargs
assert "top_k" not in kwargs
# ===========================================================================
@@ -110,18 +110,18 @@ class TestSearchThreshold:
# ===========================================================================
# SearchRequest: limit + threshold together
# SearchRequest: top_k + threshold together
# ===========================================================================
class TestSearchLimitAndThreshold:
def test_both_forwarded(self, client, mock_memory):
resp = client.post("/search", json={
"query": "food", "user_id": "u1", "limit": 10, "threshold": 0.5
"query": "food", "user_id": "u1", "top_k": 10, "threshold": 0.5
})
assert resp.status_code == 200
_, kwargs = mock_memory.search.call_args
assert kwargs["limit"] == 10
assert kwargs["top_k"] == 10
assert kwargs["threshold"] == 0.5
@@ -334,8 +334,8 @@ class TestOpenAPISchema:
def test_search_schema_includes_limit(self, client):
schema = client.get("/openapi.json").json()
search_props = schema["components"]["schemas"]["SearchRequest"]["properties"]
assert "limit" in search_props
assert search_props["limit"]["description"] == "Maximum number of results to return."
assert "top_k" in search_props
assert search_props["top_k"]["description"] == "Maximum number of results to return."
def test_search_schema_includes_threshold(self, client):
schema = client.get("/openapi.json").json()
@@ -367,7 +367,7 @@ class TestTypeValidation:
def test_limit_string_rejected(self, client):
resp = client.post("/search", json={
"query": "food", "user_id": "u1", "limit": "not_a_number",
"query": "food", "user_id": "u1", "top_k": "not_a_number",
})
assert resp.status_code == 422
@@ -399,7 +399,7 @@ class TestTypeValidation:
def test_limit_float_rejected(self, client):
resp = client.post("/search", json={
"query": "food", "user_id": "u1", "limit": 5.7,
"query": "food", "user_id": "u1", "top_k": 5.7,
})
assert resp.status_code == 422
@@ -422,11 +422,11 @@ class TestExplicitNull:
def test_limit_null_uses_memory_default(self, client, mock_memory):
resp = client.post("/search", json={
"query": "food", "user_id": "u1", "limit": None,
"query": "food", "user_id": "u1", "top_k": None,
})
assert resp.status_code == 200
_, kwargs = mock_memory.search.call_args
assert "limit" not in kwargs
assert "top_k" not in kwargs
def test_infer_null_uses_memory_default(self, client, mock_memory):
resp = client.post("/memories", json={
@@ -462,12 +462,12 @@ class TestCallSignatureMatch:
resp = client.post("/search", json={
"query": "food", "user_id": "u1", "agent_id": "a1",
"run_id": "r1", "filters": {"k": "v"},
"limit": 10, "threshold": 0.5,
"top_k": 10, "threshold": 0.5,
})
assert resp.status_code == 200
# The handler passes query= as a keyword arg, so it appears in kwargs too
_, kwargs = mock_memory.search.call_args
valid_params = {"query", "user_id", "agent_id", "run_id", "limit", "filters", "threshold", "rerank"}
valid_params = {"query", "user_id", "agent_id", "run_id", "top_k", "filters", "threshold", "rerank"}
for key in kwargs:
assert key in valid_params, f"Unexpected kwarg '{key}' forwarded to Memory.search()"
+368 -3
View File
@@ -1,4 +1,6 @@
import json
import os
import pickle
import tempfile
from unittest.mock import Mock, patch
@@ -6,7 +8,13 @@ import faiss
import numpy as np
import pytest
from mem0.vector_stores.faiss import FAISS, OutputData
from mem0.vector_stores.faiss import (
FAISS,
OutputData,
SafeUnpickler,
_safe_pickle_load,
_validate_docstore_structure,
)
@pytest.fixture
@@ -273,8 +281,8 @@ def test_delete_col(faiss_instance):
# Call delete_col
faiss_instance.delete_col()
# Verify os.remove was called twice (for index and docstore files)
assert mock_remove.call_count == 2
# Verify os.remove was called for index, json docstore, and legacy pkl files
assert mock_remove.call_count == 3
# Verify the internal state was reset
assert faiss_instance.index is None
@@ -299,3 +307,360 @@ def test_normalize_L2(faiss_instance, mock_faiss_index):
# Verify faiss.normalize_L2 was called
mock_normalize.assert_called_once()
# =============================================================================
# Security Tests for Pickle Deserialization Vulnerability Fix
# =============================================================================
class TestSafeUnpickler:
"""Tests for the SafeUnpickler class that prevents arbitrary code execution."""
def test_safe_unpickler_allows_basic_types(self):
"""SafeUnpickler should allow basic Python types."""
# Create a legitimate pickle with basic types
data = (
{"key1": "value1", "key2": {"nested": "dict"}},
{0: "id1", 1: "id2"},
)
pickled = pickle.dumps(data)
# Should load successfully
import io
result = SafeUnpickler(io.BytesIO(pickled)).load()
assert result == data
def test_safe_unpickler_blocks_os_system(self):
"""SafeUnpickler should block os.system execution attempts."""
# Generate the malicious payload dynamically to ensure correct format
import io
class Evil:
def __reduce__(self):
return (os.system, ("echo pwned",))
malicious_payload = pickle.dumps(Evil())
with pytest.raises(pickle.UnpicklingError) as exc_info:
SafeUnpickler(io.BytesIO(malicious_payload)).load()
assert "Unsafe pickle" in str(exc_info.value)
assert "posix.system" in str(exc_info.value)
def test_safe_unpickler_blocks_subprocess(self):
"""SafeUnpickler should block subprocess execution attempts."""
import subprocess
# Create a malicious pickle that tries to use subprocess
class MaliciousSubprocess:
def __reduce__(self):
return (subprocess.call, (["echo", "pwned"],))
malicious_payload = pickle.dumps(MaliciousSubprocess())
import io
with pytest.raises(pickle.UnpicklingError) as exc_info:
SafeUnpickler(io.BytesIO(malicious_payload)).load()
assert "Unsafe pickle" in str(exc_info.value)
def test_safe_unpickler_blocks_eval(self):
"""SafeUnpickler should block eval/exec attempts."""
# Create a malicious pickle that tries to use eval
class MaliciousEval:
def __reduce__(self):
return (eval, ("__import__('os').system('touch pwned')",))
malicious_payload = pickle.dumps(MaliciousEval())
import io
with pytest.raises(pickle.UnpicklingError) as exc_info:
SafeUnpickler(io.BytesIO(malicious_payload)).load()
assert "Unsafe pickle" in str(exc_info.value)
def test_safe_unpickler_blocks_arbitrary_modules(self):
"""SafeUnpickler should block imports from arbitrary modules."""
# Create a pickle that tries to load a class from a non-builtins module
class ArbitraryClass:
def __reduce__(self):
return (type, ("Evil", (), {}))
malicious_payload = pickle.dumps(ArbitraryClass())
import io
# This should either work (type is a builtin) or fail safely
# The key is it shouldn't execute arbitrary code
try:
result = SafeUnpickler(io.BytesIO(malicious_payload)).load()
# If it loads, verify it's just a benign type object
assert isinstance(result, type)
except pickle.UnpicklingError:
# This is also acceptable - blocking unknown patterns
pass
class TestSafePickleLoad:
"""Tests for the _safe_pickle_load function."""
def test_safe_pickle_load_with_valid_file(self):
"""_safe_pickle_load should load valid pickle files."""
with tempfile.NamedTemporaryFile(mode="wb", suffix=".pkl", delete=False) as f:
data = ({"id1": {"data": "test"}}, {0: "id1"})
pickle.dump(data, f)
temp_path = f.name
try:
result = _safe_pickle_load(temp_path)
assert result == data
finally:
os.unlink(temp_path)
def test_safe_pickle_load_blocks_malicious_file(self):
"""_safe_pickle_load should block malicious pickle files."""
# Generate the malicious payload dynamically
class Evil:
def __reduce__(self):
return (os.system, ("echo pwned",))
malicious_payload = pickle.dumps(Evil())
with tempfile.NamedTemporaryFile(mode="wb", suffix=".pkl", delete=False) as f:
f.write(malicious_payload)
temp_path = f.name
try:
with pytest.raises(pickle.UnpicklingError) as exc_info:
_safe_pickle_load(temp_path)
assert "Unsafe pickle" in str(exc_info.value)
finally:
os.unlink(temp_path)
class TestValidateDocstoreStructure:
"""Tests for the _validate_docstore_structure function."""
def test_valid_structure(self):
"""Should accept valid docstore structure."""
data = ({"id1": {"data": "test"}}, {0: "id1"})
docstore, index_to_id = _validate_docstore_structure(data)
assert docstore == {"id1": {"data": "test"}}
assert index_to_id == {0: "id1"}
def test_invalid_tuple_length(self):
"""Should reject tuples with wrong length."""
with pytest.raises(ValueError, match="expected tuple"):
_validate_docstore_structure(({}, {}, {}))
def test_invalid_docstore_type(self):
"""Should reject non-dict docstore."""
with pytest.raises(ValueError, match="docstore must be a dict"):
_validate_docstore_structure(("not a dict", {}))
def test_invalid_index_to_id_type(self):
"""Should reject non-dict index_to_id."""
with pytest.raises(ValueError, match="index_to_id must be a dict"):
_validate_docstore_structure(({}, "not a dict"))
def test_invalid_docstore_key_type(self):
"""Should reject non-string docstore keys."""
with pytest.raises(ValueError, match="Invalid docstore key type"):
_validate_docstore_structure(({123: {"data": "test"}}, {0: "id1"}))
def test_invalid_index_to_id_key_type(self):
"""Should reject non-int index_to_id keys."""
with pytest.raises(ValueError, match="Invalid index_to_id key type"):
_validate_docstore_structure(({"id1": {"data": "test"}}, {"0": "id1"}))
class TestFAISSSecurityIntegration:
"""Integration tests for FAISS security fixes."""
def test_faiss_saves_as_json(self):
"""FAISS should save docstore as JSON, not pickle."""
with tempfile.TemporaryDirectory() as temp_dir:
mock_index = Mock()
mock_index.d = 128
mock_index.ntotal = 0
with patch("mem0.vector_stores.faiss.faiss.IndexFlatL2", return_value=mock_index):
with patch("mem0.vector_stores.faiss.faiss.write_index"):
faiss_store = FAISS(
collection_name="test_security",
path=os.path.join(temp_dir, "test_faiss"),
distance_strategy="euclidean",
)
faiss_store.index = mock_index
# Insert some data
faiss_store.docstore = {"id1": {"data": "test"}}
faiss_store.index_to_id = {0: "id1"}
faiss_store._save()
# Verify JSON file was created
json_path = os.path.join(temp_dir, "test_faiss", "test_security.json")
pkl_path = os.path.join(temp_dir, "test_faiss", "test_security.pkl")
assert os.path.exists(json_path), "JSON docstore file should be created"
assert not os.path.exists(pkl_path), "Pickle file should NOT be created"
# Verify JSON content
with open(json_path, "r") as f:
data = json.load(f)
assert data["docstore"] == {"id1": {"data": "test"}}
assert data["index_to_id"] == {"0": "id1"}
def test_faiss_loads_json_preferentially(self):
"""FAISS should prefer JSON over pickle when both exist."""
with tempfile.TemporaryDirectory() as temp_dir:
faiss_path = os.path.join(temp_dir, "test_faiss")
os.makedirs(faiss_path)
# Create both JSON and pickle files with different data
json_data = {"docstore": {"id1": {"source": "json"}}, "index_to_id": {"0": "id1"}}
pkl_data = ({"id1": {"source": "pickle"}}, {0: "id1"})
with open(os.path.join(faiss_path, "test_pref.json"), "w") as f:
json.dump(json_data, f)
with open(os.path.join(faiss_path, "test_pref.pkl"), "wb") as f:
pickle.dump(pkl_data, f)
mock_index = Mock()
mock_index.d = 128
mock_index.ntotal = 1
with patch("mem0.vector_stores.faiss.faiss.read_index", return_value=mock_index):
with patch("mem0.vector_stores.faiss.faiss.write_index"):
faiss_store = FAISS.__new__(FAISS)
faiss_store.collection_name = "test_pref"
faiss_store.path = faiss_path
faiss_store.index = None
faiss_store.docstore = {}
faiss_store.index_to_id = {}
faiss_store._load(
os.path.join(faiss_path, "test_pref.faiss"),
os.path.join(faiss_path, "test_pref.pkl"),
)
# Should have loaded from JSON, not pickle
assert faiss_store.docstore == {"id1": {"source": "json"}}
def test_faiss_blocks_malicious_pickle_on_load(self):
"""FAISS should block loading of malicious pickle files."""
with tempfile.TemporaryDirectory() as temp_dir:
faiss_path = os.path.join(temp_dir, "test_faiss")
os.makedirs(faiss_path)
# Create a malicious pickle file (RCE payload)
class Evil:
def __reduce__(self):
return (os.system, (f"touch {temp_dir}/pwned",))
malicious_payload = pickle.dumps(Evil())
with open(os.path.join(faiss_path, "malicious.pkl"), "wb") as f:
f.write(malicious_payload)
mock_index = Mock()
mock_index.ntotal = 1
with patch("mem0.vector_stores.faiss.faiss.read_index", return_value=mock_index):
faiss_store = FAISS.__new__(FAISS)
faiss_store.collection_name = "malicious"
faiss_store.path = faiss_path
faiss_store.index = None
faiss_store.docstore = {}
faiss_store.index_to_id = {}
# Should raise an error, not execute the malicious payload
with pytest.raises(ValueError) as exc_info:
faiss_store._load(
os.path.join(faiss_path, "malicious.faiss"),
os.path.join(faiss_path, "malicious.pkl"),
)
assert "malicious pickle" in str(exc_info.value).lower() or "unsafe" in str(exc_info.value).lower()
# Verify the malicious command was NOT executed
pwned_file = os.path.join(temp_dir, "pwned")
assert not os.path.exists(pwned_file), "Malicious payload should NOT have been executed!"
def test_faiss_migrates_legacy_pickle_to_json(self):
"""FAISS should auto-migrate valid pickle files to JSON format."""
with tempfile.TemporaryDirectory() as temp_dir:
faiss_path = os.path.join(temp_dir, "test_faiss")
os.makedirs(faiss_path)
# Create a legitimate legacy pickle file
pkl_data = ({"id1": {"data": "legacy"}}, {0: "id1"})
with open(os.path.join(faiss_path, "legacy.pkl"), "wb") as f:
pickle.dump(pkl_data, f)
mock_index = Mock()
mock_index.d = 128
mock_index.ntotal = 1
with patch("mem0.vector_stores.faiss.faiss.read_index", return_value=mock_index):
with patch("mem0.vector_stores.faiss.faiss.write_index"):
faiss_store = FAISS.__new__(FAISS)
faiss_store.collection_name = "legacy"
faiss_store.path = faiss_path
faiss_store.index = None
faiss_store.docstore = {}
faiss_store.index_to_id = {}
faiss_store._load(
os.path.join(faiss_path, "legacy.faiss"),
os.path.join(faiss_path, "legacy.pkl"),
)
# Data should be loaded correctly
assert faiss_store.docstore == {"id1": {"data": "legacy"}}
assert faiss_store.index_to_id == {0: "id1"}
# JSON file should now exist (auto-migrated)
json_path = os.path.join(faiss_path, "legacy.json")
assert os.path.exists(json_path), "JSON file should be created during migration"
def test_delete_col_removes_json_and_pkl(self):
"""delete_col should remove both JSON and legacy pickle files."""
with tempfile.TemporaryDirectory() as temp_dir:
faiss_path = os.path.join(temp_dir, "test_faiss")
os.makedirs(faiss_path)
# Create both file types
json_path = os.path.join(faiss_path, "test_del.json")
pkl_path = os.path.join(faiss_path, "test_del.pkl")
faiss_index_path = os.path.join(faiss_path, "test_del.faiss")
with open(json_path, "w") as f:
json.dump({"docstore": {}, "index_to_id": {}}, f)
with open(pkl_path, "wb") as f:
pickle.dump(({}, {}), f)
with open(faiss_index_path, "w") as f:
f.write("dummy")
with patch("faiss.IndexFlatL2"):
faiss_store = FAISS.__new__(FAISS)
faiss_store.collection_name = "test_del"
faiss_store.path = faiss_path
faiss_store.index = Mock()
faiss_store.docstore = {}
faiss_store.index_to_id = {}
faiss_store.delete_col()
# Both files should be deleted
assert not os.path.exists(json_path), "JSON file should be deleted"
assert not os.path.exists(pkl_path), "PKL file should be deleted"
assert not os.path.exists(faiss_index_path), "FAISS index should be deleted"
+120 -6
View File
@@ -222,9 +222,7 @@ def test_update_with_none_vector_preserves_embedding(valkey_db, mock_valkey_clie
mock_valkey_client.hset.assert_called_once()
args, kwargs = mock_valkey_client.hset.call_args
assert "embedding" not in kwargs["mapping"], (
"embedding should not be in hash_data when vector is None"
)
assert "embedding" not in kwargs["mapping"], "embedding should not be in hash_data when vector is None"
assert kwargs["mapping"]["memory_id"] == "test_id"
assert kwargs["mapping"]["memory"] == "updated_data"
@@ -243,9 +241,7 @@ def test_update_with_vector_includes_embedding(valkey_db, mock_valkey_client):
mock_valkey_client.hset.assert_called_once()
args, kwargs = mock_valkey_client.hset.call_args
assert "embedding" in kwargs["mapping"], (
"embedding should be in hash_data when vector is provided"
)
assert "embedding" in kwargs["mapping"], "embedding should be in hash_data when vector is provided"
expected_bytes = np.array(vector, dtype=np.float32).tobytes()
assert kwargs["mapping"]["embedding"] == expected_bytes
@@ -906,3 +902,121 @@ def test_list_with_missing_fields_and_defaults(valkey_db, mock_valkey_client):
assert result.id == "fallback_id"
assert "hash" in result.payload
assert "data" in result.payload # memory is renamed to data
# Cluster mode tests
@pytest.fixture
def mock_valkey_cluster_client():
"""Create a mock ValkeyCluster client."""
with patch("valkey.cluster.ValkeyCluster.from_url") as mock_from_url:
mock_client = MagicMock()
mock_ft = MagicMock()
mock_client.ft = MagicMock(return_value=mock_ft)
mock_client.execute_command = MagicMock()
mock_client.hset = MagicMock()
mock_client.hgetall = MagicMock()
mock_client.delete = MagicMock()
mock_from_url.return_value = mock_client
yield mock_client
@pytest.fixture
def valkey_db_cluster(mock_valkey_cluster_client):
"""Create a ValkeyDB instance in cluster mode with a mock client."""
valkey_db = ValkeyDB(
valkey_url="valkey://localhost:7000",
collection_name="test_cluster",
embedding_model_dims=1536,
cluster_mode=True,
)
valkey_db.client = mock_valkey_cluster_client
return valkey_db
def test_cluster_mode_init(mock_valkey_cluster_client):
"""Test that cluster_mode=True uses ValkeyCluster client."""
db = ValkeyDB(
valkey_url="valkey://localhost:7000",
collection_name="test_cluster",
embedding_model_dims=1536,
cluster_mode=True,
)
assert db.cluster_mode is True
def test_cluster_mode_create_index(valkey_db_cluster, mock_valkey_cluster_client):
"""Test that index creation works in cluster mode (server handles propagation)."""
mock_valkey_cluster_client.execute_command.reset_mock()
mock_valkey_cluster_client.ft.return_value.info.side_effect = ResponseError("not found")
valkey_db_cluster._create_index(1536)
call_args = mock_valkey_cluster_client.execute_command.call_args
assert call_args is not None
assert "FT.CREATE" in call_args.args
def test_cluster_mode_drop_index(valkey_db_cluster, mock_valkey_cluster_client):
"""Test that dropping index works in cluster mode."""
mock_valkey_cluster_client.execute_command.reset_mock()
valkey_db_cluster._drop_index("test_cluster")
call_args = mock_valkey_cluster_client.execute_command.call_args
assert "FT.DROPINDEX" in call_args.args
def test_cluster_mode_search(valkey_db_cluster, mock_valkey_cluster_client):
"""Test that search in cluster mode uses ft().search() (server handles cross-shard fan-out)."""
ts = str(int(datetime.now().timestamp()))
mock_doc = MagicMock()
mock_doc.memory_id = "id1"
mock_doc.hash = "h1"
mock_doc.memory = "data1"
mock_doc.created_at = ts
mock_doc.metadata = "{}"
mock_doc.vector_score = "0.1"
mock_results = MagicMock()
mock_results.docs = [mock_doc]
mock_valkey_cluster_client.ft.return_value.search.return_value = mock_results
results = valkey_db_cluster.search("test", np.random.rand(1536).tolist(), top_k=5)
assert len(results) == 1
assert results[0].id == "id1"
mock_valkey_cluster_client.ft.return_value.search.assert_called_once()
def test_cluster_mode_insert(valkey_db_cluster, mock_valkey_cluster_client):
"""Test that insert works in cluster mode (ValkeyCluster handles routing)."""
vectors = [np.random.rand(1536).tolist()]
payloads = [{"hash": "h1", "data": "test", "user_id": "u1"}]
ids = ["id1"]
valkey_db_cluster.insert(vectors=vectors, payloads=payloads, ids=ids)
mock_valkey_cluster_client.hset.assert_called_once()
def test_cluster_mode_get(valkey_db_cluster, mock_valkey_cluster_client):
"""Test that get works in cluster mode."""
mock_valkey_cluster_client.hgetall.return_value = {
"memory_id": "id1",
"hash": "h1",
"memory": "test_data",
"created_at": str(int(datetime.now().timestamp())),
"metadata": "{}",
}
result = valkey_db_cluster.get("id1")
assert result.id == "id1"
assert result.payload["data"] == "test_data"
def test_cluster_mode_delete(valkey_db_cluster, mock_valkey_cluster_client):
"""Test that delete works in cluster mode."""
valkey_db_cluster.delete("id1")
mock_valkey_cluster_client.delete.assert_called_once_with("mem0:test_cluster:id1")