Commit Graph
26 Commits
Author SHA1 Message Date
Soumil Rathi 7410d190fc Merge branch 'feat/v3-pipeline' of https://github.com/mem0ai/mem0 into feat/v3-pipeline 2026-04-13 13:19:34 -07:00
Soumil RathiandClaude Opus 4.6 7c313add10 fix: update search() return type to match v3 response format
Change TS client search() return type from Promise<Array<Memory>>
to Promise<{ results: Array<Memory> }> to match v3 API envelope.
Update unit test mocks to return {results: []} and fix assertions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 13:19:26 -07:00
Soumil RathiandClaude Opus 4.6 7f19c499ca fix: delete remaining graph test files
Remove test_neo4j_cypher_syntax.py and test_graph_delete_docker.py
which reference deleted graph_memory module.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 13:15:03 -07:00
Soumil RathiandClaude Opus 4.6 c5e748b469 fix: remove all graph references from tests/test_main.py
Strip graph_memory patches, GraphStoreFactory mocks, graph_store config,
enable_graph parametrize, _add_to_graph assertions, and "relations"
assertions from all test fixtures and test functions. Graph store has
been removed from the codebase.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 13:14:40 -07:00
Soumil RathiandClaude Opus 4.6 b9d7ae5fa7 fix: update integration tests for v3 search response format
v3 search returns {results: [...]} instead of a bare array.
Updated waitForSearchResults helper and search edge case tests
to unwrap the v3 envelope.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 13:10:17 -07:00
Soumil RathiandClaude Opus 4.6 78cc11ef1f fix: restore lmstudio/deepseek providers in TS factory, add langchain to extras
- Restore LMStudioEmbedder, LMStudioLLM, DeepSeekLLM in TS factory.ts
  (accidentally removed during graph store cleanup — unrelated to graphs)
- Add langchain>=0.1.0 to Python extras dep group (was transitively
  provided by langchain-neo4j in the removed graph group, still needed
  by mem0/llms/langchain.py and mem0/embeddings/langchain.py)
- Add pytest.importorskip("langchain") to test_langchain.py for graceful
  skip when langchain isn't installed

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 12:55:36 -07:00
Soumil RathiandClaude Opus 4.6 1a2edf739d fix: delete graph test files that import removed graph modules
These test files import mem0.memory.graph_memory, kuzu_memory,
memgraph_memory, apache_age_memory, and mem0.graphs.neptune which
were deleted in the graph store removal commit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 12:17:35 -07:00
Soumil RathiandClaude Opus 4.6 66446bd118 fix: address reviewer bugs — metadata mutation, test args, embedding dict, output_format
1. Fix _create_procedural_memory metadata mutation: use spread operator
   instead of in-place dict assignment (sync + async)
2. Fix test_add assertion: remove trailing None arg that doesn't match
   _add_to_vector_store(messages, metadata, filters, infer) signature
3. Restore output_format: "v1.1" in TS client search payload as fallback
4. Fix infer=False path: pass {text: embedding} dict to _create_memory
   instead of raw embedding list to avoid redundant embed() calls (sync + async)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 11:50:56 -07:00
Soumil RathiandClaude Opus 4.6 163444e33b fix: update TS client tests for v3 API endpoints
Update mock URLs in memoryClient.crud.test.ts and
memoryClient.search.test.ts to match v3 endpoints:
- add: /v1/memories/ -> /v3/memories/
- search: /v2/memories/search/ -> /v3/memories/search/
- deleteAll: stays on /v1/memories/ (no v3 equivalent)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 11:34:52 -07:00
Soumil RathiandClaude Opus 4.6 e1e033daef style: run prettier on TS SDK files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 11:29:06 -07:00
Soumil RathiandClaude Opus 4.6 3ae22be8a2 fix: resolve CI failures — lint errors, pytz in redis, lockfile mismatch
- Remove unused imports: get_update_memory_messages, get_fact_retrieval_messages
- Fix pytz.timezone("US/Pacific") → timezone.utc in redis vector store
- Remove unused entity_types variable in test_entity_extraction
- Regenerate pnpm-lock.yaml after neo4j-driver removal

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 11:24:14 -07:00
Soumil RathiandClaude Opus 4.6 730b2ff422 fix: restore complete _SENSITIVE_FIELDS_EXACT list from original commit
Was missing 13 entries (credentials, token, access_token, refresh_token,
auth_token, session_token, client_secret, auth_client_secret,
azure_client_secret, service_account_json, aws_session_token, credential,
secret). Now matches the original list from commit 46b4b2e9.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 11:03:49 -07:00
Soumil RathiandClaude Opus 4.6 573ca54637 fix: restore _is_sensitive_field and sensitive field redaction in _safe_deepcopy_config
The function and associated constants (_SENSITIVE_FIELDS_EXACT,
_SENSITIVE_SUFFIXES) were accidentally dropped during the merge.
Restores telemetry-safe config cloning that redacts API keys,
passwords, and other secrets while preserving runtime auth objects.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 10:53:56 -07:00
Soumil RathiandClaude Opus 4.6 59880a6f8f feat: TS SDK v3 port, graph store removal, client v3 API migration
TypeScript SDK v3 pipeline (full parity with Python):
- Single-pass additive extraction with ADDITIVE_EXTRACTION_PROMPT
- Hybrid search (semantic + BM25 + entity boost) with additive scoring
- 8-phase batch pipeline (batch embed, persist, entity linking)
- New utils: scoring.ts, lemmatization.ts (natural), entity_extraction.ts (compromise)
- keywordSearch() on 8 vector stores (3 full: PGVector, Memory, Azure AI Search)
- Message persistence in SQLiteManager (rolling window of 10)
- Entity store as second vector collection
- MAX_BATCH=100 chunking guard on OpenAI/Azure embedBatch
- Updated default LLM model to gpt-4.1-nano-2025-04-14
- compromise + natural added as peer dependencies

Graph store removal (Python + TypeScript):
- Removed Neo4j, Memgraph, Kuzu, Neptune, Apache AGE integrations
- Deleted 18 graph-related files across both SDKs
- Removed GraphStoreFactory, GraphStoreConfig, graph_store config field
- Removed "relations" key from all API responses
- Removed graph optional dependency group from pyproject.toml
- Removed neo4j-driver from TS peerDependencies
- Simplified add/search/delete/reset (no more parallel graph operations)

Client SDK v3 API migration:
- add() endpoint: /v1/memories/ -> /v3/memories/ (async response)
- search() endpoint: /v2/memories/search/ -> /v3/memories/search/
- Removed output_format injection and v1.1 unwrapping logic
- Applied to both Python (sync + async) and TypeScript clients

Review feedback fixes:
- Removed deprecated custom_update_memory_prompt from MemoryConfig
- Added MAX_BATCH=100 chunking to Python + TS embed_batch
- Moved all inline imports to top level in main.py
- Cleaned up GraphStoreError dead code from exceptions.py

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 10:45:40 -07:00
Soumil RathiandClaude Opus 4.6 3402c67805 fix: address PR review feedback
- Remove deprecated custom_update_memory_prompt from MemoryConfig entirely
- Add MAX_BATCH=100 chunking guard to embed_batch in openai.py and azure_openai.py
- Move all inline imports (mem0.utils.*) to top-level in main.py
- Remove custom_update_memory_prompt from test fixture

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 09:01:07 -07:00
Soumil Rathi c0343aec1b Merge remote-tracking branch 'origin/feat/v3-pipeline' into feat/v3-pipeline
# Conflicts:
#	mem0-ts/src/oss/src/utils/factory.ts
#	mem0/vector_stores/milvus.py
#	mem0/vector_stores/qdrant.py
2026-04-12 16:30:05 -07:00
Soumil RathiandClaude Opus 4.6 b9e26acf4d merge: integrate main into feat/v3-pipeline + port AsyncMemory to v3
Merge 190 commits from main including:
- limit→top_k rename across all vector store APIs
- custom_fact_extraction_prompt→custom_instructions rename
- enable_graph flag removal (use self.graph truthiness)
- US/Pacific→timezone.utc normalization
- 18 Category A bug fixes (partial updates, graph cleanup on delete,
  telemetry gating, metadata mutation prevention, filter operator
  merging, delete timestamps, actor_id preservation, etc.)

Conflict resolutions:
- openai.py: merged _pass_dimensions_to_api with embed_batch
- milvus.py: merged partial-update logic with BM25 text field
- qdrant.py: took main's enhanced _create_filter, merged BM25
  sparse vectors with partial-update endpoints
- test_main.py: adopted main's test structure, removed obsolete
  test_custom_prompts, kept hybrid search over-fetch assertion

AsyncMemory v3 port (full parity):
- Rewrote AsyncMemory._add_to_vector_store with the same 8-phase
  batch pipeline as sync Memory (single LLM call, batch embed,
  hash dedup, batch persist, batch entity linking, message saving)
- All blocking calls wrapped in await asyncio.to_thread()
- Moved _build_session_scope to module level for shared access

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 16:26:10 -07:00
Soumil RathiandClaude Sonnet 4.6 90f9e6a550 Fix: use caller-provided timestamp for memory created_at instead of always using datetime.now()
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 12:52:43 -07:00
Soumil Rathi a6130c5436 perf: skip ThreadPoolExecutor when graph store is disabled
Avoids spawning 2 threads per add() call when graph is not configured.
Fixes thread leakage under high concurrency (was accumulating 3500+ threads
per worker). Calls _add_to_vector_store directly in the current thread.

Applied to both sync Memory and async AsyncMemory classes.
2026-04-03 07:42:32 -07:00
Soumil Rathi cb64d978fc perf: batch entity linking — embed, search, insert in bulk instead of per-entity
- Replace sequential _upsert_entity() loop with batched pipeline:
  1. Global dedup: collect unique entities across all memories
  2. Single embed_batch() call for all entity texts
  3. Batch search via search_batch() (Qdrant query_batch_points)
  4. Batch insert for new entities, individual updates for existing
- Add search_batch() to VectorStoreBase (sequential fallback)
- Add native search_batch() to Qdrant using query_batch_points
- ~6-10x reduction in entity linking network round trips
2026-04-02 19:32:29 -07:00
Soumil RathiandClaude Opus 4.6 41532d8850 refactor(oss): trim and update ADDITIVE_EXTRACTION_PROMPT
Streamline prompt guidelines and update examples for clarity.
Remove redundant sections and verbose explanations.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 12:11:52 -07:00
Soumil RathiandClaude Opus 4.6 9c9b3d54e2 feat(oss): make spaCy an optional dependency
Move spacy from core dependencies to optional [nlp] extra.
The system gracefully degrades without spaCy:
- Lemmatization falls back to raw text
- Entity extraction returns empty (no entity boost)
- BM25 keyword search still works on raw text
- Semantic search and additive scoring unaffected

Install with NLP support: pip install mem0ai[nlp]

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 11:04:38 -07:00
Soumil Rathi 70c62ea54f fix: add text_lemmatized to create/update memory + fix keyword search across vector stores
- Add text_lemmatized to _create_memory() and _update_memory() (sync + async)
  so BM25/keyword search works for all code paths, not just the batch pipeline
- pgvector: add GIN index on text_lemmatized for fast full-text search
- pgvector: add error logging to keyword_search
- milvus: add sparse BM25 field + function to create_col() so keyword_search works
- mongodb: auto-create Atlas Search text index in create_col()
- azure_mysql: add generated column + FULLTEXT index for keyword_search
2026-04-02 10:34:03 -07:00
Soumil Rathi 4534acdfbe feat(qdrant): implement client-side BM25 sparse vectors via fastembed
- Add sparse_vectors_config with 'bm25' named vector to create_col()
- Compute BM25 sparse vectors from text_lemmatized during insert/update
- Use client-side fastembed Qdrant/bm25 encoder instead of server-side inference
- Graceful fallback if fastembed not installed
- Fixes keyword_search() returning None on self-hosted Qdrant
2026-04-02 10:23:17 -07:00
Soumil RathiandClaude Opus 4.6 80749b92bf fix(oss): align API naming with platform v3
- Rename `observation_date` param to `timestamp` (matches platform)
- Change `rerank` default from True to False (matches platform)
- Internal prompt builder still uses "Observation Date" section heading
  for LLM compatibility with ADDITIVE_EXTRACTION_PROMPT

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 21:49:21 -07:00
Soumil RathiandClaude Opus 4.6 a33c557a25 feat(oss): port v3 pipeline with hybrid search, entity extraction, and additive scoring
Replace the 2-LLM-call add pipeline with a single-pass additive extraction
using ADDITIVE_EXTRACTION_PROMPT. Memories now accumulate (ADD-only) instead
of being updated/deleted during extraction.

Search pipeline upgraded to hybrid scoring combining three signals:
- Semantic similarity (vector search)
- BM25 keyword matching (native per vector store, 15 stores supported)
- Entity boost (spaCy NER with entity collection linking)

Combined score = (semantic + bm25 + entity_boost) / max_possible, where
max_possible adapts based on which signals are active.

Key changes:
- Add spaCy-based lemmatization for BM25 keyword search
- Add spaCy-based entity extraction (PROPER, QUOTED, COMPOUND, NOUN types)
- Add entity store as second vector collection ({collection}_entities)
- Add native keyword_search() to 15 vector store adapters
- Add batch embedding support (embed_batch) for OpenAI and Azure OpenAI
- Add message persistence in SQLite (rolling window of 10 per scope)
- Add additive scoring with adaptive normalization
- Add observation_date parameter to add()
- Add custom_instructions config field
- Default search threshold changed to 0.1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 14:09:37 -07:00