- Replace sequential _upsert_entity() loop with batched pipeline:
1. Global dedup: collect unique entities across all memories
2. Single embed_batch() call for all entity texts
3. Batch search via search_batch() (Qdrant query_batch_points)
4. Batch insert for new entities, individual updates for existing
- Add search_batch() to VectorStoreBase (sequential fallback)
- Add native search_batch() to Qdrant using query_batch_points
- ~6-10x reduction in entity linking network round trips
Streamline prompt guidelines and update examples for clarity.
Remove redundant sections and verbose explanations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move spacy from core dependencies to optional [nlp] extra.
The system gracefully degrades without spaCy:
- Lemmatization falls back to raw text
- Entity extraction returns empty (no entity boost)
- BM25 keyword search still works on raw text
- Semantic search and additive scoring unaffected
Install with NLP support: pip install mem0ai[nlp]
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add text_lemmatized to _create_memory() and _update_memory() (sync + async)
so BM25/keyword search works for all code paths, not just the batch pipeline
- pgvector: add GIN index on text_lemmatized for fast full-text search
- pgvector: add error logging to keyword_search
- milvus: add sparse BM25 field + function to create_col() so keyword_search works
- mongodb: auto-create Atlas Search text index in create_col()
- azure_mysql: add generated column + FULLTEXT index for keyword_search
- Add sparse_vectors_config with 'bm25' named vector to create_col()
- Compute BM25 sparse vectors from text_lemmatized during insert/update
- Use client-side fastembed Qdrant/bm25 encoder instead of server-side inference
- Graceful fallback if fastembed not installed
- Fixes keyword_search() returning None on self-hosted Qdrant
- Rename `observation_date` param to `timestamp` (matches platform)
- Change `rerank` default from True to False (matches platform)
- Internal prompt builder still uses "Observation Date" section heading
for LLM compatibility with ADDITIVE_EXTRACTION_PROMPT
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the 2-LLM-call add pipeline with a single-pass additive extraction
using ADDITIVE_EXTRACTION_PROMPT. Memories now accumulate (ADD-only) instead
of being updated/deleted during extraction.
Search pipeline upgraded to hybrid scoring combining three signals:
- Semantic similarity (vector search)
- BM25 keyword matching (native per vector store, 15 stores supported)
- Entity boost (spaCy NER with entity collection linking)
Combined score = (semantic + bm25 + entity_boost) / max_possible, where
max_possible adapts based on which signals are active.
Key changes:
- Add spaCy-based lemmatization for BM25 keyword search
- Add spaCy-based entity extraction (PROPER, QUOTED, COMPOUND, NOUN types)
- Add entity store as second vector collection ({collection}_entities)
- Add native keyword_search() to 15 vector store adapters
- Add batch embedding support (embed_batch) for OpenAI and Azure OpenAI
- Add message persistence in SQLite (rolling window of 10 per scope)
- Add additive scoring with adaptive normalization
- Add observation_date parameter to add()
- Add custom_instructions config field
- Default search threshold changed to 0.1
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Soumil Rathi