feat(oss): port v3 pipeline with hybrid search, entity extraction, and additive scoring

Replace the 2-LLM-call add pipeline with a single-pass additive extraction
using ADDITIVE_EXTRACTION_PROMPT. Memories now accumulate (ADD-only) instead
of being updated/deleted during extraction.

Search pipeline upgraded to hybrid scoring combining three signals:
- Semantic similarity (vector search)
- BM25 keyword matching (native per vector store, 15 stores supported)
- Entity boost (spaCy NER with entity collection linking)

Combined score = (semantic + bm25 + entity_boost) / max_possible, where
max_possible adapts based on which signals are active.

Key changes:
- Add spaCy-based lemmatization for BM25 keyword search
- Add spaCy-based entity extraction (PROPER, QUOTED, COMPOUND, NOUN types)
- Add entity store as second vector collection ({collection}_entities)
- Add native keyword_search() to 15 vector store adapters
- Add batch embedding support (embed_batch) for OpenAI and Azure OpenAI
- Add message persistence in SQLite (rolling window of 10 per scope)
- Add additive scoring with adaptive normalization
- Add observation_date parameter to add()
- Add custom_instructions config field
- Default search threshold changed to 0.1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Soumil Rathi
2026-03-31 14:09:37 -07:00
co-authored by Claude Opus 4.6
parent 861cbb7289
commit a33c557a25
34 changed files with 2980 additions and 255 deletions
+3 -2
View File
@@ -14,13 +14,14 @@ license = "Apache-2.0"
license-files = ["LICENSE"]
requires-python = ">=3.9,<4.0"
dependencies = [
"qdrant-client>=1.9.1",
"qdrant-client>=1.12.0",
"pydantic>=2.7.3",
"openai>=1.90.0",
"posthog>=3.5.0",
"pytz>=2024.1",
"sqlalchemy>=2.0.31",
"protobuf>=5.29.0,<6.0.0",
"spacy>=3.7.0",
]
[project.optional-dependencies]
@@ -40,7 +41,7 @@ vector_stores = [
"pinecone<=7.3.0",
"pinecone-text>=0.10.0",
"faiss-cpu>=1.7.4",
"upstash-vector>=0.1.0",
"upstash-vector>=0.6.0",
"azure-search-documents>=11.4.0b8",
"psycopg>=3.2.8",
"psycopg-pool>=3.2.6,<4.0.0",