Soumil RathiandClaude Opus 4.6
a33c557a25
Replace the 2-LLM-call add pipeline with a single-pass additive extraction
using ADDITIVE_EXTRACTION_PROMPT. Memories now accumulate (ADD-only) instead
of being updated/deleted during extraction.
Search pipeline upgraded to hybrid scoring combining three signals:
- Semantic similarity (vector search)
- BM25 keyword matching (native per vector store, 15 stores supported)
- Entity boost (spaCy NER with entity collection linking)
Combined score = (semantic + bm25 + entity_boost) / max_possible, where
max_possible adapts based on which signals are active.
Key changes:
- Add spaCy-based lemmatization for BM25 keyword search
- Add spaCy-based entity extraction (PROPER, QUOTED, COMPOUND, NOUN types)
- Add entity store as second vector collection ({collection}_entities)
- Add native keyword_search() to 15 vector store adapters
- Add batch embedding support (embed_batch) for OpenAI and Azure OpenAI
- Add message persistence in SQLite (rolling window of 10 per scope)
- Add additive scoring with adaptive normalization
- Add observation_date parameter to add()
- Add custom_instructions config field
- Default search threshold changed to 0.1
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
60 lines
2.2 KiB
Python
60 lines
2.2 KiB
Python
import os
|
|
import warnings
|
|
from typing import Literal, Optional
|
|
|
|
from openai import OpenAI
|
|
|
|
from mem0.configs.embeddings.base import BaseEmbedderConfig
|
|
from mem0.embeddings.base import EmbeddingBase
|
|
|
|
|
|
class OpenAIEmbedding(EmbeddingBase):
|
|
def __init__(self, config: Optional[BaseEmbedderConfig] = None):
|
|
super().__init__(config)
|
|
|
|
self.config.model = self.config.model or "text-embedding-3-small"
|
|
self.config.embedding_dims = self.config.embedding_dims or 1536
|
|
|
|
api_key = self.config.api_key or os.getenv("OPENAI_API_KEY")
|
|
base_url = (
|
|
self.config.openai_base_url
|
|
or os.getenv("OPENAI_API_BASE")
|
|
or os.getenv("OPENAI_BASE_URL")
|
|
or "https://api.openai.com/v1"
|
|
)
|
|
if os.environ.get("OPENAI_API_BASE"):
|
|
warnings.warn(
|
|
"The environment variable 'OPENAI_API_BASE' is deprecated and will be removed in the 0.1.80. "
|
|
"Please use 'OPENAI_BASE_URL' instead.",
|
|
DeprecationWarning,
|
|
)
|
|
|
|
self.client = OpenAI(api_key=api_key, base_url=base_url)
|
|
|
|
def embed(self, text, memory_action: Optional[Literal["add", "search", "update"]] = None):
|
|
"""
|
|
Get the embedding for the given text using OpenAI.
|
|
|
|
Args:
|
|
text (str): The text to embed.
|
|
memory_action (optional): The type of embedding to use. Must be one of "add", "search", or "update". Defaults to None.
|
|
Returns:
|
|
list: The embedding vector.
|
|
"""
|
|
text = text.replace("\n", " ")
|
|
return (
|
|
self.client.embeddings.create(input=[text], model=self.config.model, dimensions=self.config.embedding_dims)
|
|
.data[0]
|
|
.embedding
|
|
)
|
|
|
|
def embed_batch(self, texts, memory_action="add"):
|
|
"""Embed multiple texts in a single OpenAI API call."""
|
|
texts = [text.replace("\n", " ") for text in texts]
|
|
response = self.client.embeddings.create(
|
|
input=texts,
|
|
model=self.config.model,
|
|
dimensions=self.config.embedding_dims,
|
|
)
|
|
return [item.embedding for item in sorted(response.data, key=lambda x: x.index)]
|