Aspen theme for 1.x (#3473)
This commit is contained in:
Parshva Daftari
@@ -0,0 +1,116 @@
|
||||
---
|
||||
title: Cohere
|
||||
---
|
||||
|
||||
Cohere provides state-of-the-art reranking models that can significantly improve the relevance of search results. Cohere's rerankers are optimized for various languages and use cases.
|
||||
|
||||
## Usage
|
||||
|
||||
To use Cohere's reranker with Mem0:
|
||||
|
||||
```python
|
||||
import os
|
||||
from mem0 import Memory
|
||||
|
||||
os.environ["COHERE_API_KEY"] = "your-cohere-api-key"
|
||||
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "cohere",
|
||||
"config": {
|
||||
"api_key": "your-cohere-api-key", # Can also use environment variable
|
||||
"model": "rerank-english-v3.0",
|
||||
"top_n": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
memory = Memory.from_config(config)
|
||||
|
||||
# Use memory as usual
|
||||
memory.add("I love playing basketball", user_id="alice")
|
||||
memory.add("I enjoy watching movies", user_id="alice")
|
||||
|
||||
# Search will now use Cohere reranking
|
||||
results = memory.search("What sports does Alice like?", user_id="alice")
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `api_key` | Cohere API key | Required |
|
||||
| `model` | Cohere rerank model | `rerank-english-v3.0` |
|
||||
| `top_n` | Number of results to return | `10` |
|
||||
|
||||
## Available Models
|
||||
|
||||
- `rerank-english-v3.0`: Latest English reranking model
|
||||
- `rerank-multilingual-v3.0`: Multilingual reranking model
|
||||
- `rerank-english-v2.0`: Previous English model
|
||||
- `rerank-multilingual-v2.0`: Previous multilingual model
|
||||
|
||||
## Example with Different Models
|
||||
|
||||
### English Reranker
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "cohere",
|
||||
"config": {
|
||||
"api_key": "your-cohere-api-key",
|
||||
"model": "rerank-english-v3.0",
|
||||
"top_n": 5
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Multilingual Reranker
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "cohere",
|
||||
"config": {
|
||||
"api_key": "your-cohere-api-key",
|
||||
"model": "rerank-multilingual-v3.0",
|
||||
"top_n": 8
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
You can set your Cohere API key as an environment variable:
|
||||
|
||||
```bash
|
||||
export COHERE_API_KEY="your-cohere-api-key"
|
||||
```
|
||||
|
||||
Then use the config without specifying the API key:
|
||||
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "cohere",
|
||||
"config": {
|
||||
"model": "rerank-english-v3.0",
|
||||
"top_n": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Getting Your API Key
|
||||
|
||||
1. Sign up at [Cohere](https://cohere.ai/)
|
||||
2. Navigate to the API keys section in your dashboard
|
||||
3. Generate a new API key
|
||||
4. Use this key in your configuration
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- Cohere rerankers work best with 10-100 candidate documents
|
||||
- Higher `top_n` values provide more comprehensive reranking but may increase latency
|
||||
- The v3.0 models generally provide better performance than v2.0 models
|
||||
@@ -0,0 +1,352 @@
|
||||
---
|
||||
title: Hugging Face Reranker
|
||||
description: 'Access thousands of reranking models from Hugging Face Hub'
|
||||
icon: "face-smile"
|
||||
iconType: "solid"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The Hugging Face reranker provider gives you access to thousands of reranking models available on the Hugging Face Hub. This includes popular models like BAAI's BGE rerankers and other state-of-the-art cross-encoder models.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Basic Setup
|
||||
|
||||
```python
|
||||
from mem0 import Memory
|
||||
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-base",
|
||||
"device": "cpu"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
m = Memory.from_config(config)
|
||||
```
|
||||
|
||||
### Configuration Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `model` | str | Required | Hugging Face model identifier |
|
||||
| `device` | str | "cpu" | Device to run model on ("cpu", "cuda", "mps") |
|
||||
| `batch_size` | int | 32 | Batch size for processing |
|
||||
| `max_length` | int | 512 | Maximum input sequence length |
|
||||
| `trust_remote_code` | bool | False | Allow remote code execution |
|
||||
|
||||
### Advanced Configuration
|
||||
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-large",
|
||||
"device": "cuda",
|
||||
"batch_size": 16,
|
||||
"max_length": 512,
|
||||
"trust_remote_code": False,
|
||||
"model_kwargs": {
|
||||
"torch_dtype": "float16"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Popular Models
|
||||
|
||||
### BGE Rerankers (Recommended)
|
||||
|
||||
```python
|
||||
# Base model - good balance of speed and quality
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-base",
|
||||
"device": "cuda"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Large model - better quality, slower
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-large",
|
||||
"device": "cuda"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# v2 models - latest improvements
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-v2-m3",
|
||||
"device": "cuda"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Multilingual Models
|
||||
|
||||
```python
|
||||
# Multilingual BGE reranker
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-v2-multilingual",
|
||||
"device": "cuda"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Domain-Specific Models
|
||||
|
||||
```python
|
||||
# For code search
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "microsoft/codebert-base",
|
||||
"device": "cuda"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# For biomedical content
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "dmis-lab/biobert-base-cased-v1.1",
|
||||
"device": "cuda"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from mem0 import Memory
|
||||
|
||||
m = Memory.from_config(config)
|
||||
|
||||
# Add some memories
|
||||
m.add("I love hiking in the mountains", user_id="alice")
|
||||
m.add("Pizza is my favorite food", user_id="alice")
|
||||
m.add("I enjoy reading science fiction books", user_id="alice")
|
||||
|
||||
# Search with reranking
|
||||
results = m.search(
|
||||
"What outdoor activities do I enjoy?",
|
||||
user_id="alice",
|
||||
rerank=True
|
||||
)
|
||||
|
||||
for result in results["results"]:
|
||||
print(f"Memory: {result['memory']}")
|
||||
print(f"Score: {result['score']:.3f}")
|
||||
```
|
||||
|
||||
### Batch Processing
|
||||
|
||||
```python
|
||||
# Process multiple queries efficiently
|
||||
queries = [
|
||||
"What are my hobbies?",
|
||||
"What food do I like?",
|
||||
"What books interest me?"
|
||||
]
|
||||
|
||||
results = []
|
||||
for query in queries:
|
||||
result = m.search(query, user_id="alice", rerank=True)
|
||||
results.append(result)
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### GPU Acceleration
|
||||
|
||||
```python
|
||||
# Use GPU for better performance
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-base",
|
||||
"device": "cuda",
|
||||
"batch_size": 64, # Increase batch size for GPU
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Memory Optimization
|
||||
|
||||
```python
|
||||
# For limited memory environments
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-base",
|
||||
"device": "cpu",
|
||||
"batch_size": 8, # Smaller batch size
|
||||
"max_length": 256, # Shorter sequences
|
||||
"model_kwargs": {
|
||||
"torch_dtype": "float16" # Half precision
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Model Comparison
|
||||
|
||||
| Model | Size | Quality | Speed | Memory | Best For |
|
||||
|-------|------|---------|-------|---------|----------|
|
||||
| bge-reranker-base | 278M | Good | Fast | Low | General use |
|
||||
| bge-reranker-large | 560M | Better | Medium | Medium | High quality needs |
|
||||
| bge-reranker-v2-m3 | 568M | Best | Medium | Medium | Latest improvements |
|
||||
| bge-reranker-v2-multilingual | 568M | Good | Medium | Medium | Multiple languages |
|
||||
|
||||
## Error Handling
|
||||
|
||||
```python
|
||||
try:
|
||||
results = m.search(
|
||||
"test query",
|
||||
user_id="alice",
|
||||
rerank=True
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"Reranking failed: {e}")
|
||||
# Fall back to vector search only
|
||||
results = m.search(
|
||||
"test query",
|
||||
user_id="alice",
|
||||
rerank=False
|
||||
)
|
||||
```
|
||||
|
||||
## Custom Models
|
||||
|
||||
### Using Private Models
|
||||
|
||||
```python
|
||||
# Use a private model from Hugging Face
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "your-org/custom-reranker",
|
||||
"device": "cuda",
|
||||
"use_auth_token": "your-hf-token"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Local Model Path
|
||||
|
||||
```python
|
||||
# Use a locally downloaded model
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "/path/to/local/model",
|
||||
"device": "cuda"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Choose the Right Model**: Balance quality vs speed based on your needs
|
||||
2. **Use GPU**: Significantly faster than CPU for larger models
|
||||
3. **Optimize Batch Size**: Tune based on your hardware capabilities
|
||||
4. **Monitor Memory**: Watch GPU/CPU memory usage with large models
|
||||
5. **Cache Models**: Download once and reuse to avoid repeated downloads
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Out of Memory Error**
|
||||
```python
|
||||
# Reduce batch size and sequence length
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-base",
|
||||
"batch_size": 4,
|
||||
"max_length": 256
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Model Download Issues**
|
||||
```python
|
||||
# Set cache directory
|
||||
import os
|
||||
os.environ["TRANSFORMERS_CACHE"] = "/path/to/cache"
|
||||
|
||||
# Or use offline mode
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-base",
|
||||
"local_files_only": True
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**CUDA Not Available**
|
||||
```python
|
||||
import torch
|
||||
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "huggingface",
|
||||
"config": {
|
||||
"model": "BAAI/bge-reranker-base",
|
||||
"device": "cuda" if torch.cuda.is_available() else "cpu"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Reranker Overview" icon="sort" href="/components/rerankers/overview">
|
||||
Learn about reranking concepts
|
||||
</Card>
|
||||
<Card title="Configuration Guide" icon="gear" href="/components/rerankers/config">
|
||||
Detailed configuration options
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,491 @@
|
||||
---
|
||||
title: LLM Reranker
|
||||
description: 'Use any language model as a reranker with custom prompts'
|
||||
icon: "robot"
|
||||
iconType: "solid"
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The LLM reranker allows you to use any supported language model as a reranker. This approach uses prompts to instruct the LLM to score and rank memories based on their relevance to the query. While slower than specialized rerankers, it offers maximum flexibility and can be fine-tuned with custom prompts.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Basic Setup
|
||||
|
||||
```python
|
||||
from mem0 import Memory
|
||||
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {
|
||||
"llm": {
|
||||
"provider": "openai",
|
||||
"config": {
|
||||
"model": "gpt-4",
|
||||
"api_key": "your-openai-api-key"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
m = Memory.from_config(config)
|
||||
```
|
||||
|
||||
### Configuration Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `llm` | dict | Required | LLM configuration object |
|
||||
| `top_k` | int | 10 | Number of results to rerank |
|
||||
| `temperature` | float | 0.0 | LLM temperature for consistency |
|
||||
| `custom_prompt` | str | None | Custom reranking prompt |
|
||||
| `score_range` | tuple | (0, 10) | Score range for relevance |
|
||||
|
||||
### Advanced Configuration
|
||||
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {
|
||||
"llm": {
|
||||
"provider": "anthropic",
|
||||
"config": {
|
||||
"model": "claude-3-sonnet-20240229",
|
||||
"api_key": "your-anthropic-api-key"
|
||||
}
|
||||
},
|
||||
"top_k": 15,
|
||||
"temperature": 0.0,
|
||||
"score_range": (1, 5),
|
||||
"custom_prompt": """
|
||||
Rate the relevance of each memory to the query on a scale of 1-5.
|
||||
Consider semantic similarity, context, and practical utility.
|
||||
Only provide the numeric score.
|
||||
"""
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Supported LLM Providers
|
||||
|
||||
### OpenAI
|
||||
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {
|
||||
"llm": {
|
||||
"provider": "openai",
|
||||
"config": {
|
||||
"model": "gpt-4",
|
||||
"api_key": "your-openai-api-key",
|
||||
"temperature": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Anthropic
|
||||
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {
|
||||
"llm": {
|
||||
"provider": "anthropic",
|
||||
"config": {
|
||||
"model": "claude-3-sonnet-20240229",
|
||||
"api_key": "your-anthropic-api-key"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Ollama (Local)
|
||||
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {
|
||||
"llm": {
|
||||
"provider": "ollama",
|
||||
"config": {
|
||||
"model": "llama2",
|
||||
"ollama_base_url": "http://localhost:11434"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Azure OpenAI
|
||||
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {
|
||||
"llm": {
|
||||
"provider": "azure_openai",
|
||||
"config": {
|
||||
"model": "gpt-4",
|
||||
"api_key": "your-azure-api-key",
|
||||
"azure_endpoint": "https://your-resource.openai.azure.com/",
|
||||
"azure_deployment": "gpt-4-deployment"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Custom Prompts
|
||||
|
||||
### Default Prompt Behavior
|
||||
|
||||
The default prompt asks the LLM to score relevance on a 0-10 scale:
|
||||
|
||||
```
|
||||
Given a query and a memory, rate how relevant the memory is to answering the query.
|
||||
Score from 0 (completely irrelevant) to 10 (perfectly relevant).
|
||||
Only provide the numeric score.
|
||||
|
||||
Query: {query}
|
||||
Memory: {memory}
|
||||
Score:
|
||||
```
|
||||
|
||||
### Custom Prompt Examples
|
||||
|
||||
#### Domain-Specific Scoring
|
||||
|
||||
```python
|
||||
custom_prompt = """
|
||||
You are a medical information specialist. Rate how relevant each memory is for answering the medical query.
|
||||
Consider clinical accuracy, specificity, and practical applicability.
|
||||
Rate from 1-10 where:
|
||||
- 1-3: Irrelevant or potentially harmful
|
||||
- 4-6: Somewhat relevant but incomplete
|
||||
- 7-8: Relevant and helpful
|
||||
- 9-10: Highly relevant and clinically useful
|
||||
|
||||
Query: {query}
|
||||
Memory: {memory}
|
||||
Score:
|
||||
"""
|
||||
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {
|
||||
"llm": {
|
||||
"provider": "openai",
|
||||
"config": {
|
||||
"model": "gpt-4",
|
||||
"api_key": "your-api-key"
|
||||
}
|
||||
},
|
||||
"custom_prompt": custom_prompt
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Contextual Relevance
|
||||
|
||||
```python
|
||||
contextual_prompt = """
|
||||
Rate how well this memory answers the specific question asked.
|
||||
Consider:
|
||||
- Direct relevance to the question
|
||||
- Completeness of information
|
||||
- Recency and accuracy
|
||||
- Practical usefulness
|
||||
|
||||
Rate 1-5:
|
||||
1 = Not relevant
|
||||
2 = Slightly relevant
|
||||
3 = Moderately relevant
|
||||
4 = Very relevant
|
||||
5 = Perfectly answers the question
|
||||
|
||||
Query: {query}
|
||||
Memory: {memory}
|
||||
Score:
|
||||
"""
|
||||
```
|
||||
|
||||
#### Conversational Context
|
||||
|
||||
```python
|
||||
conversation_prompt = """
|
||||
You are helping evaluate which memories are most useful for a conversational AI assistant.
|
||||
Rate how helpful this memory would be for generating a relevant response.
|
||||
|
||||
Consider:
|
||||
- Direct relevance to user's intent
|
||||
- Emotional appropriateness
|
||||
- Factual accuracy
|
||||
- Conversation flow
|
||||
|
||||
Rate 0-10:
|
||||
Query: {query}
|
||||
Memory: {memory}
|
||||
Score:
|
||||
"""
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from mem0 import Memory
|
||||
|
||||
m = Memory.from_config(config)
|
||||
|
||||
# Add memories
|
||||
m.add("I'm allergic to peanuts", user_id="alice")
|
||||
m.add("I love Italian food", user_id="alice")
|
||||
m.add("I'm vegetarian", user_id="alice")
|
||||
|
||||
# Search with LLM reranking
|
||||
results = m.search(
|
||||
"What foods should I avoid?",
|
||||
user_id="alice",
|
||||
rerank=True
|
||||
)
|
||||
|
||||
for result in results["results"]:
|
||||
print(f"Memory: {result['memory']}")
|
||||
print(f"LLM Score: {result['score']:.2f}")
|
||||
```
|
||||
|
||||
### Batch Processing with Error Handling
|
||||
|
||||
```python
|
||||
def safe_llm_rerank_search(query, user_id, max_retries=3):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
return m.search(query, user_id=user_id, rerank=True)
|
||||
except Exception as e:
|
||||
print(f"Attempt {attempt + 1} failed: {e}")
|
||||
if attempt == max_retries - 1:
|
||||
# Fall back to vector search
|
||||
return m.search(query, user_id=user_id, rerank=False)
|
||||
|
||||
# Use the safe function
|
||||
results = safe_llm_rerank_search("What are my preferences?", "alice")
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Speed vs Quality Trade-offs
|
||||
|
||||
| Model Type | Speed | Quality | Cost | Best For |
|
||||
|------------|-------|---------|------|----------|
|
||||
| GPT-3.5 Turbo | Fast | Good | Low | High-volume applications |
|
||||
| GPT-4 | Medium | Excellent | Medium | Quality-critical applications |
|
||||
| Claude 3 Sonnet | Medium | Excellent | Medium | Balanced performance |
|
||||
| Ollama Local | Variable | Good | Free | Privacy-sensitive applications |
|
||||
|
||||
### Optimization Strategies
|
||||
|
||||
```python
|
||||
# Fast configuration for high-volume use
|
||||
fast_config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {
|
||||
"llm": {
|
||||
"provider": "openai",
|
||||
"config": {
|
||||
"model": "gpt-3.5-turbo",
|
||||
"api_key": "your-api-key"
|
||||
}
|
||||
},
|
||||
"top_k": 5, # Limit candidates
|
||||
"temperature": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# High-quality configuration
|
||||
quality_config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {
|
||||
"llm": {
|
||||
"provider": "openai",
|
||||
"config": {
|
||||
"model": "gpt-4",
|
||||
"api_key": "your-api-key"
|
||||
}
|
||||
},
|
||||
"top_k": 15,
|
||||
"temperature": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Advanced Use Cases
|
||||
|
||||
### Multi-Step Reasoning
|
||||
|
||||
```python
|
||||
reasoning_prompt = """
|
||||
Evaluate this memory's relevance using multi-step reasoning:
|
||||
|
||||
1. What is the main intent of the query?
|
||||
2. What key information does the memory contain?
|
||||
3. How directly does the memory address the query?
|
||||
4. What additional context might be needed?
|
||||
|
||||
Based on this analysis, rate relevance 1-10:
|
||||
|
||||
Query: {query}
|
||||
Memory: {memory}
|
||||
|
||||
Analysis:
|
||||
Step 1 (Intent):
|
||||
Step 2 (Information):
|
||||
Step 3 (Directness):
|
||||
Step 4 (Context):
|
||||
Final Score:
|
||||
"""
|
||||
```
|
||||
|
||||
### Comparative Ranking
|
||||
|
||||
```python
|
||||
comparative_prompt = """
|
||||
You will see a query and multiple memories. Rank them in order of relevance.
|
||||
Consider which memories best answer the question and would be most helpful.
|
||||
|
||||
Query: {query}
|
||||
|
||||
Memories to rank:
|
||||
{memories}
|
||||
|
||||
Provide scores 1-10 for each memory, considering their relative usefulness.
|
||||
"""
|
||||
```
|
||||
|
||||
### Emotional Intelligence
|
||||
|
||||
```python
|
||||
emotional_prompt = """
|
||||
Consider both factual relevance and emotional appropriateness.
|
||||
Rate how suitable this memory is for responding to the user's query.
|
||||
|
||||
Factors to consider:
|
||||
- Factual accuracy and relevance
|
||||
- Emotional tone and sensitivity
|
||||
- User's likely emotional state
|
||||
- Appropriateness of response
|
||||
|
||||
Query: {query}
|
||||
Memory: {memory}
|
||||
Emotional Context: {context}
|
||||
Score (1-10):
|
||||
"""
|
||||
```
|
||||
|
||||
## Error Handling and Fallbacks
|
||||
|
||||
```python
|
||||
class RobustLLMReranker:
|
||||
def __init__(self, primary_config, fallback_config=None):
|
||||
self.primary = Memory.from_config(primary_config)
|
||||
self.fallback = Memory.from_config(fallback_config) if fallback_config else None
|
||||
|
||||
def search(self, query, user_id, max_retries=2):
|
||||
# Try primary LLM reranker
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
return self.primary.search(query, user_id=user_id, rerank=True)
|
||||
except Exception as e:
|
||||
print(f"Primary reranker attempt {attempt + 1} failed: {e}")
|
||||
|
||||
# Try fallback reranker
|
||||
if self.fallback:
|
||||
try:
|
||||
return self.fallback.search(query, user_id=user_id, rerank=True)
|
||||
except Exception as e:
|
||||
print(f"Fallback reranker failed: {e}")
|
||||
|
||||
# Final fallback: vector search only
|
||||
return self.primary.search(query, user_id=user_id, rerank=False)
|
||||
|
||||
# Usage
|
||||
primary_config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {"llm": {"provider": "openai", "config": {"model": "gpt-4"}}}
|
||||
}
|
||||
}
|
||||
|
||||
fallback_config = {
|
||||
"reranker": {
|
||||
"provider": "llm_reranker",
|
||||
"config": {"llm": {"provider": "openai", "config": {"model": "gpt-3.5-turbo"}}}
|
||||
}
|
||||
}
|
||||
|
||||
reranker = RobustLLMReranker(primary_config, fallback_config)
|
||||
results = reranker.search("What are my preferences?", "alice")
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use Specific Prompts**: Tailor prompts to your domain and use case
|
||||
2. **Set Temperature to 0**: Ensure consistent scoring across runs
|
||||
3. **Limit Top-K**: Don't rerank too many candidates to control costs
|
||||
4. **Implement Fallbacks**: Always have a backup plan for API failures
|
||||
5. **Monitor Costs**: Track API usage, especially with expensive models
|
||||
6. **Cache Results**: Consider caching reranking results for repeated queries
|
||||
7. **Test Prompts**: Experiment with different prompts to find what works best
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Inconsistent Scores**
|
||||
- Set temperature to 0.0
|
||||
- Use more specific prompts
|
||||
- Consider using multiple calls and averaging
|
||||
|
||||
**API Rate Limits**
|
||||
- Implement exponential backoff
|
||||
- Use cheaper models for high-volume scenarios
|
||||
- Add retry logic with delays
|
||||
|
||||
**Poor Ranking Quality**
|
||||
- Refine your custom prompt
|
||||
- Try different LLM models
|
||||
- Add examples to your prompt
|
||||
|
||||
## Next Steps
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Custom Prompts Guide" icon="pencil" href="/components/rerankers/custom-prompts">
|
||||
Learn to craft effective reranking prompts
|
||||
</Card>
|
||||
<Card title="Performance Optimization" icon="bolt" href="/components/rerankers/optimization">
|
||||
Optimize LLM reranker performance
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,162 @@
|
||||
---
|
||||
title: Sentence Transformer
|
||||
---
|
||||
|
||||
Sentence Transformer rerankers use cross-encoder models that are specifically designed for ranking tasks. These models can run locally and provide good reranking performance without external API calls.
|
||||
|
||||
## Usage
|
||||
|
||||
To use Sentence Transformer reranker with Mem0:
|
||||
|
||||
```python
|
||||
from mem0 import Memory
|
||||
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "sentence_transformer",
|
||||
"config": {
|
||||
"model": "cross-encoder/ms-marco-MiniLM-L-6-v2",
|
||||
"device": "cpu",
|
||||
"top_n": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
memory = Memory.from_config(config)
|
||||
|
||||
# Use memory as usual
|
||||
memory.add("I love playing basketball", user_id="alice")
|
||||
memory.add("I enjoy watching movies", user_id="alice")
|
||||
|
||||
# Search will now use Sentence Transformer reranking
|
||||
results = memory.search("What sports does Alice like?", user_id="alice")
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `model` | Sentence Transformer cross-encoder model | `cross-encoder/ms-marco-MiniLM-L-6-v2` |
|
||||
| `device` | Device to run on (`cpu`, `cuda`, `mps`) | `cpu` |
|
||||
| `top_n` | Number of results to return | `10` |
|
||||
|
||||
## Popular Models
|
||||
|
||||
### Lightweight Models
|
||||
- `cross-encoder/ms-marco-MiniLM-L-6-v2`: Fast and efficient
|
||||
- `cross-encoder/ms-marco-MiniLM-L-4-v2`: Even faster, slightly lower accuracy
|
||||
- `cross-encoder/ms-marco-MiniLM-L-2-v2`: Fastest, good for real-time applications
|
||||
|
||||
### High-Performance Models
|
||||
- `cross-encoder/ms-marco-electra-base`: Better accuracy, larger model
|
||||
- `ms-marco-MiniLM-L-12-v2`: Balanced performance and speed
|
||||
- `cross-encoder/qnli-electra-base`: Good for question-answering tasks
|
||||
|
||||
## Device Configuration
|
||||
|
||||
### CPU Usage
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "sentence_transformer",
|
||||
"config": {
|
||||
"model": "cross-encoder/ms-marco-MiniLM-L-6-v2",
|
||||
"device": "cpu",
|
||||
"top_n": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### GPU Usage (CUDA)
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "sentence_transformer",
|
||||
"config": {
|
||||
"model": "cross-encoder/ms-marco-electra-base",
|
||||
"device": "cuda",
|
||||
"top_n": 15
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Apple Silicon (MPS)
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "sentence_transformer",
|
||||
"config": {
|
||||
"model": "cross-encoder/ms-marco-MiniLM-L-6-v2",
|
||||
"device": "mps",
|
||||
"top_n": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
The sentence-transformers library is required:
|
||||
|
||||
```bash
|
||||
pip install sentence-transformers
|
||||
```
|
||||
|
||||
For GPU support with CUDA:
|
||||
```bash
|
||||
pip install sentence-transformers torch
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Model Selection
|
||||
- Use MiniLM models for faster inference
|
||||
- Use larger models (electra-base) for better accuracy
|
||||
- Consider the trade-off between speed and quality
|
||||
|
||||
### Device Optimization
|
||||
- Use GPU (`cuda` or `mps`) for larger models
|
||||
- CPU is sufficient for MiniLM models
|
||||
- Batch processing improves GPU utilization
|
||||
|
||||
### Memory Considerations
|
||||
```python
|
||||
# For memory-constrained environments
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "sentence_transformer",
|
||||
"config": {
|
||||
"model": "cross-encoder/ms-marco-MiniLM-L-2-v2", # Smallest model
|
||||
"device": "cpu",
|
||||
"top_n": 5 # Fewer results to process
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Custom Models
|
||||
|
||||
You can use any Sentence Transformer cross-encoder model:
|
||||
|
||||
```python
|
||||
config = {
|
||||
"reranker": {
|
||||
"provider": "sentence_transformer",
|
||||
"config": {
|
||||
"model": "your-custom-model-name",
|
||||
"device": "cpu",
|
||||
"top_n": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Advantages
|
||||
|
||||
- **Local Processing**: No external API calls required
|
||||
- **Privacy**: Data stays on your infrastructure
|
||||
- **Cost Effective**: No per-request charges
|
||||
- **Fast**: Especially with GPU acceleration
|
||||
- **Customizable**: Can fine-tune on your specific data
|
||||
Reference in New Issue
Block a user