Update Docs (#3315)
This commit is contained in:
Deshraj Yadav
@@ -5,13 +5,17 @@ icon: "image"
|
||||
iconType: "solid"
|
||||
---
|
||||
|
||||
<Snippet file="blank-notif.mdx" />
|
||||
|
||||
Mem0 extends its capabilities beyond text by supporting multimodal data. With this feature, users can seamlessly integrate images into their interactions—allowing Mem0 to extract relevant information.
|
||||
Mem0 extends its capabilities beyond text by supporting multimodal data. With this feature, you can seamlessly integrate images into your interactions—allowing Mem0 to extract relevant information and context from visual content.
|
||||
|
||||
## How It Works
|
||||
|
||||
When a user submits an image, Mem0 processes it to extract textual information and other pertinent details. These details are then added to the user's memory, enhancing the system's ability to understand and recall multimodal inputs.
|
||||
When you submit an image, Mem0:
|
||||
1. **Processes the visual content** using advanced vision models
|
||||
2. **Extracts textual information** and relevant details from the image
|
||||
3. **Stores the extracted information** as searchable memories
|
||||
4. **Maintains context** between visual and textual interactions
|
||||
|
||||
This enables more comprehensive understanding of user interactions that include both text and visual elements.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
@@ -62,7 +66,246 @@ client.add(messages, user_id="alice")
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Using these methods, you can seamlessly incorporate various media types into your interactions, further enhancing Mem0's multimodal capabilities.
|
||||
## Supported Image Formats
|
||||
|
||||
Mem0 supports common image formats:
|
||||
- **JPEG/JPG** - Standard photos and images
|
||||
- **PNG** - Images with transparency support
|
||||
- **WebP** - Modern web-optimized format
|
||||
- **GIF** - Animated and static graphics
|
||||
|
||||
## Local Files vs URLs
|
||||
|
||||
### Using Image URLs
|
||||
Images can be referenced via publicly accessible URLs:
|
||||
|
||||
```python
|
||||
content = {
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://example.com/my-image.jpg"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Using Local Files
|
||||
For local images, convert them to base64 format:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
import base64
|
||||
from mem0 import Memory
|
||||
|
||||
def encode_image(image_path):
|
||||
with open(image_path, "rb") as image_file:
|
||||
return base64.b64encode(image_file.read()).decode('utf-8')
|
||||
|
||||
client = Memory()
|
||||
|
||||
# Encode local image
|
||||
base64_image = encode_image("path/to/your/image.jpg")
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "What's in this image?"
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": f"data:image/jpeg;base64,{base64_image}"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
client.add(messages, user_id="alice")
|
||||
```
|
||||
|
||||
```javascript JavaScript
|
||||
import fs from 'fs';
|
||||
import { Memory } from 'mem0ai';
|
||||
|
||||
function encodeImage(imagePath) {
|
||||
const imageBuffer = fs.readFileSync(imagePath);
|
||||
return imageBuffer.toString('base64');
|
||||
}
|
||||
|
||||
const client = new Memory();
|
||||
|
||||
// Encode local image
|
||||
const base64Image = encodeImage("path/to/your/image.jpg");
|
||||
|
||||
const messages = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{
|
||||
type: "text",
|
||||
text: "What's in this image?"
|
||||
},
|
||||
{
|
||||
type: "image_url",
|
||||
image_url: {
|
||||
url: `data:image/jpeg;base64,${base64Image}`
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
];
|
||||
|
||||
await client.add(messages, { user_id: "alice" });
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Advanced Examples
|
||||
|
||||
### Restaurant Menu Analysis
|
||||
```python
|
||||
from mem0 import Memory
|
||||
|
||||
client = Memory()
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "I'm looking at this restaurant menu. Help me remember my preferences."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": {
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://example.com/restaurant-menu.jpg"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "I'm allergic to peanuts and prefer vegetarian options."
|
||||
}
|
||||
]
|
||||
|
||||
result = client.add(messages, user_id="user123")
|
||||
print(result)
|
||||
```
|
||||
|
||||
### Document Analysis
|
||||
```python
|
||||
# Analyzing receipts, invoices, or documents
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Store this receipt information for my expense tracking."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": {
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://example.com/receipt.jpg"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
client.add(messages, user_id="user123")
|
||||
```
|
||||
|
||||
## File Size and Performance Considerations
|
||||
|
||||
### Image Size Limits
|
||||
- **Maximum file size**: 20MB per image
|
||||
- **Recommended size**: Under 5MB for optimal performance
|
||||
- **Resolution**: Images are automatically resized if needed
|
||||
|
||||
### Performance Tips
|
||||
1. **Compress large images** before sending to reduce processing time
|
||||
2. **Use appropriate formats** - JPEG for photos, PNG for graphics with text
|
||||
3. **Batch processing** - Send multiple images in separate requests for better reliability
|
||||
|
||||
## Error Handling
|
||||
|
||||
Handle common errors when working with images:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
from mem0 import Memory
|
||||
from mem0.exceptions import InvalidImageError, FileSizeError
|
||||
|
||||
client = Memory()
|
||||
|
||||
try:
|
||||
messages = [{
|
||||
"role": "user",
|
||||
"content": {
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://example.com/image.jpg"}
|
||||
}
|
||||
}]
|
||||
|
||||
result = client.add(messages, user_id="user123")
|
||||
print("Image processed successfully")
|
||||
|
||||
except InvalidImageError:
|
||||
print("Invalid image format or corrupted file")
|
||||
except FileSizeError:
|
||||
print("Image file too large")
|
||||
except Exception as e:
|
||||
print(f"Unexpected error: {e}")
|
||||
```
|
||||
|
||||
```javascript JavaScript
|
||||
import { Memory } from 'mem0ai';
|
||||
|
||||
const client = new Memory();
|
||||
|
||||
try {
|
||||
const messages = [{
|
||||
role: "user",
|
||||
content: {
|
||||
type: "image_url",
|
||||
image_url: { url: "https://example.com/image.jpg" }
|
||||
}
|
||||
}];
|
||||
|
||||
const result = await client.add(messages, { user_id: "user123" });
|
||||
console.log("Image processed successfully");
|
||||
|
||||
} catch (error) {
|
||||
if (error.type === 'invalid_image') {
|
||||
console.log("Invalid image format or corrupted file");
|
||||
} else if (error.type === 'file_size_exceeded') {
|
||||
console.log("Image file too large");
|
||||
} else {
|
||||
console.log(`Unexpected error: ${error.message}`);
|
||||
}
|
||||
}
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Image Selection
|
||||
- **Use high-quality images** with clear, readable text and details
|
||||
- **Ensure good lighting** in photos for better text extraction
|
||||
- **Avoid heavily stylized fonts** that may be difficult to read
|
||||
|
||||
### Memory Context
|
||||
- **Provide context** about what information you want extracted
|
||||
- **Combine with text** to give Mem0 better understanding of the image's purpose
|
||||
- **Be specific** about what aspects of the image are important
|
||||
|
||||
### Privacy and Security
|
||||
- **Avoid sensitive information** in images (SSN, passwords, private data)
|
||||
- **Use secure image hosting** for URLs to prevent unauthorized access
|
||||
- **Consider local processing** for highly sensitive visual content
|
||||
|
||||
Using these methods, you can seamlessly incorporate various visual content types into your interactions, further enhancing Mem0's multimodal capabilities for more comprehensive memory management.
|
||||
|
||||
If you have any questions, please feel free to reach out to us using one of the following methods:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user