JDSS/mem0
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: align Databricks docs with config and fix query mode selection (#4477)

Browse Source 
Co-authored-by: utkarsh240799 <utkarsh240799@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Utkarsh
2026-03-23 19:20:11 +05:30
committed by GitHub
parent 316dc67a0a
commit d8a6960b4a
3 changed files with 588 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/components/vectordbs/dbs/databricks.mdx
+29 -20
View File
@@ -17,8 +17,10 @@ config = {
"workspace_url": "https://your-workspace.databricks.com",
"access_token": "your-access-token",
"endpoint_name": "your-vector-search-endpoint",
"index_name": "catalog.schema.index_name",
"source_table_name": "catalog.schema.source_table",
"catalog": "your_catalog",
"schema": "your_schema",
"table_name": "your_table",
"collection_name": "your_index_name",
"embedding_dimension": 1536
}
}
@@ -42,17 +44,22 @@ Here are the parameters available for configuring Databricks Vector Search:
| --- | --- | --- |
| `workspace_url` | The URL of your Databricks workspace | **Required** |
| `access_token` | Personal Access Token for authentication | `None` |
| `service_principal_client_id` | Service principal client ID (alternative to access_token) | `None` |
| `service_principal_client_secret` | Service principal client secret (required with client_id) | `None` |
| `client_id` | Service principal client ID (alternative to access_token) | `None` |
| `client_secret` | Service principal client secret (required with client_id) | `None` |
| `azure_client_id` | Azure AD application client ID (for Azure Databricks) | `None` |
| `azure_client_secret` | Azure AD application client secret (for Azure Databricks) | `None` |
| `endpoint_name` | Name of the Vector Search endpoint | **Required** |
| `index_name` | Name of the vector index (Unity Catalog format: catalog.schema.index) | **Required** |
| `source_table_name` | Name of the source Delta table (Unity Catalog format: catalog.schema.table) | **Required** |
| `embedding_dimension` | Dimension of self-managed embeddings | `1536` |
| `embedding_source_column` | Column name for text when using Databricks-computed embeddings | `None` |
| `catalog` | Unity Catalog catalog name | **Required** |
| `schema` | Unity Catalog schema name | **Required** |
| `table_name` | Source Delta table name | **Required** |
| `collection_name` | Vector search index name | `mem0` |
| `index_type` | Index type: `DELTA_SYNC` or `DIRECT_ACCESS` | `DELTA_SYNC` |
| `embedding_model_endpoint_name` | Databricks serving endpoint for embeddings | `None` |
| `embedding_vector_column` | Column name for self-managed embedding vectors | `embedding` |
| `embedding_dimension` | Dimension of self-managed embeddings | `1536` |
| `endpoint_type` | Type of endpoint (`STANDARD` or `STORAGE_OPTIMIZED`) | `STANDARD` |
| `sync_computed_embeddings` | Whether to sync computed embeddings automatically | `True` |
| `pipeline_type` | Sync pipeline type: `TRIGGERED` or `CONTINUOUS` | `TRIGGERED` |
| `warehouse_name` | Databricks SQL warehouse name (if using SQL warehouse) | `None` |
| `query_type` | Query type: `ANN` or `HYBRID` | `ANN` |
### Authentication
@@ -65,11 +72,13 @@ config = {
"provider": "databricks",
"config": {
"workspace_url": "https://your-workspace.databricks.com",
"service_principal_client_id": "your-service-principal-id",
"service_principal_client_secret": "your-service-principal-secret",
"client_id": "your-service-principal-id",
"client_secret": "your-service-principal-secret",
"endpoint_name": "your-endpoint",
"index_name": "catalog.schema.index_name",
"source_table_name": "catalog.schema.source_table"
"catalog": "your_catalog",
"schema": "your_schema",
"table_name": "your_table",
"collection_name": "your_index_name",
}
}
}
@@ -84,8 +93,10 @@ config = {
"workspace_url": "https://your-workspace.databricks.com",
"access_token": "your-personal-access-token",
"endpoint_name": "your-endpoint",
"index_name": "catalog.schema.index_name",
"source_table_name": "catalog.schema.source_table"
"catalog": "your_catalog",
"schema": "your_schema",
"table_name": "your_table",
"collection_name": "your_index_name",
}
}
}
@@ -103,7 +114,6 @@ config = {
"config": {
# ... authentication config ...
"embedding_dimension": 768, # Match your embedding model
"embedding_vector_column": "embedding"
}
}
}
@@ -118,7 +128,6 @@ config = {
"provider": "databricks",
"config": {
# ... authentication config ...
"embedding_source_column": "text",
"embedding_model_endpoint_name": "e5-small-v2"
}
}
@@ -127,8 +136,8 @@ config = {
### Important Notes
- **Delta Sync Index**: This implementation uses Delta Sync Index, which automatically syncs with your source Delta table. Direct vector insertion/deletion/update operations will log warnings as they're not supported with Delta Sync.
- **Unity Catalog**: Both the source table and index must be in Unity Catalog format (`catalog.schema.table_name`).
- **Index Types**: This implementation supports both `DELTA_SYNC` (auto-syncs with source Delta table) and `DIRECT_ACCESS` (manage vectors directly) index types.
- **Unity Catalog**: The source table and index are created under the specified `catalog.schema` namespace.
- **Endpoint Auto-Creation**: If the specified endpoint doesn't exist, it will be created automatically.
- **Index Auto-Creation**: If the specified index doesn't exist, it will be created automatically with the provided configuration.
- **Filter Support**: Supports filtering by metadata fields, with different syntax for STANDARD vs STORAGE_OPTIMIZED endpoints.