[version] bump package version and minor cleanup (#909 )

Add yaml config validation (#890 )
Beautify JSON docs (#906 )
2023-11-05 16:34:36 -08:00 · 2023-11-04 22:23:55 -07:00 · 2023-11-03 13:52:33 -07:00 · 2023-11-03 13:40:40 -07:00 · 2023-11-03 13:40:16 -07:00 · 2023-11-03 10:00:27 -07:00
54 changed files with 2028 additions and 239 deletions
@@ -76,7 +76,6 @@ docs/_build/
 target/

 # Jupyter Notebook
-*.yaml

 # IPython
 profile_default/
@@ -26,6 +26,14 @@ Book a [1-on-1 Session](https://cal.com/taranjeetio/ec) with Taranjeet, the foun
 pip install --upgrade embedchain
 ```

+To run Embedchain as a REST API server run the following command:
+
+```bash
+docker run -d --name embedchain -p 8080:8080 embedchain/rest-api:latest
+```
+
+Navigate to http://0.0.0.0:8080/docs to interact with the API.
+
 ## 🔍 Demo

 Try out embedchain in your browser:
@@ -1,7 +1,7 @@
 llm:
  provider: anthropic
-  model: 'claude-instant-1'
  config:
+    model: 'claude-instant-1'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -4,8 +4,8 @@ app:

 llm:
  provider: azure_openai
-  model: gpt-35-turbo
  config:
+    model: gpt-35-turbo
    deployment_name: your_llm_deployment_name
    temperature: 0.5
    max_tokens: 1000
@@ -5,8 +5,8 @@ app:

 llm:
  provider: openai
-  model: 'gpt-3.5-turbo'
  config:
+    model: 'gpt-3.5-turbo'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -23,4 +23,4 @@ embedder:
  provider: openai
  config:
    model: 'text-embedding-ada-002'
-    deployment_name: null
+    deployment_name: 'test-deployment'
@@ -1,7 +1,7 @@
 llm:
  provider: cohere
-  model: large
  config:
+    model: large
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -4,8 +4,8 @@ app:

 llm:
  provider: openai
-  model: 'gpt-3.5-turbo'
  config:
+    model: 'gpt-3.5-turbo'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -1,7 +1,7 @@
 llm:
  provider: huggingface
-  model: 'google/flan-t5-xxl'
  config:
+    model: 'google/flan-t5-xxl'
    temperature: 0.5
    max_tokens: 1000
    top_p: 0.5
@@ -1,7 +1,7 @@
 llm:
  provider: llama2
-  model: 'a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5'
  config:
+    model: 'a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5'
    temperature: 0.5
    max_tokens: 1000
    top_p: 0.5
@@ -7,8 +7,8 @@ app:

 llm:
  provider: openai
-  model: 'gpt-3.5-turbo'
  config:
+    model: 'gpt-3.5-turbo'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -23,4 +23,4 @@ vectordb:
 embedder:
  provider: gpt4all
  config:
-    deployment_name: null
+    deployment_name: 'test-deployment'
@@ -1,6 +1,6 @@
 llm:
  provider: vertexai
-  model: 'chat-bison'
  config:
+    model: 'chat-bison'
    temperature: 0.5
    top_p: 0.5
@@ -2,8 +2,25 @@
 title: '📃 JSON'
 ---

-To add any json file, use the data_type as `json`. `json` allows remote urls and conventional file paths. Headers are included for each line, so if you have an `age` column, `18` will be added as `age: 18`. Eg:
+To add any json file, use the data_type as `json`. Headers are included for each line, so if you have an `age` column, `18` will be added as `age: 18`. Eg:

+Here are the supported sources for loading `json`:
+```
+1. URL - valid url to json file that ends with ".json" extension.
+2. Local file - valid url to local json file that ends with ".json" extension.
+3. String - valid json string (e.g. - app.add('{"foo": "bar"}'))
+```
+
+If you would like to add other data structures (e.x. list, dict etc.), do:
+```python
+    import json
+    a = {"foo": "bar"}
+    valid_json_string_data = json.dumps(a, indent=0)
+
+    b = [{"foo": "bar"}]
+    valid_json_string_data = json.dumps(b, indent=0)
+```
+Example:
 ```python
 import os

@@ -25,8 +42,8 @@ response = app.query("What is the net worth of Elon Musk as of October 2023?")
 print(response)
 "As of October 2023, Elon Musk's net worth is $255.2 billion."
 ```
-
-```temp.json
+temp.json
+```json
 {
    "question": "What is your net worth, Elon Musk?",
    "answer": "As of October 2023, Elon Musk's net worth is $255.2 billion, making him one of the wealthiest individuals in the world."
@@ -1,93 +0,0 @@
---
-title: '🌍 API Server'
---
-
-The API server example can be found [here](https://github.com/embedchain/embedchain/tree/main/examples/api_server).
-
-It is a Flask based server that integrates the `embedchain` package, offering endpoints to add, query, and chat to engage in conversations with a chatbot using JSON requests.
-
-### 🐳 Docker Setup
-
- Open variables.env, and edit it to add your 🔑 `OPENAI_API_KEY`.
- To setup your api server using docker, run the following command inside this folder using your terminal.
-
-```bash
-docker-compose up --build
-```
-
-📝 Note: The build command might take a while to install all the packages depending on your system resources.
-
-### 🚀 Usage Instructions
-
- Your api server is running on [http://localhost:5000/](http://localhost:5000/)
- To use the api server, make an api call to the endpoints `/add`, `/query` and `/chat` using the json formats discussed below.
- To add data sources to the bot (/add):
-```json
-// Request
-{
-  "data_type": "your_data_type_here",
-  "url_or_text": "your_url_or_text_here"
-}
-
-// Response
-{
-  "data": "Added data_type: url_or_text"
-}
-```
- To ask queries from the bot (/query):
-```json
-// Request
-{
-  "question": "your_question_here"
-}
-
-// Response
-{
-  "data": "your_answer_here"
-}
-```
- To chat with the bot (/chat):
-```json
-// Request
-{
-  "question": "your_question_here"
-}
-
-// Response
-{
-  "data": "your_answer_here"
-}
-```
-
-### 📡 Curl Call Formats
-
- To add data sources to the bot (/add):
-```bash
-curl -X POST \
-     -H "Content-Type: application/json" \
-     -d '{
-         "data_type": "your_data_type_here",
-         "url_or_text": "your_url_or_text_here"
-     }' \
-     http://localhost:5000/add
-```
- To ask queries from the bot (/query):
-```bash
-curl -X POST \
-     -H "Content-Type: application/json" \
-     -d '{
-         "question": "your_question_here"
-     }' \
-     http://localhost:5000/query
-```
- To chat with the bot (/chat):
-```bash
-curl -X POST \
-     -H "Content-Type: application/json" \
-     -d '{
-         "question": "your_question_here"
-     }' \
-     http://localhost:5000/chat
-```
-
-🎉 Happy Chatting! 🎉
@@ -14,6 +14,7 @@
      "dark": "#020415"
    }
  },
+  "openapi": ["/rest-api.json"],
  "metadata": {
    "og:image": "/images/og.png",
    "twitter:site": "@embedchain"
@@ -28,8 +29,8 @@
      "url": "https://twitter.com/embedchain"
    },
    {
-      "name":"Slack",
-      "url":"https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw"
+      "name": "Slack",
+      "url": "https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw"
    },
    {
      "name": "Discord",
@@ -46,11 +47,20 @@
  "navigation": [
    {
      "group": "Get started",
-      "pages": ["get-started/quickstart", "get-started/introduction", "get-started/faq", "get-started/examples"]
+      "pages": [
+        "get-started/quickstart",
+        "get-started/introduction",
+        "get-started/faq",
+        "get-started/examples"
+      ]
    },
    {
      "group": "Components",
-      "pages": ["components/llms", "components/embedding-models", "components/vector-databases"]
+      "pages": [
+        "components/llms",
+        "components/embedding-models",
+        "components/vector-databases"
+      ]
    },
    {
      "group": "Data sources",
@@ -81,16 +91,34 @@
      "group": "Advanced",
      "pages": ["advanced/configuration"]
    },
+    {
+      "group": "REST API",
+      "pages": [
+        "rest-api/getting-started",
+        "rest-api/create",
+        "rest-api/get-all-apps",
+        "rest-api/add-data",
+        "rest-api/get-data",
+        "rest-api/query",
+        "rest-api/deploy",
+        "rest-api/delete",
+        "rest-api/check-status"
+      ]
+    },
    {
      "group": "Examples",
-      "pages": ["examples/full_stack", "examples/api_server", "examples/discord_bot", "examples/slack_bot", "examples/telegram_bot", "examples/whatsapp_bot", "examples/poe_bot"]
+      "pages": [
+        "examples/full_stack",
+        "examples/discord_bot",
+        "examples/slack_bot",
+        "examples/telegram_bot",
+        "examples/whatsapp_bot",
+        "examples/poe_bot"
+      ]
    },
    {
      "group": "Community",
-      "pages": [
-        "community/connect-with-us",
-        "community/showcase"
-      ]
+      "pages": ["community/connect-with-us", "community/showcase"]
    },
    {
      "group": "Integrations",
@@ -108,16 +136,13 @@
    },
    {
      "group": "Product",
-      "pages": [
-        "product/release-notes"
-      ]
+      "pages": ["product/release-notes"]
    }
-
  ],
  "footerSocials": {
    "website": "https://embedchain.ai",
    "github": "https://github.com/embedchain/embedchain",
-    "slack":"https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw",
+    "slack": "https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw",
    "discord": "https://discord.gg/6PzXDgEjG5",
    "twitter": "https://twitter.com/embedchain",
    "linkedin": "https://www.linkedin.com/company/embedchain"
@@ -136,5 +161,8 @@
  },
  "search": {
    "prompt": "✨ Search embedchain docs..."
+  },
+  "api": {
+    "baseUrl": "http://localhost:8080"
  }
 }
@@ -0,0 +1,427 @@
+{
+    "openapi": "3.1.0",
+    "info": {
+      "title": "Embedchain REST API",
+      "description": "This is the REST API for Embedchain.",
+      "license": {
+        "name": "Apache 2.0",
+        "url": "https://github.com/embedchain/embedchain/blob/main/LICENSE"
+      },
+      "version": "0.0.1"
+    },
+    "paths": {
+      "/ping": {
+        "get": {
+          "tags": ["Utility"],
+          "summary": "Check status",
+          "description": "Endpoint to check the status of the API",
+          "operationId": "check_status_ping_get",
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": { "application/json": { "schema": {} } }
+            }
+          }
+        }
+      },
+      "/apps": {
+        "get": {
+          "tags": ["Apps"],
+          "summary": "Get all apps",
+          "description": "Get all applications",
+          "operationId": "get_all_apps_apps_get",
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": { "application/json": { "schema": {} } }
+            }
+          }
+        }
+      },
+      "/create": {
+        "post": {
+          "tags": ["Apps"],
+          "summary": "Create app",
+          "description": "Create a new app using App ID",
+          "operationId": "create_app_using_default_config_create_post",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "query",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "requestBody": {
+            "content": {
+              "multipart/form-data": {
+                "schema": {
+                  "allOf": [
+                    {
+                      "$ref": "#/components/schemas/Body_create_app_using_default_config_create_post"
+                    }
+                  ],
+                  "title": "Body"
+                }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/data": {
+        "get": {
+          "tags": ["Apps"],
+          "summary": "Get data",
+          "description": "Get all data sources for an app",
+          "operationId": "get_datasources_associated_with_app_id__app_id__data_get",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": { "application/json": { "schema": {} } }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/add": {
+        "post": {
+          "tags": ["Apps"],
+          "summary": "Add data",
+          "description": "Add a data source to an app.",
+          "operationId": "add_datasource_to_an_app__app_id__add_post",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": { "$ref": "#/components/schemas/SourceApp" }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/query": {
+        "post": {
+          "tags": ["Apps"],
+          "summary": "Query app",
+          "description": "Query an app",
+          "operationId": "query_an_app__app_id__query_post",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": { "$ref": "#/components/schemas/QueryApp" }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/chat": {
+        "post": {
+          "tags": ["Apps"],
+          "summary": "Chat",
+          "description": "Chat with an app.\n\napp_id: The ID of the app. Use \"default\" for the default app.\n\nmessage: The message that you want to send to the app.",
+          "operationId": "chat_with_an_app__app_id__chat_post",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": { "$ref": "#/components/schemas/MessageApp" }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/deploy": {
+        "post": {
+          "tags": ["Apps"],
+          "summary": "Deploy App",
+          "description": "Deploy an existing app.",
+          "operationId": "deploy_app__app_id__deploy_post",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": { "$ref": "#/components/schemas/DeployAppRequest" }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/delete": {
+        "delete": {
+          "tags": ["Apps"],
+          "summary": "Delete app",
+          "description": "Delete an existing app",
+          "operationId": "delete_app__app_id__delete_delete",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "components": {
+      "schemas": {
+        "Body_create_app_using_default_config_create_post": {
+          "properties": {
+            "config": { "type": "string", "format": "binary", "title": "Config" }
+          },
+          "type": "object",
+          "title": "Body_create_app_using_default_config_create_post"
+        },
+        "DefaultResponse": {
+          "properties": { "response": { "type": "string", "title": "Response" } },
+          "type": "object",
+          "required": ["response"],
+          "title": "DefaultResponse"
+        },
+        "DeployAppRequest": {
+          "properties": {
+            "api_key": {
+              "type": "string",
+              "title": "Api Key",
+              "description": "The Embedchain API key for app deployments. You get the api key on the Embedchain platform by visiting [https://app.embedchain.ai](https://app.embedchain.ai)",
+              "default": ""
+            }
+          },
+          "type": "object",
+          "title": "DeployAppRequest",
+          "example":{
+            "api_key":"ec-xxx"
+         }
+        },
+        "HTTPValidationError": {
+          "properties": {
+            "detail": {
+              "items": { "$ref": "#/components/schemas/ValidationError" },
+              "type": "array",
+              "title": "Detail"
+            }
+          },
+          "type": "object",
+          "title": "HTTPValidationError"
+        },
+        "MessageApp": {
+          "properties": {
+            "message": {
+              "type": "string",
+              "title": "Message",
+              "description": "The message that you want to send to the App.",
+              "default": ""
+            }
+          },
+          "type": "object",
+          "title": "MessageApp"
+        },
+        "QueryApp": {
+          "properties": {
+            "query": {
+              "type": "string",
+              "title": "Query",
+              "description": "The query that you want to ask the App.",
+              "default": ""
+            }
+          },
+          "type": "object",
+          "title": "QueryApp",
+          "example":{
+            "query":"Who is Elon Musk?"
+         }
+        },
+        "SourceApp": {
+          "properties": {
+            "source": {
+              "type": "string",
+              "title": "Source",
+              "description": "The source that you want to add to the App.",
+              "default": ""
+            },
+            "data_type": {
+              "anyOf": [{ "type": "string" }, { "type": "null" }],
+              "title": "Data Type",
+              "description": "The type of data to add, remove it if you want Embedchain to detect it automatically.",
+              "default": ""
+            }
+          },
+          "type": "object",
+          "title": "SourceApp",
+          "example":{
+            "source":"https://en.wikipedia.org/wiki/Elon_Musk"
+         }
+        },
+        "ValidationError": {
+          "properties": {
+            "loc": {
+              "items": { "anyOf": [{ "type": "string" }, { "type": "integer" }] },
+              "type": "array",
+              "title": "Location"
+            },
+            "msg": { "type": "string", "title": "Message" },
+            "type": { "type": "string", "title": "Error Type" }
+          },
+          "type": "object",
+          "required": ["loc", "msg", "type"],
+          "title": "ValidationError"
+        }
+      }
+    }
+  }
@@ -0,0 +1,22 @@
+---
+openapi: post /{app_id}/add
+---
+
+<RequestExample>
+
+```bash Request
+curl --request POST \
+  --url http://localhost:8080/{app_id}/add \
+  -d "source=https://www.forbes.com/profile/elon-musk" \
+  -d "data_type=web_page"
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "response": "fec7fe91e6b2d732938a2ec2e32bfe3f" }
+```
+
+</ResponseExample>
@@ -0,0 +1,3 @@
+---
+openapi: post /{app_id}/chat
+---
@@ -0,0 +1,20 @@
+---
+openapi: get /ping
+---
+
+<RequestExample>
+
+```bash Request
+  curl --request GET \
+    --url http://localhost:8080/ping
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "ping": "pong" }
+```
+
+</ResponseExample>
@@ -0,0 +1,95 @@
+---
+openapi: post /create
+---
+
+<RequestExample>
+
+```bash Request
+curl --request POST \
+  --url http://localhost:8080/create?app_id=app1 \
+  -F "config=@/path/to/config.yaml"
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "response": "App created successfully. App ID: app1" }
+```
+
+</ResponseExample>
+
+By default we will use the opensource **gpt4all** model to get started. You can also specify your own config by uploading a config YAML file.
+
+For example, create a `config.yaml` file (adjust according to your requirements):
+
+```yaml
+app:
+  config:
+    id: "default-app"
+
+llm:
+  provider: openai
+  config:
+    model: "gpt-3.5-turbo"
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+    template: |
+      Use the following pieces of context to answer the query at the end.
+      If you don't know the answer, just say that you don't know, don't try to make up an answer.
+
+      $context
+
+      Query: $query
+
+      Helpful Answer:
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: "rest-api-app"
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: openai
+  config:
+    model: "text-embedding-ada-002"
+```
+
+To learn more about custom configurations, check out the [custom configurations docs](https://docs.embedchain.ai/advanced/configuration). To explore more examples of config yamls for embedchain, visit [embedchain/configs](https://github.com/embedchain/embedchain/tree/main/configs).
+
+Now, you can upload this config file in the request body.
+
+For example,
+
+```bash Request
+curl --request POST \
+  --url http://localhost:8080/create?app_id=my-app \
+  -F "config=@/path/to/config.yaml"
+```
+
+**Note:** To use custom models, an **API key** might be required. Refer to the table below to determine the necessary API key for your provider.
+
+| Keys                       | Providers                      |
+| -------------------------- | ------------------------------ |
+| `OPENAI_API_KEY `          | OpenAI, Azure OpenAI, Jina etc |
+| `OPENAI_API_TYPE`          | Azure OpenAI                   |
+| `OPENAI_API_BASE`          | Azure OpenAI                   |
+| `OPENAI_API_VERSION`       | Azure OpenAI                   |
+| `COHERE_API_KEY`           | Cohere                         |
+| `ANTHROPIC_API_KEY`        | Anthropic                      |
+| `JINACHAT_API_KEY`         | Jina                           |
+| `HUGGINGFACE_ACCESS_TOKEN` | Huggingface                    |
+| `REPLICATE_API_TOKEN`      | LLAMA2                         |
+
+To add env variables, you can simply run the docker command with the `-e` flag.
+
+For example,
+
+```bash
+docker run --name embedchain -p 8080:8080 -e OPENAI_API_KEY=<YOUR_OPENAI_API_KEY> embedchain/rest-api:latest
+```
@@ -0,0 +1,21 @@
+---
+openapi: delete /{app_id}/delete
+---
+
+
+<RequestExample>
+
+```bash Request
+  curl --request DELETE \
+    --url http://localhost:8080/{app_id}/delete
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "response": "App with id {app_id} deleted successfully." }
+```
+
+</ResponseExample>
@@ -0,0 +1,22 @@
+---
+openapi: post /{app_id}/deploy
+---
+
+
+<RequestExample>
+
+```bash Request
+curl --request POST \
+  --url http://localhost:8080/{app_id}/deploy \
+  -d "api_key=ec-xxxx"
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "response": "App deployed successfully." }
+```
+
+</ResponseExample>
@@ -0,0 +1,33 @@
+---
+openapi: get /apps
+---
+
+<RequestExample>
+
+```bash Request
+curl --request GET \
+  --url http://localhost:8080/apps
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{
+  "results": [
+    {
+      "config": "config1.yaml",
+      "id": 1,
+      "app_id": "app1"
+    },
+    {
+      "config": "config2.yaml",
+      "id": 2,
+      "app_id": "app2"
+    }
+  ]
+}
+```
+
+</ResponseExample>
@@ -0,0 +1,28 @@
+---
+openapi: get /{app_id}/data
+---
+
+<RequestExample>
+
+```bash Request
+curl --request GET \
+  --url http://localhost:8080/{app_id}/data
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{
+  "results": [
+    {
+      "data_type": "web_page",
+      "data_value": "https://www.forbes.com/profile/elon-musk/",
+      "metadata": "null"
+    }
+  ]
+}
+```
+
+</ResponseExample>
@@ -0,0 +1,294 @@
+---
+title: "🌍 Getting Started"
+---
+
+## Quickstart
+
+To use Embedchain as a REST API service, run the following command:
+
+```bash
+docker run --name embedchain -p 8080:8080 embedchain/rest-api:latest
+```
+
+Navigate to [http://localhost:8080/docs](http://localhost:8080/docs) to interact with the API. There is a full-fledged Swagger docs playground with all the information about the API endpoints.
+
+![Swagger Docs Screenshot](https://github.com/embedchain/embedchain/assets/73601258/299d81e5-a0df-407c-afc2-6fa2c4286844)
+
+## ⚡ Steps to get started
+
+<Steps>
+  <Step title="⚙️ Create an app">
+    <Tabs>
+      <Tab title="cURL">
+      ```bash
+      curl --request POST "http://localhost:8080/create?app_id=my-app" \
+       -H "accept: application/json"
+      ```
+      </Tab>
+      <Tab title="python">
+      ```python
+      import requests
+
+      url = "http://localhost:8080/create?app_id=my-app"
+
+      payload={}
+
+      response = requests.request("POST", url, data=payload)
+
+      print(response)
+      ```
+      </Tab>
+      <Tab title="javascript">
+      ```javascript
+      const data = fetch("http://localhost:8080/create?app_id=my-app", {
+        method: "POST",
+      }).then((res) => res.json());
+
+      console.log(data);
+      ```
+      </Tab>
+      <Tab title="go">
+      ```go
+      package main
+
+      import (
+        "fmt"
+        "net/http"
+        "io/ioutil"
+      )
+
+      func main() {
+
+        url := "http://localhost:8080/create?app_id=my-app"
+
+        payload := strings.NewReader("")
+
+        req, _ := http.NewRequest("POST", url, payload)
+
+        req.Header.Add("Content-Type", "application/json")
+
+        res, _ := http.DefaultClient.Do(req)
+
+        defer res.Body.Close()
+        body, _ := ioutil.ReadAll(res.Body)
+
+        fmt.Println(res)
+        fmt.Println(string(body))
+
+      }
+      ```
+      </Tab>
+    </Tabs>
+
+  </Step>
+  <Step title="🗃️ Add data sources">
+    <Tabs>
+      <Tab title="cURL">
+        ```bash
+        curl --request POST \
+          --url http://localhost:8080/my-app/add \
+          -d "source=https://www.forbes.com/profile/elon-musk" \
+          -d "data_type=web_page"
+          ```
+      </Tab>
+      <Tab title="python">
+        ```python
+        import requests
+
+        url = "http://localhost:8080/my-app/add"
+
+        payload = "source=https://www.forbes.com/profile/elon-musk&data_type=web_page"
+        headers = {}
+
+        response = requests.request("POST", url, headers=headers, data=payload)
+
+        print(response)
+        ```
+      </Tab>
+      <Tab title="javascript">
+        ```javascript
+        const data = fetch("http://localhost:8080/my-app/add", {
+          method: "POST",
+          body: "source=https://www.forbes.com/profile/elon-musk&data_type=web_page",
+        }).then((res) => res.json());
+
+        console.log(data);
+        ```
+        </Tab>
+      <Tab title="go">
+        ```go
+        package main
+
+        import (
+          "fmt"
+          "strings"
+          "net/http"
+          "io/ioutil"
+        )
+
+        func main() {
+
+          url := "http://localhost:8080/my-app/add"
+
+          payload := strings.NewReader("source=https://www.forbes.com/profile/elon-musk&data_type=web_page")
+
+          req, _ := http.NewRequest("POST", url, payload)
+
+          req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
+
+          res, _ := http.DefaultClient.Do(req)
+
+          defer res.Body.Close()
+          body, _ := ioutil.ReadAll(res.Body)
+
+          fmt.Println(res)
+          fmt.Println(string(body))
+
+        }
+        ```
+      </Tab>
+      </Tabs>
+
+  </Step>
+  <Step title="💬 Query on your data">
+    <Tabs>
+      <Tab title="cURL">
+        ```bash
+        curl --request POST \
+          --url http://localhost:8080/my-app/query \
+          -d "query=Who is Elon Musk?"
+        ```
+      </Tab>
+      <Tab title="python">
+        ```python
+        import requests
+
+        url = "http://localhost:8080/my-app/query"
+
+        payload = "query=Who is Elon Musk?"
+        headers = {}
+
+        response = requests.request("POST", url, headers=headers, data=payload)
+
+        print(response)
+        ```
+      </Tab>
+      <Tab title="javascript">
+        ```javascript
+        const data = fetch("http://localhost:8080/my-app/query", {
+          method: "POST",
+          body: "query=Who is Elon Musk?",
+        }).then((res) => res.json());
+
+        console.log(data);
+        ```
+        </Tab>
+        <Tab title="go">
+        ```go
+        package main
+
+        import (
+          "fmt"
+          "strings"
+          "net/http"
+          "io/ioutil"
+        )
+
+        func main() {
+
+          url := "http://localhost:8080/my-app/query"
+
+          payload := strings.NewReader("query=Who is Elon Musk?")
+
+          req, _ := http.NewRequest("POST", url, payload)
+
+          req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
+
+          res, _ := http.DefaultClient.Do(req)
+
+          defer res.Body.Close()
+          body, _ := ioutil.ReadAll(res.Body)
+
+          fmt.Println(res)
+          fmt.Println(string(body))
+
+        }
+        ```
+      </Tab>
+    </Tabs>
+
+  </Step>
+  <Step title="🚀 (Optional) Deploy your app to Embedchain Platform">
+    <Tabs>
+      <Tab title="cURL">
+        ```bash
+        curl --request POST \
+          --url http://localhost:8080/my-app/deploy \
+          -d "api_key=ec-xxxx"
+          ```
+      </Tab>
+      <Tab title="python">
+        ```python
+        import requests
+
+        url = "http://localhost:8080/my-app/deploy"
+
+        payload = "api_key=ec-xxxx"
+
+        response = requests.request("POST", url, data=payload)
+
+        print(response)
+        ```
+      </Tab>
+      <Tab title="javascript">
+        ```javascript
+        const data = fetch("http://localhost:8080/my-app/deploy", {
+          method: "POST",
+          body: "api_key=ec-xxxx",
+        }).then((res) => res.json());
+
+        console.log(data);
+        ```
+      </Tab>
+      <Tab title="go">
+        ```go
+        package main
+
+        import (
+          "fmt"
+          "strings"
+          "net/http"
+          "io/ioutil"
+        )
+
+        func main() {
+
+          url := "http://localhost:8080/my-app/deploy"
+
+          payload := strings.NewReader("api_key=ec-xxxx")
+
+          req, _ := http.NewRequest("POST", url, payload)
+
+          req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
+
+          res, _ := http.DefaultClient.Do(req)
+
+          defer res.Body.Close()
+          body, _ := ioutil.ReadAll(res.Body)
+
+          fmt.Println(res)
+          fmt.Println(string(body))
+
+        }
+        ```
+      </Tab>
+    </Tabs>
+
+  </Step>
+</Steps>
+
+And you're ready! 🎉
+
+If you run into issues, please feel free to contact us using below links:
+
+<Snippet file="get-help.mdx" />
@@ -0,0 +1,21 @@
+---
+openapi: post /{app_id}/query
+---
+
+<RequestExample>
+
+```bash Request
+curl --request POST \
+  --url http://localhost:8080/{app_id}/query \
+  -d "query=who is Elon Musk?"
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "response": "Net worth of Elon Musk is $218 Billion." }
+```
+
+</ResponseExample>
@@ -11,6 +11,7 @@ from embedchain.factory import EmbedderFactory, LlmFactory, VectorDBFactory
 from embedchain.helper.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm
 from embedchain.llm.openai import OpenAILlm
+from embedchain.utils import validate_yaml_config
 from embedchain.vectordb.base import BaseVectorDB
 from embedchain.vectordb.chroma import ChromaDB

@@ -127,10 +128,15 @@ class App(EmbedChain):
        with open(yaml_path, "r") as file:
            config_data = yaml.safe_load(file)

+        try:
+            validate_yaml_config(config_data)
+        except Exception as e:
+            raise Exception(f"❌ Error occurred while validating the YAML config. Error: {str(e)}")
+
        app_config_data = config_data.get("app", {})
        llm_config_data = config_data.get("llm", {})
        db_config_data = config_data.get("vectordb", {})
-        embedder_config_data = config_data.get("embedder", {})
+        embedding_model_config_data = config_data.get("embedding_model", config_data.get("embedder", {}))

        app_config = AppConfig(**app_config_data.get("config", {}))

@@ -140,6 +146,6 @@ class App(EmbedChain):
        db_provider = db_config_data.get("provider", "chroma")
        db = VectorDBFactory.create(db_provider, db_config_data.get("config", {}))

-        embedder_provider = embedder_config_data.get("provider", "openai")
-        embedder = EmbedderFactory.create(embedder_provider, embedder_config_data.get("config", {}))
+        embedder_provider = embedding_model_config_data.get("provider", "openai")
+        embedder = EmbedderFactory.create(embedder_provider, embedding_model_config_data.get("config", {}))
        return cls(config=app_config, llm=llm, db=db, embedder=embedder)
@@ -20,7 +20,7 @@ from embedchain.loaders.base_loader import BaseLoader
 from embedchain.models.data_type import (DataType, DirectDataType,
                                         IndirectDataType, SpecialDataType)
 from embedchain.telemetry.posthog import AnonymousTelemetry
-from embedchain.utils import detect_datatype
+from embedchain.utils import detect_datatype, is_valid_json_string
 from embedchain.vectordb.base import BaseVectorDB

 load_dotenv()
@@ -175,11 +175,27 @@ class EmbedChain(JSONSerializable):
        if data_type:
            try:
                data_type = DataType(data_type)
+                if data_type == DataType.JSON:
+                    if isinstance(source, str):
+                        if not is_valid_json_string(source):
+                            raise ValueError(
+                                f"Invalid json input: {source}",
+                                "Provide the correct JSON formatted source, \
+                                    refer `https://docs.embedchain.ai/data-sources/json`",
+                            )
+                    elif not isinstance(source, str):
+                        raise ValueError(
+                            "Invaid content input. \
+                            If you want to upload (list, dict, etc.), do \
+                                `json.dump(data, indent=0)` and add the stringified JSON. \
+                                    Check - `https://docs.embedchain.ai/data-sources/json`"
+                        )
            except ValueError:
                raise ValueError(
                    f"Invalid data_type: '{data_type}'.",
                    f"Please use one of the following: {[data_type.value for data_type in DataType]}",
                ) from None
+
        if not data_type:
            data_type = detect_datatype(source)

@@ -287,6 +303,10 @@ class EmbedChain(JSONSerializable):
            # These types have a indirect source reference
            # As long as the reference is the same, they can be updated.
            where = {"url": src}
+            if chunker.data_type == DataType.JSON and is_valid_json_string(src):
+                url = hashlib.sha256((src).encode("utf-8")).hexdigest()
+                where = {"url": url}
+
            if self.config.id is not None:
                where.update({"app_id": self.config.id})

@@ -368,6 +388,10 @@ class EmbedChain(JSONSerializable):

        # get existing ids, and discard doc if any common id exist.
        where = {"url": src}
+        if chunker.data_type == DataType.JSON and is_valid_json_string(src):
+            url = hashlib.sha256((src).encode("utf-8")).hexdigest()
+            where = {"url": url}
+
        # if data type is qna_pair, we check for question
        if chunker.data_type == DataType.QNA_PAIR:
            where = {"question": src[0]}
@@ -6,33 +6,37 @@ import re
 import requests

 from embedchain.loaders.base_loader import BaseLoader
-from embedchain.utils import clean_string
+from embedchain.utils import clean_string, is_valid_json_string

 VALID_URL_PATTERN = "^https:\/\/[0-9A-z.]+.[0-9A-z.]+.[a-z]+\/.*\.json$"


 class JSONLoader(BaseLoader):
    @staticmethod
-    def load_data(content):
-        """Load a json file. Each data point is a key value pair."""
+    def _get_llama_hub_loader():
        try:
            from llama_hub.jsondata.base import \
-                JSONDataReader as LLHBUBJSONLoader
-        except ImportError:
+                JSONDataReader as LLHUBJSONLoader
+        except ImportError as e:
            raise Exception(
-                f"Couldn't import the required packages to load {content}, \
-                Do `pip install --upgrade 'embedchain[json]`"
+                f"Failed to install required packages: {e}, \
+                install them using `pip install --upgrade 'embedchain[json]`"
            )

-        loader = LLHBUBJSONLoader()
+        return LLHUBJSONLoader()

-        if not isinstance(content, str):
-            print(f"Invaid content input. Provide the correct path to the json file saved locally in {content}")
+    @staticmethod
+    def load_data(content):
+        """Load a json file. Each data point is a key value pair."""
+
+        loader = JSONLoader._get_llama_hub_loader()

        data = []
        data_content = []

-        # Load json data from various sources. TODO: add support for dictionary
+        content_url_str = content
+
+        # Load json data from various sources.
        if os.path.isfile(content):
            with open(content, "r", encoding="utf-8") as json_file:
                json_data = json.load(json_file)
@@ -45,13 +49,17 @@ class JSONLoader(BaseLoader):
                    f"Loading data from the given url: {content} failed. \
                    Make sure the url is working."
                )
+        elif is_valid_json_string(content):
+            json_data = content
+            content_url_str = hashlib.sha256((content).encode("utf-8")).hexdigest()
        else:
            raise ValueError(f"Invalid content to load json data from: {content}")

        docs = loader.load_data(json_data)
        for doc in docs:
            doc_content = clean_string(doc.text)
-            data.append({"content": doc_content, "meta_data": {"url": content}})
+            data.append({"content": doc_content, "meta_data": {"url": content_url_str}})
            data_content.append(doc_content)
-        doc_id = hashlib.sha256((content + ", ".join(data_content)).encode()).hexdigest()
+
+        doc_id = hashlib.sha256((content_url_str + ", ".join(data_content)).encode()).hexdigest()
        return {"doc_id": doc_id, "data": data}
@@ -7,7 +7,6 @@ import uuid

 import requests
 import yaml
-from fastapi import FastAPI, HTTPException

 from embedchain import Client
 from embedchain.config import PipelineConfig
@@ -19,6 +18,7 @@ from embedchain.helper.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm
 from embedchain.llm.openai import OpenAILlm
 from embedchain.telemetry.posthog import AnonymousTelemetry
+from embedchain.utils import validate_yaml_config
 from embedchain.vectordb.base import BaseVectorDB
 from embedchain.vectordb.chroma import ChromaDB

@@ -313,6 +313,15 @@ class Pipeline(EmbedChain):
        )
        self.connection.commit()

+    def get_data_sources(self):
+        db_data = self.cursor.execute("SELECT * FROM data_sources WHERE pipeline_id = ?", (self.local_id,)).fetchall()
+
+        data_sources = []
+        for data in db_data:
+            data_sources.append({"data_type": data[2], "data_value": data[3], "metadata": data[4]})
+
+        return data_sources
+
    def deploy(self):
        if self.client is None:
            self._init_client()
@@ -348,9 +357,14 @@ class Pipeline(EmbedChain):
        with open(yaml_path, "r") as file:
            config_data = yaml.safe_load(file)

-        pipeline_config_data = config_data.get("pipeline", {}).get("config", {})
+        try:
+            validate_yaml_config(config_data)
+        except Exception as e:
+            raise Exception(f"❌ Error occurred while validating the YAML config. Error: {str(e)}")
+
+        pipeline_config_data = config_data.get("app", {}).get("config", {})
        db_config_data = config_data.get("vectordb", {})
-        embedding_model_config_data = config_data.get("embedding_model", {})
+        embedding_model_config_data = config_data.get("embedding_model", config_data.get("embedder", {}))
        llm_config_data = config_data.get("llm", {})

        pipeline_config = PipelineConfig(**pipeline_config_data)
@@ -381,33 +395,3 @@ class Pipeline(EmbedChain):
            yaml_path=yaml_path,
            auto_deploy=auto_deploy,
        )
-
-    def start(self, host="0.0.0.0", port=8000):
-        app = FastAPI()
-
-        @app.post("/add")
-        async def add_document(data_value: str, data_type: str = None):
-            """
-            Add a document to the pipeline.
-            """
-            try:
-                document = {"data_value": data_value, "data_type": data_type}
-                self.add(document)
-                return {"message": "Document added successfully"}
-            except Exception as e:
-                raise HTTPException(status_code=500, detail=str(e))
-
-        @app.post("/query")
-        async def query_documents(query: str, num_documents: int = 3):
-            """
-            Query for similar documents in the pipeline.
-            """
-            try:
-                results = self.search(query, num_documents)
-                return results
-            except Exception as e:
-                raise HTTPException(status_code=500, detail=str(e))
-
-        import uvicorn
-
-        uvicorn.run(app, host=host, port=port)
@@ -1,9 +1,12 @@
+import json
 import logging
 import os
 import re
 import string
 from typing import Any

+from schema import Optional, Or, Schema
+
 from embedchain.models.data_type import DataType


@@ -135,8 +138,7 @@ def detect_datatype(source: Any) -> DataType:
    formatted_source = format_source(str(source), 30)

    if url:
-        from langchain.document_loaders.youtube import \
-            ALLOWED_NETLOCK as YOUTUBE_ALLOWED_NETLOCS
+        from langchain.document_loaders.youtube import ALLOWED_NETLOCK as YOUTUBE_ALLOWED_NETLOCS

        if url.netloc in YOUTUBE_ALLOWED_NETLOCS:
            logging.debug(f"Source of `{formatted_source}` detected as `youtube_video`.")
@@ -261,6 +263,94 @@ def detect_datatype(source: Any) -> DataType:

        # TODO: check if source is gmail query

+        # check if the source is valid json string
+        if is_valid_json_string(source):
+            logging.debug(f"Source of `{formatted_source}` detected as `json`.")
+            return DataType.JSON
+
        # Use text as final fallback.
        logging.debug(f"Source of `{formatted_source}` detected as `text`.")
        return DataType.TEXT
+
+
+# check if the source is valid json string
+def is_valid_json_string(source: str):
+    try:
+        _ = json.loads(source)
+        return True
+    except json.JSONDecodeError:
+        logging.error(
+            "Insert valid string format of JSON. \
+            Check the docs to see the supported formats - `https://docs.embedchain.ai/data-sources/json`"
+        )
+        return False
+
+
+def validate_yaml_config(config_data):
+    schema = Schema(
+        {
+            Optional("app"): {
+                Optional("config"): {
+                    Optional("id"): str,
+                    Optional("name"): str,
+                    Optional("log_level"): Or("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"),
+                    Optional("collect_metrics"): bool,
+                    Optional("collection_name"): str,
+                }
+            },
+            Optional("llm"): {
+                Optional("provider"): Or(
+                    "openai",
+                    "azure_openai",
+                    "anthropic",
+                    "huggingface",
+                    "cohere",
+                    "gpt4all",
+                    "jina",
+                    "llama2",
+                    "vertex_ai",
+                ),
+                Optional("config"): {
+                    Optional("model"): str,
+                    Optional("number_documents"): int,
+                    Optional("temperature"): float,
+                    Optional("max_tokens"): int,
+                    Optional("top_p"): Or(float, int),
+                    Optional("stream"): bool,
+                    Optional("template"): str,
+                    Optional("system_prompt"): str,
+                    Optional("deployment_name"): str,
+                    Optional("where"): dict,
+                    Optional("query_type"): str,
+                },
+            },
+            Optional("vectordb"): {
+                Optional("provider"): Or(
+                    "chroma", "elasticsearch", "opensearch", "pinecone", "qdrant", "weaviate", "zilliz"
+                ),
+                Optional("config"): {
+                    Optional("collection_name"): str,
+                    Optional("dir"): str,
+                    Optional("allow_reset"): bool,
+                    Optional("host"): str,
+                    Optional("port"): str,
+                },
+            },
+            Optional("embedder"): {
+                Optional("provider"): Or("openai", "gpt4all", "huggingface", "vertexai"),
+                Optional("config"): {
+                    Optional("model"): Optional(str),
+                    Optional("deployment_name"): Optional(str),
+                },
+            },
+            Optional("embedding_model"): {
+                Optional("provider"): Or("openai", "gpt4all", "huggingface", "vertexai"),
+                Optional("config"): {
+                    Optional("model"): str,
+                    Optional("deployment_name"): str,
+                },
+            },
+        }
+    )
+
+    return schema.validate(config_data)
@@ -0,0 +1,4 @@
+.env
+app.db
+configs/**.yaml
+db
@@ -0,0 +1,4 @@
+.env
+app.db
+configs/**.yaml
+db
@@ -0,0 +1,15 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+COPY requirements.txt /app/
+
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY . /app
+
+EXPOSE 8080
+
+ENV NAME embedchain
+
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]
@@ -0,0 +1,21 @@
+## Single command to rule them all,
+
+```bash
+docker run -d --name embedchain -p 8080:8080 embedchain/rest-api:latest
+```
+
+### To run the app locally,
+
+```bash
+# will help reload on changes
+DEVELOPMENT=True && python -m main
+```
+
+Using docker (locally),
+
+```bash
+docker build -t embedchain/rest-api:latest .
+docker run -d --name embedchain -p 8080:8080 embedchain/rest-api:latest
+docker image push embedchain/rest-api:latest
+```
+
@@ -0,0 +1,5 @@
+{
+  "version": "1",
+  "name": "ec-rest-api",
+  "type": "collection"
+}
@@ -0,0 +1,18 @@
+meta {
+  name: default_add
+  type: http
+  seq: 3
+}
+
+post {
+  url: http://localhost:8080/add
+  body: json
+  auth: none
+}
+
+body:json {
+  {
+    "source": "source_url",
+    "data_type": "data_type"
+  }
+}
@@ -0,0 +1,17 @@
+meta {
+  name: default_chat
+  type: http
+  seq: 4
+}
+
+post {
+  url: http://localhost:8080/chat
+  body: json
+  auth: none
+}
+
+body:json {
+  {
+    "message": "message"
+  }
+}
@@ -0,0 +1,17 @@
+meta {
+  name: default_query
+  type: http
+  seq: 2
+}
+
+post {
+  url: http://localhost:8080/query
+  body: json
+  auth: none
+}
+
+body:json {
+  {
+    "query": "Who is Elon Musk?"
+  }
+}
@@ -0,0 +1,11 @@
+meta {
+  name: ping
+  type: http
+  seq: 1
+}
+
+get {
+  url: http://localhost:8080/ping
+  body: json
+  auth: none
+}
@@ -0,0 +1,3 @@
+### Config directory
+
+Here, all the YAML files will get stored.
@@ -0,0 +1,11 @@
+from sqlalchemy import create_engine
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm import sessionmaker
+
+SQLALCHEMY_DATABASE_URI = "sqlite:///./app.db"
+
+engine = create_engine(SQLALCHEMY_DATABASE_URI, connect_args={"check_same_thread": False})
+
+SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
+
+Base = declarative_base()
@@ -0,0 +1,17 @@
+app:
+  config:
+    id: 'default'
+
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b.ggmlv3.q4_0.bin'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: gpt4all
+  config:
+    model: 'all-MiniLM-L6-v2'
@@ -0,0 +1,323 @@
+import logging
+import os
+
+import yaml
+from database import Base, SessionLocal, engine
+from fastapi import Depends, FastAPI, HTTPException, UploadFile
+from models import DefaultResponse, DeployAppRequest, QueryApp, SourceApp
+from services import get_app, get_apps, remove_app, save_app
+from sqlalchemy.orm import Session
+from utils import generate_error_message_for_api_keys
+
+from embedchain import Pipeline as App
+from embedchain.client import Client
+
+Base.metadata.create_all(bind=engine)
+
+
+def get_db():
+    db = SessionLocal()
+    try:
+        yield db
+    finally:
+        db.close()
+
+
+app = FastAPI(
+    title="Embedchain REST API",
+    description="This is the REST API for Embedchain.",
+    version="0.0.1",
+    license_info={
+        "name": "Apache 2.0",
+        "url": "https://github.com/embedchain/embedchain/blob/main/LICENSE",
+    },
+)
+
+
+@app.get("/ping", tags=["Utility"])
+def check_status():
+    """
+    Endpoint to check the status of the API
+    """
+    return {"ping": "pong"}
+
+
+@app.get("/apps", tags=["Apps"])
+async def get_all_apps(db: Session = Depends(get_db)):
+    """
+    Get all apps.
+    """
+    apps = get_apps(db)
+    return {"results": apps}
+
+
+@app.post("/create", tags=["Apps"], response_model=DefaultResponse)
+async def create_app_using_default_config(app_id: str, config: UploadFile = None, db: Session = Depends(get_db)):
+    """
+    Create a new app using App ID.
+    If you don't provide a config file, Embedchain will use the default config file\n
+    which uses opensource GPT4ALL model.\n
+    app_id: The ID of the app.\n
+    config: The YAML config file to create an App.\n
+    """
+    try:
+        if app_id is None:
+            raise HTTPException(detail="App ID not provided.", status_code=400)
+
+        if get_app(db, app_id) is not None:
+            raise HTTPException(detail=f"App with id '{app_id}' already exists.", status_code=400)
+
+        yaml_path = "default.yaml"
+        if config is not None:
+            contents = await config.read()
+            try:
+                yaml.safe_load(contents)
+                # TODO: validate the config yaml file here
+                yaml_path = f"configs/{app_id}.yaml"
+                with open(yaml_path, "w") as file:
+                    file.write(str(contents, "utf-8"))
+            except yaml.YAMLError as exc:
+                raise HTTPException(detail=f"Error parsing YAML: {exc}", status_code=400)
+
+        save_app(db, app_id, yaml_path)
+
+        return DefaultResponse(response=f"App created successfully. App ID: {app_id}")
+    except Exception as e:
+        logging.warn(str(e))
+        raise HTTPException(detail=f"Error creating app: {str(e)}", status_code=400)
+
+
+@app.get(
+    "/{app_id}/data",
+    tags=["Apps"],
+)
+async def get_datasources_associated_with_app_id(app_id: str, db: Session = Depends(get_db)):
+    """
+    Get all data sources for an app.\n
+    app_id: The ID of the app. Use "default" for the default app.\n
+    """
+    try:
+        if app_id is None:
+            raise HTTPException(
+                detail="App ID not provided. If you want to use the default app, use 'default' as the app_id.",
+                status_code=400,
+            )
+
+        db_app = get_app(db, app_id)
+
+        if db_app is None:
+            raise HTTPException(detail=f"App with id {app_id} does not exist, please create it first.", status_code=400)
+
+        app = App.from_config(yaml_path=db_app.config)
+
+        response = app.get_data_sources()
+        return {"results": response}
+    except ValueError as ve:
+        logging.warn(str(ve))
+        raise HTTPException(
+            detail=generate_error_message_for_api_keys(ve),
+            status_code=400,
+        )
+    except Exception as e:
+        logging.warn(str(e))
+        raise HTTPException(detail=f"Error occurred: {str(e)}", status_code=400)
+
+
+@app.post(
+    "/{app_id}/add",
+    tags=["Apps"],
+    response_model=DefaultResponse,
+)
+async def add_datasource_to_an_app(body: SourceApp, app_id: str, db: Session = Depends(get_db)):
+    """
+    Add a source to an existing app.\n
+    app_id: The ID of the app. Use "default" for the default app.\n
+    source: The source to add.\n
+    data_type: The data type of the source. Remove it if you want Embedchain to detect it automatically.\n
+    """
+    try:
+        if app_id is None:
+            raise HTTPException(
+                detail="App ID not provided. If you want to use the default app, use 'default' as the app_id.",
+                status_code=400,
+            )
+
+        db_app = get_app(db, app_id)
+
+        if db_app is None:
+            raise HTTPException(detail=f"App with id {app_id} does not exist, please create it first.", status_code=400)
+
+        app = App.from_config(yaml_path=db_app.config)
+
+        response = app.add(source=body.source, data_type=body.data_type)
+        return DefaultResponse(response=response)
+    except ValueError as ve:
+        logging.warn(str(ve))
+        raise HTTPException(
+            detail=generate_error_message_for_api_keys(ve),
+            status_code=400,
+        )
+    except Exception as e:
+        logging.warn(str(e))
+        raise HTTPException(detail=f"Error occurred: {str(e)}", status_code=400)
+
+
+@app.post(
+    "/{app_id}/query",
+    tags=["Apps"],
+    response_model=DefaultResponse,
+)
+async def query_an_app(body: QueryApp, app_id: str, db: Session = Depends(get_db)):
+    """
+    Query an existing app.\n
+    app_id: The ID of the app. Use "default" for the default app.\n
+    query: The query that you want to ask the App.\n
+    """
+    try:
+        if app_id is None:
+            raise HTTPException(
+                detail="App ID not provided. If you want to use the default app, use 'default' as the app_id.",
+                status_code=400,
+            )
+
+        db_app = get_app(db, app_id)
+
+        if db_app is None:
+            raise HTTPException(detail=f"App with id {app_id} does not exist, please create it first.", status_code=400)
+
+        app = App.from_config(yaml_path=db_app.config)
+
+        response = app.query(body.query)
+        return DefaultResponse(response=response)
+    except ValueError as ve:
+        logging.warn(str(ve))
+        raise HTTPException(
+            detail=generate_error_message_for_api_keys(ve),
+            status_code=400,
+        )
+    except Exception as e:
+        logging.warn(str(e))
+        raise HTTPException(detail=f"Error occurred: {str(e)}", status_code=400)
+
+
+# FIXME: The chat implementation of Embedchain needs to be modified to work with the REST API.
+# @app.post(
+#     "/{app_id}/chat",
+#     tags=["Apps"],
+#     response_model=DefaultResponse,
+# )
+# async def chat_with_an_app(body: MessageApp, app_id: str, db: Session = Depends(get_db)):
+#     """
+#     Query an existing app.\n
+#     app_id: The ID of the app. Use "default" for the default app.\n
+#     message: The message that you want to send to the App.\n
+#     """
+#     try:
+#         if app_id is None:
+#             raise HTTPException(
+#                 detail="App ID not provided. If you want to use the default app, use 'default' as the app_id.",
+#                 status_code=400,
+#             )
+
+#         db_app = get_app(db, app_id)
+
+#         if db_app is None:
+#             raise HTTPException(
+#               detail=f"App with id {app_id} does not exist, please create it first.",
+#               status_code=400
+#             )
+
+#         app = App.from_config(yaml_path=db_app.config)
+
+#         response = app.chat(body.message)
+#         return DefaultResponse(response=response)
+#     except ValueError as ve:
+#             raise HTTPException(
+#                 detail=generate_error_message_for_api_keys(ve),
+#                 status_code=400,
+#             )
+#     except Exception as e:
+#         raise HTTPException(detail=f"Error occurred: {str(e)}", status_code=400)
+
+
+@app.post(
+    "/{app_id}/deploy",
+    tags=["Apps"],
+    response_model=DefaultResponse,
+)
+async def deploy_app(body: DeployAppRequest, app_id: str, db: Session = Depends(get_db)):
+    """
+    Query an existing app.\n
+    app_id: The ID of the app. Use "default" for the default app.\n
+    api_key: The API key to use for deployment. If not provided,
+    Embedchain will use the API key previously used (if any).\n
+    """
+    try:
+        if app_id is None:
+            raise HTTPException(
+                detail="App ID not provided. If you want to use the default app, use 'default' as the app_id.",
+                status_code=400,
+            )
+
+        db_app = get_app(db, app_id)
+
+        if db_app is None:
+            raise HTTPException(detail=f"App with id {app_id} does not exist, please create it first.", status_code=400)
+
+        app = App.from_config(yaml_path=db_app.config)
+
+        api_key = body.api_key
+        # this will save the api key in the embedchain.db
+        Client(api_key=api_key)
+
+        app.deploy()
+        return DefaultResponse(response="App deployed successfully.")
+    except ValueError as ve:
+        logging.warn(str(ve))
+        raise HTTPException(
+            detail=generate_error_message_for_api_keys(ve),
+            status_code=400,
+        )
+    except Exception as e:
+        logging.warn(str(e))
+        raise HTTPException(detail=f"Error occurred: {str(e)}", status_code=400)
+
+
+@app.delete(
+    "/{app_id}/delete",
+    tags=["Apps"],
+    response_model=DefaultResponse,
+)
+async def delete_app(app_id: str, db: Session = Depends(get_db)):
+    """
+    Delete an existing app.\n
+    app_id: The ID of the app to be deleted.
+    """
+    try:
+        if app_id is None:
+            raise HTTPException(
+                detail="App ID not provided. If you want to use the default app, use 'default' as the app_id.",
+                status_code=400,
+            )
+
+        db_app = get_app(db, app_id)
+
+        if db_app is None:
+            raise HTTPException(detail=f"App with id {app_id} does not exist, please create it first.", status_code=400)
+
+        app = App.from_config(yaml_path=db_app.config)
+
+        # reset app.db
+        app.db.reset()
+
+        remove_app(db, app_id)
+        return DefaultResponse(response=f"App with id {app_id} deleted successfully.")
+    except Exception as e:
+        raise HTTPException(detail=f"Error occurred: {str(e)}", status_code=400)
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    is_dev = os.getenv("DEVELOPMENT", "False")
+    uvicorn.run("main:app", host="0.0.0.0", port=8080, reload=bool(is_dev))
@@ -0,0 +1,46 @@
+from typing import Optional
+
+from database import Base
+from pydantic import BaseModel, Field
+from sqlalchemy import Column, Integer, String
+
+
+class QueryApp(BaseModel):
+    query: str = Field("", description="The query that you want to ask the App.")
+
+    model_config = {
+        "json_schema_extra": {
+            "example": {
+                "query": "Who is Elon Musk?",
+            }
+        }
+    }
+
+
+class SourceApp(BaseModel):
+    source: str = Field("", description="The source that you want to add to the App.")
+    data_type: Optional[str] = Field("", description="The type of data to add, remove it for autosense.")
+
+    model_config = {"json_schema_extra": {"example": {"source": "https://en.wikipedia.org/wiki/Elon_Musk"}}}
+
+
+class DeployAppRequest(BaseModel):
+    api_key: str = Field("", description="The Embedchain API key for App deployments.")
+
+    model_config = {"json_schema_extra": {"example": {"api_key": "ec-xxx"}}}
+
+
+class MessageApp(BaseModel):
+    message: str = Field("", description="The message that you want to send to the App.")
+
+
+class DefaultResponse(BaseModel):
+    response: str
+
+
+class AppModel(Base):
+    __tablename__ = "apps"
+
+    id = Column(Integer, primary_key=True, index=True)
+    app_id = Column(String, unique=True, index=True)
+    config = Column(String, unique=True, index=True)
@@ -0,0 +1,6 @@
+fastapi==0.104.0
+uvicorn==0.23.2
+embedchain==0.0.91
+embedchain[streamlit, community, opensource, elasticsearch, opensearch, poe, discord, slack, whatsapp, weaviate, pinecone, qdrant, images, huggingface_hub, cohere, milvus, dataloaders, vertexai, llama2, gmail, json]==0.0.91
+sqlalchemy==2.0.22
+python-multipart==0.0.6
@@ -0,0 +1,33 @@
+app:
+  config:
+    id: 'default-app'
+
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+    template: |
+      Use the following pieces of context to answer the query at the end.
+      If you don't know the answer, just say that you don't know, don't try to make up an answer.
+
+      $context
+
+      Query: $query
+
+      Helpful Answer:
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'rest-api-app'
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-ada-002'
@@ -0,0 +1,25 @@
+from models import AppModel
+from sqlalchemy.orm import Session
+
+
+def get_app(db: Session, app_id: str):
+    return db.query(AppModel).filter(AppModel.app_id == app_id).first()
+
+
+def get_apps(db: Session, skip: int = 0, limit: int = 100):
+    return db.query(AppModel).offset(skip).limit(limit).all()
+
+
+def save_app(db: Session, app_id: str, config: str):
+    db_app = AppModel(app_id=app_id, config=config)
+    db.add(db_app)
+    db.commit()
+    db.refresh(db_app)
+    return db_app
+
+
+def remove_app(db: Session, app_id: str):
+    db_app = db.query(AppModel).filter(AppModel.app_id == app_id).first()
+    db.delete(db_app)
+    db.commit()
+    return db_app
@@ -0,0 +1,21 @@
+def generate_error_message_for_api_keys(error: ValueError) -> str:
+    env_mapping = {
+        "OPENAI_API_KEY": "OPENAI_API_KEY",
+        "OPENAI_API_TYPE": "OPENAI_API_TYPE",
+        "OPENAI_API_BASE": "OPENAI_API_BASE",
+        "OPENAI_API_VERSION": "OPENAI_API_VERSION",
+        "COHERE_API_KEY": "COHERE_API_KEY",
+        "ANTHROPIC_API_KEY": "ANTHROPIC_API_KEY",
+        "JINACHAT_API_KEY": "JINACHAT_API_KEY",
+        "HUGGINGFACE_ACCESS_TOKEN": "HUGGINGFACE_ACCESS_TOKEN",
+        "REPLICATE_API_TOKEN": "REPLICATE_API_TOKEN",
+    }
+
+    missing_keys = [env_mapping[key] for key in env_mapping if key in str(error)]
+    if missing_keys:
+        missing_keys_str = ", ".join(missing_keys)
+        return f"""Please set the {missing_keys_str} environment variable(s) when running the Docker container.
+Example: `docker run -e {missing_keys[0]}=xxx embedchain/rest-api:latest`
+"""
+    else:
+        return "Error: " + str(error)
@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.5.1 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.6.1 and should not be changed by hand.

 [[package]]
 name = "aiofiles"
@@ -323,6 +323,26 @@ description = "The uncompromising code formatter."
 optional = false
 python-versions = ">=3.8"
 files = [
+    {file = "black-23.9.1-cp310-cp310-macosx_10_16_arm64.whl", hash = "sha256:d6bc09188020c9ac2555a498949401ab35bb6bf76d4e0f8ee251694664df6301"},
+    {file = "black-23.9.1-cp310-cp310-macosx_10_16_universal2.whl", hash = "sha256:13ef033794029b85dfea8032c9d3b92b42b526f1ff4bf13b2182ce4e917f5100"},
+    {file = "black-23.9.1-cp310-cp310-macosx_10_16_x86_64.whl", hash = "sha256:75a2dc41b183d4872d3a500d2b9c9016e67ed95738a3624f4751a0cb4818fe71"},
+    {file = "black-23.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:13a2e4a93bb8ca74a749b6974925c27219bb3df4d42fc45e948a5d9feb5122b7"},
+    {file = "black-23.9.1-cp310-cp310-win_amd64.whl", hash = "sha256:adc3e4442eef57f99b5590b245a328aad19c99552e0bdc7f0b04db6656debd80"},
+    {file = "black-23.9.1-cp311-cp311-macosx_10_16_arm64.whl", hash = "sha256:8431445bf62d2a914b541da7ab3e2b4f3bc052d2ccbf157ebad18ea126efb91f"},
+    {file = "black-23.9.1-cp311-cp311-macosx_10_16_universal2.whl", hash = "sha256:8fc1ddcf83f996247505db6b715294eba56ea9372e107fd54963c7553f2b6dfe"},
+    {file = "black-23.9.1-cp311-cp311-macosx_10_16_x86_64.whl", hash = "sha256:7d30ec46de88091e4316b17ae58bbbfc12b2de05e069030f6b747dfc649ad186"},
+    {file = "black-23.9.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:031e8c69f3d3b09e1aa471a926a1eeb0b9071f80b17689a655f7885ac9325a6f"},
+    {file = "black-23.9.1-cp311-cp311-win_amd64.whl", hash = "sha256:538efb451cd50f43aba394e9ec7ad55a37598faae3348d723b59ea8e91616300"},
+    {file = "black-23.9.1-cp38-cp38-macosx_10_16_arm64.whl", hash = "sha256:638619a559280de0c2aa4d76f504891c9860bb8fa214267358f0a20f27c12948"},
+    {file = "black-23.9.1-cp38-cp38-macosx_10_16_universal2.whl", hash = "sha256:a732b82747235e0542c03bf352c126052c0fbc458d8a239a94701175b17d4855"},
+    {file = "black-23.9.1-cp38-cp38-macosx_10_16_x86_64.whl", hash = "sha256:cf3a4d00e4cdb6734b64bf23cd4341421e8953615cba6b3670453737a72ec204"},
+    {file = "black-23.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:cf99f3de8b3273a8317681d8194ea222f10e0133a24a7548c73ce44ea1679377"},
+    {file = "black-23.9.1-cp38-cp38-win_amd64.whl", hash = "sha256:14f04c990259576acd093871e7e9b14918eb28f1866f91968ff5524293f9c573"},
+    {file = "black-23.9.1-cp39-cp39-macosx_10_16_arm64.whl", hash = "sha256:c619f063c2d68f19b2d7270f4cf3192cb81c9ec5bc5ba02df91471d0b88c4c5c"},
+    {file = "black-23.9.1-cp39-cp39-macosx_10_16_universal2.whl", hash = "sha256:6a3b50e4b93f43b34a9d3ef00d9b6728b4a722c997c99ab09102fd5efdb88325"},
+    {file = "black-23.9.1-cp39-cp39-macosx_10_16_x86_64.whl", hash = "sha256:c46767e8df1b7beefb0899c4a95fb43058fa8500b6db144f4ff3ca38eb2f6393"},
+    {file = "black-23.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:50254ebfa56aa46a9fdd5d651f9637485068a1adf42270148cd101cdf56e0ad9"},
+    {file = "black-23.9.1-cp39-cp39-win_amd64.whl", hash = "sha256:403397c033adbc45c2bd41747da1f7fc7eaa44efbee256b53842470d4ac5a70f"},
    {file = "black-23.9.1-py3-none-any.whl", hash = "sha256:6ccd59584cc834b6d127628713e4b6b968e5f79572da66284532525a042549f9"},
    {file = "black-23.9.1.tar.gz", hash = "sha256:24b6b3ff5c6d9ea08a8888f6977eae858e1f340d7260cf56d70a49823236b62d"},
 ]
@@ -849,6 +869,17 @@ humanfriendly = ">=9.1"
 [package.extras]
 cron = ["capturer (>=2.4)"]

+[[package]]
+name = "contextlib2"
+version = "21.6.0"
+description = "Backports and enhancements for the contextlib module"
+optional = false
+python-versions = ">=3.6"
+files = [
+    {file = "contextlib2-21.6.0-py2.py3-none-any.whl", hash = "sha256:3fbdb64466afd23abaf6c977627b75b6139a5a3e8ce38405c5b413aed7a0471f"},
+    {file = "contextlib2-21.6.0.tar.gz", hash = "sha256:ab1e2bfe1d01d968e1b7e8d9023bc51ef3509bba217bb730cee3827e1ee82869"},
+]
+
 [[package]]
 name = "contourpy"
 version = "1.1.1"
@@ -1647,12 +1678,12 @@ files = [
 google-auth = ">=2.14.1,<3.0.dev0"
 googleapis-common-protos = ">=1.56.2,<2.0.dev0"
 grpcio = [
-    {version = ">=1.33.2,<2.0dev", optional = true, markers = "extra == \"grpc\""},
    {version = ">=1.49.1,<2.0dev", optional = true, markers = "python_version >= \"3.11\" and extra == \"grpc\""},
+    {version = ">=1.33.2,<2.0dev", optional = true, markers = "python_version < \"3.11\" and extra == \"grpc\""},
 ]
 grpcio-status = [
-    {version = ">=1.33.2,<2.0.dev0", optional = true, markers = "extra == \"grpc\""},
    {version = ">=1.49.1,<2.0.dev0", optional = true, markers = "python_version >= \"3.11\" and extra == \"grpc\""},
+    {version = ">=1.33.2,<2.0.dev0", optional = true, markers = "python_version < \"3.11\" and extra == \"grpc\""},
 ]
 protobuf = ">=3.19.5,<3.20.0 || >3.20.0,<3.20.1 || >3.20.1,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<5.0.0.dev0"
 requests = ">=2.18.0,<3.0.0.dev0"
@@ -1740,8 +1771,8 @@ google-api-core = {version = ">=1.31.5,<2.0.dev0 || >2.3.0,<3.0.0dev", extras =
 google-cloud-core = ">=1.6.0,<3.0.0dev"
 google-resumable-media = ">=0.6.0,<3.0dev"
 grpcio = [
-    {version = ">=1.47.0,<2.0dev", markers = "python_version < \"3.11\""},
    {version = ">=1.49.1,<2.0dev", markers = "python_version >= \"3.11\""},
+    {version = ">=1.47.0,<2.0dev", markers = "python_version < \"3.11\""},
 ]
 packaging = ">=20.0.0"
 proto-plus = ">=1.15.0,<2.0.0dev"
@@ -1792,8 +1823,8 @@ files = [
 google-api-core = {version = ">=1.34.0,<2.0.dev0 || >=2.11.dev0,<3.0.0dev", extras = ["grpc"]}
 grpc-google-iam-v1 = ">=0.12.4,<1.0.0dev"
 proto-plus = [
-    {version = ">=1.22.0,<2.0.0dev", markers = "python_version < \"3.11\""},
    {version = ">=1.22.2,<2.0.0dev", markers = "python_version >= \"3.11\""},
+    {version = ">=1.22.0,<2.0.0dev", markers = "python_version < \"3.11\""},
 ]
 protobuf = ">=3.19.5,<3.20.0 || >3.20.0,<3.20.1 || >3.20.1,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<5.0.0dev"

@@ -1980,7 +2011,7 @@ files = [
    {file = "greenlet-3.0.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:0b72b802496cccbd9b31acea72b6f87e7771ccfd7f7927437d592e5c92ed703c"},
    {file = "greenlet-3.0.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:527cd90ba3d8d7ae7dceb06fda619895768a46a1b4e423bdb24c1969823b8362"},
    {file = "greenlet-3.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:37f60b3a42d8b5499be910d1267b24355c495064f271cfe74bf28b17b099133c"},
-    {file = "greenlet-3.0.0-cp311-universal2-macosx_10_9_universal2.whl", hash = "sha256:c3692ecf3fe754c8c0f2c95ff19626584459eab110eaab66413b1e7425cd84e9"},
+    {file = "greenlet-3.0.0-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:1482fba7fbed96ea7842b5a7fc11d61727e8be75a077e603e8ab49d24e234383"},
    {file = "greenlet-3.0.0-cp312-cp312-macosx_13_0_arm64.whl", hash = "sha256:be557119bf467d37a8099d91fbf11b2de5eb1fd5fc5b91598407574848dc910f"},
    {file = "greenlet-3.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:73b2f1922a39d5d59cc0e597987300df3396b148a9bd10b76a058a2f2772fc04"},
    {file = "greenlet-3.0.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d1e22c22f7826096ad503e9bb681b05b8c1f5a8138469b255eb91f26a76634f2"},
@@ -1990,7 +2021,6 @@ files = [
    {file = "greenlet-3.0.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:952256c2bc5b4ee8df8dfc54fc4de330970bf5d79253c863fb5e6761f00dda35"},
    {file = "greenlet-3.0.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:269d06fa0f9624455ce08ae0179430eea61085e3cf6457f05982b37fd2cefe17"},
    {file = "greenlet-3.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:9adbd8ecf097e34ada8efde9b6fec4dd2a903b1e98037adf72d12993a1c80b51"},
-    {file = "greenlet-3.0.0-cp312-universal2-macosx_10_9_universal2.whl", hash = "sha256:553d6fb2324e7f4f0899e5ad2c427a4579ed4873f42124beba763f16032959af"},
    {file = "greenlet-3.0.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c6b5ce7f40f0e2f8b88c28e6691ca6806814157ff05e794cdd161be928550f4c"},
    {file = "greenlet-3.0.0-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:ecf94aa539e97a8411b5ea52fc6ccd8371be9550c4041011a091eb8b3ca1d810"},
    {file = "greenlet-3.0.0-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:80dcd3c938cbcac986c5c92779db8e8ce51a89a849c135172c88ecbdc8c056b7"},
@@ -3004,6 +3034,16 @@ files = [
    {file = "MarkupSafe-2.1.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:5bbe06f8eeafd38e5d0a4894ffec89378b6c6a625ff57e3028921f8ff59318ac"},
    {file = "MarkupSafe-2.1.3-cp311-cp311-win32.whl", hash = "sha256:dd15ff04ffd7e05ffcb7fe79f1b98041b8ea30ae9234aed2a9168b5797c3effb"},
    {file = "MarkupSafe-2.1.3-cp311-cp311-win_amd64.whl", hash = "sha256:134da1eca9ec0ae528110ccc9e48041e0828d79f24121a1a146161103c76e686"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:f698de3fd0c4e6972b92290a45bd9b1536bffe8c6759c62471efaa8acb4c37bc"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:aa57bd9cf8ae831a362185ee444e15a93ecb2e344c8e52e4d721ea3ab6ef1823"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ffcc3f7c66b5f5b7931a5aa68fc9cecc51e685ef90282f4a82f0f5e9b704ad11"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47d4f1c5f80fc62fdd7777d0d40a2e9dda0a05883ab11374334f6c4de38adffd"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1f67c7038d560d92149c060157d623c542173016c4babc0c1913cca0564b9939"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:9aad3c1755095ce347e26488214ef77e0485a3c34a50c5a5e2471dff60b9dd9c"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:14ff806850827afd6b07a5f32bd917fb7f45b046ba40c57abdb636674a8b559c"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8f9293864fe09b8149f0cc42ce56e3f0e54de883a9de90cd427f191c346eb2e1"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-win32.whl", hash = "sha256:715d3562f79d540f251b99ebd6d8baa547118974341db04f5ad06d5ea3eb8007"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-win_amd64.whl", hash = "sha256:1b8dd8c3fd14349433c79fa8abeb573a55fc0fdd769133baac1f5e07abf54aeb"},
    {file = "MarkupSafe-2.1.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:8e254ae696c88d98da6555f5ace2279cf7cd5b3f52be2b5cf97feafe883b58d2"},
    {file = "MarkupSafe-2.1.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cb0932dc158471523c9637e807d9bfb93e06a95cbf010f1a38b98623b929ef2b"},
    {file = "MarkupSafe-2.1.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9402b03f1a1b4dc4c19845e5c749e3ab82d5078d16a2a4c2cd2df62d57bb0707"},
@@ -3768,13 +3808,11 @@ files = [

 [package.dependencies]
 numpy = [
-    {version = ">=1.21.0", markers = "python_version <= \"3.9\" and platform_system == \"Darwin\" and platform_machine == \"arm64\""},
-    {version = ">=1.21.2", markers = "python_version >= \"3.10\""},
-    {version = ">=1.21.4", markers = "python_version >= \"3.10\" and platform_system == \"Darwin\""},
-    {version = ">=1.19.3", markers = "python_version >= \"3.6\" and platform_system == \"Linux\" and platform_machine == \"aarch64\" or python_version >= \"3.9\""},
-    {version = ">=1.17.0", markers = "python_version >= \"3.7\""},
-    {version = ">=1.17.3", markers = "python_version >= \"3.8\""},
    {version = ">=1.23.5", markers = "python_version >= \"3.11\""},
+    {version = ">=1.21.0", markers = "python_version == \"3.9\" and platform_system == \"Darwin\" and platform_machine == \"arm64\""},
+    {version = ">=1.21.4", markers = "python_version >= \"3.10\" and platform_system == \"Darwin\" and python_version < \"3.11\""},
+    {version = ">=1.21.2", markers = "platform_system != \"Darwin\" and python_version >= \"3.10\" and python_version < \"3.11\""},
+    {version = ">=1.19.3", markers = "platform_system == \"Linux\" and platform_machine == \"aarch64\" and python_version >= \"3.8\" and python_version < \"3.10\" or python_version > \"3.9\" and python_version < \"3.10\" or python_version >= \"3.9\" and platform_system != \"Darwin\" and python_version < \"3.10\" or python_version >= \"3.9\" and platform_machine != \"arm64\" and python_version < \"3.10\""},
 ]

 [[package]]
@@ -3873,8 +3911,8 @@ files = [

 [package.dependencies]
 numpy = [
-    {version = ">=1.22.4", markers = "python_version < \"3.11\""},
    {version = ">=1.23.2", markers = "python_version == \"3.11\""},
+    {version = ">=1.22.4", markers = "python_version < \"3.11\""},
    {version = ">=1.26.0", markers = "python_version >= \"3.12\""},
 ]
 python-dateutil = ">=2.8.2"
@@ -4852,6 +4890,7 @@ files = [
    {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:69b023b2b4daa7548bcfbd4aa3da05b3a74b772db9e23b982788168117739938"},
    {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:81e0b275a9ecc9c0c0c07b4b90ba548307583c125f54d5b6946cfee6360c733d"},
    {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ba336e390cd8e4d1739f42dfe9bb83a3cc2e80f567d8805e11b46f4a943f5515"},
+    {file = "PyYAML-6.0.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:326c013efe8048858a6d312ddd31d56e468118ad4cdeda36c719bf5bb6192290"},
    {file = "PyYAML-6.0.1-cp310-cp310-win32.whl", hash = "sha256:bd4af7373a854424dabd882decdc5579653d7868b8fb26dc7d0e99f823aa5924"},
    {file = "PyYAML-6.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:fd1592b3fdf65fff2ad0004b5e363300ef59ced41c2e6b3a99d4089fa8c5435d"},
    {file = "PyYAML-6.0.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6965a7bc3cf88e5a1c3bd2e0b5c22f8d677dc88a455344035f03399034eb3007"},
@@ -4859,8 +4898,15 @@ files = [
    {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:42f8152b8dbc4fe7d96729ec2b99c7097d656dc1213a3229ca5383f973a5ed6d"},
    {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:062582fca9fabdd2c8b54a3ef1c978d786e0f6b3a1510e0ac93ef59e0ddae2bc"},
    {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d2b04aac4d386b172d5b9692e2d2da8de7bfb6c387fa4f801fbf6fb2e6ba4673"},
+    {file = "PyYAML-6.0.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:e7d73685e87afe9f3b36c799222440d6cf362062f78be1013661b00c5c6f678b"},
    {file = "PyYAML-6.0.1-cp311-cp311-win32.whl", hash = "sha256:1635fd110e8d85d55237ab316b5b011de701ea0f29d07611174a1b42f1444741"},
    {file = "PyYAML-6.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:bf07ee2fef7014951eeb99f56f39c9bb4af143d8aa3c21b1677805985307da34"},
+    {file = "PyYAML-6.0.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:855fb52b0dc35af121542a76b9a84f8d1cd886ea97c84703eaa6d88e37a2ad28"},
+    {file = "PyYAML-6.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:40df9b996c2b73138957fe23a16a4f0ba614f4c0efce1e9406a184b6d07fa3a9"},
+    {file = "PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6c22bec3fbe2524cde73d7ada88f6566758a8f7227bfbf93a408a9d86bcc12a0"},
+    {file = "PyYAML-6.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8d4e9c88387b0f5c7d5f281e55304de64cf7f9c0021a3525bd3b1c542da3b0e4"},
+    {file = "PyYAML-6.0.1-cp312-cp312-win32.whl", hash = "sha256:d483d2cdf104e7c9fa60c544d92981f12ad66a457afae824d146093b8c294c54"},
+    {file = "PyYAML-6.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:0d3304d8c0adc42be59c5f8a4d9e3d7379e6955ad754aa9d6ab7a398b59dd1df"},
    {file = "PyYAML-6.0.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:50550eb667afee136e9a77d6dc71ae76a44df8b3e51e41b77f6de2932bfe0f47"},
    {file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1fe35611261b29bd1de0070f0b2f47cb6ff71fa6595c077e42bd0c419fa27b98"},
    {file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:704219a11b772aea0d8ecd7058d0082713c3562b4e271b849ad7dc4a5c90c13c"},
@@ -4877,6 +4923,7 @@ files = [
    {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a0cd17c15d3bb3fa06978b4e8958dcdc6e0174ccea823003a106c7d4d7899ac5"},
    {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:28c119d996beec18c05208a8bd78cbe4007878c6dd15091efb73a30e90539696"},
    {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7e07cbde391ba96ab58e532ff4803f79c4129397514e1413a7dc761ccd755735"},
+    {file = "PyYAML-6.0.1-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:49a183be227561de579b4a36efbb21b3eab9651dd81b1858589f796549873dd6"},
    {file = "PyYAML-6.0.1-cp38-cp38-win32.whl", hash = "sha256:184c5108a2aca3c5b3d3bf9395d50893a7ab82a38004c8f61c258d4428e80206"},
    {file = "PyYAML-6.0.1-cp38-cp38-win_amd64.whl", hash = "sha256:1e2722cc9fbb45d9b87631ac70924c11d3a401b2d7f410cc0e3bbf249f2dca62"},
    {file = "PyYAML-6.0.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:9eb6caa9a297fc2c2fb8862bc5370d0303ddba53ba97e71f08023b6cd73d16a8"},
@@ -4884,6 +4931,7 @@ files = [
    {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5773183b6446b2c99bb77e77595dd486303b4faab2b086e7b17bc6bef28865f6"},
    {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b786eecbdf8499b9ca1d697215862083bd6d2a99965554781d0d8d1ad31e13a0"},
    {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bc1bf2925a1ecd43da378f4db9e4f799775d6367bdb94671027b73b393a7c42c"},
+    {file = "PyYAML-6.0.1-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:04ac92ad1925b2cff1db0cfebffb6ffc43457495c9b3c39d3fcae417d7125dc5"},
    {file = "PyYAML-6.0.1-cp39-cp39-win32.whl", hash = "sha256:faca3bdcf85b2fc05d06ff3fbc1f83e1391b3e724afa3feba7d13eeab355484c"},
    {file = "PyYAML-6.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:510c9deebc5c0225e8c96813043e62b680ba2f9c50a08d3724c7f28a747d1486"},
    {file = "PyYAML-6.0.1.tar.gz", hash = "sha256:bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43"},
@@ -5382,6 +5430,20 @@ tensorflow = ["safetensors[numpy]", "tensorflow (>=2.11.0)"]
 testing = ["h5py (>=3.7.0)", "huggingface_hub (>=0.12.1)", "hypothesis (>=6.70.2)", "pytest (>=7.2.0)", "pytest-benchmark (>=4.0.0)", "safetensors[numpy]", "setuptools_rust (>=1.5.2)"]
 torch = ["safetensors[numpy]", "torch (>=1.10)"]

+[[package]]
+name = "schema"
+version = "0.7.5"
+description = "Simple data validation library"
+optional = false
+python-versions = "*"
+files = [
+    {file = "schema-0.7.5-py2.py3-none-any.whl", hash = "sha256:f3ffdeeada09ec34bf40d7d79996d9f7175db93b7a5065de0faa7f41083c1e6c"},
+    {file = "schema-0.7.5.tar.gz", hash = "sha256:f06717112c61895cabc4707752b88716e8420a8819d71404501e114f91043197"},
+]
+
+[package.dependencies]
+contextlib2 = ">=0.5.5"
+
 [[package]]
 name = "scikit-learn"
 version = "1.3.1"
@@ -5703,18 +5765,59 @@ description = "Database Abstraction Library"
 optional = false
 python-versions = ">=3.7"
 files = [
+    {file = "SQLAlchemy-2.0.22-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:f146c61ae128ab43ea3a0955de1af7e1633942c2b2b4985ac51cc292daf33222"},
+    {file = "SQLAlchemy-2.0.22-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:875de9414393e778b655a3d97d60465eb3fae7c919e88b70cc10b40b9f56042d"},
+    {file = "SQLAlchemy-2.0.22-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:13790cb42f917c45c9c850b39b9941539ca8ee7917dacf099cc0b569f3d40da7"},
+    {file = "SQLAlchemy-2.0.22-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e04ab55cf49daf1aeb8c622c54d23fa4bec91cb051a43cc24351ba97e1dd09f5"},
+    {file = "SQLAlchemy-2.0.22-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:a42c9fa3abcda0dcfad053e49c4f752eef71ecd8c155221e18b99d4224621176"},
+    {file = "SQLAlchemy-2.0.22-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:14cd3bcbb853379fef2cd01e7c64a5d6f1d005406d877ed9509afb7a05ff40a5"},
+    {file = "SQLAlchemy-2.0.22-cp310-cp310-win32.whl", hash = "sha256:d143c5a9dada696bcfdb96ba2de4a47d5a89168e71d05a076e88a01386872f97"},
+    {file = "SQLAlchemy-2.0.22-cp310-cp310-win_amd64.whl", hash = "sha256:ccd87c25e4c8559e1b918d46b4fa90b37f459c9b4566f1dfbce0eb8122571547"},
    {file = "SQLAlchemy-2.0.22-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:4f6ff392b27a743c1ad346d215655503cec64405d3b694228b3454878bf21590"},
    {file = "SQLAlchemy-2.0.22-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f776c2c30f0e5f4db45c3ee11a5f2a8d9de68e81eb73ec4237de1e32e04ae81c"},
+    {file = "SQLAlchemy-2.0.22-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c8f1792d20d2f4e875ce7a113f43c3561ad12b34ff796b84002a256f37ce9437"},
+    {file = "SQLAlchemy-2.0.22-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d80eeb5189d7d4b1af519fc3f148fe7521b9dfce8f4d6a0820e8f5769b005051"},
+    {file = "SQLAlchemy-2.0.22-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:69fd9e41cf9368afa034e1c81f3570afb96f30fcd2eb1ef29cb4d9371c6eece2"},
+    {file = "SQLAlchemy-2.0.22-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:54bcceaf4eebef07dadfde424f5c26b491e4a64e61761dea9459103ecd6ccc95"},
+    {file = "SQLAlchemy-2.0.22-cp311-cp311-win32.whl", hash = "sha256:7ee7ccf47aa503033b6afd57efbac6b9e05180f492aeed9fcf70752556f95624"},
+    {file = "SQLAlchemy-2.0.22-cp311-cp311-win_amd64.whl", hash = "sha256:b560f075c151900587ade06706b0c51d04b3277c111151997ea0813455378ae0"},
    {file = "SQLAlchemy-2.0.22-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:2c9bac865ee06d27a1533471405ad240a6f5d83195eca481f9fc4a71d8b87df8"},
    {file = "SQLAlchemy-2.0.22-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:625b72d77ac8ac23da3b1622e2da88c4aedaee14df47c8432bf8f6495e655de2"},
+    {file = "SQLAlchemy-2.0.22-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b39a6e21110204a8c08d40ff56a73ba542ec60bab701c36ce721e7990df49fb9"},
+    {file = "SQLAlchemy-2.0.22-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:53a766cb0b468223cafdf63e2d37f14a4757476157927b09300c8c5832d88560"},
+    {file = "SQLAlchemy-2.0.22-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:0e1ce8ebd2e040357dde01a3fb7d30d9b5736b3e54a94002641dfd0aa12ae6ce"},
+    {file = "SQLAlchemy-2.0.22-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:505f503763a767556fa4deae5194b2be056b64ecca72ac65224381a0acab7ebe"},
+    {file = "SQLAlchemy-2.0.22-cp312-cp312-win32.whl", hash = "sha256:154a32f3c7b00de3d090bc60ec8006a78149e221f1182e3edcf0376016be9396"},
+    {file = "SQLAlchemy-2.0.22-cp312-cp312-win_amd64.whl", hash = "sha256:129415f89744b05741c6f0b04a84525f37fbabe5dc3774f7edf100e7458c48cd"},
    {file = "SQLAlchemy-2.0.22-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:3940677d341f2b685a999bffe7078697b5848a40b5f6952794ffcf3af150c301"},
+    {file = "SQLAlchemy-2.0.22-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:55914d45a631b81a8a2cb1a54f03eea265cf1783241ac55396ec6d735be14883"},
+    {file = "SQLAlchemy-2.0.22-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2096d6b018d242a2bcc9e451618166f860bb0304f590d205173d317b69986c95"},
+    {file = "SQLAlchemy-2.0.22-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:19c6986cf2fb4bc8e0e846f97f4135a8e753b57d2aaaa87c50f9acbe606bd1db"},
+    {file = "SQLAlchemy-2.0.22-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:6ac28bd6888fe3c81fbe97584eb0b96804bd7032d6100b9701255d9441373ec1"},
+    {file = "SQLAlchemy-2.0.22-cp37-cp37m-win32.whl", hash = "sha256:cb9a758ad973e795267da334a92dd82bb7555cb36a0960dcabcf724d26299db8"},
+    {file = "SQLAlchemy-2.0.22-cp37-cp37m-win_amd64.whl", hash = "sha256:40b1206a0d923e73aa54f0a6bd61419a96b914f1cd19900b6c8226899d9742ad"},
    {file = "SQLAlchemy-2.0.22-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:3aa1472bf44f61dd27987cd051f1c893b7d3b17238bff8c23fceaef4f1133868"},
    {file = "SQLAlchemy-2.0.22-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:56a7e2bb639df9263bf6418231bc2a92a773f57886d371ddb7a869a24919face"},
+    {file = "SQLAlchemy-2.0.22-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ccca778c0737a773a1ad86b68bda52a71ad5950b25e120b6eb1330f0df54c3d0"},
+    {file = "SQLAlchemy-2.0.22-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7c6c3e9350f9fb16de5b5e5fbf17b578811a52d71bb784cc5ff71acb7de2a7f9"},
+    {file = "SQLAlchemy-2.0.22-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:564e9f9e4e6466273dbfab0e0a2e5fe819eec480c57b53a2cdee8e4fdae3ad5f"},
+    {file = "SQLAlchemy-2.0.22-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:af66001d7b76a3fab0d5e4c1ec9339ac45748bc4a399cbc2baa48c1980d3c1f4"},
+    {file = "SQLAlchemy-2.0.22-cp38-cp38-win32.whl", hash = "sha256:9e55dff5ec115316dd7a083cdc1a52de63693695aecf72bc53a8e1468ce429e5"},
+    {file = "SQLAlchemy-2.0.22-cp38-cp38-win_amd64.whl", hash = "sha256:4e869a8ff7ee7a833b74868a0887e8462445ec462432d8cbeff5e85f475186da"},
+    {file = "SQLAlchemy-2.0.22-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:9886a72c8e6371280cb247c5d32c9c8fa141dc560124348762db8a8b236f8692"},
+    {file = "SQLAlchemy-2.0.22-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:a571bc8ac092a3175a1d994794a8e7a1f2f651e7c744de24a19b4f740fe95034"},
+    {file = "SQLAlchemy-2.0.22-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8db5ba8b7da759b727faebc4289a9e6a51edadc7fc32207a30f7c6203a181592"},
+    {file = "SQLAlchemy-2.0.22-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0b0b3f2686c3f162123adba3cb8b626ed7e9b8433ab528e36ed270b4f70d1cdb"},
+    {file = "SQLAlchemy-2.0.22-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:0c1fea8c0abcb070ffe15311853abfda4e55bf7dc1d4889497b3403629f3bf00"},
+    {file = "SQLAlchemy-2.0.22-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:4bb062784f37b2d75fd9b074c8ec360ad5df71f933f927e9e95c50eb8e05323c"},
+    {file = "SQLAlchemy-2.0.22-cp39-cp39-win32.whl", hash = "sha256:58a3aba1bfb32ae7af68da3f277ed91d9f57620cf7ce651db96636790a78b736"},
+    {file = "SQLAlchemy-2.0.22-cp39-cp39-win_amd64.whl", hash = "sha256:92e512a6af769e4725fa5b25981ba790335d42c5977e94ded07db7d641490a85"},
+    {file = "SQLAlchemy-2.0.22-py3-none-any.whl", hash = "sha256:3076740335e4aaadd7deb3fe6dcb96b3015f1613bd190a4e1634e1b99b02ec86"},
    {file = "SQLAlchemy-2.0.22.tar.gz", hash = "sha256:5434cc601aa17570d79e5377f5fd45ff92f9379e2abed0be5e8c2fba8d353d2b"},
 ]

 [package.dependencies]
-greenlet = {version = "!=0.4.17", markers = "platform_machine == \"win32\" or platform_machine == \"WIN32\" or platform_machine == \"AMD64\" or platform_machine == \"amd64\" or platform_machine == \"x86_64\" or platform_machine == \"ppc64le\" or platform_machine == \"aarch64\""}
+greenlet = {version = "!=0.4.17", markers = "platform_machine == \"aarch64\" or platform_machine == \"ppc64le\" or platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"AMD64\" or platform_machine == \"win32\" or platform_machine == \"WIN32\""}
 typing-extensions = ">=4.2.0"

 [package.extras]
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "embedchain"
-version = "0.0.90"
+version = "0.0.92"
 description = "Data platform for LLMs - Load, index, retrieve and sync any unstructured data"
 authors = [
    "Taranjeet Singh <taranjeet@embedchain.ai>",
@@ -128,6 +128,7 @@ huggingface_hub = { version = "^0.17.3", optional = true }
 pymilvus = { version = "2.3.1", optional = true }
 google-cloud-aiplatform = { version = "^1.26.1", optional = true }
 replicate = { version = "^0.15.4", optional = true }
+schema = "^0.7.5"

 [tool.poetry.group.dev.dependencies]
 black = "^23.3.0"
@@ -1,61 +1,65 @@
 import os
-import unittest
-from unittest.mock import patch
+
+import pytest
+from chromadb.api.models.Collection import Collection

 from embedchain import App
 from embedchain.config import AppConfig, ChromaDbConfig
+from embedchain.embedchain import EmbedChain
+from embedchain.llm.base import BaseLlm
+
+os.environ["OPENAI_API_KEY"] = "test-api-key"


-class TestChromaDbHostsLoglevel(unittest.TestCase):
-    os.environ["OPENAI_API_KEY"] = "test_key"
+@pytest.fixture
+def app_instance():
+    config = AppConfig(log_level="DEBUG", collect_metrics=False)
+    return App(config)

-    @patch("chromadb.api.models.Collection.Collection.add")
-    @patch("embedchain.embedchain.EmbedChain.retrieve_from_database")
-    @patch("embedchain.llm.base.BaseLlm.get_answer_from_llm")
-    @patch("embedchain.llm.base.BaseLlm.get_llm_model_answer")
-    def test_whole_app(
-        self,
-        _mock_add,
-        _mock_ec_retrieve_from_database,
-        _mock_get_answer_from_llm,
-        mock_ec_get_llm_model_answer,
-    ):
-        """
-        Test if the `App` instance is initialized without a config that does not contain default hosts and ports.
-        """
-        config = AppConfig(log_level="DEBUG", collect_metrics=False)

-        app = App(config)
+def test_whole_app(app_instance, mocker):
+    knowledge = "lorem ipsum dolor sit amet, consectetur adipiscing"

-        knowledge = "lorem ipsum dolor sit amet, consectetur adipiscing"
+    mocker.patch.object(EmbedChain, "add")
+    mocker.patch.object(EmbedChain, "retrieve_from_database")
+    mocker.patch.object(BaseLlm, "get_answer_from_llm", return_value=knowledge)
+    mocker.patch.object(BaseLlm, "get_llm_model_answer", return_value=knowledge)
+    mocker.patch.object(BaseLlm, "generate_prompt")

-        app.add(knowledge, data_type="text")
+    app_instance.add(knowledge, data_type="text")
+    app_instance.query("What text did I give you?")
+    app_instance.chat("What text did I give you?")

-        app.query("What text did I give you?")
-        app.chat("What text did I give you?")
+    assert BaseLlm.generate_prompt.call_count == 2
+    app_instance.reset()

-        self.assertEqual(mock_ec_get_llm_model_answer.call_args[1]["documents"], [knowledge])

-    def test_add_after_reset(self):
-        """
-        Test if the `App` instance is correctly reconstructed after a reset.
-        """
-        config = AppConfig(log_level="DEBUG", collect_metrics=False)
-        chroma_config = {"allow_reset": True}
-        app = App(config=config, db_config=ChromaDbConfig(**chroma_config))
-        app.reset()
+def test_add_after_reset(app_instance, mocker):
+    config = AppConfig(log_level="DEBUG", collect_metrics=False)
+    chroma_config = {"allow_reset": True}

-        # Make sure the client is still healthy
-        app.db.client.heartbeat()
-        # Make sure the collection exists, and can be added to
-        app.db.collection.add(
-            embeddings=[[1.1, 2.3, 3.2], [4.5, 6.9, 4.4], [1.1, 2.3, 3.2]],
-            metadatas=[
-                {"chapter": "3", "verse": "16"},
-                {"chapter": "3", "verse": "5"},
-                {"chapter": "29", "verse": "11"},
-            ],
-            ids=["id1", "id2", "id3"],
-        )
+    app_instance = App(config=config, db_config=ChromaDbConfig(**chroma_config))
+    app_instance.reset()

-        app.reset()
+    app_instance.db.client.heartbeat()
+
+    mocker.patch.object(Collection, "add")
+
+    app_instance.db.collection.add(
+        embeddings=[[1.1, 2.3, 3.2], [4.5, 6.9, 4.4], [1.1, 2.3, 3.2]],
+        metadatas=[
+            {"chapter": "3", "verse": "16"},
+            {"chapter": "3", "verse": "5"},
+            {"chapter": "29", "verse": "11"},
+        ],
+        ids=["id1", "id2", "id3"],
+    )
+
+    app_instance.reset()
+
+
+def test_add_with_incorrect_content(app_instance, mocker):
+    content = [{"foo": "bar"}]
+
+    with pytest.raises(ValueError):
+        app_instance.add(content, data_type="json")
@@ -40,7 +40,7 @@ def test_load_data(mocker):
 def test_load_data_url(mocker):
    content = "https://example.com/posts.json"

-    mocker.patch("os.path.isfile", return_value=False)  # Mocking os.path.isfile to simulate a URL case
+    mocker.patch("os.path.isfile", return_value=False)
    mocker.patch(
        "llama_hub.jsondata.base.JSONDataReader.load_data",
        return_value=[Document(text="content1"), Document(text="content2")],
@@ -68,11 +68,11 @@ def test_load_data_url(mocker):
    assert result["doc_id"] == expected_doc_id


-def test_load_data_invalid_content(mocker):
+def test_load_data_invalid_string_content(mocker):
    mocker.patch("os.path.isfile", return_value=False)
    mocker.patch("requests.get")

-    content = "123"
+    content = "123: 345}"

    with pytest.raises(ValueError, match="Invalid content to load json data from"):
        JSONLoader.load_data(content)
@@ -89,3 +89,30 @@ def test_load_data_invalid_url(mocker):

    with pytest.raises(ValueError, match=f"Invalid content to load json data from: {content}"):
        JSONLoader.load_data(content)
+
+
+def test_load_data_from_json_string(mocker):
+    content = '{"foo": "bar"}'
+
+    content_url_str = hashlib.sha256((content).encode("utf-8")).hexdigest()
+
+    mocker.patch("os.path.isfile", return_value=False)
+    mocker.patch(
+        "llama_hub.jsondata.base.JSONDataReader.load_data",
+        return_value=[Document(text="content1"), Document(text="content2")],
+    )
+
+    result = JSONLoader.load_data(content)
+
+    assert "doc_id" in result
+    assert "data" in result
+
+    expected_data = [
+        {"content": "content1", "meta_data": {"url": content_url_str}},
+        {"content": "content2", "meta_data": {"url": content_url_str}},
+    ]
+
+    assert result["data"] == expected_data
+
+    expected_doc_id = hashlib.sha256((content_url_str + ", ".join(["content1", "content2"])).encode()).hexdigest()
+    assert result["doc_id"] == expected_doc_id
Author	SHA1	Message	Date
Deshraj Yadav	f0d112254b	[version] bump package version and minor cleanup (#909 )	2023-11-05 16:34:36 -08:00
Sidharth Mohanty	830a7397ef	Add yaml config validation (#890 )	2023-11-04 22:23:55 -07:00
Deven Patel	5428765329	Beautify JSON docs (#906 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-03 13:52:33 -07:00
Sidharth Mohanty	23c912f2b7	Improve getting started page by adding steps (#904 )	2023-11-03 13:40:40 -07:00
Sidharth Mohanty	9c4b023297	Use either `embedder` or `embedding_model` as YAML key (#905 )	2023-11-03 13:40:16 -07:00
Deven Patel	53037b5ed8	[Feature Improvement] Update JSON Loader to support loading data from more sources (#898 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-03 10:00:27 -07:00
Sidharth Mohanty	e2546a653d	[chore] fix rest api docs and other minor fixes (#902 )	2023-11-03 09:40:48 -07:00
Deshraj Yadav	4b8cada873	[REST API] Change docker image name and update docs (#901 )	2023-11-03 01:08:27 -07:00
Deshraj Yadav	fa3ca1d08a	[REST API] Incorporate changes related to REST API docs (#900 )	2023-11-03 00:42:55 -07:00
Sidharth Mohanty	8dd5cb9602	Add `rest-api` example (#889 )	2023-11-03 00:32:51 -07:00