[Feature/Improvements] Delete data sources from metadata db when using app.delete() (#1286)

Co-authored-by: codeflash-ai[bot] <148906541+codeflash-ai[bot]@users.noreply.github.com>

.github/workflows/ci.yml

@@ -39,7 +39,7 @@ jobs:
           path: .venv
           key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
       - name: Install dependencies
-        run: poetry install --all-extras
+        run: make install_all
         if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
       - name: Lint with ruff
         run: make lint

.gitignore

@@ -165,6 +165,7 @@ cython_debug/
 # Database
 db
 test-db
+!embedchain/core/db/
 
 .vscode
 .idea/

Makefile

@@ -11,6 +11,7 @@ install:
 
 install_all:
 	poetry install --all-extras
+	poetry run pip install pinecone-text pinecone-client
 
 install_es:
 	poetry install --extras elasticsearch
@@ -37,6 +38,12 @@ clean:
 lint:
 	poetry run ruff .
 
+build:
+	poetry build
+
+publish:
+	poetry publish
+
 # for example: make test file=tests/test_factory.py
 test:
 	poetry run pytest $(file)

README.md

@@ -32,9 +32,6 @@
 
 <hr />
 
-
-> ### Checkout our latest [Sadhguru AI app](https://sadhguru-ai.streamlit.app/) built using Embedchain.
-
 ## What is Embedchain?
 
 Embedchain is an Open Source RAG Framework that makes it easy to create and deploy AI apps. At its core, Embedchain follows the design principle of being *"Conventional but Configurable"* to serve both software engineers and machine learning engineers.
@@ -64,7 +61,7 @@ For example, you can create an Elon Musk bot using the following code:
 
 ```python
 import os
-from embedchain import Pipeline as App
+from embedchain import App
 
 # Create a bot instance
 os.environ["OPENAI_API_KEY"] = "YOUR API KEY"

docs/_snippets/get-help.mdx

@@ -2,7 +2,7 @@
   <Card title="Talk to founders" icon="calendar" href="https://cal.com/taranjeetio/ec">
   Schedule a call
   </Card>
-  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
     Join our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">

docs/_snippets/missing-data-source-tip.mdx

@@ -4,7 +4,7 @@
   <Card title="Google Form" icon="file" href="https://forms.gle/NDRCKsRpUHsz2Wcm8" color="#7387d0">
     Fill out this form
   </Card>
-  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
     Let us know on our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">

docs/_snippets/missing-llm-tip.mdx

@@ -1,7 +1,7 @@
 <p>If you can't find the specific LLM you need, no need to fret. We're continuously expanding our support for additional LLMs, and you can help us prioritize by opening an issue on our GitHub or simply reaching out to us on our Slack or Discord community.</p>
 
 <CardGroup cols={2}>
-  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
     Let us know on our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">

docs/_snippets/missing-vector-db-tip.mdx

@@ -1,9 +1,9 @@
 
 
-<p>If you can't find the specific vector database, please feel free to request through one of the following channels and help us prioritize.</p>
+<p>If you can't find specific feature or run into issues, please feel free to reach out through one of the following channels.</p>
 
 <CardGroup cols={2}>
-  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
     Let us know on our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">

docs/api-reference/advanced/configuration.mdx

@@ -8,7 +8,7 @@ You can configure different components of your app (`llm`, `embedding model`, or
 
 
 <Tip>
-Embedchain applications are configurable using YAML file, JSON file or by directly passing the config dictionary. Checkout the [docs here](/api-reference/pipeline/overview#usage) on how to use other formats.
+Embedchain applications are configurable using YAML file, JSON file or by directly passing the config dictionary. Checkout the [docs here](/api-reference/app/overview#usage) on how to use other formats.
 </Tip>
 
 <CodeGroup>
@@ -200,9 +200,10 @@ Alright, let's dive into what each key means in the yaml config above:
         - `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
         - `prompt` (String): A prompt for the model to follow when generating responses, requires `$context` and `$query` variables.
         - `system_prompt` (String): A system prompt for the model to follow when generating responses, in this case, it's set to the style of William Shakespeare.
-        -  `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
+        - `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
         - `number_documents` (Integer): Number of documents to pull from the vectordb as context, defaults to 1
         - `api_key` (String): The API key for the language model.
+        - `model_kwargs` (Dict): Keyword arguments to pass to the language model. Used for `aws_bedrock` provider, since it requires different arguments for each model.
 3. `vectordb` Section:
     - `provider` (String): The provider for the vector database, set to 'chroma'. You can find the full list of vector database providers in [our docs](/components/vector-databases).
     - `config`:
@@ -214,7 +215,11 @@ Alright, let's dive into what each key means in the yaml config above:
     - `provider` (String): The provider for the embedder, set to 'openai'. You can find the full list of embedding model providers in [our docs](/components/embedding-models).
     - `config`:
         - `model` (String): The specific model used for text embedding, 'text-embedding-ada-002'.
+        - `vector_dimension` (Integer): The vector dimension of the embedding model. [Defaults](https://github.com/embedchain/embedchain/blob/e572b5a3dc1b66f1e9b3357d11a88c63b5ce06e3/embedchain/models/vector_dimensions.py)
         - `api_key` (String): The API key for the embedding model.
+        - `deployment_name` (String): The deployment name for the embedding model.
+        - `title` (String): The title for the embedding model for Google Embedder.
+        - `task_type` (String): The task type for the embedding model for Google Embedder.
 5. `chunker` Section:
     - `chunk_size` (Integer): The size of each chunk of text that is sent to the language model.
     - `chunk_overlap` (Integer): The amount of overlap between each chunk of text.

docs/api-reference/app/chat.mdx

@@ -129,3 +129,18 @@ app.chat("What is the net worth of Bill Gates?", session_id="user2")
 app.chat("What was my last question", session_id="user1")
 # 'Your last question was "What is the net worth of Elon Musk?"'
 ```
+
+### With custom context window
+
+If you want to customize the context window that you want to use during chat (default context window is 3 document chunks), you can do using the following code snippet:
+
+```python with custom chunks size
+from embedchain import App
+from embedchain.config import BaseLlmConfig
+
+app = App()
+app.add("https://www.forbes.com/profile/elon-musk")
+
+query_config = BaseLlmConfig(number_documents=5)
+app.chat("What is the net worth of Elon Musk?", config=query_config)
+```

docs/api-reference/app/delete.mdx

@@ -2,9 +2,33 @@
 title: 🗑 delete
 ---
 
+## Delete Document
+
+`delete()` method allows you to delete a document previously added to the app.
+
+### Usage
+
+```python
+from embedchain import App
+
+app = App()
+
+forbes_doc_id = app.add("https://www.forbes.com/profile/elon-musk")
+wiki_doc_id = app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+
+app.delete(forbes_doc_id)   # deletes the forbes document
+```
+
+<Note>
+    If you do not have the document id, you can use `app.db.get()` method to get the document and extract the `hash` key from `metadatas` dictionary object, which serves as the document id.
+</Note>
+
+
+## Delete Chat Session History
+
 `delete_session_chat_history()` method allows you to delete all previous messages in a chat history.
 
-## Usage
+### Usage
 
 ```python
 from embedchain import App
@@ -16,4 +40,9 @@ app.add("https://www.forbes.com/profile/elon-musk")
 app.chat("What is the net worth of Elon Musk?")
 
 app.delete_session_chat_history()
-```
+```
+
+<Note>
+    `delete_session_chat_history(session_id="session_1")` method also accepts `session_id` optional param for deleting chat history of a specific session.
+    It assumes the default session if no `session_id` is provided.
+</Note>

docs/api-reference/app/deploy.mdx

@@ -2,30 +2,4 @@
 title: 🚀 deploy
 ---
 
-Using the `deploy()` method, Embedchain allows developers to easily launch their LLM-powered applications on the [Embedchain Platform](https://app.embedchain.ai). This platform facilitates seamless access to your data's context via a free and user-friendly REST API. Once your pipeline is deployed, you can update your data sources at any time.
-
-The `deploy()` method not only deploys your pipeline but also efficiently manages LLMs, vector databases, embedding models, and data syncing, enabling you to focus on querying, chatting, or searching without the hassle of infrastructure management.
-
-## Usage
-
-```python
-from embedchain import App
-
-# Initialize app
-app = App()
-
-# Add data source
-app.add("https://www.forbes.com/profile/elon-musk")
-
-# Deploy your pipeline to Embedchain Platform
-app.deploy()
-
-# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
-# ec-xxxxxx
-
-# 🛠️ Creating pipeline on the platform...
-# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
-
-# 🛠️ Adding data to your pipeline...
-# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
-```
+The `deploy()` method is currently available on an invitation-only basis. To request access, please submit your information via the provided [Google Form](https://forms.gle/vigN11h7b4Ywat668). We will review your request and respond promptly.

docs/api-reference/app/get.mdx

@@ -0,0 +1,33 @@
+---
+title: 📄 get
+---
+
+## Get data sources
+
+`get_data_sources()` returns a list of all the data sources added in the app.
+
+
+### Usage
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add("https://www.forbes.com/profile/elon-musk")
+app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+
+data_sources = app.get_data_sources()
+# [
+#   {
+#       'data_type': 'web_page',
+#       'data_value': 'https://en.wikipedia.org/wiki/Elon_Musk',
+#       'metadata': 'null'
+#   },
+#   {
+#       'data_type': 'web_page',
+#       'data_value': 'https://www.forbes.com/profile/elon-musk',
+#       'metadata': 'null'
+#   }
+# ]
+```

docs/api-reference/app/search.mdx

@@ -12,6 +12,13 @@ title: '🔍 search'
 <ParamField path="num_documents" type="int" optional>
     Number of relevant documents to fetch. Defaults to `3`
 </ParamField>
+<ParamField path="where" type="dict" optional>
+    Key value pair for metadata filtering.
+</ParamField>
+<ParamField path="raw_filter" type="dict" optional>
+    Pass raw filter query based on your vector database.
+    Currently, `raw_filter` param is only supported for Pinecone vector database.
+</ParamField>
 
 ### Returns
 
@@ -21,37 +28,84 @@ title: '🔍 search'
 
 ## Usage
 
+### Basic
+
 Refer to the following example on how to use the search api:
 
 ```python Code example
 from embedchain import App
 
-# Initialize app
 app = App()
-
-# Add data source
 app.add("https://www.forbes.com/profile/elon-musk")
 
-# Get relevant context using semantic search
 context = app.search("What is the net worth of Elon?", num_documents=2)
 print(context)
-# Context:
-# [
-#     {
-#         'context': 'Elon Musk PROFILEElon MuskCEO, Tesla$221.9BReal Time Net Worth ...',
-#         'metadata': {
-#             'source': 'https://www.forbes.com/profile/elon-musk',
-#             'document_id': 'some_document_id',
-#             'score': 0.404,
-#         }
-#     },
-#     {
-#         'context': 'company, which is now called X.Wealth HistoryHOVER TO REVEAL NET WORTH ...',
-#         'metadata': {
-#             'source': 'https://www.forbes.com/profile/elon-musk',
-#             'document_id': 'some_document_id',
-#             'score': 0.435,
-#         }
-#     }
-# ]
+```
+
+### Advanced
+
+#### Metadata filtering using `where` params
+
+Here is an advanced example of `search()` API with metadata filtering on pinecone database:
+
+```python
+import os
+
+from embedchain import App
+
+os.environ["PINECONE_API_KEY"] = "xxx"
+
+config = {
+    "vectordb": {
+        "provider": "pinecone",
+        "config": {
+            "metric": "dotproduct",
+            "vector_dimension": 1536,
+            "index_name": "ec-test",
+            "serverless_config": {"cloud": "aws", "region": "us-west-2"},
+        },
+    }
+}
+
+app = App.from_config(config=config)
+
+app.add("https://www.forbes.com/profile/bill-gates", metadata={"type": "forbes", "person": "gates"})
+app.add("https://en.wikipedia.org/wiki/Bill_Gates", metadata={"type": "wiki", "person": "gates"})
+
+results = app.search("What is the net worth of Bill Gates?", where={"person": "gates"})
+print("Num of search results: ", len(results))
+```
+
+#### Metadata filtering using `raw_filter` params
+
+Following is an example of metadata filtering by passing the raw filter query that pinecone vector database follows:
+
+```python
+import os
+
+from embedchain import App
+
+os.environ["PINECONE_API_KEY"] = "xxx"
+
+config = {
+    "vectordb": {
+        "provider": "pinecone",
+        "config": {
+            "metric": "dotproduct",
+            "vector_dimension": 1536,
+            "index_name": "ec-test",
+            "serverless_config": {"cloud": "aws", "region": "us-west-2"},
+        },
+    }
+}
+
+app = App.from_config(config=config)
+
+app.add("https://www.forbes.com/profile/bill-gates", metadata={"year": 2022, "person": "gates"})
+app.add("https://en.wikipedia.org/wiki/Bill_Gates", metadata={"year": 2024, "person": "gates"})
+
+print("Filter with person: gates and year > 2023")
+raw_filter = {"$and": [{"person": "gates"}, {"year": {"$gt": 2023}}]}
+results = app.search("What is the net worth of Bill Gates?", raw_filter=raw_filter)
+print("Num of search results: ", len(results))
 ```

docs/community/connect-with-us.mdx

@@ -8,7 +8,7 @@ We believe in building a vibrant and supportive community around embedchain. The
   <Card title="Twitter" icon="twitter" href="https://twitter.com/embedchain">
     Follow us on Twitter
   </Card>
-  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
     Join our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">

docs/components/data-sources/custom.mdx

@@ -7,11 +7,12 @@ When we say "custom", we mean that you can customize the loader and chunker to y
 ```python
 from embedchain import App
 import your_loader
-import your_chunker
+from my_module import CustomLoader
+from my_module import CustomChunker
 
 app = App()
-loader = your_loader()
-chunker = your_chunker()
+loader = CustomLoader()
+chunker = CustomChunker()
 
 app.add("source", data_type="custom", loader=loader, chunker=chunker)
 ```

docs/components/data-sources/dropbox.mdx

@@ -22,7 +22,7 @@ Following is an example of how to use the dropbox loader:
 
 ```python
 import os
-from embedchain import Pipeline as App
+from embedchain import App
 
 os.environ["DROPBOX_ACCESS_TOKEN"] = "sl.xxx"
 os.environ["OPENAI_API_KEY"] = "sk-xxx"

docs/components/data-sources/google-drive.mdx

@@ -19,10 +19,10 @@ The first time you use the loader, you will be prompted to enter your Google acc
 
 
 ```python
-from embedchain import Pipeline as App
+from embedchain import App
 
 app = App()
 
 url = "https://drive.google.com/drive/u/0/folders/xxx-xxx"
 app.add(url, data_type="google_drive")
-```
+```

docs/components/data-sources/overview.mdx

@@ -5,34 +5,35 @@ title: Overview
 Embedchain comes with built-in support for various data sources. We handle the complexity of loading unstructured data from these data sources, allowing you to easily customize your app through a user-friendly interface.
 
 <CardGroup cols={4}>
-  <Card title="📰 PDF file" href="/components/data-sources/pdf-file"></Card>
-  <Card title="📊 CSV file" href="/components/data-sources/csv"></Card>
-  <Card title="📃 JSON file" href="/components/data-sources/json"></Card>
-  <Card title="📝 Text" href="/components/data-sources/text"></Card>
-  <Card title="📁 Directory/ Folder" href="/components/data-sources/directory"></Card>
-  <Card title="🌐 HTML Web page" href="/components/data-sources/web-page"></Card>
-  <Card title="📽️ Youtube Channel" href="/components/data-sources/youtube-channel"></Card>
-  <Card title="📺 Youtube Video" href="/components/data-sources/youtube-video"></Card>
-  <Card title="📚 Docs website" href="/components/data-sources/docs-site"></Card>
-  <Card title="📝 MDX file" href="/components/data-sources/mdx"></Card>
-  <Card title="📄 DOCX file" href="/components/data-sources/docx"></Card>
-  <Card title="📓 Notion" href="/components/data-sources/notion"></Card>
-  <Card title="🗺️ Sitemap" href="/components/data-sources/sitemap"></Card>
-  <Card title="🧾 XML file" href="/components/data-sources/xml"></Card>
-  <Card title="❓💬 Q&A pair" href="/components/data-sources/qna"></Card>
-  <Card title="🙌 OpenAPI" href="/components/data-sources/openapi"></Card>
-  <Card title="📬 Gmail" href="/components/data-sources/gmail"></Card>
-  <Card title="📝 Github" href="/components/data-sources/github"></Card>
-  <Card title="🐘 Postgres" href="/components/data-sources/postgres"></Card>
-  <Card title="🐬 MySQL" href="/components/data-sources/mysql"></Card>
-  <Card title="🤖 Slack" href="/components/data-sources/slack"></Card>
-  <Card title="💬 Discord" href="/components/data-sources/discord"></Card>
-  <Card title="🗨️ Discourse" href="/components/data-sources/discourse"></Card>
-  <Card title="📝 Substack" href="/components/data-sources/substack"></Card>
-  <Card title="🐝 Beehiiv" href="/components/data-sources/beehiiv"></Card>
-  <Card title="💾 Dropbox" href="/components/data-sources/dropbox"></Card>
-  <Card title="🖼️ Image" href="/components/data-sources/image"></Card>
-  <Card title="⚙️ Custom" href="/components/data-sources/custom"></Card>
+  <Card title="PDF file" href="/components/data-sources/pdf-file"></Card>
+  <Card title="CSV file" href="/components/data-sources/csv"></Card>
+  <Card title="JSON file" href="/components/data-sources/json"></Card>
+  <Card title="Text" href="/components/data-sources/text"></Card>
+  <Card title="Directory" href="/components/data-sources/directory"></Card>
+  <Card title="Web page" href="/components/data-sources/web-page"></Card>
+  <Card title="Youtube Channel" href="/components/data-sources/youtube-channel"></Card>
+  <Card title="Youtube Video" href="/components/data-sources/youtube-video"></Card>
+  <Card title="Docs website" href="/components/data-sources/docs-site"></Card>
+  <Card title="MDX file" href="/components/data-sources/mdx"></Card>
+  <Card title="DOCX file" href="/components/data-sources/docx"></Card>
+  <Card title="Notion" href="/components/data-sources/notion"></Card>
+  <Card title="Sitemap" href="/components/data-sources/sitemap"></Card>
+  <Card title="XML file" href="/components/data-sources/xml"></Card>
+  <Card title="Q&A pair" href="/components/data-sources/qna"></Card>
+  <Card title="OpenAPI" href="/components/data-sources/openapi"></Card>
+  <Card title="Gmail" href="/components/data-sources/gmail"></Card>
+  <Card title="Google Drive" href="/components/data-sources/google-drive"></Card>
+  <Card title="GitHub" href="/components/data-sources/github"></Card>
+  <Card title="Postgres" href="/components/data-sources/postgres"></Card>
+  <Card title="MySQL" href="/components/data-sources/mysql"></Card>
+  <Card title="Slack" href="/components/data-sources/slack"></Card>
+  <Card title="Discord" href="/components/data-sources/discord"></Card>
+  <Card title="Discourse" href="/components/data-sources/discourse"></Card>
+  <Card title="Substack" href="/components/data-sources/substack"></Card>
+  <Card title="Beehiiv" href="/components/data-sources/beehiiv"></Card>
+  <Card title="Dropbox" href="/components/data-sources/dropbox"></Card>
+  <Card title="Image" href="/components/data-sources/image"></Card>
+  <Card title="Custom" href="/components/data-sources/custom"></Card>
 </CardGroup>
 
 <br/ >

docs/components/data-sources/pdf-file.mdx

@@ -4,13 +4,6 @@ title: '📰 PDF'
 
 You can load any pdf file from your local file system or through a URL.
 
-## Setup
-Install the following packages for loading youtube videos which help in transcription.
-
-```bash
-pip install pytube youtube-transcript-api
-```
-
 ## Usage
 
 ### Load from a local file
@@ -29,7 +22,7 @@ app = App()
 app.add('https://arxiv.org/pdf/1706.03762.pdf', data_type='pdf_file')
 app.query("What is the paper 'attention is all you need' about?", citations=True)
 # Answer: The paper "Attention Is All You Need" proposes a new network architecture called the Transformer, which is based solely on attention mechanisms. It suggests that complex recurrent or convolutional neural networks can be replaced with a simpler architecture that connects the encoder and decoder through attention. The paper discusses how this approach can improve sequence transduction models, such as neural machine translation.
-# Contexts: 
+# Contexts:
 # [
 #     (
 #         'Provided proper attribution is ...',

docs/components/data-sources/youtube-channel.mdx

@@ -2,15 +2,17 @@
 title: '📽️ Youtube Channel'
 ---
 
-To add all the videos from a youtube channel to your app, use the data_type as `youtube_channel`.
+## Setup
 
-<Note>
 Make sure you have all the required packages installed before using this data type. You can install them by running the following command in your terminal.
 
 ```bash
-pip install -u "embedchain[youtube]"
+pip install -U "embedchain[youtube]"
 ```
-</Note>
+
+## Usage
+
+To add all the videos from a youtube channel to your app, use the data_type as `youtube_channel`.
 
 ```python
 from embedchain import App

docs/components/data-sources/youtube-video.mdx

@@ -2,6 +2,16 @@
 title: '📺 Youtube Video'
 ---
 
+## Setup
+
+Make sure you have all the required packages installed before using this data type. You can install them by running the following command in your terminal.
+
+```bash
+pip install -U "embedchain[youtube]"
+```
+
+## Usage
+
 To add any youtube video to your app, use the data_type as `youtube_video`. Eg:
 
 ```python

docs/components/embedding-models.mdx

@@ -40,7 +40,27 @@ app.query("What is OpenAI?")
 embedder:
   provider: openai
   config:
-    model: 'text-embedding-ada-002'
+    model: 'text-embedding-3-small'
+```
+
+</CodeGroup>
+
+* OpenAI announced two new embedding models: `text-embedding-3-small` and `text-embedding-3-large`. Embedchain supports both these models. Below you can find YAML config for both:
+
+<CodeGroup>
+
+```yaml text-embedding-3-small.yaml
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-3-small'
+```
+
+```yaml text-embedding-3-large.yaml
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-3-large'
 ```
 
 </CodeGroup>

docs/components/introduction.mdx

@@ -9,4 +9,5 @@ You can configure following components
 * [Data Source](/components/data-sources/overview)
 * [LLM](/components/llms)
 * [Embedding Model](/components/embedding-models)
-* [Vector Database](/components/vector-databases)
+* [Vector Database](/components/vector-databases)
+* [Evaluation](/components/evaluation)

docs/components/llms.mdx

@@ -20,6 +20,9 @@ Embedchain comes with built-in support for various popular large language models
   <Card title="Hugging Face" href="#hugging-face"></Card>
   <Card title="Llama2" href="#llama2"></Card>
   <Card title="Vertex AI" href="#vertex-ai"></Card>
+  <Card title="Mistral AI" href="#mistral-ai"></Card>
+  <Card title="AWS Bedrock" href="#aws-bedrock"></Card>
+  <Card title="Groq" href="#groq"></Card>
 </CardGroup>
 
 ## OpenAI
@@ -66,125 +69,75 @@ llm:
 </CodeGroup>
 
 ### Function Calling
-To enable [function calling](https://platform.openai.com/docs/guides/function-calling) in your application using embedchain and OpenAI, you need to pass functions into `OpenAILlm` class as an array of functions. Here are several ways in which you can achieve that:
+Embedchain supports OpenAI [Function calling](https://platform.openai.com/docs/guides/function-calling) with a single function. It accepts inputs in accordance with the [Langchain interface](https://python.langchain.com/docs/modules/model_io/chat/function_calling#legacy-args-functions-and-function_call).
 
-Examples:
-<Accordion title="Using Pydantic Models">
+<Accordion title="Pydantic Model">
   ```python
-import os
-from embedchain import App
-from embedchain.llm.openai import OpenAILlm
-import requests
-from pydantic import BaseModel, Field, ValidationError, field_validator
+  from pydantic import BaseModel
 
-os.environ["OPENAI_API_KEY"] = "sk-xxx"
+  class multiply(BaseModel):
+      """Multiply two integers together."""
 
-class QA(BaseModel):
-    """
-    A question and answer pair.
-    """
-
-    question: str = Field(
-        ..., description="The question.", example="What is a mountain?"
-    )
-    answer: str = Field(
-        ..., description="The answer.", example="A mountain is a hill."
-    )
-    person_who_is_asking: str = Field(
-        ..., description="The person who is asking the question.", example="John"
-    )
-
-    @field_validator("question")
-    def question_must_end_with_a_question_mark(cls, v):
-        """
-        Validate that the question ends with a question mark.
-        """
-        if not v.endswith("?"):
-            raise ValueError("question must end with a question mark")
-        return v
-
-    @field_validator("answer")
-    def answer_must_end_with_a_period(cls, v):
-        """
-        Validate that the answer ends with a period.
-        """
-        if not v.endswith("."):
-            raise ValueError("answer must end with a period")
-        return v
-
-llm = OpenAILlm(config=None,functions=[QA])
-app = App(llm=llm)
-
-result = app.query("Hey I am Sid. What is a mountain? A mountain is a hill.")
-
-print(result)
+      a: int = Field(..., description="First integer")
+      b: int = Field(..., description="Second integer")
   ```
-  </Accordion>
+</Accordion>
   
-  <Accordion title="Using OpenAI JSON schema">
+<Accordion title="Python function">
+  ```python
+  def multiply(a: int, b: int) -> int:
+      """Multiply two integers together.
+
+      Args:
+          a: First integer
+          b: Second integer
+      """
+      return a * b
+  ```
+</Accordion>
+<Accordion title="OpenAI tool dictionary">
+  ```python
+  multiply = {
+    "type": "function",
+    "function": {
+      "name": "multiply",
+      "description": "Multiply two integers together.",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "a": {
+            "description": "First integer",
+            "type": "integer"
+          },
+          "b": {
+            "description": "Second integer",
+            "type": "integer"
+          }
+        },
+        "required": [
+          "a",
+          "b"
+        ]
+      }
+    }
+  }
+  ```
+</Accordion>
+
+With any of the previous inputs, the OpenAI LLM can be queried to provide the appropriate arguments for the function.
+
 ```python
 import os
 from embedchain import App
 from embedchain.llm.openai import OpenAILlm
-import requests
-from pydantic import BaseModel, Field, ValidationError, field_validator
 
 os.environ["OPENAI_API_KEY"] = "sk-xxx"
 
-json_schema = {
-    "name": "get_qa",
-    "description": "A question and answer pair and the user who is asking the question.",
-    "parameters": {
-        "type": "object",
-        "properties": {
-            "question": {"type": "string", "description": "The question."},
-            "answer": {"type": "string", "description": "The answer."},
-            "person_who_is_asking": {
-                "type": "string",
-                "description": "The person who is asking the question.",
-            }
-        },
-        "required": ["question", "answer", "person_who_is_asking"],
-    },
-}
-
-llm = OpenAILlm(config=None,functions=[json_schema])
+llm = OpenAILlm(tools=multiply)
 app = App(llm=llm)
 
-result = app.query("Hey I am Sid. What is a mountain? A mountain is a hill.")
-
-print(result)
-  ```
-  </Accordion>
-  <Accordion title="Using actual python functions">
-  ```python
-import os
-from embedchain import App
-from embedchain.llm.openai import OpenAILlm
-import requests
-from pydantic import BaseModel, Field, ValidationError, field_validator
-
-os.environ["OPENAI_API_KEY"] = "sk-xxx"
-
-def find_info_of_pokemon(pokemon: str):
-    """
-    Find the information of the given pokemon.
-    Args:
-        pokemon: The pokemon.
-    """
-    req = requests.get(f"https://pokeapi.co/api/v2/pokemon/{pokemon}")
-    if req.status_code == 404:
-        raise ValueError("pokemon not found")
-    return req.json()
-
-llm = OpenAILlm(config=None,functions=[find_info_of_pokemon])
-app = App(llm=llm)
-
-result = app.query("Tell me more about the pokemon pikachu.")
-
-print(result)
+result = app.query("What is the result of 125 multiplied by fifteen?")
 ```
-</Accordion>
 
 ## Google AI
 
@@ -250,7 +203,7 @@ app = App.from_config(config_path="config.yaml")
 llm:
   provider: azure_openai
   config:
-    model: gpt-35-turbo
+    model: gpt-3.5-turbo
     deployment_name: your_llm_deployment_name
     temperature: 0.5
     max_tokens: 1000
@@ -620,5 +573,142 @@ llm:
 ```
 </CodeGroup>
 
+
+## Mistral AI
+
+Obtain the Mistral AI api key from their [console](https://console.mistral.ai/).
+
+<CodeGroup>
+ 
+ ```python main.py
+os.environ["MISTRAL_API_KEY"] = "xxx"
+
+app = App.from_config(config_path="config.yaml")
+
+app.add("https://www.forbes.com/profile/elon-musk")
+
+response = app.query("what is the net worth of Elon Musk?")
+# As of January 16, 2024, Elon Musk's net worth is $225.4 billion.
+
+response = app.chat("which companies does elon own?")
+# Elon Musk owns Tesla, SpaceX, Boring Company, Twitter, and X.
+
+response = app.chat("what question did I ask you already?")
+# You have asked me several times already which companies Elon Musk owns, specifically Tesla, SpaceX, Boring Company, Twitter, and X.
+```
+  
+```yaml config.yaml
+llm:
+  provider: mistralai
+  config:
+    model: mistral-tiny
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+embedder:
+  provider: mistralai
+  config:
+    model: mistral-embed
+```
+</CodeGroup>
+
+
+## AWS Bedrock
+
+### Setup
+- Before using the AWS Bedrock LLM, make sure you have the appropriate model access from [Bedrock Console](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/modelaccess).
+- You will also need to authenticate the `boto3` client by using a method in the [AWS documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials)
+- You can optionally export an `AWS_REGION`
+
+
+### Usage
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["AWS_ACCESS_KEY_ID"] = "xxx"
+os.environ["AWS_SECRET_ACCESS_KEY"] = "xxx"
+os.environ["AWS_REGION"] = "us-west-2"
+
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: aws_bedrock
+  config:
+    model: amazon.titan-text-express-v1
+    # check notes below for model_kwargs
+    model_kwargs:
+      temperature: 0.5
+      topP: 1
+      maxTokenCount: 1000
+```
+</CodeGroup>
+
+<br />
+<Note>
+  The model arguments are different for each providers. Please refer to the [AWS Bedrock Documentation](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/providers) to find the appropriate arguments for your model.
+</Note>
+
 <br/ >
+
+## Groq
+
+[Groq](https://groq.com/) is the creator of the world's first Language Processing Unit (LPU), providing exceptional speed performance for AI workloads running on their LPU Inference Engine.
+
+
+### Usage
+
+In order to use LLMs from Groq, go to their [platform](https://console.groq.com/keys) and get the API key.
+
+Set the API key as `GROQ_API_KEY` environment variable or pass in your app configuration to use the model as given below in the example.
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+# Set your API key here or pass as the environment variable
+groq_api_key = "gsk_xxxx"
+
+config = {
+    "llm": {
+        "provider": "groq",
+        "config": {
+            "model": "mixtral-8x7b-32768",
+            "api_key": groq_api_key,
+            "stream": True
+        }
+    }
+}
+
+app = App.from_config(config=config)
+# Add your data source here
+app.add("https://docs.embedchain.ai/sitemap.xml", data_type="sitemap")
+app.query("Write a poem about Embedchain")
+
+# In the realm of data, vast and wide,
+# Embedchain stands with knowledge as its guide.
+# A platform open, for all to try,
+# Building bots that can truly fly.
+
+# With REST API, data in reach,
+# Deployment a breeze, as easy as a speech.
+# Updating data sources, anytime, anyday,
+# Embedchain's power, never sway.
+
+# A knowledge base, an assistant so grand,
+# Connecting to platforms, near and far.
+# Discord, WhatsApp, Slack, and more,
+# Embedchain's potential, never a bore.
+```
+</CodeGroup>
+
+<br/ >
+
 <Snippet file="missing-llm-tip.mdx" />

docs/components/vector-databases.mdx

@@ -17,216 +17,4 @@ Utilizing a vector database alongside Embedchain is a seamless process. All you
   <Card title="Weaviate" href="#weaviate"></Card>
 </CardGroup>
 
-## ChromaDB
-
-<CodeGroup>
-
-```python main.py
-from embedchain import App
-
-# load chroma configuration from yaml file
-app = App.from_config(config_path="config1.yaml")
-```
-
-```yaml config1.yaml
-vectordb:
-  provider: chroma
-  config:
-    collection_name: 'my-collection'
-    dir: db
-    allow_reset: true
-```
-
-```yaml config2.yaml
-vectordb:
-  provider: chroma
-  config:
-    collection_name: 'my-collection'
-    host: localhost
-    port: 5200
-    allow_reset: true
-```
-
-</CodeGroup>
-
-
-## Elasticsearch
-
-Install related dependencies using the following command:
-
-```bash
-pip install --upgrade 'embedchain[elasticsearch]'
-```
-
-<Note>
-You can configure the Elasticsearch connection by providing either `es_url` or `cloud_id`. If you are using the Elasticsearch Service on Elastic Cloud, you can find the `cloud_id` on the [Elastic Cloud dashboard](https://cloud.elastic.co/deployments).
-</Note>
-
-You can authorize the connection to Elasticsearch by providing either `basic_auth`, `api_key`, or `bearer_auth`.
-
-<CodeGroup>
-
-```python main.py
-from embedchain import App
-
-# load elasticsearch configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: elasticsearch
-  config:
-    collection_name: 'es-index'
-    cloud_id: 'deployment-name:xxxx'
-    basic_auth:
-      - elastic
-      - <your_password>
-    verify_certs: false
-```
-</CodeGroup>
-
-## OpenSearch
-
-Install related dependencies using the following command:
-
-```bash
-pip install --upgrade 'embedchain[opensearch]'
-```
-
-<CodeGroup>
-
-```python main.py
-from embedchain import App
-
-# load opensearch configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: opensearch
-  config:
-    collection_name: 'my-app'
-    opensearch_url: 'https://localhost:9200'
-    http_auth:
-      - admin
-      - admin
-    vector_dimension: 1536
-    use_ssl: false
-    verify_certs: false
-```
-
-</CodeGroup>
-
-## Zilliz
-
-Install related dependencies using the following command:
-
-```bash
-pip install --upgrade 'embedchain[milvus]'
-```
-
-Set the Zilliz environment variables `ZILLIZ_CLOUD_URI` and `ZILLIZ_CLOUD_TOKEN` which you can find it on their [cloud platform](https://cloud.zilliz.com/).
-
-<CodeGroup>
-
-```python main.py
-import os
-from embedchain import App
-
-os.environ['ZILLIZ_CLOUD_URI'] = 'https://xxx.zillizcloud.com'
-os.environ['ZILLIZ_CLOUD_TOKEN'] = 'xxx'
-
-# load zilliz configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: zilliz
-  config:
-    collection_name: 'zilliz_app'
-    uri: https://xxxx.api.gcp-region.zillizcloud.com
-    token: xxx
-    vector_dim: 1536
-    metric_type: L2
-```
-
-</CodeGroup>
-
-## LanceDB
-
-_Coming soon_
-
-## Pinecone
-
-Install pinecone related dependencies using the following command:
-
-```bash
-pip install --upgrade 'embedchain[pinecone]'
-```
-
-In order to use Pinecone as vector database, set the environment variables `PINECONE_API_KEY` and `PINECONE_ENV` which you can find on [Pinecone dashboard](https://app.pinecone.io/).
-
-<CodeGroup>
-
-```python main.py
-from embedchain import App
-
-# load pinecone configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: pinecone
-  config:
-    metric: cosine
-    vector_dimension: 1536
-    collection_name: my-pinecone-index
-```
-
-</CodeGroup>
-
-## Qdrant
-
-In order to use Qdrant as a vector database, set the environment variables `QDRANT_URL` and `QDRANT_API_KEY` which you can find on [Qdrant Dashboard](https://cloud.qdrant.io/).
-
-<CodeGroup>
-```python main.py
-from embedchain import App
-
-# load qdrant configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: qdrant
-  config:
-    collection_name: my_qdrant_index
-```
-</CodeGroup>
-
-## Weaviate
-
-In order to use Weaviate as a vector database, set the environment variables `WEAVIATE_ENDPOINT` and `WEAVIATE_API_KEY` which you can find on [Weaviate dashboard](https://console.weaviate.cloud/dashboard).
-
-<CodeGroup>
-```python main.py
-from embedchain import App
-
-# load weaviate configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: weaviate
-  config:
-    collection_name: my_weaviate_index
-```
-</CodeGroup>
-
 <Snippet file="missing-vector-db-tip.mdx" />

docs/components/vector-databases/chromadb.mdx

@@ -0,0 +1,35 @@
+---
+title: ChromaDB
+---
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load chroma configuration from yaml file
+app = App.from_config(config_path="config1.yaml")
+```
+
+```yaml config1.yaml
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-collection'
+    dir: db
+    allow_reset: true
+```
+
+```yaml config2.yaml
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-collection'
+    host: localhost
+    port: 5200
+    allow_reset: true
+```
+
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />

docs/components/vector-databases/elasticsearch.mdx

@@ -0,0 +1,39 @@
+---
+title: Elasticsearch
+---
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[elasticsearch]'
+```
+
+<Note>
+You can configure the Elasticsearch connection by providing either `es_url` or `cloud_id`. If you are using the Elasticsearch Service on Elastic Cloud, you can find the `cloud_id` on the [Elastic Cloud dashboard](https://cloud.elastic.co/deployments).
+</Note>
+
+You can authorize the connection to Elasticsearch by providing either `basic_auth`, `api_key`, or `bearer_auth`.
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load elasticsearch configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: elasticsearch
+  config:
+    collection_name: 'es-index'
+    cloud_id: 'deployment-name:xxxx'
+    basic_auth:
+      - elastic
+      - <your_password>
+    verify_certs: false
+```
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />

docs/components/vector-databases/opensearch.mdx

@@ -0,0 +1,36 @@
+---
+title: OpenSearch
+---
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[opensearch]'
+```
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load opensearch configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: opensearch
+  config:
+    collection_name: 'my-app'
+    opensearch_url: 'https://localhost:9200'
+    http_auth:
+      - admin
+      - admin
+    vector_dimension: 1536
+    use_ssl: false
+    verify_certs: false
+```
+
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />

docs/components/vector-databases/pinecone.mdx

@@ -0,0 +1,106 @@
+---
+title: Pinecone
+---
+
+## Overview
+
+Install pinecone related dependencies using the following command:
+
+```bash
+pip install --upgrade 'pinecone-client pinecone-text'
+```
+
+In order to use Pinecone as vector database, set the environment variable `PINECONE_API_KEY` which you can find on [Pinecone dashboard](https://app.pinecone.io/).
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# Load pinecone configuration from yaml file
+app = App.from_config(config_path="pod_config.yaml")
+# Or
+app = App.from_config(config_path="serverless_config.yaml")
+```
+
+```yaml pod_config.yaml
+vectordb:
+  provider: pinecone
+  config:
+    metric: cosine
+    vector_dimension: 1536
+    index_name: my-pinecone-index
+    pod_config:
+      environment: gcp-starter
+      metadata_config:
+        indexed:
+          - "url"
+          - "hash"
+```
+
+```yaml serverless_config.yaml
+vectordb:
+  provider: pinecone
+  config:
+    metric: cosine
+    vector_dimension: 1536
+    index_name: my-pinecone-index
+    serverless_config:
+      cloud: aws
+      region: us-west-2
+```
+
+</CodeGroup>
+
+<br />
+<Note>
+You can find more information about Pinecone configuration [here](https://docs.pinecone.io/docs/manage-indexes#create-a-pod-based-index).
+You can also optionally provide `index_name` as a config param in yaml file to specify the index name. If not provided, the index name will be `{collection_name}-{vector_dimension}`.
+</Note>
+
+## Usage
+
+### Hybrid search
+
+Here is an example of how you can do hybrid search using Pinecone as a vector database through Embedchain.
+
+```python
+import os
+
+from embedchain import App
+
+config = {
+    'app': {
+        "config": {
+            "id": "ec-docs-hybrid-search"
+        }
+    },
+    'vectordb': {
+        'provider': 'pinecone',
+        'config': {
+            'metric': 'dotproduct',
+            'vector_dimension': 1536,
+            'index_name': 'my-index',
+            'serverless_config': {
+                'cloud': 'aws',
+                'region': 'us-west-2'
+            },
+            'hybrid_search': True, # Remember to set this for hybrid search
+        }
+    }
+}
+
+# Initialize app
+app = App.from_config(config=config)
+
+# Add documents
+app.add("/path/to/file.pdf", data_type="pdf_file", namespace="my-namespace")
+
+# Query
+app.query("<YOUR QUESTION HERE>", namespace="my-namespace")
+```
+
+Under the hood, Embedchain fetches the relevant chunks from the documents you added by doing hybrid search on the pinecone index.
+If you have questions on how pinecone hybrid search works, please refer to their [offical documentation here](https://docs.pinecone.io/docs/hybrid-search).
+
+<Snippet file="missing-vector-db-tip.mdx" />

docs/components/vector-databases/qdrant.mdx

@@ -0,0 +1,23 @@
+---
+title: Qdrant
+---
+
+In order to use Qdrant as a vector database, set the environment variables `QDRANT_URL` and `QDRANT_API_KEY` which you can find on [Qdrant Dashboard](https://cloud.qdrant.io/).
+
+<CodeGroup>
+```python main.py
+from embedchain import App
+
+# load qdrant configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: qdrant
+  config:
+    collection_name: my_qdrant_index
+```
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />

docs/components/vector-databases/weaviate.mdx

@@ -0,0 +1,24 @@
+---
+title: Weaviate
+---
+
+
+In order to use Weaviate as a vector database, set the environment variables `WEAVIATE_ENDPOINT` and `WEAVIATE_API_KEY` which you can find on [Weaviate dashboard](https://console.weaviate.cloud/dashboard).
+
+<CodeGroup>
+```python main.py
+from embedchain import App
+
+# load weaviate configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: weaviate
+  config:
+    collection_name: my_weaviate_index
+```
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />

docs/components/vector-databases/zilliz.mdx

@@ -0,0 +1,39 @@
+---
+title: Zilliz
+---
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[milvus]'
+```
+
+Set the Zilliz environment variables `ZILLIZ_CLOUD_URI` and `ZILLIZ_CLOUD_TOKEN` which you can find it on their [cloud platform](https://cloud.zilliz.com/).
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ['ZILLIZ_CLOUD_URI'] = 'https://xxx.zillizcloud.com'
+os.environ['ZILLIZ_CLOUD_TOKEN'] = 'xxx'
+
+# load zilliz configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: zilliz
+  config:
+    collection_name: 'zilliz_app'
+    uri: https://xxxx.api.gcp-region.zillizcloud.com
+    token: xxx
+    vector_dim: 1536
+    metric_type: L2
+```
+
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />

docs/deployment/embedchain_ai.mdx

@@ -5,31 +5,10 @@ description: 'Deploy your RAG application to embedchain.ai platform'
 
 ## Deploy on Embedchain Platform
 
-Embedchain enables developers to deploy their LLM-powered apps in production using the [Embedchain platform](https://app.embedchain.ai). The platform offers free access to context on your data through its REST API. Once the pipeline is deployed, you can update your data sources anytime after deployment.
+Embedchain enables developers to deploy their LLM-powered apps in production using the Embedchain platform. The platform offers free access to context on your data through its REST API. Once the pipeline is deployed, you can update your data sources anytime after deployment.
 
-See the example below on how to use the deploy your app (for free):
+Deployment to Embedchain Platform is currently available on an invitation-only basis. To request access, please submit your information via the provided [Google Form](https://forms.gle/vigN11h7b4Ywat668). We will review your request and respond promptly.
 
-```python
-from embedchain import App
-
-# Initialize app
-app = App()
-
-# Add data source
-app.add("https://www.forbes.com/profile/elon-musk")
-
-# Deploy your pipeline to Embedchain Platform
-app.deploy()
-
-# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
-# ec-xxxxxx
-
-# 🛠️ Creating pipeline on the platform...
-# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
-
-# 🛠️ Adding data to your pipeline...
-# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
-```
 
 ## Seeking help?
 

docs/deployment/railway.mdx

@@ -0,0 +1,86 @@
+---
+title: 'Railway.app'
+description: 'Deploy your RAG application to railway.app'
+---
+
+It's easy to host your Embedchain-powered apps and APIs on railway.
+
+Follow the instructions given below to deploy your first application quickly:
+
+## Step-1: Create RAG app
+
+```bash Install embedchain
+pip install embedchain
+```
+
+<Tip>
+**Create a full stack app using Embedchain CLI**
+
+To use your hosted embedchain RAG app, you can easily set up a FastAPI server that can be used anywhere.
+To easily set up a FastAPI server, check out [Get started with Full stack](https://docs.embedchain.ai/get-started/full-stack) page.
+
+Hosting this server on railway is super easy!
+
+</Tip>
+
+## Step-2: Set up your project
+
+### With Docker
+
+You can create a `Dockerfile` in the root of the project, with all the instructions. However, this method is sometimes slower in deployment.
+
+### Without Docker
+
+By default, Railway uses Python 3.7. Embedchain requires the python version to be >3.9 in order to install.
+
+To fix this, create a `.python-version` file in the root directory of your project and specify the correct version
+
+```bash .python-version
+3.10
+```
+
+You also need to create a `requirements.txt` file to specify the requirements.
+
+```bash requirements.txt
+python-dotenv
+embedchain
+fastapi==0.108.0
+uvicorn==0.25.0
+embedchain
+beautifulsoup4
+sentence-transformers
+```
+
+## Step-3: Deploy to Railway 🚀
+
+1. Go to https://railway.app and create an account.
+2. Create a project by clicking on the "Start a new project" button
+
+### With Github
+
+Select `Empty Project` or `Deploy from Github Repo`. 
+
+You should be all set!
+
+### Without Github
+
+You can also use the railway CLI to deploy your apps from the terminal, if you don't want to connect a git repository.
+
+To do this, just run this command in your terminal
+
+```bash Install and set up railway CLI
+npm i -g @railway/cli
+railway login
+railway link [projectID]
+```
+
+Finally, run `railway up` to deploy your app.
+```bash Deploy
+railway up
+```
+
+## Seeking help?
+
+If you run into issues with deployment, please feel free to reach out to us via any of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/examples/showcase.mdx

@@ -20,7 +20,7 @@ Embedchain community has been super active in creating demos on top of Embedchai
 - [Create Instant ChatBot 🤖 using embedchain](https://databutton.com/v/h3e680h9) by Avra, ([Tweet](https://twitter.com/Avra_b/status/1674704745154641920/))
 - [JOBO 🤖 — The AI-driven sidekick to craft your resume](https://try-jobo.com/) by Enrico Willemse, ([LinkedIn Post](https://www.linkedin.com/posts/enrico-willemse_jobai-gptfun-embedchain-activity-7090340080879374336-ueLB/))
 - [Explore Your Knowledge Base: Interactive chats over various forms of documents](https://chatdocs.dkedar.com/) by Kedar Dabhadkar, ([LinkedIn Post](https://www.linkedin.com/posts/dkedar7_machinelearning-llmops-activity-7092524836639424513-2O3L/))
-- [Chatbot trained on 1000+ videos of Ester hicks the co-author behind the famous book Secret](https://ask-abraham.thoughtseed.repl.co) by Mohan Kumar
+- [Chatbot trained on 1000+ videos of Ester hicks the co-author behind the famous book Secret](https://askabraham.tokenofme.io/) by Mohan Kumar
 
 
 ## Templates

docs/get-started/deployment.mdx

@@ -9,6 +9,7 @@ After successfully setting up and testing your RAG app locally, the next step is
   <Card title="Fly.io" href="/deployment/fly_io"></Card>
   <Card title="Modal.com" href="/deployment/modal_com"></Card>
   <Card title="Render.com" href="/deployment/render_com"></Card>
+  <Card title="Railway.app" href="/deployment/railway"></Card>
   <Card title="Streamlit.io" href="/deployment/streamlit_io"></Card>
   <Card title="Gradio.app" href="/deployment/gradio_app"></Card>
   <Card title="Huggingface.co" href="/deployment/huggingface_spaces"></Card>

docs/get-started/full-stack.mdx

@@ -8,6 +8,9 @@ Get started with full-stack RAG applications using Embedchain's easy-to-use CLI
 
 Choose your setup method:
 
+* [Without docker](#without-docker)
+* [With Docker](#with-docker)
+
 ### Without Docker
 
 Ensure these are installed:
@@ -21,6 +24,14 @@ Install Docker from [Docker's official website](https://docs.docker.com/engine/i
 
 ## Quick Start Guide
 
+### Install the package
+
+Before proceeding, make sure you have the Embedchain package installed.
+
+```bash
+pip install embedchain -U
+```
+
 ### Setting Up
 
 For the purpose of the demo, you have to set `OPENAI_API_KEY` to start with but you can choose any llm by changing the configuration easily.
@@ -60,3 +71,11 @@ Open http://localhost:3000 to view the chat UI.
 Check out the Embedchain admin panel to see the document chunks for your RAG application.
 
 ![full stack chunks](/images/fullstack-chunks.png)
+
+### API Server
+
+If you want to access the API server, you can do so at http://localhost:8000/docs.
+
+![API Server](/images/fullstack-api-server.png)
+
+You can customize the UI and code as per your requirements.

docs/get-started/quickstart.mdx

@@ -47,7 +47,7 @@ app.query("What is the net worth of Elon Musk today?")
 llm:
   provider: huggingface
   config:
-    model: 'mistralai/Mistral-7B-v0.1'
+    model: 'mistralai/Mistral-7B-Instruct-v0.2'
     top_p: 0.5
 embedder:
   provider: huggingface
@@ -80,4 +80,4 @@ Now that you have created your first app, you can follow any of the links:
 * [Introduction](/get-started/introduction)
 * [Customization](/components/introduction)
 * [Use cases](/use-cases/introduction)
-* [Deployment](/get-started/deployment)
+* [Deployment](/get-started/deployment)

docs/mint.json

@@ -88,9 +88,8 @@
       "pages": [
         "components/introduction",
         {
-          "group": "Data sources",
+          "group": "🗂️ Data sources",
           "pages": [
-
             "components/data-sources/overview",
             {
               "group": "Data types",
@@ -129,8 +128,19 @@
             "components/data-sources/data-type-handling"
           ]
         },
+        {
+          "group": "🗄️ Vector databases",
+          "pages": [
+                "components/vector-databases/chromadb",
+                "components/vector-databases/elasticsearch",
+                "components/vector-databases/pinecone",
+                "components/vector-databases/opensearch",
+                "components/vector-databases/qdrant",
+                "components/vector-databases/weaviate",
+                "components/vector-databases/zilliz"
+          ]
+        },
         "components/llms",
-        "components/vector-databases",
         "components/embedding-models",
         "components/evaluation"
       ]
@@ -142,6 +152,7 @@
         "deployment/fly_io",
         "deployment/modal_com",
         "deployment/render_com",
+        "deployment/railway",
         "deployment/streamlit_io",
         "deployment/gradio_app",
         "deployment/huggingface_spaces",
@@ -199,18 +210,19 @@
     {
       "group": "API Reference",
       "pages": [
-        "api-reference/pipeline/overview",
+        "api-reference/app/overview",
         {
-          "group": "Pipeline methods",
+          "group": "App methods",
           "pages": [
-            "api-reference/pipeline/add",
-            "api-reference/pipeline/query",
-            "api-reference/pipeline/chat",
-            "api-reference/pipeline/search",
-            "api-reference/pipeline/deploy",
-            "api-reference/pipeline/reset",
-            "api-reference/pipeline/delete",
-            "api-reference/pipeline/evaluate"
+            "api-reference/app/add",
+            "api-reference/app/query",
+            "api-reference/app/chat",
+            "api-reference/app/search",
+            "api-reference/app/get",
+            "api-reference/app/evaluate",
+            "api-reference/app/deploy",
+            "api-reference/app/reset",
+            "api-reference/app/delete"
           ]
         },
         "api-reference/store/openai-assistant",
@@ -238,7 +250,7 @@
   "footerSocials": {
     "website": "https://embedchain.ai",
     "github": "https://github.com/embedchain/embedchain",
-    "slack": "https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw",
+    "slack": "https://embedchain.ai/slack",
     "discord": "https://discord.gg/6PzXDgEjG5",
     "twitter": "https://twitter.com/embedchain",
     "linkedin": "https://www.linkedin.com/company/embedchain"

docs/platform/faq.mdx

@@ -1,3 +0,0 @@
----
-title: 'FAQs'
----

docs/platform/overview.mdx

@@ -1,3 +0,0 @@
----
-title: 'Overview'
----

docs/platform/quickstart.mdx

@@ -1,3 +0,0 @@
----
-title: 'Quickstart'
----

docs/platform/roadmap.mdx

@@ -1,3 +0,0 @@
----
-title: 'Roadmap'
----

docs/platform/security.mdx

@@ -1,3 +0,0 @@
----
-title: 'Security'
----

embedchain/__init__.py

@@ -7,4 +7,4 @@ from embedchain.client import Client  # noqa: F401
 from embedchain.pipeline import Pipeline  # noqa: F401
 
 # Setup the user directory if doesn't exist already
-Client.setup_dir()
+Client.setup()

embedchain/alembic.ini

@@ -0,0 +1,116 @@
+# A generic, single database configuration.
+
+[alembic]
+# path to migration scripts
+script_location = embedchain:migrations
+
+# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
+# Uncomment the line below if you want the files to be prepended with date and time
+# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
+# for all available tokens
+# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
+
+# sys.path path, will be prepended to sys.path if present.
+# defaults to the current working directory.
+prepend_sys_path = .
+
+# timezone to use when rendering the date within the migration file
+# as well as the filename.
+# If specified, requires the python>=3.9 or backports.zoneinfo library.
+# Any required deps can installed by adding `alembic[tz]` to the pip requirements
+# string value is passed to ZoneInfo()
+# leave blank for localtime
+# timezone =
+
+# max length of characters to apply to the
+# "slug" field
+# truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+# set to 'true' to allow .pyc and .pyo files without
+# a source .py file to be detected as revisions in the
+# versions/ directory
+# sourceless = false
+
+# version location specification; This defaults
+# to alembic/versions.  When using multiple version
+# directories, initial revisions must be specified with --version-path.
+# The path separator used here should be the separator specified by "version_path_separator" below.
+# version_locations = %(here)s/bar:%(here)s/bat:alembic/versions
+
+# version path separator; As mentioned above, this is the character used to split
+# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
+# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
+# Valid values for version_path_separator are:
+#
+# version_path_separator = :
+# version_path_separator = ;
+# version_path_separator = space
+version_path_separator = os  # Use os.pathsep. Default configuration used for new projects.
+
+# set to 'true' to search source files recursively
+# in each "version_locations" directory
+# new in Alembic version 1.10
+# recursive_version_locations = false
+
+# the output encoding used when revision files
+# are written from script.py.mako
+# output_encoding = utf-8
+
+sqlalchemy.url = driver://user:pass@localhost/dbname
+
+
+[post_write_hooks]
+# post_write_hooks defines scripts or Python functions that are run
+# on newly generated revision scripts.  See the documentation for further
+# detail and examples
+
+# format using "black" - use the console_scripts runner, against the "black" entrypoint
+# hooks = black
+# black.type = console_scripts
+# black.entrypoint = black
+# black.options = -l 79 REVISION_SCRIPT_FILENAME
+
+# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
+# hooks = ruff
+# ruff.type = exec
+# ruff.executable = %(here)s/.venv/bin/ruff
+# ruff.options = --fix REVISION_SCRIPT_FILENAME
+
+# Logging configuration
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARN
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARN
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = WARN
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

embedchain/app.py

@@ -3,7 +3,6 @@ import concurrent.futures
 import json
 import logging
 import os
-import sqlite3
 import uuid
 from typing import Any, Optional, Union
 
@@ -16,7 +15,8 @@ from embedchain.cache import (Config, ExactMatchEvaluation,
                               gptcache_data_manager, gptcache_pre_function)
 from embedchain.client import Client
 from embedchain.config import AppConfig, CacheConfig, ChunkerConfig
-from embedchain.constants import SQLITE_PATH
+from embedchain.core.db.database import get_session, init_db, setup_engine
+from embedchain.core.db.models import DataSource
 from embedchain.embedchain import EmbedChain
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.embedder.openai import OpenAIEmbedder
@@ -33,9 +33,6 @@ from embedchain.utils.misc import validate_config
 from embedchain.vectordb.base import BaseVectorDB
 from embedchain.vectordb.chroma import ChromaDB
 
-# Set up the user directory if it doesn't exist already
-Client.setup_dir()
-
 
 @register_deserializable
 class App(EmbedChain):
@@ -89,15 +86,18 @@ class App(EmbedChain):
 
         logging.basicConfig(level=log_level, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s")
         self.logger = logging.getLogger(__name__)
+
+        # Initialize the metadata db for the app
+        setup_engine(database_uri=os.environ.get("EMBEDCHAIN_DB_URI"))
+        init_db()
+
         self.auto_deploy = auto_deploy
         # Store the dict config as an attribute to be able to send it
         self.config_data = config_data if (config_data and validate_config(config_data)) else None
         self.client = None
         # pipeline_id from the backend
         self.id = None
-        self.chunker = None
-        if chunker:
-            self.chunker = ChunkerConfig(**chunker)
+        self.chunker = ChunkerConfig(**chunker) if chunker else None
         self.cache_config = cache_config
 
         self.config = config or AppConfig()
@@ -120,6 +120,9 @@ class App(EmbedChain):
         self.llm = llm or OpenAILlm()
         self._init_db()
 
+        # Session for the metadata db
+        self.db_session = get_session()
+
         # If cache_config is provided, initializing the cache ...
         if self.cache_config is not None:
             self._init_cache()
@@ -127,27 +130,6 @@ class App(EmbedChain):
         # Send anonymous telemetry
         self._telemetry_props = {"class": self.__class__.__name__}
         self.telemetry = AnonymousTelemetry(enabled=self.config.collect_metrics)
-
-        # Establish a connection to the SQLite database
-        self.connection = sqlite3.connect(SQLITE_PATH, check_same_thread=False)
-        self.cursor = self.connection.cursor()
-
-        # Create the 'data_sources' table if it doesn't exist
-        self.cursor.execute(
-            """
-            CREATE TABLE IF NOT EXISTS data_sources (
-                pipeline_id TEXT,
-                hash TEXT,
-                type TEXT,
-                value TEXT,
-                metadata TEXT,
-                is_uploaded INTEGER DEFAULT 0,
-                PRIMARY KEY (pipeline_id, hash)
-            )
-        """
-        )
-        self.connection.commit()
-        # Send anonymous telemetry
         self.telemetry.capture(event_name="init", properties=self._telemetry_props)
 
         self.user_asks = []
@@ -250,30 +232,6 @@ class App(EmbedChain):
         r.raise_for_status()
         return r.json()
 
-    def search(self, query, num_documents=3):
-        """
-        Search for similar documents related to the query in the vector database.
-        """
-        # Send anonymous telemetry
-        self.telemetry.capture(event_name="search", properties=self._telemetry_props)
-
-        # TODO: Search will call the endpoint rather than fetching the data from the db itself when deploy=True.
-        if self.id is None:
-            where = {"app_id": self.local_id}
-            context = self.db.query(
-                query,
-                n_results=num_documents,
-                where=where,
-                citations=True,
-            )
-            result = []
-            for c in context:
-                result.append({"context": c[0], "metadata": c[1]})
-            return result
-        else:
-            # Make API call to the backend to get the results
-            NotImplementedError("Search is not implemented yet for the prod mode.")
-
     def _upload_file_to_presigned_url(self, presigned_url, file_path):
         try:
             with open(file_path, "rb") as file:
@@ -331,20 +289,14 @@ class App(EmbedChain):
             return False
 
     def _mark_data_as_uploaded(self, data_hash):
-        self.cursor.execute(
-            "UPDATE data_sources SET is_uploaded = 1 WHERE hash = ? AND pipeline_id = ?",
-            (data_hash, self.local_id),
-        )
-        self.connection.commit()
+        self.db_session.query(DataSource).filter_by(hash=data_hash, app_id=self.local_id).update({"is_uploaded": 1})
 
     def get_data_sources(self):
-        db_data = self.cursor.execute("SELECT * FROM data_sources WHERE pipeline_id = ?", (self.local_id,)).fetchall()
-
-        data_sources = []
-        for data in db_data:
-            data_sources.append({"data_type": data[2], "data_value": data[3], "metadata": data[4]})
-
-        return data_sources
+        data_sources = self.db_session.query(DataSource).filter_by(app_id=self.local_id).all()
+        results = []
+        for row in data_sources:
+            results.append({"data_type": row.data_type, "data_value": row.data_value, "metadata": row.metadata})
+        return results
 
     def deploy(self):
         if self.client is None:
@@ -353,14 +305,11 @@ class App(EmbedChain):
         pipeline_data = self._create_pipeline()
         self.id = pipeline_data["id"]
 
-        results = self.cursor.execute(
-            "SELECT * FROM data_sources WHERE pipeline_id = ? AND is_uploaded = 0", (self.local_id,)  # noqa:E501
-        ).fetchall()
-
+        results = self.db_session.query(DataSource).filter_by(app_id=self.local_id, is_uploaded=0).all()
         if len(results) > 0:
             print("🛠️ Adding data to your pipeline...")
         for result in results:
-            data_hash, data_type, data_value = result[1], result[2], result[3]
+            data_hash, data_type, data_value = result.hash, result.data_type, result.data_value
             self._process_and_upload_data(data_hash, data_type, data_value)
 
         # Send anonymous telemetry
@@ -375,18 +324,18 @@ class App(EmbedChain):
         yaml_path: Optional[str] = None,
     ):
         """
-        Instantiate a Pipeline object from a configuration.
+        Instantiate a App object from a configuration.
 
         :param config_path: Path to the YAML or JSON configuration file.
         :type config_path: Optional[str]
         :param config: A dictionary containing the configuration.
         :type config: Optional[dict[str, Any]]
-        :param auto_deploy: Whether to deploy the pipeline automatically, defaults to False
+        :param auto_deploy: Whether to deploy the app automatically, defaults to False
         :type auto_deploy: bool, optional
         :param yaml_path: (Deprecated) Path to the YAML configuration file. Use config_path instead.
         :type yaml_path: Optional[str]
-        :return: An instance of the Pipeline class.
-        :rtype: Pipeline
+        :return: An instance of the App class.
+        :rtype: App
         """
         # Backward compatibility for yaml_path
         if yaml_path and not config_path:
@@ -420,7 +369,7 @@ class App(EmbedChain):
             raise Exception(f"Error occurred while validating the config. Error: {str(e)}")
 
         app_config_data = config_data.get("app", {}).get("config", {})
-        db_config_data = config_data.get("vectordb", {})
+        vector_db_config_data = config_data.get("vectordb", {})
         embedding_model_config_data = config_data.get("embedding_model", config_data.get("embedder", {}))
         llm_config_data = config_data.get("llm", {})
         chunker_config_data = config_data.get("chunker", {})
@@ -428,10 +377,14 @@ class App(EmbedChain):
 
         app_config = AppConfig(**app_config_data)
 
-        db_provider = db_config_data.get("provider", "chroma")
-        db = VectorDBFactory.create(db_provider, db_config_data.get("config", {}))
+        vector_db_provider = vector_db_config_data.get("provider", "chroma")
+        vector_db = VectorDBFactory.create(vector_db_provider, vector_db_config_data.get("config", {}))
 
         if llm_config_data:
+            # Initialize the metadata db for the app here since llmfactory needs it for initialization of
+            # the llm memory
+            setup_engine(database_uri=os.environ.get("EMBEDCHAIN_DB_URI"))
+            init_db()
             llm_provider = llm_config_data.get("provider", "openai")
             llm = LlmFactory.create(llm_provider, llm_config_data.get("config", {}))
         else:
@@ -447,14 +400,10 @@ class App(EmbedChain):
         else:
             cache_config = None
 
-        # Send anonymous telemetry
-        event_properties = {"init_type": "config_data"}
-        AnonymousTelemetry().capture(event_name="init", properties=event_properties)
-
         return cls(
             config=app_config,
             llm=llm,
-            db=db,
+            db=vector_db,
             embedding_model=embedding_model,
             config_data=config_data,
             auto_deploy=auto_deploy,

embedchain/chunkers/base_chunker.py

@@ -39,11 +39,14 @@ class BaseChunker(JSONSerializable):
         for data in data_records:
             content = data["content"]
 
-            meta_data = data["meta_data"]
+            metadata = data["meta_data"]
             # add data type to meta data to allow query using data type
-            meta_data["data_type"] = self.data_type.value
-            meta_data["doc_id"] = doc_id
-            url = meta_data["url"]
+            metadata["data_type"] = self.data_type.value
+            metadata["doc_id"] = doc_id
+
+            # TODO: Currently defaulting to the src as the url. This is done intentianally since some
+            # of the data types like 'gmail' loader doesn't have the url in the meta data.
+            url = metadata.get("url", src)
 
             chunks = self.get_chunks(content)
             for chunk in chunks:
@@ -53,7 +56,7 @@ class BaseChunker(JSONSerializable):
                     id_map[chunk_id] = True
                     chunk_ids.append(chunk_id)
                     documents.append(chunk)
-                    metadatas.append(meta_data)
+                    metadatas.append(metadata)
         return {
             "documents": documents,
             "ids": chunk_ids,

embedchain/client.py

@@ -31,7 +31,7 @@ class Client:
                 )
 
     @classmethod
-    def setup_dir(cls):
+    def setup(cls):
         """
         Loads the user id from the config file if it exists, otherwise generates a new
         one and saves it to the config file.
@@ -40,6 +40,7 @@ class Client:
         :rtype: str
         """
         os.makedirs(CONFIG_DIR, exist_ok=True)
+
         if os.path.exists(CONFIG_FILE):
             with open(CONFIG_FILE, "r") as f:
                 data = json.load(f)
@@ -53,7 +54,7 @@ class Client:
     @classmethod
     def load_config(cls):
         if not os.path.exists(CONFIG_FILE):
-            cls.setup_dir()
+            cls.setup()
 
         with open(CONFIG_FILE, "r") as config_file:
             return json.load(config_file)

embedchain/config/base_app_config.py

@@ -61,4 +61,3 @@ class BaseAppConfig(BaseConfig, JSONSerializable):
 
         logging.basicConfig(format="%(asctime)s [%(name)s] [%(levelname)s] %(message)s", level=level)
         self.logger = logging.getLogger(__name__)
-        return

embedchain/config/embedder/base.py

@@ -6,7 +6,11 @@ from embedchain.helpers.json_serializable import register_deserializable
 @register_deserializable
 class BaseEmbedderConfig:
     def __init__(
-        self, model: Optional[str] = None, deployment_name: Optional[str] = None, api_key: Optional[str] = None
+        self,
+        model: Optional[str] = None,
+        deployment_name: Optional[str] = None,
+        vector_dimension: Optional[int] = None,
+        api_key: Optional[str] = None,
     ):
         """
         Initialize a new instance of an embedder config class.
@@ -18,4 +22,5 @@ class BaseEmbedderConfig:
         """
         self.model = model
         self.deployment_name = deployment_name
+        self.vector_dimension = vector_dimension
         self.api_key = api_key

embedchain/config/llm/base.py

@@ -7,39 +7,60 @@ from embedchain.config.base_config import BaseConfig
 from embedchain.helpers.json_serializable import register_deserializable
 
 DEFAULT_PROMPT = """
-  Use the following pieces of context to answer the query at the end.
-  If you don't know the answer, just say that you don't know, don't try to make up an answer.
+You are a Q&A expert system. Your responses must always be rooted in the context provided for each query. Here are some guidelines to follow:
 
-  $context
+1. Refrain from explicitly mentioning the context provided in your response.
+2. The context should silently guide your answers without being directly acknowledged.
+3. Do not use phrases such as 'According to the context provided', 'Based on the context, ...' etc.
 
-  Query: $query
+Context information:
+----------------------
+$context
+----------------------
 
-  Helpful Answer:
+Query: $query
+Answer:
 """  # noqa:E501
 
 DEFAULT_PROMPT_WITH_HISTORY = """
-  Use the following pieces of context to answer the query at the end.
-  If you don't know the answer, just say that you don't know, don't try to make up an answer.
-  I will provide you with our conversation history.
+You are a Q&A expert system. Your responses must always be rooted in the context provided for each query. You are also provided with the conversation history with the user. Make sure to use relevant context from conversation history as needed.
 
-  $context
+Here are some guidelines to follow:
 
-  History: $history
+1. Refrain from explicitly mentioning the context provided in your response.
+2. The context should silently guide your answers without being directly acknowledged.
+3. Do not use phrases such as 'According to the context provided', 'Based on the context, ...' etc.
 
-  Query: $query
+Context information:
+----------------------
+$context
+----------------------
 
-  Helpful Answer:
+Conversation history:
+----------------------
+$history
+----------------------
+
+Query: $query
+Answer:
 """  # noqa:E501
 
 DOCS_SITE_DEFAULT_PROMPT = """
-  Use the following pieces of context to answer the query at the end.
-  If you don't know the answer, just say that you don't know, don't try to make up an answer. Wherever possible, give complete code snippet. Dont make up any code snippet on your own.
+You are an expert AI assistant for developer support product. Your responses must always be rooted in the context provided for each query. Wherever possible, give complete code snippet. Dont make up any code snippet on your own.
 
-  $context
+Here are some guidelines to follow:
 
-  Query: $query
+1. Refrain from explicitly mentioning the context provided in your response.
+2. The context should silently guide your answers without being directly acknowledged.
+3. Do not use phrases such as 'According to the context provided', 'Based on the context, ...' etc.
 
-  Helpful Answer:
+Context information:
+----------------------
+$context
+----------------------
+
+Query: $query
+Answer:
 """  # noqa:E501
 
 DEFAULT_PROMPT_TEMPLATE = Template(DEFAULT_PROMPT)

embedchain/config/vectordb/pinecone.py

@@ -1,3 +1,4 @@
+import os
 from typing import Optional
 
 from embedchain.config.vectordb.base import BaseVectorDbConfig
@@ -8,13 +9,37 @@ from embedchain.helpers.json_serializable import register_deserializable
 class PineconeDBConfig(BaseVectorDbConfig):
     def __init__(
         self,
-        collection_name: Optional[str] = None,
-        dir: Optional[str] = None,
+        index_name: Optional[str] = None,
+        api_key: Optional[str] = None,
         vector_dimension: int = 1536,
         metric: Optional[str] = "cosine",
+        pod_config: Optional[dict[str, any]] = None,
+        serverless_config: Optional[dict[str, any]] = None,
+        hybrid_search: bool = False,
+        bm25_encoder: any = None,
         **extra_params: dict[str, any],
     ):
         self.metric = metric
+        self.api_key = api_key
+        self.index_name = index_name
         self.vector_dimension = vector_dimension
         self.extra_params = extra_params
-        super().__init__(collection_name=collection_name, dir=dir)
+        self.hybrid_search = hybrid_search
+        self.bm25_encoder = bm25_encoder
+        if pod_config is None and serverless_config is None:
+            # If no config is provided, use the default pod spec config
+            pod_environment = os.environ.get("PINECONE_ENV", "gcp-starter")
+            self.pod_config = {"environment": pod_environment, "metadata_config": {"indexed": ["*"]}}
+        else:
+            self.pod_config = pod_config
+        self.serverless_config = serverless_config
+
+        if self.pod_config and self.serverless_config:
+            raise ValueError("Only one of pod_config or serverless_config can be provided.")
+
+        if self.hybrid_search and self.metric != "dotproduct":
+            raise ValueError(
+                "Hybrid search is only supported with dotproduct metric in Pinecone. See full docs here: https://docs.pinecone.io/docs/hybrid-search#limitations"
+            )  # noqa:E501
+
+        super().__init__(collection_name=self.index_name, dir=None)

embedchain/constants.py

@@ -6,3 +6,6 @@ HOME_DIR = str(Path.home())
 CONFIG_DIR = os.path.join(HOME_DIR, ".embedchain")
 CONFIG_FILE = os.path.join(CONFIG_DIR, "config.json")
 SQLITE_PATH = os.path.join(CONFIG_DIR, "embedchain.db")
+
+# Set the environment variable for the database URI
+os.environ.setdefault("EMBEDCHAIN_DB_URI", f"sqlite:///{SQLITE_PATH}")

embedchain/core/db/database.py

@@ -0,0 +1,88 @@
+import os
+
+from alembic import command
+from alembic.config import Config
+from sqlalchemy import create_engine
+from sqlalchemy.engine.base import Engine
+from sqlalchemy.orm import Session as SQLAlchemySession
+from sqlalchemy.orm import scoped_session, sessionmaker
+
+from .models import Base
+
+
+class DatabaseManager:
+    def __init__(self, echo: bool = False):
+        self.database_uri = os.environ.get("EMBEDCHAIN_DB_URI")
+        self.echo = echo
+        self.engine: Engine = None
+        self._session_factory = None
+
+    def setup_engine(self) -> None:
+        """Initializes the database engine and session factory."""
+        if not self.database_uri:
+            raise RuntimeError("Database URI is not set. Set the EMBEDCHAIN_DB_URI environment variable.")
+        connect_args = {}
+        if self.database_uri.startswith("sqlite"):
+            connect_args["check_same_thread"] = False
+        self.engine = create_engine(self.database_uri, echo=self.echo, connect_args=connect_args)
+        self._session_factory = scoped_session(sessionmaker(bind=self.engine))
+        Base.metadata.bind = self.engine
+
+    def init_db(self) -> None:
+        """Creates all tables defined in the Base metadata."""
+        if not self.engine:
+            raise RuntimeError("Database engine is not initialized. Call setup_engine() first.")
+        Base.metadata.create_all(self.engine)
+
+    def get_session(self) -> SQLAlchemySession:
+        """Provides a session for database operations."""
+        if not self._session_factory:
+            raise RuntimeError("Session factory is not initialized. Call setup_engine() first.")
+        return self._session_factory()
+
+    def close_session(self) -> None:
+        """Closes the current session."""
+        if self._session_factory:
+            self._session_factory.remove()
+
+    def execute_transaction(self, transaction_block):
+        """Executes a block of code within a database transaction."""
+        session = self.get_session()
+        try:
+            transaction_block(session)
+            session.commit()
+        except Exception as e:
+            session.rollback()
+            raise e
+        finally:
+            self.close_session()
+
+
+# Singleton pattern to use throughout the application
+database_manager = DatabaseManager()
+
+
+# Convenience functions for backward compatibility and ease of use
+def setup_engine(database_uri: str, echo: bool = False) -> None:
+    database_manager.database_uri = database_uri
+    database_manager.echo = echo
+    database_manager.setup_engine()
+
+
+def alembic_upgrade() -> None:
+    """Upgrades the database to the latest version."""
+    alembic_config_path = os.path.join(os.path.dirname(__file__), "..", "..", "alembic.ini")
+    alembic_cfg = Config(alembic_config_path)
+    command.upgrade(alembic_cfg, "head")
+
+
+def init_db() -> None:
+    alembic_upgrade()
+
+
+def get_session() -> SQLAlchemySession:
+    return database_manager.get_session()
+
+
+def execute_transaction(transaction_block):
+    database_manager.execute_transaction(transaction_block)

embedchain/core/db/models.py

@@ -0,0 +1,31 @@
+import uuid
+
+from sqlalchemy import TIMESTAMP, Column, Integer, String, Text, func
+from sqlalchemy.orm import declarative_base
+
+Base = declarative_base()
+metadata = Base.metadata
+
+
+class DataSource(Base):
+    __tablename__ = "ec_data_sources"
+
+    id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
+    app_id = Column(Text, index=True)
+    hash = Column(Text, index=True)
+    type = Column(Text, index=True)
+    value = Column(Text)
+    meta_data = Column(Text, name="metadata")
+    is_uploaded = Column(Integer, default=0)
+
+
+class ChatHistory(Base):
+    __tablename__ = "ec_chat_history"
+
+    app_id = Column(String, primary_key=True)
+    id = Column(String, primary_key=True)
+    session_id = Column(String, primary_key=True, index=True)
+    question = Column(Text)
+    answer = Column(Text)
+    meta_data = Column(Text, name="metadata")
+    created_at = Column(TIMESTAMP, default=func.current_timestamp(), index=True)

embedchain/deployment/fly.io/app.py

@@ -2,12 +2,12 @@ from dotenv import load_dotenv
 from fastapi import FastAPI, responses
 from pydantic import BaseModel
 
-from embedchain import Pipeline
+from embedchain import App
 
 load_dotenv(".env")
 
 app = FastAPI(title="Embedchain FastAPI App")
-embedchain_app = Pipeline()
+embedchain_app = App()
 
 
 class SourceModel(BaseModel):

embedchain/deployment/modal.com/app.py

@@ -2,7 +2,7 @@ from dotenv import load_dotenv
 from fastapi import Body, FastAPI, responses
 from modal import Image, Secret, Stub, asgi_app
 
-from embedchain import Pipeline
+from embedchain import App
 
 load_dotenv(".env")
 
@@ -18,7 +18,7 @@ stub = Stub(
 )
 
 web_app = FastAPI()
-embedchain_app = Pipeline(name="embedchain-modal-app")
+embedchain_app = App(name="embedchain-modal-app")
 
 
 @web_app.post("/add")

embedchain/deployment/render.com/app.py

@@ -1,10 +1,10 @@
 from fastapi import FastAPI, responses
 from pydantic import BaseModel
 
-from embedchain import Pipeline
+from embedchain import App
 
 app = FastAPI(title="Embedchain FastAPI App")
-embedchain_app = Pipeline()
+embedchain_app = App()
 
 
 class SourceModel(BaseModel):

embedchain/embedchain.py

@@ -1,7 +1,6 @@
 import hashlib
 import json
 import logging
-import sqlite3
 from typing import Any, Optional, Union
 
 from dotenv import load_dotenv
@@ -13,7 +12,7 @@ from embedchain.cache import (adapt, get_gptcache_session,
 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config import AddConfig, BaseLlmConfig, ChunkerConfig
 from embedchain.config.base_app_config import BaseAppConfig
-from embedchain.constants import SQLITE_PATH
+from embedchain.core.db.models import ChatHistory, DataSource
 from embedchain.data_formatter import DataFormatter
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.helpers.json_serializable import JSONSerializable
@@ -21,7 +20,6 @@ from embedchain.llm.base import BaseLlm
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.models.data_type import (DataType, DirectDataType,
                                          IndirectDataType, SpecialDataType)
-from embedchain.telemetry.posthog import AnonymousTelemetry
 from embedchain.utils.misc import detect_datatype, is_valid_json_string
 from embedchain.vectordb.base import BaseVectorDB
 
@@ -53,7 +51,6 @@ class EmbedChain(JSONSerializable):
         :type system_prompt: Optional[str], optional
         :raises ValueError: No database or embedder provided.
         """
-
         self.config = config
         self.cache_config = None
         # Llm
@@ -85,30 +82,6 @@ class EmbedChain(JSONSerializable):
         self.user_asks = []
 
         self.chunker: Optional[ChunkerConfig] = None
-        # Send anonymous telemetry
-        self._telemetry_props = {"class": self.__class__.__name__}
-        self.telemetry = AnonymousTelemetry(enabled=self.config.collect_metrics)
-        # Establish a connection to the SQLite database
-        self.connection = sqlite3.connect(SQLITE_PATH, check_same_thread=False)
-        self.cursor = self.connection.cursor()
-
-        # Create the 'data_sources' table if it doesn't exist
-        self.cursor.execute(
-            """
-            CREATE TABLE IF NOT EXISTS data_sources (
-                pipeline_id TEXT,
-                hash TEXT,
-                type TEXT,
-                value TEXT,
-                metadata TEXT,
-                is_uploaded INTEGER DEFAULT 0,
-                PRIMARY KEY (pipeline_id, hash)
-            )
-        """
-        )
-        self.connection.commit()
-        # Send anonymous telemetry
-        self.telemetry.capture(event_name="init", properties=self._telemetry_props)
 
     @property
     def collect_metrics(self):
@@ -204,17 +177,21 @@ class EmbedChain(JSONSerializable):
         if data_type in {DataType.DOCS_SITE}:
             self.is_docs_site_instance = True
 
-        # Insert the data into the 'data' table
-        self.cursor.execute(
-            """
-            INSERT OR REPLACE INTO data_sources (hash, pipeline_id, type, value, metadata)
-            VALUES (?, ?, ?, ?, ?)
-        """,
-            (source_hash, self.config.id, data_type.value, str(source), json.dumps(metadata)),
+        # Insert the data into the 'ec_data_sources' table
+        self.db_session.add(
+            DataSource(
+                hash=source_hash,
+                app_id=self.config.id,
+                type=data_type.value,
+                value=source,
+                metadata=json.dumps(metadata),
+            )
         )
-
-        # Commit the transaction
-        self.connection.commit()
+        try:
+            self.db_session.commit()
+        except Exception as e:
+            logging.error(f"Error adding data source: {e}")
+            self.db_session.rollback()
 
         if dry_run:
             data_chunks_info = {"chunks": documents, "metadata": metadatas, "count": len(documents), "type": data_type}
@@ -237,46 +214,6 @@ class EmbedChain(JSONSerializable):
 
         return source_hash
 
-    def add_local(
-        self,
-        source: Any,
-        data_type: Optional[DataType] = None,
-        metadata: Optional[dict[str, Any]] = None,
-        config: Optional[AddConfig] = None,
-        **kwargs: Optional[dict[str, Any]],
-    ):
-        """
-        Adds the data from the given URL to the vector db.
-        Loads the data, chunks it, create embedding for each chunk
-        and then stores the embedding to vector database.
-
-        Warning:
-            This method is deprecated and will be removed in future versions. Use `add` instead.
-
-        :param source: The data to embed, can be a URL, local file or raw content, depending on the data type.
-        :type source: Any
-        :param data_type: Automatically detected, but can be forced with this argument. The type of the data to add,
-        defaults to None
-        :type data_type: Optional[DataType], optional
-        :param metadata: Metadata associated with the data source., defaults to None
-        :type metadata: Optional[dict[str, Any]], optional
-        :param config: The `AddConfig` instance to use as configuration options., defaults to None
-        :type config: Optional[AddConfig], optional
-        :raises ValueError: Invalid data type
-        :return: source_hash, a md5-hash of the source, in hexadecimal representation.
-        :rtype: str
-        """
-        logging.warning(
-            "The `add_local` method is deprecated and will be removed in future versions. Please use the `add` method for both local and remote files."  # noqa: E501
-        )
-        return self.add(
-            source=source,
-            data_type=data_type,
-            metadata=metadata,
-            config=config,
-            **kwargs,
-        )
-
     def _get_existing_doc_id(self, chunker: BaseChunker, src: Any):
         """
         Get id of existing document for a given source, based on the data type
@@ -433,10 +370,27 @@ class EmbedChain(JSONSerializable):
         # Count before, to calculate a delta in the end.
         chunks_before_addition = self.db.count()
 
-        self.db.add(documents=documents, metadatas=metadatas, ids=ids, **kwargs)
-        count_new_chunks = self.db.count() - chunks_before_addition
+        # Filter out empty documents and ensure they meet the API requirements
+        valid_documents = [doc for doc in documents if doc and isinstance(doc, str)]
 
+        documents = valid_documents
+
+        # Chunk documents into batches of 2048 and handle each batch
+        # helps wigth large loads of embeddings  that hit OpenAI limits
+        document_batches = [documents[i : i + 2048] for i in range(0, len(documents), 2048)]
+        for batch in document_batches:
+            try:
+                # Add only valid batches
+                if batch:
+                    self.db.add(documents=batch, metadatas=metadatas, ids=ids, **kwargs)
+            except Exception as e:
+                print(f"Failed to add batch due to a bad request: {e}")
+                # Handle the error, e.g., by logging, retrying, or skipping
+                pass
+
+        count_new_chunks = self.db.count() - chunks_before_addition
         print(f"Successfully saved {src} ({chunker.data_type}). New chunks count: {count_new_chunks}")
+
         return list(documents), metadatas, ids, count_new_chunks
 
     @staticmethod
@@ -634,6 +588,41 @@ class EmbedChain(JSONSerializable):
         else:
             return answer
 
+    def search(self, query, num_documents=3, where=None, raw_filter=None):
+        """
+        Search for similar documents related to the query in the vector database.
+
+        Args:
+            query (str): The query to use.
+            num_documents (int, optional): Number of similar documents to fetch. Defaults to 3.
+            where (dict[str, any], optional): Filter criteria for the search.
+            raw_filter (dict[str, any], optional): Advanced raw filter criteria for the search.
+
+        Raises:
+            ValueError: If both `raw_filter` and `where` are used simultaneously.
+
+        Returns:
+            list[dict]: A list of dictionaries, each containing the 'context' and 'metadata' of a document.
+        """
+        # Send anonymous telemetry
+        self.telemetry.capture(event_name="search", properties=self._telemetry_props)
+
+        if raw_filter and where:
+            raise ValueError("You can't use both `raw_filter` and `where` together.")
+
+        filter_type = "raw_filter" if raw_filter else "where"
+        filter_criteria = raw_filter if raw_filter else where
+
+        params = {
+            "input_query": query,
+            "n_results": num_documents,
+            "citations": True,
+            "app_id": self.config.id,
+            filter_type: filter_criteria,
+        }
+
+        return [{"context": c[0], "metadata": c[1]} for c in self.db.query(**params)]
+
     def set_collection_name(self, name: str):
         """
         Set the name of the collection. A collection is an isolated space for vectors.
@@ -654,16 +643,32 @@ class EmbedChain(JSONSerializable):
         Resets the database. Deletes all embeddings irreversibly.
         `App` does not have to be reinitialized after using this method.
         """
+        try:
+            self.db_session.query(DataSource).filter_by(app_id=self.config.id).delete()
+            self.db_session.query(ChatHistory).filter_by(app_id=self.config.id).delete()
+            self.db_session.commit()
+        except Exception as e:
+            logging.error(f"Error deleting data sources: {e}")
+            self.db_session.rollback()
+            return None
         self.db.reset()
-        self.cursor.execute("DELETE FROM data_sources WHERE pipeline_id = ?", (self.config.id,))
-        self.connection.commit()
         self.delete_all_chat_history(app_id=self.config.id)
         # Send anonymous telemetry
         self.telemetry.capture(event_name="reset", properties=self._telemetry_props)
 
-    def get_history(self, num_rounds: int = 10, display_format: bool = True, session_id: Optional[str] = "default"):
+    def get_history(
+        self,
+        num_rounds: int = 10,
+        display_format: bool = True,
+        session_id: Optional[str] = "default",
+        fetch_all: bool = False,
+    ):
         history = self.llm.memory.get(
-            app_id=self.config.id, session_id=session_id, num_rounds=num_rounds, display_format=display_format
+            app_id=self.config.id,
+            session_id=session_id,
+            num_rounds=num_rounds,
+            display_format=display_format,
+            fetch_all=fetch_all,
         )
         return history
 
@@ -674,3 +679,22 @@ class EmbedChain(JSONSerializable):
     def delete_all_chat_history(self, app_id: str):
         self.llm.memory.delete(app_id=app_id)
         self.llm.update_history(app_id=app_id)
+
+    def delete(self, source_id: str):
+        """
+        Deletes the data from the database.
+        :param source_hash: The hash of the source.
+        :type source_hash: str
+        """
+        try:
+            self.db_session.query(DataSource).filter_by(hash=source_id, app_id=self.config.id).delete()
+            self.db_session.commit()
+        except Exception as e:
+            logging.error(f"Error deleting data sources: {e}")
+            self.db_session.rollback()
+            return None
+        self.db.delete(where={"hash": source_id})
+        logging.info(f"Successfully deleted {source_id}")
+        # Send anonymous telemetry
+        if self.config.collect_metrics:
+            self.telemetry.capture(event_name="delete", properties=self._telemetry_props)

embedchain/embedder/google.py

@@ -1,4 +1,4 @@
-from typing import Optional
+from typing import Optional, Union
 
 import google.generativeai as genai
 from chromadb import EmbeddingFunction, Embeddings
@@ -13,12 +13,19 @@ class GoogleAIEmbeddingFunction(EmbeddingFunction):
         super().__init__()
         self.config = config or GoogleAIEmbedderConfig()
 
-    def __call__(self, input_: str) -> Embeddings:
+    def __call__(self, input: Union[list[str], str]) -> Embeddings:
         model = self.config.model
         title = self.config.title
         task_type = self.config.task_type
-        embeddings = genai.embed_content(model=model, content=input_, task_type=task_type, title=title)
-        return embeddings["embedding"]
+        if isinstance(input, str):
+            input_ = [input]
+        else:
+            input_ = input
+        data = genai.embed_content(model=model, content=input_, task_type=task_type, title=title)
+        embeddings = data["embedding"]
+        if isinstance(input_, str):
+            embeddings = [embeddings]
+        return embeddings
 
 
 class GoogleAIEmbedder(BaseEmbedder):
@@ -27,5 +34,5 @@ class GoogleAIEmbedder(BaseEmbedder):
         embedding_fn = GoogleAIEmbeddingFunction(config=config)
         self.set_embedding_fn(embedding_fn=embedding_fn)
 
-        vector_dimension = VectorDimensions.GOOGLE_AI.value
+        vector_dimension = self.config.vector_dimension or VectorDimensions.GOOGLE_AI.value
         self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/embedder/gpt4all.py

@@ -16,5 +16,5 @@ class GPT4AllEmbedder(BaseEmbedder):
         embedding_fn = BaseEmbedder._langchain_default_concept(embeddings)
         self.set_embedding_fn(embedding_fn=embedding_fn)
 
-        vector_dimension = VectorDimensions.GPT4ALL.value
+        vector_dimension = self.config.vector_dimension or VectorDimensions.GPT4ALL.value
         self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/embedder/huggingface.py

@@ -1,6 +1,6 @@
 from typing import Optional
 
-from langchain.embeddings import HuggingFaceEmbeddings
+from langchain_community.embeddings import HuggingFaceEmbeddings
 
 from embedchain.config import BaseEmbedderConfig
 from embedchain.embedder.base import BaseEmbedder
@@ -15,5 +15,5 @@ class HuggingFaceEmbedder(BaseEmbedder):
         embedding_fn = BaseEmbedder._langchain_default_concept(embeddings)
         self.set_embedding_fn(embedding_fn=embedding_fn)
 
-        vector_dimension = VectorDimensions.HUGGING_FACE.value
+        vector_dimension = self.config.vector_dimension or VectorDimensions.HUGGING_FACE.value
         self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/embedder/mistralai.py

@@ -0,0 +1,46 @@
+import os
+from typing import Optional, Union
+
+from chromadb import EmbeddingFunction, Embeddings
+
+from embedchain.config import BaseEmbedderConfig
+from embedchain.embedder.base import BaseEmbedder
+from embedchain.models import VectorDimensions
+
+
+class MistralAIEmbeddingFunction(EmbeddingFunction):
+    def __init__(self, config: BaseEmbedderConfig) -> None:
+        super().__init__()
+        try:
+            from langchain_mistralai import MistralAIEmbeddings
+        except ModuleNotFoundError:
+            raise ModuleNotFoundError(
+                "The required dependencies for MistralAI are not installed."
+                'Please install with `pip install --upgrade "embedchain[mistralai]"`'
+            ) from None
+        self.config = config
+        api_key = self.config.api_key or os.getenv("MISTRAL_API_KEY")
+        self.client = MistralAIEmbeddings(mistral_api_key=api_key)
+        self.client.model = self.config.model
+
+    def __call__(self, input: Union[list[str], str]) -> Embeddings:
+        if isinstance(input, str):
+            input_ = [input]
+        else:
+            input_ = input
+        response = self.client.embed_documents(input_)
+        return response
+
+
+class MistralAIEmbedder(BaseEmbedder):
+    def __init__(self, config: Optional[BaseEmbedderConfig] = None):
+        super().__init__(config)
+
+        if self.config.model is None:
+            self.config.model = "mistral-embed"
+
+        embedding_fn = MistralAIEmbeddingFunction(config=self.config)
+        self.set_embedding_fn(embedding_fn=embedding_fn)
+
+        vector_dimension = self.config.vector_dimension or VectorDimensions.MISTRAL_AI.value
+        self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/embedder/openai.py

@@ -2,7 +2,7 @@ import os
 from typing import Optional
 
 from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
-from langchain.embeddings import AzureOpenAIEmbeddings
+from langchain_community.embeddings import AzureOpenAIEmbeddings
 
 from embedchain.config import BaseEmbedderConfig
 from embedchain.embedder.base import BaseEmbedder
@@ -32,4 +32,5 @@ class OpenAIEmbedder(BaseEmbedder):
                 model_name=self.config.model,
             )
         self.set_embedding_fn(embedding_fn=embedding_fn)
-        self.set_vector_dimension(vector_dimension=VectorDimensions.OPENAI.value)
+        vector_dimension = self.config.vector_dimension or VectorDimensions.OPENAI.value
+        self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/embedder/vertexai.py

@@ -1,6 +1,6 @@
 from typing import Optional
 
-from langchain.embeddings import VertexAIEmbeddings
+from langchain_community.embeddings import VertexAIEmbeddings
 
 from embedchain.config import BaseEmbedderConfig
 from embedchain.embedder.base import BaseEmbedder
@@ -15,5 +15,5 @@ class VertexAIEmbedder(BaseEmbedder):
         embedding_fn = BaseEmbedder._langchain_default_concept(embeddings)
         self.set_embedding_fn(embedding_fn=embedding_fn)
 
-        vector_dimension = VectorDimensions.VERTEX_AI.value
+        vector_dimension = self.config.vector_dimension or VectorDimensions.VERTEX_AI.value
         self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/evaluation/metrics/groundedness.py

@@ -21,7 +21,7 @@ class Groundedness(BaseMetric):
     def __init__(self, config: Optional[GroundednessConfig] = None):
         super().__init__(name=EvalMetric.GROUNDEDNESS.value)
         self.config = config or GroundednessConfig()
-        api_key = self.config.api_key or os.environ["OPENAI_API_KEY"]
+        api_key = self.config.api_key or os.getenv("OPENAI_API_KEY")
         if not api_key:
             raise ValueError("Please set the OPENAI_API_KEY environment variable or pass the `api_key` in config.")
         self.client = OpenAI(api_key=api_key)

embedchain/factory.py

@@ -21,6 +21,9 @@ class LlmFactory:
         "openai": "embedchain.llm.openai.OpenAILlm",
         "vertexai": "embedchain.llm.vertex_ai.VertexAILlm",
         "google": "embedchain.llm.google.GoogleLlm",
+        "aws_bedrock": "embedchain.llm.aws_bedrock.AWSBedrockLlm",
+        "mistralai": "embedchain.llm.mistralai.MistralAILlm",
+        "groq": "embedchain.llm.groq.GroqLlm",
     }
     provider_to_config_class = {
         "embedchain": "embedchain.config.llm.base.BaseLlmConfig",
@@ -50,12 +53,14 @@ class EmbedderFactory:
         "openai": "embedchain.embedder.openai.OpenAIEmbedder",
         "vertexai": "embedchain.embedder.vertexai.VertexAIEmbedder",
         "google": "embedchain.embedder.google.GoogleAIEmbedder",
+        "mistralai": "embedchain.embedder.mistralai.MistralAIEmbedder",
     }
     provider_to_config_class = {
         "azure_openai": "embedchain.config.embedder.base.BaseEmbedderConfig",
         "openai": "embedchain.config.embedder.base.BaseEmbedderConfig",
         "gpt4all": "embedchain.config.embedder.base.BaseEmbedderConfig",
         "google": "embedchain.config.embedder.google.GoogleAIEmbedderConfig",
+        "huggingface": "embedchain.config.embedder.base.BaseEmbedderConfig",
     }
 
     @classmethod

embedchain/llm/anthropic.py

@@ -19,7 +19,7 @@ class AnthropicLlm(BaseLlm):
 
     @staticmethod
     def _get_answer(prompt: str, config: BaseLlmConfig) -> str:
-        from langchain.chat_models import ChatAnthropic
+        from langchain_community.chat_models import ChatAnthropic
 
         chat = ChatAnthropic(
             anthropic_api_key=os.environ["ANTHROPIC_API_KEY"], temperature=config.temperature, model=config.model

embedchain/llm/aws_bedrock.py

@@ -0,0 +1,49 @@
+import os
+from typing import Optional
+
+from langchain_community.llms import Bedrock
+
+from embedchain.config import BaseLlmConfig
+from embedchain.helpers.json_serializable import register_deserializable
+from embedchain.llm.base import BaseLlm
+
+
+@register_deserializable
+class AWSBedrockLlm(BaseLlm):
+    def __init__(self, config: Optional[BaseLlmConfig] = None):
+        super().__init__(config)
+
+    def get_llm_model_answer(self, prompt) -> str:
+        response = self._get_answer(prompt, self.config)
+        return response
+
+    def _get_answer(self, prompt: str, config: BaseLlmConfig) -> str:
+        try:
+            import boto3
+        except ModuleNotFoundError:
+            raise ModuleNotFoundError(
+                "The required dependencies for AWSBedrock are not installed."
+                'Please install with `pip install --upgrade "embedchain[aws-bedrock]"`'
+            ) from None
+
+        self.boto_client = boto3.client("bedrock-runtime", "us-west-2" or os.environ.get("AWS_REGION"))
+
+        kwargs = {
+            "model_id": config.model or "amazon.titan-text-express-v1",
+            "client": self.boto_client,
+            "model_kwargs": config.model_kwargs
+            or {
+                "temperature": config.temperature,
+            },
+        }
+
+        if config.stream:
+            from langchain.callbacks.streaming_stdout import \
+                StreamingStdOutCallbackHandler
+
+            callbacks = [StreamingStdOutCallbackHandler()]
+            llm = Bedrock(**kwargs, streaming=config.stream, callbacks=callbacks)
+        else:
+            llm = Bedrock(**kwargs)
+
+        return llm(prompt)

embedchain/llm/azure_openai.py

@@ -16,7 +16,7 @@ class AzureOpenAILlm(BaseLlm):
 
     @staticmethod
     def _get_answer(prompt: str, config: BaseLlmConfig) -> str:
-        from langchain.chat_models import AzureChatOpenAI
+        from langchain_community.chat_models import AzureChatOpenAI
 
         if not config.deployment_name:
             raise ValueError("Deployment name must be provided for Azure OpenAI")

embedchain/llm/base.py

@@ -65,6 +65,14 @@ class BaseLlm(JSONSerializable):
         self.memory.add(app_id=app_id, chat_message=chat_message, session_id=session_id)
         self.update_history(app_id=app_id, session_id=session_id)
 
+    def _format_history(self) -> str:
+        """Format history to be used in prompt
+
+        :return: Formatted history
+        :rtype: str
+        """
+        return "\n".join(self.history)
+
     def generate_prompt(self, input_query: str, contexts: list[str], **kwargs: dict[str, Any]) -> str:
         """
         Generates a prompt based on the given query and context, ready to be
@@ -84,10 +92,8 @@ class BaseLlm(JSONSerializable):
 
         prompt_contains_history = self.config._validate_prompt_history(self.config.prompt)
         if prompt_contains_history:
-            # Prompt contains history
-            # If there is no history yet, we insert `- no history -`
             prompt = self.config.prompt.substitute(
-                context=context_string, query=input_query, history=self.history or "- no history -"
+                context=context_string, query=input_query, history=self._format_history() or "No history"
             )
         elif self.history and not prompt_contains_history:
             # History is present, but not included in the prompt.
@@ -98,7 +104,7 @@ class BaseLlm(JSONSerializable):
             ):
                 # swap in the template with history
                 prompt = DEFAULT_PROMPT_WITH_HISTORY_TEMPLATE.substitute(
-                    context=context_string, query=input_query, history=self.history
+                    context=context_string, query=input_query, history=self._format_history()
                 )
             else:
                 # If we can't swap in the default, we still proceed but tell users that the history is ignored.

embedchain/llm/cohere.py

@@ -2,7 +2,7 @@ import importlib
 import os
 from typing import Optional
 
-from langchain.llms.cohere import Cohere
+from langchain_community.llms.cohere import Cohere
 
 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable

embedchain/llm/gpt4all.py

@@ -26,7 +26,8 @@ class GPT4ALLLlm(BaseLlm):
     @staticmethod
     def _get_instance(model):
         try:
-            from langchain.llms.gpt4all import GPT4All as LangchainGPT4All
+            from langchain_community.llms.gpt4all import \
+                GPT4All as LangchainGPT4All
         except ModuleNotFoundError:
             raise ModuleNotFoundError(
                 "The GPT4All python package is not installed. Please install it with `pip install --upgrade embedchain[opensource]`"  # noqa E501

embedchain/llm/groq.py

@@ -0,0 +1,43 @@
+import os
+from typing import Optional
+
+from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
+from langchain.schema import HumanMessage, SystemMessage
+
+try:
+    from langchain_groq import ChatGroq
+except ImportError:
+    raise ImportError("Groq requires extra dependencies. Install with `pip install langchain-groq`") from None
+
+
+from embedchain.config import BaseLlmConfig
+from embedchain.helpers.json_serializable import register_deserializable
+from embedchain.llm.base import BaseLlm
+
+
+@register_deserializable
+class GroqLlm(BaseLlm):
+    def __init__(self, config: Optional[BaseLlmConfig] = None):
+        super().__init__(config=config)
+
+    def get_llm_model_answer(self, prompt) -> str:
+        response = self._get_answer(prompt, self.config)
+        return response
+
+    def _get_answer(self, prompt: str, config: BaseLlmConfig) -> str:
+        messages = []
+        if config.system_prompt:
+            messages.append(SystemMessage(content=config.system_prompt))
+        messages.append(HumanMessage(content=prompt))
+        api_key = config.api_key or os.environ["GROQ_API_KEY"]
+        kwargs = {
+            "model_name": config.model or "mixtral-8x7b-32768",
+            "temperature": config.temperature,
+            "groq_api_key": api_key,
+        }
+        if config.stream:
+            callbacks = config.callbacks if config.callbacks else [StreamingStdOutCallbackHandler()]
+            chat = ChatGroq(**kwargs, streaming=config.stream, callbacks=callbacks, api_key=api_key)
+        else:
+            chat = ChatGroq(**kwargs)
+        return chat.invoke(messages).content

embedchain/llm/huggingface.py

@@ -3,8 +3,8 @@ import logging
 import os
 from typing import Optional
 
-from langchain.llms.huggingface_endpoint import HuggingFaceEndpoint
-from langchain.llms.huggingface_hub import HuggingFaceHub
+from langchain_community.llms.huggingface_endpoint import HuggingFaceEndpoint
+from langchain_community.llms.huggingface_hub import HuggingFaceHub
 
 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable

embedchain/llm/jina.py

@@ -1,8 +1,8 @@
 import os
 from typing import Optional
 
-from langchain.chat_models import JinaChat
 from langchain.schema import HumanMessage, SystemMessage
+from langchain_community.chat_models import JinaChat
 
 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable

embedchain/llm/llama2.py

@@ -2,7 +2,7 @@ import importlib
 import os
 from typing import Optional
 
-from langchain.llms.replicate import Replicate
+from langchain_community.llms.replicate import Replicate
 
 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable

embedchain/llm/mistralai.py

@@ -0,0 +1,52 @@
+import os
+from typing import Optional
+
+from embedchain.config import BaseLlmConfig
+from embedchain.helpers.json_serializable import register_deserializable
+from embedchain.llm.base import BaseLlm
+
+
+@register_deserializable
+class MistralAILlm(BaseLlm):
+    def __init__(self, config: Optional[BaseLlmConfig] = None):
+        super().__init__(config)
+        if not self.config.api_key and "MISTRAL_API_KEY" not in os.environ:
+            raise ValueError("Please set the MISTRAL_API_KEY environment variable or pass it in the config.")
+
+    def get_llm_model_answer(self, prompt):
+        return MistralAILlm._get_answer(prompt=prompt, config=self.config)
+
+    @staticmethod
+    def _get_answer(prompt: str, config: BaseLlmConfig):
+        try:
+            from langchain_core.messages import HumanMessage, SystemMessage
+            from langchain_mistralai.chat_models import ChatMistralAI
+        except ModuleNotFoundError:
+            raise ModuleNotFoundError(
+                "The required dependencies for MistralAI are not installed."
+                'Please install with `pip install --upgrade "embedchain[mistralai]"`'
+            ) from None
+
+        api_key = config.api_key or os.getenv("MISTRAL_API_KEY")
+        client = ChatMistralAI(mistral_api_key=api_key)
+        messages = []
+        if config.system_prompt:
+            messages.append(SystemMessage(content=config.system_prompt))
+        messages.append(HumanMessage(content=prompt))
+        kwargs = {
+            "model": config.model or "mistral-tiny",
+            "temperature": config.temperature,
+            "max_tokens": config.max_tokens,
+            "top_p": config.top_p,
+        }
+
+        # TODO: Add support for streaming
+        if config.stream:
+            answer = ""
+            for chunk in client.stream(**kwargs, input=messages):
+                answer += chunk.content
+            return answer
+        else:
+            response = client.invoke(**kwargs, input=messages)
+            answer = response.content
+            return answer

embedchain/llm/ollama.py

@@ -4,7 +4,7 @@ from typing import Optional, Union
 from langchain.callbacks.manager import CallbackManager
 from langchain.callbacks.stdout import StdOutCallbackHandler
 from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
-from langchain.llms.ollama import Ollama
+from langchain_community.llms.ollama import Ollama
 
 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable

embedchain/llm/openai.py

@@ -1,9 +1,12 @@
 import json
 import os
-from typing import Any, Optional
+from typing import Any, Callable, Dict, Optional, Type, Union
 
-from langchain.chat_models import ChatOpenAI
-from langchain.schema import AIMessage, HumanMessage, SystemMessage
+from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
+from langchain.schema import BaseMessage, HumanMessage, SystemMessage
+from langchain_core.tools import BaseTool
+from langchain_openai import ChatOpenAI
+from pydantic import BaseModel
 
 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -12,8 +15,12 @@ from embedchain.llm.base import BaseLlm
 
 @register_deserializable
 class OpenAILlm(BaseLlm):
-    def __init__(self, config: Optional[BaseLlmConfig] = None, functions: Optional[dict[str, Any]] = None):
-        self.functions = functions
+    def __init__(
+        self,
+        config: Optional[BaseLlmConfig] = None,
+        tools: Optional[Union[Dict[str, Any], Type[BaseModel], Callable[..., Any], BaseTool]] = None,
+    ):
+        self.tools = tools
         super().__init__(config=config)
 
     def get_llm_model_answer(self, prompt) -> str:
@@ -35,26 +42,27 @@ class OpenAILlm(BaseLlm):
         if config.top_p:
             kwargs["model_kwargs"]["top_p"] = config.top_p
         if config.stream:
-            from langchain.callbacks.streaming_stdout import \
-                StreamingStdOutCallbackHandler
-
             callbacks = config.callbacks if config.callbacks else [StreamingStdOutCallbackHandler()]
             chat = ChatOpenAI(**kwargs, streaming=config.stream, callbacks=callbacks, api_key=api_key)
         else:
             chat = ChatOpenAI(**kwargs, api_key=api_key)
+        if self.tools:
+            return self._query_function_call(chat, self.tools, messages)
 
-        if self.functions is not None:
-            from langchain.chains.openai_functions import \
-                create_openai_fn_runnable
-            from langchain.prompts import ChatPromptTemplate
+        return chat.invoke(messages).content
 
-            structured_prompt = ChatPromptTemplate.from_messages(messages)
-            runnable = create_openai_fn_runnable(functions=self.functions, prompt=structured_prompt, llm=chat)
-            fn_res = runnable.invoke(
-                {
-                    "input": prompt,
-                }
-            )
-            messages.append(AIMessage(content=json.dumps(fn_res)))
+    def _query_function_call(
+        self,
+        chat: ChatOpenAI,
+        tools: Optional[Union[Dict[str, Any], Type[BaseModel], Callable[..., Any], BaseTool]],
+        messages: list[BaseMessage],
+    ) -> str:
+        from langchain.output_parsers.openai_tools import JsonOutputToolsParser
+        from langchain_core.utils.function_calling import convert_to_openai_tool
 
-        return chat(messages).content
+        openai_tools = [convert_to_openai_tool(tools)]
+        chat = chat.bind(tools=openai_tools).pipe(JsonOutputToolsParser())
+        try:
+            return json.dumps(chat.invoke(messages)[0])
+        except IndexError:
+            return "Input could not be mapped to the function!"

embedchain/llm/together.py

@@ -2,7 +2,7 @@ import importlib
 import os
 from typing import Optional
 
-from langchain.llms import Together
+from langchain_community.llms import Together
 
 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable

embedchain/llm/vertex_ai.py

@@ -2,6 +2,9 @@ import importlib
 import logging
 from typing import Optional
 
+from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
+from langchain_google_vertexai import ChatVertexAI
+
 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm
@@ -24,13 +27,17 @@ class VertexAILlm(BaseLlm):
 
     @staticmethod
     def _get_answer(prompt: str, config: BaseLlmConfig) -> str:
-        from langchain.chat_models import ChatVertexAI
-
-        chat = ChatVertexAI(temperature=config.temperature, model=config.model)
-
         if config.top_p and config.top_p != 1:
             logging.warning("Config option `top_p` is not supported by this model.")
 
         messages = BaseLlm._get_messages(prompt, system_prompt=config.system_prompt)
 
-        return chat(messages).content
+        if config.stream:
+            callbacks = config.callbacks if config.callbacks else [StreamingStdOutCallbackHandler()]
+            llm = ChatVertexAI(
+                temperature=config.temperature, model=config.model, callbacks=callbacks, streaming=config.stream
+            )
+        else:
+            llm = ChatVertexAI(temperature=config.temperature, model=config.model)
+
+        return llm.invoke(messages).content

embedchain/loaders/discord.py

@@ -131,7 +131,7 @@ class DiscordLoader(BaseLoader):
         client = DiscordClient(intents=intents)
         client.run(self.token)
 
-        meta_data = {
+        metadata = {
             "url": channel_id,
         }
 
@@ -144,7 +144,7 @@ class DiscordLoader(BaseLoader):
             "data": [
                 {
                     "content": messages,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                 }
             ],
         }

embedchain/loaders/discourse.py

@@ -14,19 +14,19 @@ class DiscourseLoader(BaseLoader):
         super().__init__()
         if not config:
             raise ValueError(
-                "DiscourseLoader requires a config. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+                "DiscourseLoader requires a config. Check the documentation for the correct format - `https://docs.embedchain.ai/components/data-sources/discourse`"  # noqa: E501
             )
 
         self.domain = config.get("domain")
         if not self.domain:
             raise ValueError(
-                "DiscourseLoader requires a domain. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+                "DiscourseLoader requires a domain. Check the documentation for the correct format - `https://docs.embedchain.ai/components/data-sources/discourse`"  # noqa: E501
             )
 
     def _check_query(self, query):
         if not query or not isinstance(query, str):
             raise ValueError(
-                "DiscourseLoader requires a query. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+                "DiscourseLoader requires a query. Check the documentation for the correct format - `https://docs.embedchain.ai/components/data-sources/discourse`"  # noqa: E501
             )
 
     def _load_post(self, post_id):
@@ -39,7 +39,7 @@ class DiscourseLoader(BaseLoader):
             return
         response_data = response.json()
         post_contents = clean_string(response_data.get("raw"))
-        meta_data = {
+        metadata = {
             "url": post_url,
             "created_at": response_data.get("created_at", ""),
             "username": response_data.get("username", ""),
@@ -48,7 +48,7 @@ class DiscourseLoader(BaseLoader):
         }
         data = {
             "content": post_contents,
-            "meta_data": meta_data,
+            "meta_data": metadata,
         }
         return data
 

embedchain/loaders/docx_file.py

@@ -1,7 +1,7 @@
 import hashlib
 
 try:
-    from langchain.document_loaders import Docx2txtLoader
+    from langchain_community.document_loaders import Docx2txtLoader
 except ImportError:
     raise ImportError(
         'Docx file requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'
@@ -18,9 +18,9 @@ class DocxFileLoader(BaseLoader):
         output = []
         data = loader.load()
         content = data[0].page_content
-        meta_data = data[0].metadata
-        meta_data["url"] = "local"
-        output.append({"content": content, "meta_data": meta_data})
+        metadata = data[0].metadata
+        metadata["url"] = "local"
+        output.append({"content": content, "meta_data": metadata})
         doc_id = hashlib.sha256((content + url).encode()).hexdigest()
         return {
             "doc_id": doc_id,

embedchain/loaders/google_drive.py

@@ -8,8 +8,8 @@ except ImportError:
         "Google Drive requires extra dependencies. Install with `pip install embedchain[googledrive]`"
     ) from None
 
-from langchain.document_loaders import GoogleDriveLoader as Loader
-from langchain.document_loaders import UnstructuredFileIOLoader
+from langchain_community.document_loaders import GoogleDriveLoader as Loader
+from langchain_community.document_loaders import UnstructuredFileIOLoader
 
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader

embedchain/loaders/json.py

@@ -36,7 +36,9 @@ class JSONReader:
         return ["\n".join(useful_lines)]
 
 
-VALID_URL_PATTERN = "^https:\/\/[0-9A-Za-z]+(\.[0-9A-Za-z]+)*\/[0-9A-Za-z_\/]*\.json$"
+VALID_URL_PATTERN = (
+    "^https?://(?:www\.)?(?:\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|[a-zA-Z0-9.-]+)(?::\d+)?/(?:[^/\s]+/)*[^/\s]+\.json$"
+)
 
 
 class JSONLoader(BaseLoader):

embedchain/loaders/local_qna_pair.py

@@ -11,14 +11,14 @@ class LocalQnaPairLoader(BaseLoader):
         question, answer = content
         content = f"Q: {question}\nA: {answer}"
         url = "local"
-        meta_data = {"url": url, "question": question}
+        metadata = {"url": url, "question": question}
         doc_id = hashlib.sha256((content + url).encode()).hexdigest()
         return {
             "doc_id": doc_id,
             "data": [
                 {
                     "content": content,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                 }
             ],
         }

embedchain/loaders/local_text.py

@@ -9,7 +9,7 @@ class LocalTextLoader(BaseLoader):
     def load_data(self, content):
         """Load data from a local text file."""
         url = "local"
-        meta_data = {
+        metadata = {
             "url": url,
         }
         doc_id = hashlib.sha256((content + url).encode()).hexdigest()
@@ -18,7 +18,7 @@ class LocalTextLoader(BaseLoader):
             "data": [
                 {
                     "content": content,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                 }
             ],
         }

embedchain/loaders/mdx.py

@@ -10,7 +10,7 @@ class MdxLoader(BaseLoader):
         """Load data from a mdx file."""
         with open(url, "r", encoding="utf-8") as infile:
             content = infile.read()
-        meta_data = {
+        metadata = {
             "url": url,
         }
         doc_id = hashlib.sha256((content + url).encode()).hexdigest()
@@ -19,7 +19,7 @@ class MdxLoader(BaseLoader):
             "data": [
                 {
                     "content": content,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                 }
             ],
         }

embedchain/loaders/openapi.py

@@ -35,8 +35,8 @@ class OpenAPILoader(BaseLoader):
             yaml_data = yaml.load(file, Loader=yaml.SafeLoader)
             for i, (key, value) in enumerate(yaml_data.items()):
                 string_data = f"{key}: {value}"
-                meta_data = {"url": file_path, "row": i + 1}
-                data.append({"content": string_data, "meta_data": meta_data})
+                metadata = {"url": file_path, "row": i + 1}
+                data.append({"content": string_data, "meta_data": metadata})
                 data_content.append(string_data)
         doc_id = hashlib.sha256((content + ", ".join(data_content)).encode()).hexdigest()
         return {"doc_id": doc_id, "data": data}

embedchain/loaders/pdf_file.py

@@ -1,7 +1,7 @@
 import hashlib
 
 try:
-    from langchain.document_loaders import PyPDFLoader
+    from langchain_community.document_loaders import PyPDFLoader
 except ImportError:
     raise ImportError(
         'PDF File requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'
@@ -15,7 +15,10 @@ from embedchain.utils.misc import clean_string
 class PdfFileLoader(BaseLoader):
     def load_data(self, url):
         """Load data from a PDF file."""
-        loader = PyPDFLoader(url)
+        headers = {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36",  # noqa:E501
+        }
+        loader = PyPDFLoader(url, headers=headers)
         data = []
         all_content = []
         pages = loader.load_and_split()
@@ -24,12 +27,12 @@ class PdfFileLoader(BaseLoader):
         for page in pages:
             content = page.page_content
             content = clean_string(content)
-            meta_data = page.metadata
-            meta_data["url"] = url
+            metadata = page.metadata
+            metadata["url"] = url
             data.append(
                 {
                     "content": content,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                 }
             )
             all_content.append(content)

embedchain/loaders/rss_feed.py

@@ -28,7 +28,7 @@ class RSSFeedLoader(BaseLoader):
     @staticmethod
     def get_rss_content(url: str):
         try:
-            from langchain.document_loaders import \
+            from langchain_community.document_loaders import \
                 RSSFeedLoader as LangchainRSSFeedLoader
         except ImportError:
             raise ImportError(
@@ -41,12 +41,12 @@ class RSSFeedLoader(BaseLoader):
         data = loader.load()
 
         for entry in data:
-            meta_data = RSSFeedLoader.serialize_metadata(entry.metadata)
-            meta_data.update({"url": url})
+            metadata = RSSFeedLoader.serialize_metadata(entry.metadata)
+            metadata.update({"url": url})
             output.append(
                 {
                     "content": entry.page_content,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                 }
             )
 

embedchain/loaders/sitemap.py

@@ -31,10 +31,13 @@ class SitemapLoader(BaseLoader):
     def load_data(self, sitemap_source):
         output = []
         web_page_loader = WebPageLoader()
+        headers = {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36",  # noqa:E501
+        }
 
         if urlparse(sitemap_source).scheme in ("http", "https"):
             try:
-                response = requests.get(sitemap_source)
+                response = requests.get(sitemap_source, headers=headers)
                 response.raise_for_status()
                 soup = BeautifulSoup(response.text, "xml")
             except requests.RequestException as e:

embedchain/loaders/slack.py

@@ -88,16 +88,16 @@ class SlackLoader(BaseLoader):
                 content = clean_string(text)
 
                 message_meta_data_keys = ["iid", "team", "ts", "type", "user", "username"]
-                meta_data = {}
+                metadata = {}
                 for key in message.keys():
                     if key in message_meta_data_keys:
-                        meta_data[key] = message.get(key)
-                meta_data.update({"url": url})
+                        metadata[key] = message.get(key)
+                metadata.update({"url": url})
 
                 data.append(
                     {
                         "content": content,
-                        "meta_data": meta_data,
+                        "meta_data": metadata,
                     }
                 )
                 data_content.append(content)