Update version to 0.1.18 (#970 )

Update embedding_fn signature to newest chroma db's (#969 )
[Feature] Improve github and youtube channel loader (#966 )
2023-11-21 10:01:09 -08:00 · 2023-11-21 09:42:11 -08:00 · 2023-11-17 18:25:14 -08:00 · 2023-11-17 18:19:08 -08:00 · 2023-11-16 22:06:36 -08:00 · 2023-11-16 16:01:43 -08:00
117 changed files with 3349 additions and 651 deletions
@@ -1,74 +1,64 @@
-# embedchain
+<p align="center">
+  <img src="docs/logo/dark.svg" width="400px" alt="Embedchain Logo">
+</p>

-<a href="https://runacap.com/ross-index/q3-2023/" target="_blank" rel="noopener"><img style="width: 260px; height: 56px" src="https://runacap.com/wp-content/uploads/2023/10/ROSS_badge_black_Q3_2023.svg" alt="ROSS Index - Fastest Growing Open-Source Startups in Q3 2023 | Runa Capital" width="260" height="56"/></a>
+<p align="center">
+  <a href="https://runacap.com/ross-index/q3-2023/" target="_blank" rel="noopener"><img style="width: 260px; height: 56px" src="https://runacap.com/wp-content/uploads/2023/10/ROSS_badge_black_Q3_2023.svg" alt="ROSS Index - Fastest Growing Open-Source Startups in Q3 2023 | Runa Capital" width="260" height="56"/></a>
+</p>

-[![PyPI](https://img.shields.io/pypi/v/embedchain)](https://pypi.org/project/embedchain/)
-[![Slack](https://img.shields.io/badge/slack-embedchain-brightgreen.svg?logo=slack)](https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw)
-[![Discord](https://dcbadge.vercel.app/api/server/6PzXDgEjG5?style=flat)](https://discord.gg/CUU9FPhRNt)
-[![Twitter](https://img.shields.io/twitter/follow/embedchain)](https://twitter.com/embedchain)
-[![Substack](https://img.shields.io/badge/Substack-%23006f5c.svg?logo=substack)](https://embedchain.substack.com/)
-[![Open in Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/138lMWhENGeEu7Q1-6lNbNTHGLZXBBz_B?usp=sharing)
-[![codecov](https://codecov.io/gh/embedchain/embedchain/graph/badge.svg?token=EMRRHZXW1Q)](https://codecov.io/gh/embedchain/embedchain)
+<p align="center">
+  <a href="https://pypi.org/project/embedchain/">
+    <img src="https://img.shields.io/pypi/v/embedchain" alt="PyPI">
+  </a>
+  <a href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw">
+    <img src="https://img.shields.io/badge/slack-embedchain-brightgreen.svg?logo=slack" alt="Slack">
+  </a>
+  <a href="https://discord.gg/CUU9FPhRNt">
+    <img src="https://dcbadge.vercel.app/api/server/6PzXDgEjG5?style=flat" alt="Discord">
+  </a>
+  <a href="https://twitter.com/embedchain">
+    <img src="https://img.shields.io/twitter/follow/embedchain" alt="Twitter">
+  </a>
+  <a href="https://embedchain.substack.com/">
+    <img src="https://img.shields.io/badge/Substack-%23006f5c.svg?logo=substack" alt="Substack">
+  </a>
+  <a href="https://colab.research.google.com/drive/138lMWhENGeEu7Q1-6lNbNTHGLZXBBz_B?usp=sharing">
+    <img src="https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667" alt="Open in Colab">
+  </a>
+  <a href="https://codecov.io/gh/embedchain/embedchain">
+    <img src="https://codecov.io/gh/embedchain/embedchain/graph/badge.svg?token=EMRRHZXW1Q" alt="codecov">
+  </a>
+</p>

-Embedchain is a Data Platform for LLMs - load, index, retrieve, and sync any unstructured data. Using embedchain, you can easily create LLM powered apps over any data. If you want a javascript version, check out [embedchain-js](https://github.com/embedchain/embedchain/tree/main/embedchain-js)
+<hr />

-## Community
-
-* Join embedchain community on slack by accepting [this invite](https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw)
-
-## 🤝 Schedule a 1-on-1 Session
-
-Book a [1-on-1 Session](https://cal.com/taranjeetio/ec) with Taranjeet, the founder, to discuss any issues, provide feedback, or explore how we can improve Embedchain for you.
+## What is Embedchain?
+Embedchain is a Data Platform for Large Language Models (LLMs). Seamlessly load, index, retrieve, and sync unstructured data to build dynamic, LLM-powered applications. Check out [embedchain-js](https://github.com/embedchain/embedchain/tree/main/embedchain-js) for a JavaScript implementation.

 ## 🔧 Quick install

+### Python API
 ```bash
 pip install --upgrade embedchain
 ```

-To run Embedchain as a REST API server run the following command:
+### REST API
+You can also run Embedchain as a REST API server using the following command:

 ```bash
-docker run -d --name embedchain -p 8080:8080 embedchain/rest-api:latest
+docker run --name embedchain -p 8080:8080 embedchain/rest-api:latest
 ```

-Navigate to http://0.0.0.0:8080/docs to interact with the API.
+Then, navigate to http://0.0.0.0:8080/docs to interact with the API.

-## 🔍 Demo
+## 🔍 Usage and Demo

-Try out embedchain in your browser:
+<!-- Demo GIF or Image -->
+<p align="center">
+  <img src="docs/images/cover.gif" width="900px" alt="Embedchain Demo">
+</p>

-[![Open in Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/17ON1LPonnXAtLaZEebnOktstB_1cJJmh?usp=sharing)
-
-## 📖 Documentation
-
-The documentation for embedchain can be found at [docs.embedchain.ai](https://docs.embedchain.ai).
-
-## 💻 Usage
-
-Embedchain empowers you to create ChatGPT like apps, on your own dynamic dataset.
-
-### Data types supported
-
-* Youtube video
-* PDF file
-* CSV file
-* Web page
-* MDX file
-* XML file
-* Sitemap
-* Doc file
-* Notion
-* JSON file
-* OpenAPI specs
-* Code docs website
-* Unstructured file loader and many more
-
-You can find the full list of data types on [our documentation](https://docs.embedchain.ai/data-sources/).
-
-### Queries
-
-For example, you can use Embedchain to create an Elon Musk bot using the following code:
+For example, you can create an Elon Musk bot using the following code:

 ```python
 import os
@@ -99,35 +89,27 @@ app.deploy()
 # ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
 ```

-## Examples
+You can also try it in your browser with Google Colab:

-| LLM          | Google Colab  | Replit   |
-|--------------|---------------|----------|
-| OpenAI       | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/openai.ipynb)           | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/openai#main.py)      |
-| Anthropic    | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/anthropic.ipynb)        | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/anthropic#main.py)   |
-| Azure OpenAI | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/azure-openai.ipynb)     | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/azureopenai#main.py) |
-| VertexAI     | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/vertex_ai.ipynb)        | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/vertexai#main.py)    |
-| Cohere       | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/cohere.ipynb)           | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/cohere#main.py)      |
-| Hugging Face | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/hugging_face_hub.ipynb) | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/huggingface#main.py) |
-| JinaChat     | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/jina.ipynb)             | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/jina#main.py)        |
-| GPT4All      | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/gpt4all.ipynb)          | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/gpt4all#main.py)     |
-| Llama2       | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/llama2.ipynb)           | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/llama2#main.py)      |
+[![Open in Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/17ON1LPonnXAtLaZEebnOktstB_1cJJmh?usp=sharing)

-| Embedding model     | Google Colab                                                                                                                                                                            | Replit                                                                                                                        |
-| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
-| OpenAI       | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/openai.ipynb)           | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/openai#main.py)      |
-| VertexAI     | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/vertex_ai.ipynb)        | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/vertexai#main.py)    |
-| GPT4All      | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/gpt4all.ipynb)          | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/gpt4all#main.py)     |
-| Hugging Face | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/hugging_face_hub.ipynb) | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/huggingface#main.py) |
+## 📖 Documentation
+Comprehensive guides and API documentation are available to help you get the most out of Embedchain:

-| Vector DB     | Google Colab                                                                                                                                                                         | Replit                                                                                                                          |
-| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
-| ChromaDB      | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/chromadb.ipynb)      | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/chromadb#main.py)      |
-| Elasticsearch | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/elasticsearch.ipynb) | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/elasticsearchdb#main.py) |
-| Opensearch    | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/opensearch.ipynb)    | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/opensearchdb#main.py)    |
-| Pinecone      | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/pinecone.ipynb)      | [![Try with Replit Badge](https://replit.com/badge?caption=Try%20with%20Replit&variant=small)](https://replit.com/@taranjeetio/pineconedb#main.py)      |
+- [Getting Started](https://docs.embedchain.ai/get-started/quickstart)
+- [Introduction](https://docs.embedchain.ai/get-started/introduction#what-is-embedchain)
+- [Examples](https://docs.embedchain.ai/get-started/examples)
+- [Supported data types](https://docs.embedchain.ai/data-sources/)

-## 🤝 Contributing
+## 🔗 Join the Community
+
+Connect with fellow developers and users by joining our [Slack Workspace](https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw). Dive into discussions, ask questions, and share your experiences.
+
+## 🤝 Schedule a 1-on-1 Session
+
+Book a [1-on-1 Session](https://cal.com/taranjeetio/ec) with Taranjeet, the founder, to discuss any issues, provide feedback, or explore how we can improve Embedchain for you.
+
+## 🌐 Contributing

 Contributions are welcome! Please check out the issues on the repository, and feel free to open a pull request.
 For more information, please see the [contributing guidelines](CONTRIBUTING.md).
@@ -0,0 +1,4 @@
+chunker:
+  chunk_size: 100
+  chunk_overlap: 20
+  length_function: 'len'
@@ -2,6 +2,11 @@ app:
  config:
    id: 'full-stack-app'

+chunker:
+  chunk_size: 100
+  chunk_overlap: 20
+  length_function: 'len'
+
 llm:
  provider: openai
  config:
@@ -0,0 +1,8 @@
+llm:
+  provider: openai
+  config:
+    model: 'gpt-4'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
@@ -1,7 +1,7 @@
 llm:
  provider: gpt4all
  config:
-    model: 'orca-mini-3b.ggmlv3.q4_0.bin'
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -1,7 +1,7 @@
 app:
  config:
    id: 'my-app'
-    log_level: 'WARN'
+    log_level: 'WARNING'
    collect_metrics: true
    collection_name: 'my-app'

@@ -30,4 +30,4 @@ embedder:
  provider: openai
  config:
    model: 'text-embedding-ada-002'
-    deployment_name: null
+    deployment_name: 'my-app'
@@ -7,7 +7,7 @@ app:
 llm:
  provider: gpt4all
  config:
-    model: 'orca-mini-3b.ggmlv3.q4_0.bin'
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -13,7 +13,7 @@ vectordb:
 llm:
  provider: gpt4all
  config:
-    model: 'orca-mini-3b.ggmlv3.q4_0.bin'
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -1,5 +1,4 @@
-<Tip>
-If you can't find the specific data source, please feel free to request through one of the following channels and help us prioritize.
+<p>If you can't find the specific data source, please feel free to request through one of the following channels and help us prioritize.</p>

 <CardGroup cols={2}>
  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
@@ -15,4 +14,3 @@ If you can't find the specific data source, please feel free to request through
  Schedule a call with Embedchain founder
  </Card>
 </CardGroup>
-</Tip>
@@ -1,5 +1,4 @@
-<Tip>
-If you can't find the specific LLM you need, no need to fret. We're continuously expanding our support for additional LLMs, and you can help us prioritize by opening an issue on our GitHub or simply reaching out to us on our Slack or Discord community.
+<p>If you can't find the specific LLM you need, no need to fret. We're continuously expanding our support for additional LLMs, and you can help us prioritize by opening an issue on our GitHub or simply reaching out to us on our Slack or Discord community.</p>

 <CardGroup cols={2}>
  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
@@ -15,4 +14,3 @@ If you can't find the specific LLM you need, no need to fret. We're continuously
  Schedule a call with Embedchain founder
  </Card>
 </CardGroup>
-</Tip>
@@ -1,5 +1,6 @@
-<Tip>
-If you can't find the specific vector database, please feel free to request through one of the following channels and help us prioritize.
+
+
+<p>If you can't find the specific vector database, please feel free to request through one of the following channels and help us prioritize.</p>

 <CardGroup cols={2}>
  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
@@ -15,4 +16,3 @@ If you can't find the specific vector database, please feel free to request thro
  Schedule a call with Embedchain founder
  </Card>
 </CardGroup>
-</Tip>
@@ -11,6 +11,11 @@ app:
  config:
    id: 'full-stack-app'

+chunker:
+  chunk_size: 100
+  chunk_overlap: 20
+  length_function: 'len'
+
 llm:
  provider: openai
  config:
@@ -49,7 +54,11 @@ Alright, let's dive into what each key means in the yaml config above:
 1. `app` Section:
    - `config`:
        - `id` (String): The ID or name of your full-stack application.
-2. `llm` Section:
+2. `chunker` Section:
+    - `chunk_size` (Integer): The size of each chunk of text that is sent to the language model.
+    - `chunk_overlap` (Integer): The amount of overlap between each chunk of text.
+    - `length_function` (String): The function used to calculate the length of each chunk of text. In this case, it's set to 'len'. You can also use any function import directly as a string here.
+3. `llm` Section:
    - `provider` (String): The provider for the language model, which is set to 'openai'. You can find the full list of llm providers in [our docs](/components/llms).
    - `model` (String): The specific model being used, 'gpt-3.5-turbo'.
    - `config`:
@@ -59,13 +68,13 @@ Alright, let's dive into what each key means in the yaml config above:
        - `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
        - `template` (String): A custom template for the prompt that the model uses to generate responses.
        - `system_prompt` (String): A system prompt for the model to follow when generating responses, in this case, it's set to the style of William Shakespeare.
-3. `vectordb` Section:
+4. `vectordb` Section:
    - `provider` (String): The provider for the vector database, set to 'chroma'. You can find the full list of vector database providers in [our docs](/components/vector-databases).
    - `config`:
        - `collection_name` (String): The initial collection name for the database, set to 'full-stack-app'.
        - `dir` (String): The directory for the database, set to 'db'.
        - `allow_reset` (Boolean): Indicates whether resetting the database is allowed, set to true.
-4. `embedder` Section:
+5. `embedder` Section:
    - `provider` (String): The provider for the embedder, set to 'openai'. You can find the full list of embedding model providers in [our docs](/components/embedding-models).
    - `config`:
        - `model` (String): The specific model used for text embedding, 'text-embedding-ada-002'.
@@ -100,7 +100,7 @@ app = App.from_config(yaml_path="config.yaml")
 llm:
  provider: gpt4all
  config:
-    model: 'orca-mini-3b.ggmlv3.q4_0.bin'
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -190,7 +190,7 @@ app = App.from_config(yaml_path="config.yaml")
 llm:
  provider: gpt4all
  config:
-    model: 'orca-mini-3b.ggmlv3.q4_0.bin'
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -237,7 +237,7 @@ llm:
 Install related dependencies using the following command:

 ```bash
-pip install --upgrade 'embedchain[huggingface_hub]'
+pip install --upgrade 'embedchain[huggingface-hub]'
 ```

 First, set `HUGGINGFACE_ACCESS_TOKEN` in environment variable which you can obtain from [their platform](https://huggingface.co/settings/tokens).
@@ -138,7 +138,7 @@ app = App.from_config(yaml_path="config.yaml")
 vectordb:
  provider: zilliz
  config:
-    collection_name: 'zilliz-app'
+    collection_name: 'zilliz_app'
    uri: https://xxxx.api.gcp-region.zillizcloud.com
    token: xxx
    vector_dim: 1536
@@ -0,0 +1,44 @@
+---
+title: '🗨️ Discourse'
+---
+
+You can now easily load data from your community built with [Discourse](https://discourse.org/).
+
+## Example
+
+1. Setup the Discourse Loader with your community url.
+```Python
+from embedchain.loaders.discourse import DiscourseLoader
+
+dicourse_loader = DiscourseLoader(config={"domain": "https://community.openai.com"})
+```
+
+2. Once you setup the loader, you can create an app and load data using the above discourse loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+
+app.add("openai after:2023-10-1", data_type="discourse", loader=dicourse_loader)
+
+question = "Where can I find the OpenAI API status page?"
+app.query(question)
+# Answer: You can find the OpenAI API status page at https:/status.openai.com/.
+```
+
+NOTE: The `add` function of the app will accept any executable search query to load data. Refer [Discourse API Docs](https://docs.discourse.org/#tag/Search) to learn more about search queries.
+
+3. We automatically create a chunker to chunk your discourse data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+
+from embedchain.chunkers.discourse import DiscourseChunker
+from embedchain.config.add_config import ChunkerConfig
+
+discourse_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+discourse_chunker = DiscourseChunker(config=discourse_chunker_config)
+
+app.add("openai", data_type='discourse', loader=dicourse_loader, chunker=discourse_chunker)
+```
@@ -24,12 +24,11 @@ To use this you need to save `credentials.json` in the directory from where you
 12. Put the `.json` file in your current directory and rename it to `credentials.json`

 ```python
-import os
-from embedchain.apps.app import App
-from embedchain.models.data_type import DataType
+from embedchain import Pipeline as App
+
 app = App()

-query = "to: me label:inbox"
-app.add(query, data_type=DataType.GMAIL)
+gmail_filter = "to: me label:inbox"
+app.add(gmail_filter, data_type="gmail")
 app.query("Summarize my email conversations")
 ```
@@ -2,52 +2,43 @@
 title: '📃 JSON'
 ---

-To add any json file, use the data_type as `json`. Headers are included for each line, so if you have an `age` column, `18` will be added as `age: 18`. Eg:
+To add any json file, use the data_type as `json`. Headers are included for each line, so for example if you have a json like `{"age": 18}`, then it will be added as `age: 18`.

 Here are the supported sources for loading `json`:
+
 ```
 1. URL - valid url to json file that ends with ".json" extension.
 2. Local file - valid url to local json file that ends with ".json" extension.
 3. String - valid json string (e.g. - app.add('{"foo": "bar"}'))
 ```

-If you would like to add other data structures (e.x. list, dict etc.), do:
-```python
-    import json
-    a = {"foo": "bar"}
-    valid_json_string_data = json.dumps(a, indent=0)
+<Tip>
+If you would like to add other data structures (e.g. list, dict etc.), convert it to a valid json first using `json.dumps()` function.
+</Tip>

-    b = [{"foo": "bar"}]
-    valid_json_string_data = json.dumps(b, indent=0)
-```
-Example:
-```python
-import os
+## Example

-from embedchain.apps.app import App
+<CodeGroup>

-os.environ["OPENAI_API_KEY"] = "openai_api_key"
+```python python
+from embedchain import Pipeline as App

 app = App()

-response = app.query("What is the net worth of Elon Musk as of October 2023?")
+# Add json file
+app.add("temp.json")

-print(response)
-"I'm sorry, but I don't have access to real-time information or future predictions. Therefore, I don't know the net worth of Elon Musk as of October 2023."
-
-source_id = app.add("temp.json")
-
-response = app.query("What is the net worth of Elon Musk as of October 2023?")
-
-print(response)
-"As of October 2023, Elon Musk's net worth is $255.2 billion."
+app.query("What is the net worth of Elon Musk as of October 2023?")
+# As of October 2023, Elon Musk's net worth is $255.2 billion.
 ```
-temp.json
-```json
+
+
+```json temp.json
 {
    "question": "What is your net worth, Elon Musk?",
    "answer": "As of October 2023, Elon Musk's net worth is $255.2 billion, making him one of the wealthiest individuals in the world."
 }
 ```
+</CodeGroup>


@@ -0,0 +1,48 @@
+---
+title: '🐬 MySQL'
+---
+
+1. Setup the MySQL loader by configuring the SQL db.
+```Python
+from embedchain.loaders.mysql import MySQLLoader
+
+config = {
+    "host": "host",
+    "port": "port",
+    "database": "database",
+    "user": "username",
+    "password": "password",
+}
+
+mysql_loader = MySQLLoader(config=config)
+```
+
+For more details on how to setup with valid config, check MySQL [documentation](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html).
+
+2. Once you setup the loader, you can create an app and load data using the above MySQL loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+app = App()
+
+app.add("SELECT * FROM table_name;", data_type='mysql', loader=mysql_loader)
+# Adds `(1, 'What is your net worth, Elon Musk?', "As of October 2023, Elon Musk's net worth is $255.2 billion.")`
+
+response = app.query(question)
+# Answer: As of October 2023, Elon Musk's net worth is $255.2 billion.
+```
+
+NOTE: The `add` function of the app will accept any executable query to load data. DO NOT pass the `CREATE`, `INSERT` queries in `add` function.
+
+3. We automatically create a chunker to chunk your SQL data, however if you wish to provide your own chunker class. Here is how you can do that:
+``Python
+
+from embedchain.chunkers.mysql import MySQLChunker
+from embedchain.config.add_config import ChunkerConfig
+
+mysql_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+mysql_chunker = MySQLChunker(config=mysql_chunker_config)
+
+app.add("SELECT * FROM table_name;", data_type='mysql', loader=mysql_loader, chunker=mysql_chunker)
+```
@@ -2,13 +2,10 @@
 title: 🙌 OpenAPI
 ---

-To add any OpenAPI spec yaml file (currently the json file will be detected as JSON data type), use the data_type as 'openapi'. 'openapi' allows remote urls and conventional file paths. Headers are included for each line, so if you have an `age` column, `18` will be added as `age: 18`. Eg:
+To add any OpenAPI spec yaml file (currently the json file will be detected as JSON data type), use the data_type as 'openapi'. 'openapi' allows remote urls and conventional file paths.

 ```python
-from embedchain.apps.app import App
-import os
-
-os.environ["OPENAI_API_KEY"] = "sk-xxx"
+from embedchain import Pipeline as App

 app = App()

@@ -16,8 +13,10 @@ app.add("https://github.com/openai/openai-openapi/blob/master/openapi.yaml", dat
 # Or add using the local file path
 # app.add("configs/openai_openapi.yaml", data_type="openapi")

-response = app.query("What can OpenAI API endpoint do? Can you list the things it can learn from?")
+app.query("What can OpenAI API endpoint do? Can you list the things it can learn from?")
 # Answer: The OpenAI API endpoint allows users to interact with OpenAI's models and perform various tasks such as generating text, answering questions, summarizing documents, translating languages, and more. The specific capabilities and tasks that the API can learn from may vary depending on the models and features provided by OpenAI. For more detailed information, it is recommended to refer to the OpenAI API documentation at https://platform.openai.com/docs/api-reference.
 ```

-NOTE: The yaml file added to the App must have the required OpenAPI fields otherwise the adding OpenAPI spec will fail. Please refer to [OpenAPI Spec Doc](https://spec.openapis.org/oas/v3.1.0)
+<Note>
+The yaml file added to the App must have the required OpenAPI fields otherwise the adding OpenAPI spec will fail. Please refer to [OpenAPI Spec Doc](https://spec.openapis.org/oas/v3.1.0)
+</Note>
@@ -18,8 +18,12 @@ Embedchain comes with built-in support for various data sources. We handle the c
  <Card title="🌐📄 web page" href="/data-sources/web-page"></Card>
  <Card title="🧾 xml" href="/data-sources/xml"></Card>
  <Card title="🙌 OpenAPI" href="/data-sources/openapi"></Card>
-  <Card title="🎥📺 youtube video" href="/data-sources/youtube-video"></Card>
+  <Card title="📺 youtube video" href="/data-sources/youtube-video"></Card>
  <Card title="📬 Gmail" href="/data-sources/gmail"></Card>
+  <Card title="🐘 Postgres" href="/data-sources/postgres"></Card>
+  <Card title="🐬 MySQL" href="/data-sources/mysql"></Card>
+  <Card title="🤖 Slack" href="/data-sources/slack"></Card>
+  <Card title="🗨️ Discourse" href="/data-sources/discourse"></Card>
 </CardGroup>

 <br/ >
@@ -0,0 +1,64 @@
+---
+title: '🐘 Postgres'
+---
+
+1. Setup the Postgres loader by configuring the postgres db.
+```Python
+from embedchain.loaders.postgres import PostgresLoader
+
+config = {
+    "host": "host_address",
+    "port": "port_number",
+    "dbname": "database_name",
+    "user": "username",
+    "password": "password",
+}
+
+"""
+config = {
+    "url": "your_postgres_url"
+}
+"""
+
+postgres_loader = PostgresLoader(config=config)
+
+```
+
+You can either setup the loader by passing the postgresql url or by providing the config data.
+For more details on how to setup with valid url and config, check postgres [documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING:~:text=34.1.1.%C2%A0Connection%20Strings-,%23,-Several%20libpq%20functions).
+
+NOTE: if you provide the `url` field in config, all other fields will be ignored.
+
+2. Once you setup the loader, you can create an app and load data using the above postgres loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+
+question = "What is Elon Musk's networth?"
+response = app.query(question)
+# Answer: As of September 2021, Elon Musk's net worth is estimated to be around $250 billion, making him one of the wealthiest individuals in the world. However, please note that net worth can fluctuate over time due to various factors such as stock market changes and business ventures.
+
+app.add("SELECT * FROM table_name;", data_type='postgres', loader=postgres_loader)
+# Adds `(1, 'What is your net worth, Elon Musk?', "As of October 2023, Elon Musk's net worth is $255.2 billion.")`
+
+response = app.query(question)
+# Answer: As of October 2023, Elon Musk's net worth is $255.2 billion.
+```
+
+NOTE: The `add` function of the app will accept any executable query to load data. DO NOT pass the `CREATE`, `INSERT` queries in `add` function as they will result in not adding any data, so it is pointless.
+
+3. We automatically create a chunker to chunk your postgres data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+
+from embedchain.chunkers.postgres import PostgresChunker
+from embedchain.config.add_config import ChunkerConfig
+
+postgres_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+postgres_chunker = PostgresChunker(config=postgres_chunker_config)
+
+app.add("SELECT * FROM table_name;", data_type='postgres', loader=postgres_loader, chunker=postgres_chunker)
+```
@@ -0,0 +1,54 @@
+---
+title: '🤖 Slack'
+---
+
+## Pre-requisite
+- Download required packages by running `pip install --upgrade "embedchain[slack]"`.
+- Configure your slack bot token as environment variable `SLACK_USER_TOKEN`.
+    - Find your user token on your [Slack Account](https://api.slack.com/authentication/token-types)
+    - Make sure your slack user token includes [search](https://api.slack.com/scopes/search:read) scope.
+
+## Example
+1. Setup the Slack loader by configuring the Slack Webclient.
+```Python
+from embedchain.loaders.slack import SlackLoader
+
+os.environ["SLACK_USER_TOKEN"] = "xoxp-*"
+
+loader = SlackLoader()
+
+"""
+config = {
+    'base_url': slack_app_url,
+    'headers': web_headers,
+    'team_id': slack_team_id,
+}
+
+loader = SlackLoader(config)
+"""
+```
+
+NOTE: you can also pass the `config` with `base_url`, `headers`, `team_id` to setup your SlackLoader.
+
+2. Once you setup the loader, you can create an app and load data using the above slack loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+app = App()
+
+app.add("in:random", data_type="slack", loader=loader)
+question = "Which bots are available in the slack workspace's random channel?"
+# Answer: The available bot in the slack workspace's random channel is the Embedchain bot.
+```
+
+3. We automatically create a chunker to chunk your slack data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+from embedchain.chunkers.slack import SlackChunker
+from embedchain.config.add_config import ChunkerConfig
+
+slack_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+slack_chunker = SlackChunker(config=slack_chunker_config)
+
+app.add(slack_chunker, data_type="slack", loader=loader, chunker=slack_chunker)
+```
@@ -0,0 +1,16 @@
+---
+title: "📝 Substack"
+---
+
+To add any Substack data sources to your app, just add the sitemap.xml of that url as the source and set the data_type to `substack`.
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+# source: for any substack just add the sitemap.xml url
+app.add('https://www.lennysnewsletter.com/sitemap.xml', data_type='substack')
+app.query("Who is Brian Chesky?")
+# Answer: Brian Chesky is the co-founder and CEO of Airbnb.
+```
@@ -1,5 +1,5 @@
 ---
-title: '🎥📺 Youtube video'
+title: '📺 Youtube video'
 ---


@@ -1,5 +1,5 @@
 ---
-title: '🤖 Discord Bot'
+title: "🤖 Discord Bot"
 ---

 ### 🔑 Keys Setup
@@ -12,9 +12,11 @@ title: '🤖 Discord Bot'
 - On the left sidebar, click on `OAuth2` and go to `General`.
 - Set `Authorization Method` to `In-app Authorization`. Under `Scopes` select `bot`.
 - Under `Bot Permissions` allow the following and then click on `Save Changes`.
+
 ```text
 Send Messages (under Text Permissions)
 ```
+
 - Now under `OAuth2` and go to `URL Generator`. Under `Scopes` select `bot`.
 - Under `Bot Permissions` set the same permissions as above.
 - Now scroll down and copy the `Generated URL`. Paste it in a browser window and select the Server where you want to add the bot.
@@ -23,40 +25,46 @@ Send Messages (under Text Permissions)

 ### Take the bot online

-1. Install embedchain python package:
+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name discord-bot -e OPENAI_API_KEY=sk-xxx -e DISCORD_BOT_TOKEN=xxx -p 8080:8080 embedchain/discord-bot:latest
+        ```
+    </Tab>
+    <Tab title="python">
+        ```bash
+        pip install --upgrade "embedchain[discord]"

-```bash
-pip install --upgrade "embedchain[discord]"
-```
+        python -m embedchain.bots.discord

-2. Launch your Discord bot:
-
-
-```bash
-python -m embedchain.bots.discord
-```
-
-If you prefer to see the question and not only the answer, run it with
-
-```bash
-python -m embedchain.bots.discord --include-question
-```
+        # or if you prefer to see the question and not only the answer, run it with
+        python -m embedchain.bots.discord --include-question
+        ```
+    </Tab>
+</Tabs>

 ### 🚀 Usage Instructions

 - Go to the server where you have added your bot.
+  ![Slash commands interaction with bot](https://github.com/embedchain/embedchain/assets/73601258/bf1414e3-d408-4863-b0d2-ef382a76467e)
 - You can add data sources to the bot using the slash command:
+
 ```text
-/add <data_type> <url_or_text>
+/ec add <data_type> <url_or_text>
 ```
+
 - You can ask your queries from the bot using the slash command:
+
 ```text
-/query <question>
+/ec query <question>
 ```
+
 - You can chat with the bot using the slash command:
+
 ```text
-/chat <question>
+/ec chat <question>
 ```
+
 📝 Note: To use the bot privately, you can message the bot directly by right clicking the bot and selecting `Message`.

 🎉 Happy Chatting! 🎉
@@ -8,14 +8,45 @@ This guide will help you setup the full stack app on your local machine.

 ### 🐳 Docker Setup

- To setup full stack app using docker, run the following command inside this folder using your terminal.
+- Create a `docker-compose.yml` file and paste the following code in it.
+
+```yaml
+version: "3.9"
+
+services:
+  backend:
+    container_name: embedchain-backend
+    restart: unless-stopped
+    build:
+      context: backend
+      dockerfile: Dockerfile
+    image: embedchain/backend
+    ports:
+      - "8000:8000"
+
+  frontend:
+    container_name: embedchain-frontend
+    restart: unless-stopped
+    build:
+      context: frontend
+      dockerfile: Dockerfile
+    image: embedchain/frontend
+    ports:
+      - "3000:3000"
+    depends_on:
+      - "backend"
+```
+
+- Run the following command,

 ```bash
-docker-compose up --build
+docker-compose up
 ```

 📝 Note: The build command might take a while to install all the packages depending on your system resources.

+![Fullstack App](https://github.com/embedchain/embedchain/assets/73601258/c7c04bbb-9be7-4669-a6af-039e7e972a13)
+
 ### 🚀 Usage Instructions

 - Go to [http://localhost:3000/](http://localhost:3000/) in your browser to view the dashboard.
@@ -15,8 +15,21 @@ channels:read
 chat:write
 ```
 5. Now select the option `Install to Workspace` and after it's done, copy the `Bot User OAuth Token` and set it in your secrets as `SLACK_BOT_TOKEN`.
-6. Run your bot now with `python3 -m embedchain.bots.slack`
-7. Expose your bot to the internet. Default port is `5000`, which can be changed by adding `port --8080` to the startup command. You can use your machine's public IP or DNS. Otherwise, employ a proxy server like [ngrok](https://ngrok.com/) to make your local bot accessible.
+6. Run your bot now,
+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name slack-bot -e OPENAI_API_KEY=sk-xxx -e SLACK_BOT_TOKEN=xxx -p 8000:8000 embedchain/slack-bot
+        ```
+    </Tab>
+    <Tab title="python">
+        ```bash
+        pip install --upgrade "embedchain[slack]"
+        python3 -m embedchain.bots.slack --port 8000
+        ```
+</Tab>
+</Tabs>
+7. Expose your bot to the internet. You can use your machine's public IP or DNS. Otherwise, employ a proxy server like [ngrok](https://ngrok.com/) to make your local bot accessible.
 8. On the Slack API website go to `Event Subscriptions` on the left Sidebar and turn on `Enable Events`.
 9. In `Request URL`, enter your server or ngrok address.
 10. After it gets verified, click on `Subscribe to bot events`, add `message.channels` Bot User Event and click on `Save Changes`.
@@ -1,21 +1,46 @@
 ---
-title: '📱 Telegram Bot'
+title: "📱 Telegram Bot"
 ---

 ### 🖼️ Template Setup

- Fork [this](https://replit.com/@taranjeetio/EC-Telegram-Bot-Template?v=1#README.md) replit template.
- Set your `OPENAI_API_KEY` in Secrets.
 - Open the Telegram app and search for the `BotFather` user.
 - Start a chat with BotFather and use the `/newbot` command to create a new bot.
 - Follow the instructions to choose a name and username for your bot.
 - Once the bot is created, BotFather will provide you with a unique token for your bot.
- Set this token as `TELEGRAM_BOT_TOKEN` in Secrets.
+
+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name telegram-bot -e OPENAI_API_KEY=sk-xxx -e TELEGRAM_BOT_TOKEN=xxx -p 8000:8000 embedchain/telegram-bot
+        ```
+
+    <Note>
+    If you wish to use **Docker**, you would need to host your bot on a server.
+    You can use [ngrok](https://ngrok.com/) to expose your localhost to the
+    internet and then set the webhook using the ngrok URL.
+    </Note>
+
+    </Tab>
+    <Tab title="replit">
+    <Card>
+        Fork <ins>**[this](https://replit.com/@taranjeetio/EC-Telegram-Bot-Template?v=1#README.md)**</ins> replit template.
+    </Card>
+
+    - Set your `OPENAI_API_KEY` in Secrets.
+    - Set the unique token as `TELEGRAM_BOT_TOKEN` in Secrets.
+
+    </Tab>
+
+</Tabs>
+
 - Click on `Run` in the replit container and a URL will get generated for your bot.
 - Now set your webhook by running the following link in your browser:
+
 ```url
 https://api.telegram.org/bot<Your_Telegram_Bot_Token>/setWebhook?url=<Replit_Generated_URL>
 ```
+
 - When you get a successful response in your browser, your bot is ready to be used.

 ### 🚀 Usage Instructions
@@ -12,10 +12,19 @@ pip install --upgrade embedchain

 2. Launch your WhatsApp bot:

+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name whatsapp-bot -e OPENAI_API_KEY=sk-xxx -p 8000:8000 embedchain/whatsapp-bot
+        ```
+    </Tab>
+    <Tab title="python">
+        ```bash
+        python -m embedchain.bots.whatsapp --port 5000
+        ```
+    </Tab>
+</Tabs>

-```bash
-python -m embedchain.bots.whatsapp --port 5000
-```

 If your bot needs to be accessible online, use your machine's public IP or DNS. Otherwise, employ a proxy server like [ngrok](https://ngrok.com/) to make your local bot accessible.

@@ -3,6 +3,38 @@ title: ❓ FAQs
 description: 'Collections of all the frequently asked questions'
 ---

+#### Does Embedchain support OpenAI's Assistant APIs?
+
+Yes, it does. Please refer to the [OpenAI Assistant docs page](/get-started/openai-assistant).
+
+#### How to use `gpt-4-turbo` model released on OpenAI DevDay?
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ['OPENAI_API_KEY'] = 'xxx'
+
+# load llm configuration from gpt4_turbo.yaml file
+app = App.from_config(yaml_path="gpt4_turbo.yaml")
+```
+
+```yaml gpt4_turbo.yaml
+llm:
+  provider: openai
+  config:
+    model: 'gpt-4-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+```
+
+</CodeGroup>
+
+
 #### How to use GPT-4 as the LLM model?

 <CodeGroup>
@@ -48,7 +80,7 @@ app = App.from_config(yaml_path="opensource.yaml")
 llm:
  provider: gpt4all
  config:
-    model: 'orca-mini-3b.ggmlv3.q4_0.bin'
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -0,0 +1,95 @@
+---
+title: '🤖 OpenAI Assistant'
+---
+
+<img src="https://blogs.swarthmore.edu/its/wp-content/uploads/2022/05/openai.jpg"  align="center" width="500" alt="OpenAI Logo"/>
+
+Embedchain now supports [OpenAI Assistants API](https://platform.openai.com/docs/assistants/overview) which allows you to build AI assistants within your own applications. An Assistant has instructions and can leverage models, tools, and knowledge to respond to user queries.
+
+At a high level, an integration of the Assistants API has the following flow:
+
+1. Create an Assistant in the API by defining custom instructions and picking a model
+2. Create a Thread when a user starts a conversation
+3. Add Messages to the Thread as the user ask questions
+4. Run the Assistant on the Thread to trigger responses. This automatically calls the relevant tools.
+
+Creating an OpenAI Assistant using Embedchain is very simple 3 step process.
+
+## Step 1: Create OpenAI Assistant
+
+Make sure that you have `OPENAI_API_KEY` set in the environment variable.
+
+```python Initialize
+from embedchain.store.assistants import OpenAIAssistant
+
+assistant = OpenAIAssistant(
+    name="OpenAI DevDay Assistant",
+    instructions="You are an organizer of OpenAI DevDay",
+)
+```
+
+If you want to use the existing assistant, you can do something like this:
+
+```python Initialize
+# Load an assistant and create a new thread
+assistant = OpenAIAssistant(assistant_id="asst_xxx")
+
+# Load a specific thread for an assistant
+assistant = OpenAIAssistant(assistant_id="asst_xxx", thread_id="thread_xxx")
+```
+
+### Arguments
+
+<ResponseField name="name" type="string">
+  Name for your AI assistant
+</ResponseField>
+
+<ResponseField name="instructions" type="string">
+  how the Assistant and model should behave or respond
+</ResponseField>
+
+<ResponseField name="assistant_id" type="string">
+  Load existing OpenAI Assistant. If you pass this, you don't have to pass other arguments.
+</ResponseField>
+
+<ResponseField name="thread_id" type="string">
+  Existing OpenAI thread id if exists
+</ResponseField>
+
+<ResponseField name="model" type="str" default="gpt-4-1106-preview">
+  OpenAI model to use
+</ResponseField>
+
+<ResponseField name="tools" type="list">
+  OpenAI tools to use. Default set to `[{"type": "retrieval"}]`
+</ResponseField>
+
+<ResponseField name="data_sources" type="list" default="[]">
+  Add data sources to your assistant. You can add in the following format: `[{"source": "https://example.com", "data_type": "web_page"}]`
+</ResponseField>
+
+<ResponseField name="telemetry" type="boolean" default="True">
+  Anonymous telemetry (doesn't collect any user information or user's files). Used to improve the Embedchain package utilization. Default is `True`.
+</ResponseField>
+
+## Step-2: Add data to thread
+
+You can add any custom data source that is supported by Embedchain. Else, you can directly pass the file path on your local system and Embedchain propagates it to OpenAI Assistant.
+```python Add data
+assistant.add("/path/to/file.pdf")
+assistant.add("https://www.youtube.com/watch?v=U9mJuUkhUzk")
+assistant.add("https://openai.com/blog/new-models-and-developer-products-announced-at-devday")
+```
+
+## Step-3: Chat with your Assistant
+```python Chat
+assistant.chat("How much OpenAI credits were offered to attendees during OpenAI DevDay?")
+# Response: 'Every attendee of OpenAI DevDay 2023 was offered $500 in OpenAI credits.'
+```
+
+You can try it out yourself using the following Google Colab notebook:
+
+<a href="https://colab.research.google.com/drive/1BKlXZYSl6AFRgiHZ5XIzXrXC_24kDYHQ?usp=sharing">
+    <img src="https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667" alt="Open in Colab" />
+</a>
+
@@ -11,6 +11,10 @@ Install embedchain python package:
 pip install embedchain
 ```

+<Tip>
+Embedchain now supports OpenAI's latest `gpt-4-turbo` model. Checkout the [docs here](/get-started/faq#how-to-use-gpt-4-turbo-model-released-on-openai-devday) on how to use it.
+</Tip>
+
 Creating an app involves 3 steps:

 <Steps>
@@ -79,3 +83,9 @@ app.deploy()
 # 🛠️ Adding data to your pipeline...
 # ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
 ```
+
+You can try it out yourself using the following Google Colab notebook:
+
+<a href="https://colab.research.google.com/drive/17ON1LPonnXAtLaZEebnOktstB_1cJJmh?usp=sharing">
+    <img src="https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667" alt="Open in Colab" />
+</a>
@@ -3,38 +3,43 @@
  "name": "Embedchain",
  "logo": {
    "dark": "/logo/dark.svg",
-    "light": "/logo/light.svg"
+    "light": "/logo/light.svg",
+    "href": "https://embedchain.ai/"
  },
  "favicon": "/favicon.png",
  "colors": {
-    "primary": "#2B48EE",
-    "light": "#2B48EE",
-    "dark": "#2B48EE",
+    "primary": "#3B2FC9",
+    "light": "#6673FF",
+    "dark": "#3B2FC9",
    "background": {
-      "dark": "#020415"
+      "dark": "#0f1117",
+      "light": "#fff"
    }
  },
+  "modeToggle": {
+    "default": "dark"
+  },
  "openapi": ["/rest-api.json"],
  "metadata": {
    "og:image": "/images/og.png",
    "twitter:site": "@embedchain"
  },
-  "topAnchor": {
-    "name": "Documentation",
-    "icon": "book-open"
-  },
+  "anchors": [
+    {
+      "name": "Embedchain Platform",
+      "icon": "tv",
+      "url": "https://app.embedchain.ai/"
+    },
+    {
+      "name": "Join our slack",
+      "icon": "slack",
+      "url": "https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw"
+    }
+  ],
  "topbarLinks": [
    {
-      "name": "Twitter",
-      "url": "https://twitter.com/embedchain"
-    },
-    {
-      "name": "Slack",
-      "url": "https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw"
-    },
-    {
-      "name": "Discord",
-      "url": "https://discord.gg/6PzXDgEjG5"
+      "name": "Create account",
+      "url": "https://app.embedchain.ai/login/"
    }
  ],
  "topbarCtaButton": {
@@ -50,6 +55,7 @@
      "pages": [
        "get-started/quickstart",
        "get-started/introduction",
+        "get-started/openai-assistant",
        "get-started/faq",
        "get-started/examples"
      ]
@@ -81,7 +87,9 @@
            "data-sources/text",
            "data-sources/web-page",
            "data-sources/openapi",
-            "data-sources/youtube-video"
+            "data-sources/youtube-video",
+            "data-sources/discourse",
+            "data-sources/substack"
          ]
        },
        "data-sources/data-type-handling"
@@ -106,7 +114,7 @@
      ]
    },
    {
-      "group": "Examples",
+      "group": "Use Cases",
      "pages": [
        "examples/full_stack",
        "examples/discord_bot",
@@ -2,7 +2,9 @@ from typing import Optional

 import yaml

-from embedchain.config import AppConfig, BaseEmbedderConfig, BaseLlmConfig
+from embedchain.client import Client
+from embedchain.config import (AppConfig, BaseEmbedderConfig, BaseLlmConfig,
+                               ChunkerConfig)
 from embedchain.config.vectordb.base import BaseVectorDbConfig
 from embedchain.embedchain import EmbedChain
 from embedchain.embedder.base import BaseEmbedder
@@ -38,6 +40,7 @@ class App(EmbedChain):
        embedder: BaseEmbedder = None,
        embedder_config: Optional[BaseEmbedderConfig] = None,
        system_prompt: Optional[str] = None,
+        chunker: Optional[ChunkerConfig] = None,
    ):
        """
        Initialize a new `App` instance.
@@ -65,6 +68,9 @@ class App(EmbedChain):
        :type system_prompt: Optional[str], optional
        :raises TypeError: LLM, database or embedder or their config is not a valid class instance.
        """
+        # Setup user directory if it doesn't exist already
+        Client.setup_dir()
+
        # Type check configs
        if config and not isinstance(config, AppConfig):
            raise TypeError(
@@ -97,6 +103,9 @@ class App(EmbedChain):
        if embedder is None:
            embedder = OpenAIEmbedder(config=embedder_config)

+        self.chunker = None
+        if chunker:
+            self.chunker = ChunkerConfig(**chunker)
        # Type check assignments
        if not isinstance(llm, BaseLlm):
            raise TypeError(
@@ -125,6 +134,9 @@ class App(EmbedChain):
        :return: An instance of the App class.
        :rtype: App
        """
+        # Setup user directory if it doesn't exist already
+        Client.setup_dir()
+
        with open(yaml_path, "r") as file:
            config_data = yaml.safe_load(file)

@@ -137,6 +149,7 @@ class App(EmbedChain):
        llm_config_data = config_data.get("llm", {})
        db_config_data = config_data.get("vectordb", {})
        embedding_model_config_data = config_data.get("embedding_model", config_data.get("embedder", {}))
+        chunker_config_data = config_data.get("chunker", {})

        app_config = AppConfig(**app_config_data.get("config", {}))

@@ -148,4 +161,4 @@ class App(EmbedChain):

        embedder_provider = embedding_model_config_data.get("provider", "openai")
        embedder = EmbedderFactory.create(embedder_provider, embedding_model_config_data.get("config", {}))
-        return cls(config=app_config, llm=llm, db=db, embedder=embedder)
+        return cls(config=app_config, llm=llm, db=db, embedder=embedder, chunker=chunker_config_data)
@@ -1,7 +1,7 @@
 from typing import Any

-from embedchain import App
-from embedchain.config import AddConfig, AppConfig, BaseLlmConfig
+from embedchain import Pipeline as App
+from embedchain.config import AddConfig, BaseLlmConfig, PipelineConfig
 from embedchain.embedder.openai import OpenAIEmbedder
 from embedchain.helper.json_serializable import (JSONSerializable,
                                                 register_deserializable)
@@ -12,7 +12,7 @@ from embedchain.vectordb.chroma import ChromaDB
@register_deserializable
 class BaseBot(JSONSerializable):
    def __init__(self):
-        self.app = App(config=AppConfig(), llm=OpenAILlm(), db=ChromaDB(), embedder=OpenAIEmbedder())
+        self.app = App(config=PipelineConfig(), llm=OpenAILlm(), db=ChromaDB(), embedding_model=OpenAIEmbedder())

    def add(self, data: Any, config: AddConfig = None):
        """
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helper.json_serializable import register_deserializable
+
+
+@register_deserializable
+class CommonChunker(BaseChunker):
+    """Common chunker for all loaders."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helper.json_serializable import register_deserializable
+
+
+@register_deserializable
+class DiscourseChunker(BaseChunker):
+    """Chunker for discourse."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helper.json_serializable import register_deserializable
+
+
+@register_deserializable
+class MySQLChunker(BaseChunker):
+    """Chunker for json."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helper.json_serializable import register_deserializable
+
+
+@register_deserializable
+class PostgresChunker(BaseChunker):
+    """Chunker for postgres."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helper.json_serializable import register_deserializable
+
+
+@register_deserializable
+class SlackChunker(BaseChunker):
+    """Chunker for postgres."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helper.json_serializable import register_deserializable
+
+
+@register_deserializable
+class SubstackChunker(BaseChunker):
+    """Chunker for Substack."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -5,7 +5,7 @@ import uuid

 import requests

-from embedchain.embedchain import CONFIG_DIR, CONFIG_FILE
+from embedchain.constants import CONFIG_DIR, CONFIG_FILE


 class Client:
@@ -1,3 +1,5 @@
+import builtins
+from importlib import import_module
 from typing import Callable, Optional

 from embedchain.config.base_config import BaseConfig
@@ -18,7 +20,18 @@ class ChunkerConfig(BaseConfig):
    ):
        self.chunk_size = chunk_size if chunk_size else 2000
        self.chunk_overlap = chunk_overlap if chunk_overlap else 0
-        self.length_function = length_function if length_function else len
+        if isinstance(length_function, str):
+            self.length_function = self.load_func(length_function)
+        else:
+            self.length_function = length_function if length_function else len
+
+    def load_func(self, dotpath: str):
+        if "." not in dotpath:
+            return getattr(builtins, dotpath)
+        else:
+            module_, func = dotpath.rsplit(".", maxsplit=1)
+            m = import_module(module_)
+            return getattr(m, func)


@register_deserializable
@@ -0,0 +1,8 @@
+import os
+from pathlib import Path
+
+ABS_PATH = os.getcwd()
+HOME_DIR = str(Path.home())
+CONFIG_DIR = os.path.join(HOME_DIR, ".embedchain")
+CONFIG_FILE = os.path.join(CONFIG_DIR, "config.json")
+SQLITE_PATH = os.path.join(CONFIG_DIR, "embedchain.db")
@@ -1,4 +1,5 @@
 from importlib import import_module
+from typing import Any, Dict

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config import AddConfig
@@ -15,7 +16,7 @@ class DataFormatter(JSONSerializable):
    .add or .add_local method call
    """

-    def __init__(self, data_type: DataType, config: AddConfig):
+    def __init__(self, data_type: DataType, config: AddConfig, kwargs: Dict[str, Any]):
        """
        Initialize a dataformatter, set data type and chunker based on datatype.

@@ -24,15 +25,15 @@ class DataFormatter(JSONSerializable):
        :param config: AddConfig instance with nested loader and chunker config attributes.
        :type config: AddConfig
        """
-        self.loader = self._get_loader(data_type=data_type, config=config.loader)
-        self.chunker = self._get_chunker(data_type=data_type, config=config.chunker)
+        self.loader = self._get_loader(data_type=data_type, config=config.loader, kwargs=kwargs)
+        self.chunker = self._get_chunker(data_type=data_type, config=config.chunker, kwargs=kwargs)

    def _lazy_load(self, module_path: str):
        module_path, class_name = module_path.rsplit(".", 1)
        module = import_module(module_path)
        return getattr(module, class_name)

-    def _get_loader(self, data_type: DataType, config: LoaderConfig) -> BaseLoader:
+    def _get_loader(self, data_type: DataType, config: LoaderConfig, kwargs: Dict[str, Any]) -> BaseLoader:
        """
        Returns the appropriate data loader for the given data type.

@@ -62,14 +63,35 @@ class DataFormatter(JSONSerializable):
            DataType.OPENAPI: "embedchain.loaders.openapi.OpenAPILoader",
            DataType.GMAIL: "embedchain.loaders.gmail.GmailLoader",
            DataType.NOTION: "embedchain.loaders.notion.NotionLoader",
+            DataType.SUBSTACK: "embedchain.loaders.substack.SubstackLoader",
+            DataType.GITHUB: "embedchain.loaders.github.GithubLoader",
+            DataType.YOUTUBE_CHANNEL: "embedchain.loaders.youtube_channel.YoutubeChannelLoader",
        }
+
+        custom_loaders = set(
+            [
+                DataType.POSTGRES,
+                DataType.MYSQL,
+                DataType.SLACK,
+                DataType.DISCOURSE,
+            ]
+        )
+
        if data_type in loaders:
            loader_class: type = self._lazy_load(loaders[data_type])
            return loader_class()
-        else:
-            raise ValueError(f"Unsupported data type: {data_type}")
+        elif data_type in custom_loaders:
+            loader_class: type = kwargs.get("loader", None)
+            if loader_class is not None:
+                return loader_class

-    def _get_chunker(self, data_type: DataType, config: ChunkerConfig) -> BaseChunker:
+        raise ValueError(
+            f"Cant find the loader for {data_type}.\
+                    We recommend to pass the loader to use data_type: {data_type},\
+                        check `https://docs.embedchain.ai/data-sources/overview`."
+        )
+
+    def _get_chunker(self, data_type: DataType, config: ChunkerConfig, kwargs: Dict[str, Any]) -> BaseChunker:
        """Returns the appropriate chunker for the given data type (updated for lazy loading)."""
        chunker_classes = {
            DataType.YOUTUBE_VIDEO: "embedchain.chunkers.youtube_video.YoutubeVideoChunker",
@@ -89,12 +111,27 @@ class DataFormatter(JSONSerializable):
            DataType.OPENAPI: "embedchain.chunkers.openapi.OpenAPIChunker",
            DataType.GMAIL: "embedchain.chunkers.gmail.GmailChunker",
            DataType.NOTION: "embedchain.chunkers.notion.NotionChunker",
+            DataType.POSTGRES: "embedchain.chunkers.postgres.PostgresChunker",
+            DataType.MYSQL: "embedchain.chunkers.mysql.MySQLChunker",
+            DataType.SLACK: "embedchain.chunkers.slack.SlackChunker",
+            DataType.DISCOURSE: "embedchain.chunkers.discourse.DiscourseChunker",
+            DataType.SUBSTACK: "embedchain.chunkers.substack.SubstackChunker",
+            DataType.GITHUB: "embedchain.chunkers.common_chunker.CommonChunker",
+            DataType.YOUTUBE_CHANNEL: "embedchain.chunkers.common_chunker.CommonChunker",
        }

        if data_type in chunker_classes:
-            chunker_class = self._lazy_load(chunker_classes[data_type])
+            if "chunker" in kwargs:
+                chunker_class = kwargs.get("chunker")
+            else:
+                chunker_class = self._lazy_load(chunker_classes[data_type])
+
            chunker = chunker_class(config)
            chunker.set_data_type(data_type)
            return chunker
        else:
-            raise ValueError(f"Unsupported data type: {data_type}")
+            raise ValueError(
+                f"Cant find the chunker for {data_type}.\
+                    We recommend to pass the chunker to use data_type: {data_type},\
+                        check `https://docs.embedchain.ai/data-sources/overview`."
+            )
@@ -1,17 +1,16 @@
 import hashlib
 import json
 import logging
-import os
 import sqlite3
-from pathlib import Path
 from typing import Any, Dict, List, Optional, Tuple, Union

 from dotenv import load_dotenv
 from langchain.docstore.document import Document

 from embedchain.chunkers.base_chunker import BaseChunker
-from embedchain.config import AddConfig, BaseLlmConfig
+from embedchain.config import AddConfig, BaseLlmConfig, ChunkerConfig
 from embedchain.config.apps.base_app_config import BaseAppConfig
+from embedchain.constants import SQLITE_PATH
 from embedchain.data_formatter import DataFormatter
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.helper.json_serializable import JSONSerializable
@@ -25,12 +24,6 @@ from embedchain.vectordb.base import BaseVectorDB

 load_dotenv()

-ABS_PATH = os.getcwd()
-HOME_DIR = str(Path.home())
-CONFIG_DIR = os.path.join(HOME_DIR, ".embedchain")
-CONFIG_FILE = os.path.join(CONFIG_DIR, "config.json")
-SQLITE_PATH = os.path.join(CONFIG_DIR, "embedchain.db")
-

 class EmbedChain(JSONSerializable):
    def __init__(
@@ -81,14 +74,18 @@ class EmbedChain(JSONSerializable):
        if system_prompt:
            self.llm.config.system_prompt = system_prompt

+        # Fetch the history from the database if exists
+        self.llm.update_history(app_id=self.config.id)
+
        # Attributes that aren't subclass related.
        self.user_asks = []

+        self.chunker: ChunkerConfig = None
        # Send anonymous telemetry
        self._telemetry_props = {"class": self.__class__.__name__}
        self.telemetry = AnonymousTelemetry(enabled=self.config.collect_metrics)
        # Establish a connection to the SQLite database
-        self.connection = sqlite3.connect(SQLITE_PATH)
+        self.connection = sqlite3.connect(SQLITE_PATH, check_same_thread=False)
        self.cursor = self.connection.cursor()

        # Create the 'data_sources' table if it doesn't exist
@@ -136,6 +133,7 @@ class EmbedChain(JSONSerializable):
        metadata: Optional[Dict[str, Any]] = None,
        config: Optional[AddConfig] = None,
        dry_run=False,
+        **kwargs: Dict[str, Any],
    ):
        """
        Adds the data from the given URL to the vector db.
@@ -157,7 +155,11 @@ class EmbedChain(JSONSerializable):
        :return: source_hash, a md5-hash of the source, in hexadecimal representation.
        :rtype: str
        """
-        if config is None:
+        if config is not None:
+            pass
+        elif self.chunker is not None:
+            config = AddConfig(chunker=self.chunker)
+        else:
            config = AddConfig()

        try:
@@ -175,21 +177,6 @@ class EmbedChain(JSONSerializable):
        if data_type:
            try:
                data_type = DataType(data_type)
-                if data_type == DataType.JSON:
-                    if isinstance(source, str):
-                        if not is_valid_json_string(source):
-                            raise ValueError(
-                                f"Invalid json input: {source}",
-                                "Provide the correct JSON formatted source, \
-                                    refer `https://docs.embedchain.ai/data-sources/json`",
-                            )
-                    elif not isinstance(source, str):
-                        raise ValueError(
-                            "Invaid content input. \
-                            If you want to upload (list, dict, etc.), do \
-                                `json.dump(data, indent=0)` and add the stringified JSON. \
-                                    Check - `https://docs.embedchain.ai/data-sources/json`"
-                        )
            except ValueError:
                raise ValueError(
                    f"Invalid data_type: '{data_type}'.",
@@ -213,9 +200,10 @@ class EmbedChain(JSONSerializable):
            print(f"Data with hash {source_hash} already exists. Skipping addition.")
            return source_hash

-        data_formatter = DataFormatter(data_type, config)
        self.user_asks.append([source, data_type.value, metadata])
-        documents, metadatas, _ids, new_chunks = self.load_and_embed(
+
+        data_formatter = DataFormatter(data_type, config, kwargs)
+        documents, metadatas, _ids, new_chunks = self._load_and_embed(
            data_formatter.loader, data_formatter.chunker, source, metadata, source_hash, dry_run
        )
        if data_type in {DataType.DOCS_SITE}:
@@ -260,6 +248,7 @@ class EmbedChain(JSONSerializable):
        data_type: Optional[DataType] = None,
        metadata: Optional[Dict[str, Any]] = None,
        config: Optional[AddConfig] = None,
+        **kwargs: Dict[str, Any],
    ):
        """
        Adds the data from the given URL to the vector db.
@@ -285,7 +274,13 @@ class EmbedChain(JSONSerializable):
        logging.warning(
            "The `add_local` method is deprecated and will be removed in future versions. Please use the `add` method for both local and remote files."  # noqa: E501
        )
-        return self.add(source=source, data_type=data_type, metadata=metadata, config=config)
+        return self.add(
+            source=source,
+            data_type=data_type,
+            metadata=metadata,
+            config=config,
+            kwargs=kwargs,
+        )

    def _get_existing_doc_id(self, chunker: BaseChunker, src: Any):
        """
@@ -345,7 +340,7 @@ class EmbedChain(JSONSerializable):
                "When it should be  DirectDataType, IndirectDataType or SpecialDataType."
            )

-    def load_and_embed(
+    def _load_and_embed(
        self,
        loader: BaseLoader,
        chunker: BaseChunker,
@@ -462,7 +457,7 @@ class EmbedChain(JSONSerializable):
            )
        ]

-    def retrieve_from_database(
+    def _retrieve_from_database(
        self, input_query: str, config: Optional[BaseLlmConfig] = None, where=None, citations: bool = False
    ) -> Union[List[Tuple[str, str, str]], List[str]]:
        """
@@ -483,13 +478,13 @@ class EmbedChain(JSONSerializable):
        query_config = config or self.llm.config
        if where is not None:
            where = where
-        elif query_config is not None and query_config.where is not None:
-            where = query_config.where
        else:
            where = {}
+            if query_config is not None and query_config.where is not None:
+                where = query_config.where

-        if self.config.id is not None:
-            where.update({"app_id": self.config.id})
+            if self.config.id is not None:
+                where.update({"app_id": self.config.id})

        # We cannot query the database with the input query in case of an image search. This is because we need
        # to bring down both the image and text to the same dimension to be able to compare them.
@@ -542,7 +537,9 @@ class EmbedChain(JSONSerializable):
        :rtype: str, if citations is False, otherwise Tuple[str,List[Tuple[str,str,str]]]
        """
        citations = kwargs.get("citations", False)
-        contexts = self.retrieve_from_database(input_query=input_query, config=config, where=where, citations=citations)
+        contexts = self._retrieve_from_database(
+            input_query=input_query, config=config, where=where, citations=citations
+        )
        if citations and len(contexts) > 0 and isinstance(contexts[0], tuple):
            contexts_data_for_llm_query = list(map(lambda x: x[0], contexts))
        else:
@@ -593,7 +590,9 @@ class EmbedChain(JSONSerializable):
        :rtype: str, if citations is False, otherwise Tuple[str,List[Tuple[str,str,str]]]
        """
        citations = kwargs.get("citations", False)
-        contexts = self.retrieve_from_database(input_query=input_query, config=config, where=where, citations=citations)
+        contexts = self._retrieve_from_database(
+            input_query=input_query, config=config, where=where, citations=citations
+        )
        if citations and len(contexts) > 0 and isinstance(contexts[0], tuple):
            contexts_data_for_llm_query = list(map(lambda x: x[0], contexts))
        else:
@@ -603,6 +602,9 @@ class EmbedChain(JSONSerializable):
            input_query=input_query, contexts=contexts_data_for_llm_query, config=config, dry_run=dry_run
        )

+        # add conversation in memory
+        self.llm.add_history(self.config.id, input_query, answer)
+
        # Send anonymous telemetry
        self.telemetry.capture(event_name="chat", properties=self._telemetry_props)

@@ -646,5 +648,16 @@ class EmbedChain(JSONSerializable):
        self.db.reset()
        self.cursor.execute("DELETE FROM data_sources WHERE pipeline_id = ?", (self.config.id,))
        self.connection.commit()
+        self.delete_history()
        # Send anonymous telemetry
        self.telemetry.capture(event_name="reset", properties=self._telemetry_props)
+
+    def get_history(self, num_rounds: int = 10, display_format: bool = True):
+        return self.llm.memory.get_recent_memories(
+            app_id=self.config.id,
+            num_rounds=num_rounds,
+            display_format=display_format,
+        )
+
+    def delete_history(self):
+        self.llm.memory.delete_chat_history(app_id=self.config.id)
@@ -3,12 +3,20 @@ from typing import Any, Callable, Optional
 from embedchain.config.embedder.base import BaseEmbedderConfig

 try:
-    from chromadb.api.types import Documents, Embeddings
+    from chromadb.api.types import Embeddings, Embeddable, EmbeddingFunction
 except RuntimeError:
    from embedchain.utils import use_pysqlite3

    use_pysqlite3()
-    from chromadb.api.types import Documents, Embeddings
+    from chromadb.api.types import Embeddings, Embeddable, EmbeddingFunction
+
+
+class EmbeddingFunc(EmbeddingFunction):
+    def __init__(self, embedding_fn: Callable[[list[str]], list[str]]):
+        self.embedding_fn = embedding_fn
+
+    def __call__(self, input: Embeddable) -> Embeddings:
+        return self.embedding_fn(input)


 class BaseEmbedder:
@@ -66,7 +74,4 @@ class BaseEmbedder:
        :rtype: Callable
        """

-        def embed_function(texts: Documents) -> Embeddings:
-            return embeddings.embed_documents(texts)
-
-        return embed_function
+        return EmbeddingFunc(embeddings.embed_documents)
@@ -1,24 +1,18 @@
 import os
 from typing import Optional

+from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
 from langchain.embeddings import OpenAIEmbeddings

 from embedchain.config import BaseEmbedderConfig
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.models import VectorDimensions

-try:
-    from chromadb.utils import embedding_functions
-except RuntimeError:
-    from embedchain.utils import use_pysqlite3
-
-    use_pysqlite3()
-    from chromadb.utils import embedding_functions
-

 class OpenAIEmbedder(BaseEmbedder):
    def __init__(self, config: Optional[BaseEmbedderConfig] = None):
        super().__init__(config=config)
+
        if self.config.model is None:
            self.config.model = "text-embedding-ada-002"

@@ -30,11 +24,10 @@ class OpenAIEmbedder(BaseEmbedder):
                raise ValueError(
                    "OPENAI_API_KEY or OPENAI_ORGANIZATION environment variables not provided"
                )  # noqa:E501
-            embedding_fn = embedding_functions.OpenAIEmbeddingFunction(
+            embedding_fn = OpenAIEmbeddingFunction(
                api_key=os.getenv("OPENAI_API_KEY"),
                organization_id=os.getenv("OPENAI_ORGANIZATION"),
                model_name=self.config.model,
            )
-
        self.set_embedding_fn(embedding_fn=embedding_fn)
        self.set_vector_dimension(vector_dimension=VectorDimensions.OPENAI.value)
@@ -33,7 +33,7 @@ def register_deserializable(cls: Type[T]) -> Type[T]:
    Returns:
        Type: The same class, after registration.
    """
-    JSONSerializable.register_class_as_deserializable(cls)
+    JSONSerializable._register_class_as_deserializable(cls)
    return cls


@@ -183,7 +183,7 @@ class JSONSerializable:
            return cls.deserialize(json_str)

    @classmethod
-    def register_class_as_deserializable(cls, target_class: Type[T]) -> None:
+    def _register_class_as_deserializable(cls, target_class: Type[T]) -> None:
        """
        Register a class as deserializable. This is a classmethod and globally shared.

@@ -1,14 +1,15 @@
 import logging
 from typing import Any, Dict, Generator, List, Optional

-from langchain.memory import ConversationBufferMemory
-from langchain.schema import BaseMessage
+from langchain.schema import BaseMessage as LCBaseMessage

 from embedchain.config import BaseLlmConfig
 from embedchain.config.llm.base import (DEFAULT_PROMPT,
                                        DEFAULT_PROMPT_WITH_HISTORY_TEMPLATE,
                                        DOCS_SITE_PROMPT_TEMPLATE)
 from embedchain.helper.json_serializable import JSONSerializable
+from embedchain.memory.base import ECChatMemory
+from embedchain.memory.message import ChatMessage


 class BaseLlm(JSONSerializable):
@@ -23,7 +24,7 @@ class BaseLlm(JSONSerializable):
        else:
            self.config = config

-        self.memory = ConversationBufferMemory()
+        self.memory = ECChatMemory()
        self.is_docs_site_instance = False
        self.online = False
        self.history: Any = None
@@ -44,11 +45,18 @@ class BaseLlm(JSONSerializable):
        """
        self.history = history

-    def update_history(self):
+    def update_history(self, app_id: str):
        """Update class history attribute with history in memory (for chat method)"""
-        chat_history = self.memory.load_memory_variables({})["history"]
+        chat_history = self.memory.get_recent_memories(app_id=app_id, num_rounds=10)
        if chat_history:
-            self.set_history(chat_history)
+            self.set_history([str(history) for history in chat_history])
+
+    def add_history(self, app_id: str, question: str, answer: str, metadata: Optional[Dict[str, Any]] = None):
+        chat_message = ChatMessage()
+        chat_message.add_user_message(question, metadata=metadata)
+        chat_message.add_ai_message(answer, metadata=metadata)
+        self.memory.add(app_id=app_id, chat_message=chat_message)
+        self.update_history(app_id=app_id)

    def generate_prompt(self, input_query: str, contexts: List[str], **kwargs: Dict[str, Any]) -> str:
        """
@@ -165,7 +173,6 @@ class BaseLlm(JSONSerializable):
        for chunk in answer:
            streamed_answer = streamed_answer + chunk
            yield chunk
-        self.memory.chat_memory.add_ai_message(streamed_answer)
        logging.info(f"Answer: {streamed_answer}")

    def query(self, input_query: str, contexts: List[str], config: BaseLlmConfig = None, dry_run=False):
@@ -257,8 +264,6 @@ class BaseLlm(JSONSerializable):
            if self.online:
                k["web_search_result"] = self.access_search_and_get_results(input_query)

-            self.update_history()
-
            prompt = self.generate_prompt(input_query, contexts, **k)
            logging.info(f"Prompt: {prompt}")

@@ -267,16 +272,9 @@ class BaseLlm(JSONSerializable):

            answer = self.get_answer_from_llm(prompt)

-            self.memory.chat_memory.add_user_message(input_query)
-
            if isinstance(answer, str):
-                self.memory.chat_memory.add_ai_message(answer)
                logging.info(f"Answer: {answer}")

-                # NOTE: Adding to history before and after. This could be seen as redundant.
-                # If we change it, we have to change the tests (no big deal).
-                self.update_history()
-
                return answer
            else:
                # this is a streamed response and needs to be handled differently.
@@ -287,7 +285,7 @@ class BaseLlm(JSONSerializable):
                self.config: BaseLlmConfig = BaseLlmConfig.deserialize(prev_config)

    @staticmethod
-    def _get_messages(prompt: str, system_prompt: Optional[str] = None) -> List[BaseMessage]:
+    def _get_messages(prompt: str, system_prompt: Optional[str] = None) -> List[LCBaseMessage]:
        """
        Construct a list of langchain messages

@@ -13,7 +13,7 @@ class GPT4ALLLlm(BaseLlm):
    def __init__(self, config: Optional[BaseLlmConfig] = None):
        super().__init__(config=config)
        if self.config.model is None:
-            self.config.model = "orca-mini-3b.ggmlv3.q4_0.bin"
+            self.config.model = "orca-mini-3b-gguf2-q4_0.gguf"
        self.instance = GPT4ALLLlm._get_instance(self.config.model)
        self.instance.streaming = self.config.stream

@@ -20,7 +20,7 @@ class HuggingFaceLlm(BaseLlm):
        except ModuleNotFoundError:
            raise ModuleNotFoundError(
                "The required dependencies for HuggingFaceHub are not installed."
-                'Please install with `pip install --upgrade "embedchain[huggingface_hub]"`'
+                'Please install with `pip install --upgrade "embedchain[huggingface-hub]"`'
            ) from None

        super().__init__(config=config)
@@ -0,0 +1,77 @@
+import hashlib
+import logging
+import time
+from typing import Any, Dict, Optional
+
+import requests
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+
+class DiscourseLoader(BaseLoader):
+    def __init__(self, config: Optional[Dict[str, Any]] = None):
+        super().__init__()
+        if not config:
+            raise ValueError(
+                "DiscourseLoader requires a config. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+            )
+
+        self.domain = config.get("domain")
+        if not self.domain:
+            raise ValueError(
+                "DiscourseLoader requires a domain. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+            )
+
+    def _check_query(self, query):
+        if not query or not isinstance(query, str):
+            raise ValueError(
+                "DiscourseLoader requires a query. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+            )
+
+    def _load_post(self, post_id):
+        post_url = f"{self.domain}posts/{post_id}.json"
+        response = requests.get(post_url)
+        try:
+            response.raise_for_status()
+        except Exception as e:
+            logging.error(f"Failed to load post {post_id}: {e}")
+            return
+        response_data = response.json()
+        post_contents = clean_string(response_data.get("raw"))
+        meta_data = {
+            "url": post_url,
+            "created_at": response_data.get("created_at", ""),
+            "username": response_data.get("username", ""),
+            "topic_slug": response_data.get("topic_slug", ""),
+            "score": response_data.get("score", ""),
+        }
+        data = {
+            "content": post_contents,
+            "meta_data": meta_data,
+        }
+        return data
+
+    def load_data(self, query):
+        self._check_query(query)
+        data = []
+        data_contents = []
+        logging.info(f"Searching data on discourse url: {self.domain}, for query: {query}")
+        search_url = f"{self.domain}search.json?q={query}"
+        response = requests.get(search_url)
+        try:
+            response.raise_for_status()
+        except Exception as e:
+            raise ValueError(f"Failed to search query {query}: {e}")
+        response_data = response.json()
+        post_ids = response_data.get("grouped_search_result").get("post_ids")
+        for id in post_ids:
+            post_data = self._load_post(id)
+            if post_data:
+                data.append(post_data)
+                data_contents.append(post_data.get("content"))
+            # Sleep for 0.4 sec, to avoid rate limiting. Check `https://meta.discourse.org/t/api-rate-limits/208405/6`
+            time.sleep(0.4)
+        doc_id = hashlib.sha256((query + ", ".join(data_contents)).encode()).hexdigest()
+        response_data = {"doc_id": doc_id, "data": data}
+        return response_data
@@ -0,0 +1,117 @@
+import concurrent.futures
+import hashlib
+import logging
+import os
+
+from tqdm import tqdm
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.loaders.json import JSONLoader
+from embedchain.loaders.mdx import MdxLoader
+from embedchain.utils import detect_datatype
+
+
+def _load_file_data(path):
+    data = []
+    data_content = []
+    try:
+        with open(path, "rb") as f:
+            content = f.read().decode("utf-8")
+    except Exception as e:
+        print(f"Error reading file {path}: {e}")
+        raise ValueError(f"Failed to read file {path}")
+
+    meta_data = {}
+    meta_data["url"] = path
+    data.append(
+        {
+            "content": content,
+            "meta_data": meta_data,
+        }
+    )
+    data_content.append(content)
+    doc_id = hashlib.sha256((" ".join(data_content) + path).encode()).hexdigest()
+    return {
+        "doc_id": doc_id,
+        "data": data,
+    }
+
+
+class GithubLoader(BaseLoader):
+    def load_data(self, repo_url):
+        """Load data from a git repo."""
+        try:
+            from git import Repo
+        except ImportError as e:
+            raise ValueError(
+                "GithubLoader requires extra dependencies. Install with `pip install --upgrade 'embedchain[git]'`"
+            ) from e
+
+        mdx_loader = MdxLoader()
+        json_loader = JSONLoader()
+        data = []
+        data_urls = []
+
+        def _fetch_or_clone_repo(repo_url: str, local_path: str):
+            if os.path.exists(local_path):
+                logging.info("Repository already exists. Fetching updates...")
+                repo = Repo(local_path)
+                origin = repo.remotes.origin
+                origin.fetch()
+                logging.info("Fetch completed.")
+            else:
+                logging.info("Cloning repository...")
+                Repo.clone_from(repo_url, local_path)
+                logging.info("Clone completed.")
+
+        def _load_file(file_path: str):
+            try:
+                data_type = detect_datatype(file_path).value
+            except Exception:
+                data_type = "unstructured"
+
+            if data_type == "mdx":
+                data = mdx_loader.load_data(file_path)
+            elif data_type == "json":
+                data = json_loader.load_data(file_path)
+            else:
+                data = _load_file_data(file_path)
+
+            return data.get("data", [])
+
+        def _is_file_empty(file_path):
+            return os.path.getsize(file_path) == 0
+
+        def _is_whitelisted(file_path):
+            whitelisted_extensions = ["md", "txt", "html", "json", "py", "js", "jsx", "ts", "tsx", "mdx", "rst"]
+            _, file_extension = os.path.splitext(file_path)
+            return file_extension[1:] in whitelisted_extensions
+
+        def _add_repo_files(repo_path: str):
+            with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
+                future_to_file = {
+                    executor.submit(_load_file, os.path.join(root, filename)): os.path.join(root, filename)
+                    for root, _, files in os.walk(repo_path)
+                    for filename in files
+                    if _is_whitelisted(os.path.join(root, filename))
+                    and not _is_file_empty(os.path.join(root, filename))  # noqa:E501
+                }
+                for future in tqdm(concurrent.futures.as_completed(future_to_file), total=len(future_to_file)):
+                    file = future_to_file[future]
+                    try:
+                        results = future.result()
+                        if results:
+                            data.extend(results)
+                            data_urls.extend([result.get("meta_data").get("url") for result in results])
+                    except Exception as e:
+                        logging.warn(f"Failed to process {file}: {e}")
+
+        source_hash = hashlib.sha256(repo_url.encode()).hexdigest()
+        repo_path = f"/tmp/{source_hash}"
+        _fetch_or_clone_repo(repo_url=repo_url, local_path=repo_path)
+        _add_repo_files(repo_path)
+        doc_id = hashlib.sha256((repo_url + ", ".join(data_urls)).encode()).hexdigest()
+        return {
+            "doc_id": doc_id,
+            "data": data,
+        }
@@ -25,10 +25,21 @@ class JSONLoader(BaseLoader):

        return LLHUBJSONLoader()

+    @staticmethod
+    def _check_content(content):
+        if not isinstance(content, str):
+            raise ValueError(
+                "Invaid content input. \
+                If you want to upload (list, dict, etc.), do \
+                    `json.dump(data, indent=0)` and add the stringified JSON. \
+                        Check - `https://docs.embedchain.ai/data-sources/json`"
+            )
+
    @staticmethod
    def load_data(content):
        """Load a json file. Each data point is a key value pair."""

+        JSONLoader._check_content(content)
        loader = JSONLoader._get_llama_hub_loader()

        data = []
@@ -0,0 +1,64 @@
+import hashlib
+import logging
+from typing import Any, Dict, Optional
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+
+class MySQLLoader(BaseLoader):
+    def __init__(self, config: Optional[Dict[str, Any]]):
+        super().__init__()
+        if not config:
+            raise ValueError(
+                f"Invalid sql config: {config}.",
+                "Provide the correct config, refer `https://docs.embedchain.ai/data-sources/mysql`.",
+            )
+
+        self.config = config
+        self.connection = None
+        self.cursor = None
+        self._setup_loader(config=config)
+
+    def _setup_loader(self, config: Dict[str, Any]):
+        try:
+            import mysql.connector as sqlconnector
+        except ImportError as e:
+            raise ImportError(
+                "Unable to import required packages for MySQL loader. Run `pip install --upgrade 'embedchain[mysql]'`."  # noqa: E501
+            ) from e
+
+        try:
+            self.connection = sqlconnector.connection.MySQLConnection(**config)
+            self.cursor = self.connection.cursor()
+        except (sqlconnector.Error, IOError) as err:
+            logging.info(f"Connection failed: {err}")
+            raise ValueError(
+                f"Unable to connect with the given config: {config}.",
+                "Please provide the correct configuration to load data from you MySQL DB. \
+                    Refer `https://docs.embedchain.ai/data-sources/mysql`.",
+            )
+
+    def _check_query(self, query):
+        if not isinstance(query, str):
+            raise ValueError(
+                f"Invalid mysql query: {query}",
+                "Provide the valid query to add from mysql, \
+                    make sure you are following `https://docs.embedchain.ai/data-sources/mysql`",
+            )
+
+    def load_data(self, query):
+        self._check_query(query=query)
+        data = []
+        data_content = []
+        self.cursor.execute(query)
+        rows = self.cursor.fetchall()
+        for row in rows:
+            doc_content = clean_string(str(row))
+            data.append({"content": doc_content, "meta_data": {"url": query}})
+            data_content.append(doc_content)
+        doc_id = hashlib.sha256((query + ", ".join(data_content)).encode()).hexdigest()
+        return {
+            "doc_id": doc_id,
+            "data": data,
+        }
@@ -0,0 +1,71 @@
+import hashlib
+import logging
+from typing import Any, Dict, Optional
+
+from embedchain.loaders.base_loader import BaseLoader
+
+
+class PostgresLoader(BaseLoader):
+    def __init__(self, config: Optional[Dict[str, Any]] = None):
+        super().__init__()
+        if not config:
+            raise ValueError(f"Must provide the valid config. Received: {config}")
+
+        self.connection = None
+        self.cursor = None
+        self._setup_loader(config=config)
+
+    def _setup_loader(self, config: Dict[str, Any]):
+        try:
+            import psycopg
+        except ImportError as e:
+            raise ImportError(
+                "Unable to import required packages. \
+                    Run `pip install --upgrade 'embedchain[postgres]'`"
+            ) from e
+
+        config_info = ""
+        if "url" in config:
+            config_info = config.get("url")
+        else:
+            conn_params = []
+            for key, value in config.items():
+                conn_params.append(f"{key}={value}")
+            config_info = " ".join(conn_params)
+
+        logging.info(f"Connecting to postrgres sql: {config_info}")
+        self.connection = psycopg.connect(conninfo=config_info)
+        self.cursor = self.connection.cursor()
+
+    def _check_query(self, query):
+        if not isinstance(query, str):
+            raise ValueError(
+                f"Invalid postgres query: {query}. Provide the valid source to add from postgres, make sure you are following `https://docs.embedchain.ai/data-sources/postgres`",  # noqa:E501
+            )
+
+    def load_data(self, query):
+        self._check_query(query)
+        try:
+            data = []
+            data_content = []
+            self.cursor.execute(query)
+            results = self.cursor.fetchall()
+            for result in results:
+                doc_content = str(result)
+                data.append({"content": doc_content, "meta_data": {"url": query}})
+                data_content.append(doc_content)
+            doc_id = hashlib.sha256((query + ", ".join(data_content)).encode()).hexdigest()
+            return {
+                "doc_id": doc_id,
+                "data": data,
+            }
+        except Exception as e:
+            raise ValueError(f"Failed to load data using query={query} with: {e}")
+
+    def close_connection(self):
+        if self.cursor:
+            self.cursor.close()
+            self.cursor = None
+        if self.connection:
+            self.connection.close()
+            self.connection = None
@@ -1,7 +1,9 @@
+import concurrent.futures
 import hashlib
 import logging

 import requests
+from tqdm import tqdm

 try:
    from bs4 import BeautifulSoup
@@ -19,33 +21,45 @@ from embedchain.utils import is_readable

@register_deserializable
 class SitemapLoader(BaseLoader):
+    """
+    This method takes a sitemap URL as input and retrieves
+    all the URLs to use the WebPageLoader to load content
+    of each page.
+    """
+
    def load_data(self, sitemap_url):
-        """
-        This method takes a sitemap URL as input and retrieves
-        all the URLs to use the WebPageLoader to load content
-        of each page.
-        """
        output = []
        web_page_loader = WebPageLoader()
        response = requests.get(sitemap_url)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "xml")
-
        links = [link.text for link in soup.find_all("loc") if link.parent.name == "url"]
        if len(links) == 0:
-            # Get all <loc> tags as a fallback. This might include images.
            links = [link.text for link in soup.find_all("loc")]

        doc_id = hashlib.sha256((" ".join(links) + sitemap_url).encode()).hexdigest()

-        for link in links:
+        def load_link(link):
            try:
                each_load_data = web_page_loader.load_data(link)
                if is_readable(each_load_data.get("data")[0].get("content")):
-                    output.append(each_load_data.get("data"))
+                    return each_load_data.get("data")
                else:
                    logging.warning(f"Page is not readable (too many invalid characters): {link}")
            except ParserRejectedMarkup as e:
                logging.error(f"Failed to parse {link}: {e}")
-        return {"doc_id": doc_id, "data": [data[0] for data in output]}
+            return None
+
+        with concurrent.futures.ThreadPoolExecutor() as executor:
+            future_to_link = {executor.submit(load_link, link): link for link in links}
+            for future in tqdm(concurrent.futures.as_completed(future_to_link), total=len(links), desc="Loading pages"):
+                link = future_to_link[future]
+                try:
+                    data = future.result()
+                    if data:
+                        output.extend(data)
+                except Exception as e:
+                    logging.error(f"Error loading page {link}: {e}")
+
+        return {"doc_id": doc_id, "data": output}
@@ -0,0 +1,108 @@
+import hashlib
+import logging
+import os
+import ssl
+from typing import Any, Dict, Optional
+
+import certifi
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+SLACK_API_BASE_URL = "https://www.slack.com/api/"
+
+
+class SlackLoader(BaseLoader):
+    def __init__(self, config: Optional[Dict[str, Any]] = None):
+        super().__init__()
+
+        if config is not None:
+            self.config = config
+        else:
+            self.config = {"base_url": SLACK_API_BASE_URL}
+
+        self.client = None
+        self._setup_loader(self.config)
+
+    def _setup_loader(self, config: Dict[str, Any]):
+        try:
+            from slack_sdk import WebClient
+        except ImportError as e:
+            raise ImportError(
+                "Slack loader requires extra dependencies. \
+                Install with `pip install --upgrade embedchain[slack]`"
+            ) from e
+
+        if os.getenv("SLACK_USER_TOKEN") is None:
+            raise ValueError(
+                "SLACK_USER_TOKEN environment variables not provided. Check `https://docs.embedchain.ai/data-sources/slack` to learn more."  # noqa:E501
+            )
+
+        logging.info(f"Creating Slack Loader with config: {config}")
+        # get slack client config params
+        slack_bot_token = os.getenv("SLACK_USER_TOKEN")
+        ssl_cert = ssl.create_default_context(cafile=certifi.where())
+        base_url = config.get("base_url", SLACK_API_BASE_URL)
+        headers = config.get("headers")
+        # for Org-Wide App
+        team_id = config.get("team_id")
+
+        self.client = WebClient(
+            token=slack_bot_token,
+            base_url=base_url,
+            ssl=ssl_cert,
+            headers=headers,
+            team_id=team_id,
+        )
+        logging.info("Slack Loader setup successful!")
+
+    def _check_query(self, query):
+        if not isinstance(query, str):
+            raise ValueError(
+                f"Invalid query passed to Slack loader, found: {query}. Check `https://docs.embedchain.ai/data-sources/slack` to learn more."  # noqa:E501
+            )
+
+    def load_data(self, query):
+        self._check_query(query)
+        try:
+            data = []
+            data_content = []
+
+            logging.info(f"Searching slack conversations for query: {query}")
+            results = self.client.search_messages(
+                query=query,
+                sort="timestamp",
+                sort_dir="desc",
+                count=1000,
+            )
+
+            messages = results.get("messages")
+            num_message = results.get("total")
+            logging.info(f"Found {num_message} messages for query: {query}")
+
+            matches = messages.get("matches", [])
+            for message in matches:
+                url = message.get("permalink")
+                text = message.get("text")
+                content = clean_string(text)
+
+                message_meta_data_keys = ["channel", "iid", "team", "ts", "type", "user", "username"]
+                meta_data = message.fromkeys(message_meta_data_keys, "")
+                meta_data.update({"url": url})
+                data.append(
+                    {
+                        "content": content,
+                        "meta_data": meta_data,
+                    }
+                )
+                data_content.append(content)
+            doc_id = hashlib.md5((query + ", ".join(data_content)).encode()).hexdigest()
+            return {
+                "doc_id": doc_id,
+                "data": data,
+            }
+        except Exception as e:
+            logging.warning(f"Error in loading slack data: {e}")
+            raise ValueError(
+                f"Error in loading slack data: {e}. Check `https://docs.embedchain.ai/data-sources/slack` to learn more."  # noqa:E501
+            ) from e
@@ -0,0 +1,86 @@
+import hashlib
+import logging
+import time
+
+import requests
+
+from embedchain.helper.json_serializable import register_deserializable
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import is_readable
+
+
+@register_deserializable
+class SubstackLoader(BaseLoader):
+    """
+    This method takes a sitemap URL as input and retrieves
+    all the URLs to use the WebPageLoader to load content
+    of each page.
+    """
+
+    def load_data(self, url: str):
+        try:
+            from bs4 import BeautifulSoup
+            from bs4.builder import ParserRejectedMarkup
+        except ImportError:
+            raise ImportError(
+                'Substack requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'
+            ) from None
+
+        output = []
+        response = requests.get(url)
+        response.raise_for_status()
+
+        soup = BeautifulSoup(response.text, "xml")
+        links = [link.text for link in soup.find_all("loc") if link.parent.name == "url" and "/p/" in link.text]
+        if len(links) == 0:
+            links = [link.text for link in soup.find_all("loc") if "/p/" in link.text]
+
+        doc_id = hashlib.sha256((" ".join(links) + url).encode()).hexdigest()
+
+        def serialize_response(soup: BeautifulSoup):
+            data = {}
+
+            h1_els = soup.find_all("h1")
+            if h1_els is not None and len(h1_els) > 0:
+                data["title"] = h1_els[1].text
+
+            description_el = soup.find("meta", {"name": "description"})
+            if description_el is not None:
+                data["description"] = description_el["content"]
+
+            content_el = soup.find("div", {"class": "available-content"})
+            if content_el is not None:
+                data["content"] = content_el.text
+
+            like_btn = soup.find("div", {"class": "like-button-container"})
+            if like_btn is not None:
+                no_of_likes_div = like_btn.find("div", {"class": "label"})
+                if no_of_likes_div is not None:
+                    data["no_of_likes"] = no_of_likes_div.text
+
+            return data
+
+        def load_link(link: str):
+            try:
+                each_load_data = requests.get(link)
+                each_load_data.raise_for_status()
+
+                soup = BeautifulSoup(response.text, "html.parser")
+                data = serialize_response(soup)
+                data = str(data)
+                if is_readable(data):
+                    return data
+                else:
+                    logging.warning(f"Page is not readable (too many invalid characters): {link}")
+            except ParserRejectedMarkup as e:
+                logging.error(f"Failed to parse {link}: {e}")
+            return None
+
+        for link in links:
+            data = load_link(link)
+            if data:
+                output.append({"content": data, "meta_data": {"url": link}})
+            # TODO: allow users to configure this
+            time.sleep(1.0)  # added to avoid rate limiting
+
+        return {"doc_id": doc_id, "data": output}
@@ -1,11 +1,5 @@
 import hashlib

-try:
-    from langchain.document_loaders import UnstructuredFileLoader
-except ImportError:
-    raise ImportError(
-        'PDF File requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'
-    ) from None
 from embedchain.helper.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils import clean_string
@@ -15,6 +9,13 @@ from embedchain.utils import clean_string
 class UnstructuredLoader(BaseLoader):
    def load_data(self, url):
        """Load data from a Unstructured file."""
+        try:
+            from langchain.document_loaders import UnstructuredFileLoader
+        except ImportError:
+            raise ImportError(
+                'Unstructured file requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'  # noqa: E501
+            ) from None
+
        loader = UnstructuredFileLoader(url)
        data = []
        all_content = []
@@ -17,15 +17,17 @@ from embedchain.utils import clean_string

@register_deserializable
 class WebPageLoader(BaseLoader):
+    # Shared session for all instances
+    _session = requests.Session()
+
    def load_data(self, url):
-        """Load data from a web page."""
-        response = requests.get(url)
+        """Load data from a web page using a shared requests session."""
+        response = self._session.get(url, timeout=30)
+        response.raise_for_status()
        data = response.content
        content = self._get_clean_content(data, url)

-        meta_data = {
-            "url": url,
-        }
+        meta_data = {"url": url}

        doc_id = hashlib.sha256((content + url).encode()).hexdigest()
        return {
@@ -86,3 +88,7 @@ class WebPageLoader(BaseLoader):
            )

        return content
+
+    @classmethod
+    def close_session(cls):
+        cls._session.close()
@@ -0,0 +1,77 @@
+import concurrent.futures
+import hashlib
+import logging
+
+from tqdm import tqdm
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.loaders.youtube_video import YoutubeVideoLoader
+
+
+class YoutubeChannelLoader(BaseLoader):
+    """Loader for youtube channel."""
+
+    def load_data(self, channel_name):
+        try:
+            import yt_dlp
+        except ImportError as e:
+            raise ValueError(
+                "YoutubeLoader requires extra dependencies. Install with `pip install --upgrade 'embedchain[youtube_channel]'`"  # noqa: E501
+            ) from e
+
+        data = []
+        data_urls = []
+        youtube_url = f"https://www.youtube.com/{channel_name}/videos"
+        youtube_video_loader = YoutubeVideoLoader()
+
+        def _get_yt_video_links():
+            try:
+                ydl_opts = {
+                    "quiet": True,
+                    "extract_flat": True,
+                }
+                with yt_dlp.YoutubeDL(ydl_opts) as ydl:
+                    info_dict = ydl.extract_info(youtube_url, download=False)
+                    if "entries" in info_dict:
+                        videos = [entry["url"] for entry in info_dict["entries"]]
+                        return videos
+            except Exception:
+                logging.error(f"Failed to fetch youtube videos for channel: {channel_name}")
+                return []
+
+        def _load_yt_video(video_link):
+            try:
+                each_load_data = youtube_video_loader.load_data(video_link)
+                if each_load_data:
+                    return each_load_data.get("data")
+            except Exception as e:
+                logging.error(f"Failed to load youtube video {video_link}: {e}")
+            return None
+
+        def _add_youtube_channel():
+            video_links = _get_yt_video_links()
+            logging.info("Loading videos from youtube channel...")
+            with concurrent.futures.ThreadPoolExecutor() as executor:
+                # Submitting all tasks and storing the future object with the video link
+                future_to_video = {
+                    executor.submit(_load_yt_video, video_link): video_link for video_link in video_links
+                }
+
+                for future in tqdm(
+                    concurrent.futures.as_completed(future_to_video), total=len(video_links), desc="Processing videos"
+                ):
+                    video = future_to_video[future]
+                    try:
+                        results = future.result()
+                        if results:
+                            data.extend(results)
+                            data_urls.extend([result.get("meta_data").get("url") for result in results])
+                    except Exception as e:
+                        logging.error(f"Failed to process youtube video {video}: {e}")
+
+        _add_youtube_channel()
+        doc_id = hashlib.sha256((youtube_url + ", ".join(data_urls)).encode()).hexdigest()
+        return {
+            "doc_id": doc_id,
+            "data": data,
+        }
@@ -19,7 +19,7 @@ class YoutubeVideoLoader(BaseLoader):
        doc = loader.load()
        output = []
        if not len(doc):
-            raise ValueError("No data found")
+            raise ValueError(f"No data found for url: {url}")
        content = doc[0].page_content
        content = clean_string(content)
        meta_data = doc[0].metadata
@@ -0,0 +1,116 @@
+import json
+import logging
+import sqlite3
+import uuid
+from typing import Any, Dict, List, Optional
+
+from embedchain.constants import SQLITE_PATH
+from embedchain.memory.message import ChatMessage
+from embedchain.memory.utils import merge_metadata_dict
+
+CHAT_MESSAGE_CREATE_TABLE_QUERY = """
+            CREATE TABLE IF NOT EXISTS chat_history (
+                app_id TEXT,
+                id TEXT,
+                question TEXT,
+                answer TEXT,
+                metadata TEXT,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                PRIMARY KEY (id, app_id)
+            )
+            """
+
+
+class ECChatMemory:
+    def __init__(self) -> None:
+        with sqlite3.connect(SQLITE_PATH) as self.connection:
+            self.cursor = self.connection.cursor()
+
+            self.cursor.execute(CHAT_MESSAGE_CREATE_TABLE_QUERY)
+            self.connection.commit()
+
+    def add(self, app_id, chat_message: ChatMessage) -> Optional[str]:
+        memory_id = str(uuid.uuid4())
+        metadata_dict = merge_metadata_dict(chat_message.human_message.metadata, chat_message.ai_message.metadata)
+        if metadata_dict:
+            metadata = self._serialize_json(metadata_dict)
+        ADD_CHAT_MESSAGE_QUERY = """
+            INSERT INTO chat_history (app_id, id, question, answer, metadata)
+            VALUES (?, ?, ?, ?, ?)
+        """
+        self.cursor.execute(
+            ADD_CHAT_MESSAGE_QUERY,
+            (
+                app_id,
+                memory_id,
+                chat_message.human_message.content,
+                chat_message.ai_message.content,
+                metadata if metadata_dict else "{}",
+            ),
+        )
+        self.connection.commit()
+        logging.info(f"Added chat memory to db with id: {memory_id}")
+        return memory_id
+
+    def delete_chat_history(self, app_id: str):
+        DELETE_CHAT_HISTORY_QUERY = """
+            DELETE FROM chat_history WHERE app_id=?
+        """
+        self.cursor.execute(
+            DELETE_CHAT_HISTORY_QUERY,
+            (app_id,),
+        )
+        self.connection.commit()
+
+    def get_recent_memories(self, app_id, num_rounds=10, display_format=False) -> List[ChatMessage]:
+        """
+        Get the most recent num_rounds rounds of conversations
+        between human and AI, for a given app_id.
+        """
+
+        QUERY = """
+            SELECT * FROM chat_history
+            WHERE app_id=?
+            ORDER BY created_at DESC
+            LIMIT ?
+        """
+        self.cursor.execute(
+            QUERY,
+            (app_id, num_rounds),
+        )
+
+        results = self.cursor.fetchall()
+        history = []
+        for result in results:
+            app_id, _, question, answer, metadata, timestamp = result
+            metadata = self._deserialize_json(metadata=metadata)
+            # Return list of dict if display_format is True
+            if display_format:
+                history.append({"human": question, "ai": answer, "metadata": metadata, "timestamp": timestamp})
+            else:
+                memory = ChatMessage()
+                memory.add_user_message(question, metadata=metadata)
+                memory.add_ai_message(answer, metadata=metadata)
+                history.append(memory)
+        return history
+
+    def _serialize_json(self, metadata: Dict[str, Any]):
+        return json.dumps(metadata)
+
+    def _deserialize_json(self, metadata: str):
+        return json.loads(metadata)
+
+    def close_connection(self):
+        self.connection.close()
+
+    def count_history_messages(self, app_id: str):
+        QUERY = """
+        SELECT COUNT(*) FROM chat_history
+        WHERE app_id=?
+        """
+        self.cursor.execute(
+            QUERY,
+            (app_id,),
+        )
+        count = self.cursor.fetchone()[0]
+        return count
@@ -0,0 +1,72 @@
+import logging
+from typing import Any, Dict, Optional
+
+from embedchain.helper.json_serializable import JSONSerializable
+
+
+class BaseMessage(JSONSerializable):
+    """
+    The base abstract message class.
+
+    Messages are the inputs and outputs of Models.
+    """
+
+    # The string content of the message.
+    content: str
+
+    # The creator of the message. AI, Human, Bot etc.
+    by: str
+
+    # Any additional info.
+    metadata: Dict[str, Any]
+
+    def __init__(self, content: str, creator: str, metadata: Optional[Dict[str, Any]] = None) -> None:
+        super().__init__()
+        self.content = content
+        self.creator = creator
+        self.metadata = metadata
+
+    @property
+    def type(self) -> str:
+        """Type of the Message, used for serialization."""
+
+    @classmethod
+    def is_lc_serializable(cls) -> bool:
+        """Return whether this class is serializable."""
+        return True
+
+    def __str__(self) -> str:
+        return f"{self.creator}: {self.content}"
+
+
+class ChatMessage(JSONSerializable):
+    """
+    The base abstract chat message class.
+
+    Chat messages are the pair of (question, answer) conversation
+    between human and model.
+    """
+
+    human_message: Optional[BaseMessage] = None
+    ai_message: Optional[BaseMessage] = None
+
+    def add_user_message(self, message: str, metadata: Optional[dict] = None):
+        if self.human_message:
+            logging.info(
+                "Human message already exists in the chat message,\
+                overwritting it with new message."
+            )
+
+        self.human_message = BaseMessage(content=message, creator="human", metadata=metadata)
+
+    def add_ai_message(self, message: str, metadata: Optional[dict] = None):
+        if self.ai_message:
+            logging.info(
+                "AI message already exists in the chat message,\
+                overwritting it with new message."
+            )
+
+        self.ai_message = BaseMessage(content=message, creator="ai", metadata=metadata)
+
+    def __str__(self) -> str:
+        return f"{self.human_message}\n{self.ai_message}"
@@ -0,0 +1,35 @@
+from typing import Any, Dict, Optional
+
+
+def merge_metadata_dict(left: Optional[Dict[str, Any]], right: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]:
+    """
+    Merge the metadatas of two BaseMessage types.
+
+    Args:
+        left (Dict[str, Any]): metadata of human message
+        right (Dict[str, Any]): metadata of ai message
+
+    Returns:
+        Dict[str, Any]: combined metadata dict with dedup
+        to be saved in db.
+    """
+    if not left and not right:
+        return None
+    elif not left:
+        return right
+    elif not right:
+        return left
+
+    merged = left.copy()
+    for k, v in right.items():
+        if k not in merged:
+            merged[k] = v
+        elif type(merged[k]) != type(v):
+            raise ValueError(f'additional_kwargs["{k}"] already exists in this message,' " but with a different type.")
+        elif isinstance(merged[k], str):
+            merged[k] += v
+        elif isinstance(merged[k], dict):
+            merged[k] = merge_metadata_dict(merged[k], v)
+        else:
+            raise ValueError(f"Additional kwargs key {k} already exists in this message.")
+    return merged
@@ -29,6 +29,13 @@ class IndirectDataType(Enum):
    JSON = "json"
    OPENAPI = "openapi"
    GMAIL = "gmail"
+    POSTGRES = "postgres"
+    MYSQL = "mysql"
+    SLACK = "slack"
+    DISCOURSE = "discourse"
+    SUBSTACK = "substack"
+    GITHUB = "github"
+    YOUTUBE_CHANNEL = "youtube_channel"


 class SpecialDataType(Enum):
@@ -57,3 +64,10 @@ class DataType(Enum):
    JSON = IndirectDataType.JSON.value
    OPENAPI = IndirectDataType.OPENAPI.value
    GMAIL = IndirectDataType.GMAIL.value
+    POSTGRES = IndirectDataType.POSTGRES.value
+    MYSQL = IndirectDataType.MYSQL.value
+    SLACK = IndirectDataType.SLACK.value
+    DISCOURSE = IndirectDataType.DISCOURSE.value
+    SUBSTACK = IndirectDataType.SUBSTACK.value
+    GITHUB = IndirectDataType.GITHUB.value
+    YOUTUBE_CHANNEL = IndirectDataType.YOUTUBE_CHANNEL.value
@@ -9,8 +9,9 @@ import requests
 import yaml

 from embedchain import Client
-from embedchain.config import PipelineConfig
-from embedchain.embedchain import CONFIG_DIR, EmbedChain
+from embedchain.config import ChunkerConfig, PipelineConfig
+from embedchain.constants import SQLITE_PATH
+from embedchain.embedchain import EmbedChain
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.embedder.openai import OpenAIEmbedder
 from embedchain.factory import EmbedderFactory, LlmFactory, VectorDBFactory
@@ -22,8 +23,6 @@ from embedchain.utils import validate_yaml_config
 from embedchain.vectordb.base import BaseVectorDB
 from embedchain.vectordb.chroma import ChromaDB

-SQLITE_PATH = os.path.join(CONFIG_DIR, "embedchain.db")
-

@register_deserializable
 class Pipeline(EmbedChain):
@@ -42,8 +41,9 @@ class Pipeline(EmbedChain):
        embedding_model: BaseEmbedder = None,
        llm: BaseLlm = None,
        yaml_path: str = None,
-        log_level=logging.INFO,
+        log_level=logging.WARN,
        auto_deploy: bool = False,
+        chunker: ChunkerConfig = None,
    ):
        """
        Initialize a new `App` instance.
@@ -58,12 +58,15 @@ class Pipeline(EmbedChain):
        :type llm: BaseLlm, optional
        :param yaml_path: Path to the YAML configuration file, defaults to None
        :type yaml_path: str, optional
-        :param log_level: Log level to use, defaults to logging.INFO
+        :param log_level: Log level to use, defaults to logging.WARN
        :type log_level: int, optional
        :param auto_deploy: Whether to deploy the pipeline automatically, defaults to False
        :type auto_deploy: bool, optional
        :raises Exception: If an error occurs while creating the pipeline
        """
+        # Setup user directory if it doesn't exist already
+        Client.setup_dir()
+
        if id and yaml_path:
            raise Exception("Cannot provide both id and config. Please provide only one of them.")

@@ -75,18 +78,18 @@ class Pipeline(EmbedChain):

        logging.basicConfig(level=log_level, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s")
        self.logger = logging.getLogger(__name__)
-
        self.auto_deploy = auto_deploy
-
        # Store the yaml config as an attribute to be able to send it
        self.yaml_config = None
        self.client = None
        # pipeline_id from the backend
        self.id = None
+        self.chunker = None
+        if chunker:
+            self.chunker = ChunkerConfig(**chunker)

        self.config = config or PipelineConfig()
        self.name = self.config.name
-
        self.config.id = self.local_id = str(uuid.uuid4()) if self.config.id is None else self.config.id

        if yaml_path:
@@ -115,7 +118,7 @@ class Pipeline(EmbedChain):
        self.telemetry = AnonymousTelemetry(enabled=self.config.collect_metrics)

        # Establish a connection to the SQLite database
-        self.connection = sqlite3.connect(SQLITE_PATH)
+        self.connection = sqlite3.connect(SQLITE_PATH, check_same_thread=False)
        self.cursor = self.connection.cursor()

        # Create the 'data_sources' table if it doesn't exist
@@ -354,6 +357,9 @@ class Pipeline(EmbedChain):
        :return: An instance of the Pipeline class.
        :rtype: Pipeline
        """
+        # Setup user directory if it doesn't exist already
+        Client.setup_dir()
+
        with open(yaml_path, "r") as file:
            config_data = yaml.safe_load(file)

@@ -366,6 +372,7 @@ class Pipeline(EmbedChain):
        db_config_data = config_data.get("vectordb", {})
        embedding_model_config_data = config_data.get("embedding_model", config_data.get("embedder", {}))
        llm_config_data = config_data.get("llm", {})
+        chunker_config_data = config_data.get("chunker", {})

        pipeline_config = PipelineConfig(**pipeline_config_data)

@@ -394,4 +401,5 @@ class Pipeline(EmbedChain):
            embedding_model=embedding_model,
            yaml_path=yaml_path,
            auto_deploy=auto_deploy,
+            chunker=chunker_config_data,
        )
@@ -0,0 +1,204 @@
+import logging
+import os
+import re
+import tempfile
+import time
+import uuid
+from pathlib import Path
+from typing import cast
+
+from openai import OpenAI
+from openai.types.beta.threads import MessageContentText, ThreadMessage
+
+from embedchain import Pipeline
+from embedchain.config import AddConfig
+from embedchain.data_formatter import DataFormatter
+from embedchain.models.data_type import DataType
+from embedchain.telemetry.posthog import AnonymousTelemetry
+from embedchain.utils import detect_datatype
+
+logging.basicConfig(level=logging.WARN)
+
+
+class OpenAIAssistant:
+    def __init__(
+        self,
+        name=None,
+        instructions=None,
+        tools=None,
+        thread_id=None,
+        model="gpt-4-1106-preview",
+        data_sources=None,
+        assistant_id=None,
+        log_level=logging.WARN,
+        collect_metrics=True,
+    ):
+        self.name = name or "OpenAI Assistant"
+        self.instructions = instructions
+        self.tools = tools or [{"type": "retrieval"}]
+        self.model = model
+        self.data_sources = data_sources or []
+        self.log_level = log_level
+        self._client = OpenAI()
+        self._initialize_assistant(assistant_id)
+        self.thread_id = thread_id or self._create_thread()
+        self._telemetry_props = {"class": self.__class__.__name__}
+        self.telemetry = AnonymousTelemetry(enabled=collect_metrics)
+        self.telemetry.capture(event_name="init", properties=self._telemetry_props)
+
+    def add(self, source, data_type=None):
+        file_path = self._prepare_source_path(source, data_type)
+        self._add_file_to_assistant(file_path)
+
+        event_props = {
+            **self._telemetry_props,
+            "data_type": data_type or detect_datatype(source),
+        }
+        self.telemetry.capture(event_name="add", properties=event_props)
+        logging.info("Data successfully added to the assistant.")
+
+    def chat(self, message):
+        self._send_message(message)
+        self.telemetry.capture(event_name="chat", properties=self._telemetry_props)
+        return self._get_latest_response()
+
+    def delete_thread(self):
+        self._client.beta.threads.delete(self.thread_id)
+        self.thread_id = self._create_thread()
+
+    # Internal methods
+    def _initialize_assistant(self, assistant_id):
+        file_ids = self._generate_file_ids(self.data_sources)
+        self.assistant = (
+            self._client.beta.assistants.retrieve(assistant_id)
+            if assistant_id
+            else self._client.beta.assistants.create(
+                name=self.name, model=self.model, file_ids=file_ids, instructions=self.instructions, tools=self.tools
+            )
+        )
+
+    def _create_thread(self):
+        thread = self._client.beta.threads.create()
+        return thread.id
+
+    def _prepare_source_path(self, source, data_type=None):
+        if Path(source).is_file():
+            return source
+        data_type = data_type or detect_datatype(source)
+        formatter = DataFormatter(data_type=DataType(data_type), config=AddConfig(), kwargs={})
+        data = formatter.loader.load_data(source)["data"]
+        return self._save_temp_data(data=data[0]["content"].encode(), source=source)
+
+    def _add_file_to_assistant(self, file_path):
+        file_obj = self._client.files.create(file=open(file_path, "rb"), purpose="assistants")
+        self._client.beta.assistants.files.create(assistant_id=self.assistant.id, file_id=file_obj.id)
+
+    def _generate_file_ids(self, data_sources):
+        return [
+            self._add_file_to_assistant(self._prepare_source_path(ds["source"], ds.get("data_type")))
+            for ds in data_sources
+        ]
+
+    def _send_message(self, message):
+        self._client.beta.threads.messages.create(thread_id=self.thread_id, role="user", content=message)
+        self._wait_for_completion()
+
+    def _wait_for_completion(self):
+        run = self._client.beta.threads.runs.create(
+            thread_id=self.thread_id,
+            assistant_id=self.assistant.id,
+            instructions=self.instructions,
+        )
+        run_id = run.id
+        run_status = run.status
+
+        while run_status in ["queued", "in_progress", "requires_action"]:
+            time.sleep(0.1)  # Sleep before making the next API call to avoid hitting rate limits
+            run = self._client.beta.threads.runs.retrieve(thread_id=self.thread_id, run_id=run_id)
+            run_status = run.status
+            if run_status == "failed":
+                raise ValueError(f"Thread run failed with the following error: {run.last_error}")
+
+    def _get_latest_response(self):
+        history = self._get_history()
+        return self._format_message(history[0]) if history else None
+
+    def _get_history(self):
+        messages = self._client.beta.threads.messages.list(thread_id=self.thread_id, order="desc")
+        return list(messages)
+
+    def _format_message(self, thread_message):
+        thread_message = cast(ThreadMessage, thread_message)
+        content = [c.text.value for c in thread_message.content if isinstance(c, MessageContentText)]
+        return " ".join(content)
+
+    def _save_temp_data(self, data, source):
+        special_chars_pattern = r'[\\/:*?"<>|&=% ]+'
+        sanitized_source = re.sub(special_chars_pattern, "_", source)[:256]
+        temp_dir = tempfile.mkdtemp()
+        file_path = os.path.join(temp_dir, sanitized_source)
+        with open(file_path, "wb") as file:
+            file.write(data)
+        return file_path
+
+
+class AIAssistant:
+    def __init__(
+        self,
+        name=None,
+        instructions=None,
+        yaml_path=None,
+        assistant_id=None,
+        thread_id=None,
+        data_sources=None,
+        log_level=logging.WARN,
+        collect_metrics=True,
+    ):
+        logging.basicConfig(level=log_level)
+
+        self.name = name or "AI Assistant"
+        self.data_sources = data_sources or []
+        self.log_level = log_level
+        self.instructions = instructions
+        self.assistant_id = assistant_id or str(uuid.uuid4())
+        self.thread_id = thread_id or str(uuid.uuid4())
+        self.pipeline = Pipeline.from_config(yaml_path=yaml_path) if yaml_path else Pipeline()
+        self.pipeline.local_id = self.pipeline.config.id = self.thread_id
+
+        if self.instructions:
+            self.pipeline.system_prompt = self.instructions
+
+        print(
+            f"🎉 Created AI Assistant with name: {self.name}, assistant_id: {self.assistant_id}, thread_id: {self.thread_id}"  # noqa: E501
+        )
+
+        # telemetry related properties
+        self._telemetry_props = {"class": self.__class__.__name__}
+        self.telemetry = AnonymousTelemetry(enabled=collect_metrics)
+        self.telemetry.capture(event_name="init", properties=self._telemetry_props)
+
+        if self.data_sources:
+            for data_source in self.data_sources:
+                metadata = {"assistant_id": self.assistant_id, "thread_id": "global_knowledge"}
+                self.pipeline.add(data_source["source"], data_source.get("data_type"), metadata=metadata)
+
+    def add(self, source, data_type=None):
+        metadata = {"assistant_id": self.assistant_id, "thread_id": self.thread_id}
+        self.pipeline.add(source, data_type=data_type, metadata=metadata)
+        event_props = {
+            **self._telemetry_props,
+            "data_type": data_type or detect_datatype(source),
+        }
+        self.telemetry.capture(event_name="add", properties=event_props)
+
+    def chat(self, query):
+        where = {
+            "$and": [
+                {"assistant_id": {"$eq": self.assistant_id}},
+                {"thread_id": {"$in": [self.thread_id, "global_knowledge"]}},
+            ]
+        }
+        return self.pipeline.chat(query, where=where)
+
+    def delete(self):
+        self.pipeline.reset()
@@ -20,7 +20,7 @@ class AnonymousTelemetry:
        self.project_api_key = "phc_PHQDA5KwztijnSojsxJ2c1DuJd52QCzJzT2xnSGvjN2"
        self.host = host
        self.posthog = Posthog(project_api_key=self.project_api_key, host=self.host)
-        self.user_id = self.get_user_id()
+        self.user_id = self._get_user_id()
        self.enabled = enabled

        # Check if telemetry tracking is disabled via environment variable
@@ -38,7 +38,7 @@ class AnonymousTelemetry:
        posthog_logger = logging.getLogger("posthog")
        posthog_logger.disabled = True

-    def get_user_id(self):
+    def _get_user_id(self):
        if not os.path.exists(CONFIG_DIR):
            os.makedirs(CONFIG_DIR)

@@ -10,6 +10,62 @@ from schema import Optional, Or, Schema
 from embedchain.models.data_type import DataType


+def parse_content(content, type):
+    implemented = ["html.parser", "lxml", "lxml-xml", "xml", "html5lib"]
+    if type not in implemented:
+        raise ValueError(f"Parser type {type} not implemented. Please choose one of {implemented}")
+
+    from bs4 import BeautifulSoup
+
+    soup = BeautifulSoup(content, type)
+    original_size = len(str(soup.get_text()))
+
+    tags_to_exclude = [
+        "nav",
+        "aside",
+        "form",
+        "header",
+        "noscript",
+        "svg",
+        "canvas",
+        "footer",
+        "script",
+        "style",
+    ]
+    for tag in soup(tags_to_exclude):
+        tag.decompose()
+
+    ids_to_exclude = ["sidebar", "main-navigation", "menu-main-menu"]
+    for id in ids_to_exclude:
+        tags = soup.find_all(id=id)
+        for tag in tags:
+            tag.decompose()
+
+    classes_to_exclude = [
+        "elementor-location-header",
+        "navbar-header",
+        "nav",
+        "header-sidebar-wrapper",
+        "blog-sidebar-wrapper",
+        "related-posts",
+    ]
+    for class_name in classes_to_exclude:
+        tags = soup.find_all(class_=class_name)
+        for tag in tags:
+            tag.decompose()
+
+    content = soup.get_text()
+    content = clean_string(content)
+
+    cleaned_size = len(content)
+    if original_size != 0:
+        logging.info(
+            f"Cleaned page size: {cleaned_size} characters, down from {original_size} (shrunk: {original_size-cleaned_size} chars, {round((1-(cleaned_size/original_size)) * 100, 2)}%)"  # noqa:E501
+        )
+
+    return content
+
+
 def clean_string(text):
    """
    This function takes in a string and performs a series of text cleaning operations.
@@ -138,7 +194,8 @@ def detect_datatype(source: Any) -> DataType:
    formatted_source = format_source(str(source), 30)

    if url:
-        from langchain.document_loaders.youtube import ALLOWED_NETLOCK as YOUTUBE_ALLOWED_NETLOCS
+        from langchain.document_loaders.youtube import \
+            ALLOWED_NETLOCK as YOUTUBE_ALLOWED_NETLOCS

        if url.netloc in YOUTUBE_ALLOWED_NETLOCS:
            logging.debug(f"Source of `{formatted_source}` detected as `youtube_video`.")
@@ -160,6 +217,10 @@ def detect_datatype(source: Any) -> DataType:
            logging.debug(f"Source of `{formatted_source}` detected as `csv`.")
            return DataType.CSV

+        if url.path.endswith(".mdx") or url.path.endswith(".md"):
+            logging.debug(f"Source of `{formatted_source}` detected as `mdx`.")
+            return DataType.MDX
+
        if url.path.endswith(".docx"):
            logging.debug(f"Source of `{formatted_source}` detected as `docx`.")
            return DataType.DOCX
@@ -199,6 +260,10 @@ def detect_datatype(source: Any) -> DataType:
            logging.debug(f"Source of `{formatted_source}` detected as `docs_site`.")
            return DataType.DOCS_SITE

+        if "github.com" in url.netloc:
+            logging.debug(f"Source of `{formatted_source}` detected as `github`.")
+            return DataType.GITHUB
+
        # If none of the above conditions are met, it's a general web page
        logging.debug(f"Source of `{formatted_source}` detected as `web_page`.")
        return DataType.WEB_PAGE
@@ -232,6 +297,10 @@ def detect_datatype(source: Any) -> DataType:
            logging.debug(f"Source of `{formatted_source}` detected as `xml`.")
            return DataType.XML

+        if source.endswith(".mdx") or source.endswith(".md"):
+            logging.debug(f"Source of `{formatted_source}` detected as `mdx`.")
+            return DataType.MDX
+
        if source.endswith(".yaml"):
            with open(source, "r") as file:
                yaml_content = yaml.safe_load(file)
@@ -308,7 +377,7 @@ def validate_yaml_config(config_data):
                    "gpt4all",
                    "jina",
                    "llama2",
-                    "vertex_ai",
+                    "vertexai",
                ),
                Optional("config"): {
                    Optional("model"): str,
@@ -328,28 +397,27 @@ def validate_yaml_config(config_data):
                Optional("provider"): Or(
                    "chroma", "elasticsearch", "opensearch", "pinecone", "qdrant", "weaviate", "zilliz"
                ),
-                Optional("config"): {
-                    Optional("collection_name"): str,
-                    Optional("dir"): str,
-                    Optional("allow_reset"): bool,
-                    Optional("host"): str,
-                    Optional("port"): str,
-                },
+                Optional("config"): object,  # TODO: add particular config schema for each provider
            },
            Optional("embedder"): {
-                Optional("provider"): Or("openai", "gpt4all", "huggingface", "vertexai"),
+                Optional("provider"): Or("openai", "gpt4all", "huggingface", "vertexai", "azure_openai"),
                Optional("config"): {
                    Optional("model"): Optional(str),
                    Optional("deployment_name"): Optional(str),
                },
            },
            Optional("embedding_model"): {
-                Optional("provider"): Or("openai", "gpt4all", "huggingface", "vertexai"),
+                Optional("provider"): Or("openai", "gpt4all", "huggingface", "vertexai", "azure_openai"),
                Optional("config"): {
                    Optional("model"): str,
                    Optional("deployment_name"): str,
                },
            },
+            Optional("chunker"): {
+                Optional("chunk_size"): int,
+                Optional("chunk_overlap"): int,
+                Optional("length_function"): str,
+            },
        }
    )

@@ -3,6 +3,7 @@ from typing import Any, Dict, List, Optional, Tuple, Union

 from chromadb import Collection, QueryResult
 from langchain.docstore.document import Document
+from tqdm import tqdm

 from embedchain.config import ChromaDbConfig
 from embedchain.helper.json_serializable import register_deserializable
@@ -77,7 +78,7 @@ class ChromaDB(BaseVectorDB):
    def _generate_where_clause(self, where: Dict[str, any]) -> str:
        # If only one filter is supplied, return it as is
        # (no need to wrap in $and based on chroma docs)
-        if len(where.keys()) == 1:
+        if len(where.keys()) <= 1:
            return where
        where_filters = []
        for k, v in where.items():
@@ -157,8 +158,7 @@ class ChromaDB(BaseVectorDB):
                " Ids size: {}".format(len(documents), len(metadatas), len(ids))
            )

-        for i in range(0, len(documents), self.BATCH_SIZE):
-            print("Inserting batches from {} to {} in chromadb".format(i, min(len(documents), i + self.BATCH_SIZE)))
+        for i in tqdm(range(0, len(documents), self.BATCH_SIZE), desc="Inserting batches in chromadb"):
            if skip_embedding:
                self.collection.add(
                    embeddings=embeddings[i : i + self.BATCH_SIZE],
@@ -224,7 +224,7 @@ class ChromaDB(BaseVectorDB):
                        input_query,
                    ],
                    n_results=n_results,
-                    where=where,
+                    where=self._generate_where_clause(where),
                )
            else:
                result = self.collection.query(
@@ -232,7 +232,7 @@ class ChromaDB(BaseVectorDB):
                        input_query,
                    ],
                    n_results=n_results,
-                    where=where,
+                    where=self._generate_where_clause(where),
                )
        except InvalidDimensionException as e:
            raise InvalidDimensionException(
@@ -275,7 +275,7 @@ class ChromaDB(BaseVectorDB):
        return self.collection.count()

    def delete(self, where):
-        return self.collection.delete(where=where)
+        return self.collection.delete(where=self._generate_where_clause(where))

    def reset(self):
        """
@@ -1,6 +1,9 @@
 import logging
+import time
 from typing import Dict, List, Optional, Set, Tuple, Union

+from tqdm import tqdm
+
 try:
    from opensearchpy import OpenSearch
    from opensearchpy.helpers import bulk
@@ -23,6 +26,8 @@ class OpenSearchDB(BaseVectorDB):
    OpenSearch as vector database
    """

+    BATCH_SIZE = 100
+
    def __init__(self, config: OpenSearchDBConfig):
        """OpenSearch as vector database.

@@ -131,19 +136,28 @@ class OpenSearchDB(BaseVectorDB):
        :type skip_embedding: bool
        """

-        docs = []
-        if not skip_embedding:
-            embeddings = self.embedder.embedding_fn(documents)
-        for id, text, metadata, embeddings in zip(ids, documents, metadatas, embeddings):
-            docs.append(
-                {
-                    "_index": self._get_index(),
-                    "_id": id,
-                    "_source": {"text": text, "metadata": metadata, "embeddings": embeddings},
-                }
-            )
-        bulk(self.client, docs)
-        self.client.indices.refresh(index=self._get_index())
+        for i in tqdm(range(0, len(documents), self.BATCH_SIZE), desc="Inserting batches in opensearch"):
+            if not skip_embedding:
+                embeddings = self.embedder.embedding_fn(documents[i : i + self.BATCH_SIZE])
+
+            docs = []
+            for id, text, metadata, embeddings in zip(
+                ids[i : i + self.BATCH_SIZE],
+                documents[i : i + self.BATCH_SIZE],
+                metadatas[i : i + self.BATCH_SIZE],
+                embeddings[i : i + self.BATCH_SIZE],
+            ):
+                docs.append(
+                    {
+                        "_index": self._get_index(),
+                        "_id": id,
+                        "_source": {"text": text, "metadata": metadata, "embeddings": embeddings},
+                    }
+                )
+            bulk(self.client, docs)
+            self.client.indices.refresh(index=self._get_index())
+            # Sleep for 0.1 seconds to avoid rate limiting
+            time.sleep(0.1)

    def query(
        self,
@@ -222,3 +222,16 @@ class ZillizVectorDB(BaseVectorDB):
        if not isinstance(name, str):
            raise TypeError("Collection name must be a string")
        self.config.collection_name = name
+
+    def delete(self, keys: Union[list, str, int]):
+        """
+        Delete the embeddings from DB. Zilliz only support deleting with keys.
+
+
+        :param keys: Primary keys of the table entries to delete.
+        :type keys: Union[list, str, int]
+        """
+        self.client.delete(
+            collection_name=self.config.collection_name,
+            pks=keys,
+        )
@@ -1,4 +1,4 @@
-FROM python:3.11 AS backend
+FROM python:3.11-slim

 WORKDIR /usr/src/discord_bot
 COPY requirements.txt .
@@ -1,3 +1,9 @@
 # Discord Bot

-This is a docker template to create your own Discord bot using the embedchain package. To know more about the bot and how to use it, go [here](https://docs.embedchain.ai/examples/discord_bot).
+This is a docker template to create your own Discord bot using the embedchain package. To know more about the bot and how to use it, go [here](https://docs.embedchain.ai/examples/discord_bot).
+
+To run this use the following command,
+
+```bash
+docker run --name discord-bot -e OPENAI_API_KEY=sk-xxx -e DISCORD_BOT_TOKEN=xxx -p 8080:8080 embedchain/discord-bot:latest
+```
@@ -1,4 +1,4 @@
-FROM python:3.11 AS backend
+FROM python:3.11-slim AS backend

 WORKDIR /usr/src/app/backend
 COPY requirements.txt .
@@ -2,20 +2,22 @@ version: "3.9"

 services:
  backend:
-    container_name: embedchain_backend
+    container_name: embedchain-backend
    restart: unless-stopped
    build:
      context: backend
      dockerfile: Dockerfile
+    image: embedchain/backend
    ports:
      - "8000:8000"

  frontend:
-    container_name: embedchain_frontend
+    container_name: embedchain-frontend
    restart: unless-stopped
    build:
      context: frontend
      dockerfile: Dockerfile
+    image: embedchain/frontend
    ports:
      - "3000:3000"
    depends_on:
@@ -1,4 +1,4 @@
-FROM node:18 AS frontend
+FROM node:18-slim AS frontend

 WORKDIR /usr/src/app/frontend
 COPY package.json .
@@ -5,7 +5,7 @@ app:
 llm:
  provider: gpt4all
  config:
-    model: 'orca-mini-3b.ggmlv3.q4_0.bin'
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
    temperature: 0.5
    max_tokens: 1000
    top_p: 1
@@ -83,7 +83,7 @@ async def create_app_using_default_config(app_id: str, config: UploadFile = None

        return DefaultResponse(response=f"App created successfully. App ID: {app_id}")
    except Exception as e:
-        logging.warn(str(e))
+        logging.warning(str(e))
        raise HTTPException(detail=f"Error creating app: {str(e)}", status_code=400)


@@ -113,13 +113,13 @@ async def get_datasources_associated_with_app_id(app_id: str, db: Session = Depe
        response = app.get_data_sources()
        return {"results": response}
    except ValueError as ve:
-        logging.warn(str(ve))
+        logging.warning(str(ve))
        raise HTTPException(
            detail=generate_error_message_for_api_keys(ve),
            status_code=400,
        )
    except Exception as e:
-        logging.warn(str(e))
+        logging.warning(str(e))
        raise HTTPException(detail=f"Error occurred: {str(e)}", status_code=400)


@@ -152,13 +152,13 @@ async def add_datasource_to_an_app(body: SourceApp, app_id: str, db: Session = D
        response = app.add(source=body.source, data_type=body.data_type)
        return DefaultResponse(response=response)
    except ValueError as ve:
-        logging.warn(str(ve))
+        logging.warning(str(ve))
        raise HTTPException(
            detail=generate_error_message_for_api_keys(ve),
            status_code=400,
        )
    except Exception as e:
-        logging.warn(str(e))
+        logging.warning(str(e))
        raise HTTPException(detail=f"Error occurred: {str(e)}", status_code=400)


@@ -190,13 +190,13 @@ async def query_an_app(body: QueryApp, app_id: str, db: Session = Depends(get_db
        response = app.query(body.query)
        return DefaultResponse(response=response)
    except ValueError as ve:
-        logging.warn(str(ve))
+        logging.warning(str(ve))
        raise HTTPException(
            detail=generate_error_message_for_api_keys(ve),
            status_code=400,
        )
    except Exception as e:
-        logging.warn(str(e))
+        logging.warning(str(e))
        raise HTTPException(detail=f"Error occurred: {str(e)}", status_code=400)


@@ -273,13 +273,13 @@ async def deploy_app(body: DeployAppRequest, app_id: str, db: Session = Depends(
        app.deploy()
        return DefaultResponse(response="App deployed successfully.")
    except ValueError as ve:
-        logging.warn(str(ve))
+        logging.warning(str(ve))
        raise HTTPException(
            detail=generate_error_message_for_api_keys(ve),
            status_code=400,
        )
    except Exception as e:
-        logging.warn(str(e))
+        logging.warning(str(e))
        raise HTTPException(detail=f"Error occurred: {str(e)}", status_code=400)


@@ -1,6 +1,6 @@
 fastapi==0.104.0
 uvicorn==0.23.2
-embedchain==0.0.91
-embedchain[streamlit, community, opensource, elasticsearch, opensearch, poe, discord, slack, whatsapp, weaviate, pinecone, qdrant, images, huggingface_hub, cohere, milvus, dataloaders, vertexai, llama2, gmail, json]==0.0.91
+embedchain==0.1.3
+embedchain[streamlit, community, opensource, elasticsearch, opensearch, poe, discord, slack, whatsapp, weaviate, pinecone, qdrant, images, huggingface_hub, cohere, milvus, dataloaders, vertexai, llama2, gmail, json]==0.1.3
 sqlalchemy==2.0.22
 python-multipart==0.0.6
@@ -0,0 +1,11 @@
+FROM python:3.11-slim
+
+WORKDIR /usr/src/
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . .
+
+EXPOSE 8000
+
+CMD ["python", "-m", "embedchain.bots.slack", "--port", "8000"]
@@ -0,0 +1 @@
+embedchain[slack, poe]==0.1.7
@@ -0,0 +1,2 @@
+TELEGRAM_BOT_TOKEN=
+OPENAI_API_KEY=
@@ -0,0 +1,11 @@
+FROM python:3.11-slim
+
+WORKDIR /usr/src/
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . .
+
+EXPOSE 8000
+
+CMD ["python", "telegram_bot.py"]
@@ -63,4 +63,4 @@ def send_message(chat_id, text):


 if __name__ == "__main__":
-    app.run(host="0.0.0.0", port=5000, debug=False)
+    app.run(host="0.0.0.0", port=8000, debug=False)
@@ -1,2 +0,0 @@
-TELEGRAM_BOT_TOKEN=""
-OPENAI_API_KEY=""
@@ -0,0 +1 @@
+OPENAI_API_KEY=
@@ -0,0 +1,11 @@
+FROM python:3.11-slim
+
+WORKDIR /usr/src/
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . .
+
+EXPOSE 8000
+
+CMD ["python", "whatsapp_bot.py"]
@@ -1 +0,0 @@
-OPENAI_API_KEY=""
@@ -48,4 +48,4 @@ def query(message):


 if __name__ == "__main__":
-    app.run(host="0.0.0.0", port=5000, debug=False)
+    app.run(host="0.0.0.0", port=8000, debug=False)
@@ -100,7 +100,7 @@
        "llm:\n",
        "  provider: gpt4all\n",
        "  config:\n",
-        "    model: 'orca-mini-3b.ggmlv3.q4_0.bin'\n",
+        "    model: 'orca-mini-3b-gguf2-q4_0.gguf'\n",
        "    temperature: 0.5\n",
        "    max_tokens: 1000\n",
        "    top_p: 1\n",
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "embedchain"
-version = "0.0.92"
+version = "0.1.18"
 description = "Data platform for LLMs - Load, index, retrieve and sync any unstructured data"
 authors = [
    "Taranjeet Singh <taranjeet@embedchain.ai>",
@@ -88,12 +88,12 @@ exclude = '''
 color = true

 [tool.poetry.dependencies]
-python = ">=3.9,<3.13"
+python = ">=3.9,<3.12"
 python-dotenv = "^1.0.0"
-langchain = "^0.0.303"
+langchain = "^0.0.336"
 requests = "^2.31.0"
-openai = ">=0.28.0"
-chromadb = "^0.4.8"
+openai = ">=1.1.1"
+chromadb = "^0.4.17"
 posthog = "^3.0.2"
 tiktoken = { version = "^0.4.0", optional = true }
 youtube-transcript-api = { version = "^0.6.1", optional = true }
@@ -101,11 +101,12 @@ beautifulsoup4 = { version = "^4.12.2", optional = true }
 pypdf = { version = "^3.11.0", optional = true }
 pytube = { version = "^15.0.0", optional = true }
 duckduckgo-search = { version = "^3.8.5", optional = true }
-llama-hub = { version = "^0.0.29", optional = true }
+llama-hub = { version = "^0.0.43", optional = true }
+llama-index = { version = "^0.8.65", optional = true }
 sentence-transformers = { version = "^2.2.2", optional = true }
 torch = { version = "2.0.0", optional = true }
 # Torch 2.0.1 is not compatible with poetry (https://github.com/pytorch/pytorch/issues/100974)
-gpt4all = { version = "1.0.8", optional = true }
+gpt4all = { version = "2.0.2", optional = true }
 # 1.0.9 is not working for some users (https://github.com/nomic-ai/gpt4all/issues/1394)
 opensearch-py = { version = "2.3.1", optional = true }
 elasticsearch = { version = "^8.9.0", optional = true }
@@ -119,7 +120,7 @@ weaviate-client = { version = "^3.24.1", optional = true }
 docx2txt = { version = "^0.8", optional = true }
 pinecone-client = { version = "^2.2.4", optional = true }
 qdrant-client = { version = "1.6.3", optional = true }
-unstructured = {extras = ["local-inference"], version = "^0.10.18", optional = true}
+unstructured = {extras = ["local-inference", "all-docs"], version = "^0.10.18", optional = true}
 pillow = { version = "10.0.1", optional = true }
 torchvision = { version = ">=0.15.1, !=0.15.2", optional = true }
 ftfy = { version = "6.1.1", optional = true }
@@ -129,6 +130,12 @@ pymilvus = { version = "2.3.1", optional = true }
 google-cloud-aiplatform = { version = "^1.26.1", optional = true }
 replicate = { version = "^0.15.4", optional = true }
 schema = "^0.7.5"
+psycopg = { version = "^3.1.12", optional = true }
+psycopg-binary = { version = "^3.1.12", optional = true }
+psycopg-pool = { version = "^3.1.8", optional = true }
+mysql-connector-python = { version = "^8.1.0", optional = true }
+gitpython = { version = "^3.1.38", optional = true }
+yt_dlp = { version = "^2023.11.14", optional = true }

 [tool.poetry.group.dev.dependencies]
 black = "^23.3.0"
@@ -162,7 +169,7 @@ huggingface_hub=["huggingface_hub"]
 cohere = ["cohere"]
 milvus = ["pymilvus"]
 dataloaders=[
-    "youtube-transcripts-api",
+    "youtube-transcript-api",
    "beautifulsoup4",
    "docx2txt",
    "duckduckgo-search",
@@ -183,9 +190,14 @@ gmail = [
    "google-api-core",
 ]
 json = ["llama-hub"]
+postgres = ["psycopg", "psycopg-binary", "psycopg-pool"]
+mysql = ["mysql-connector-python"]
+git = ["gitpython"]
+youtube = [
+    "yt_dlp",
+    "youtube-transcript-api",
+]

 [tool.poetry.group.docs.dependencies]

-
-
 [tool.poetry.scripts]
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Sidharth Mohanty	798d3fcc5a	Update version to 0.1.18 (#970 )	2023-11-21 10:01:09 -08:00
Sidharth Mohanty	85f3ac428b	Update embedding_fn signature to newest chroma db's (#969 )	2023-11-21 09:42:11 -08:00
Deshraj Yadav	9fcf2130b5	[Feature] Improve github and youtube channel loader (#966 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-17 18:25:14 -08:00
Taranjeet Singh	51df00729e	Import beautifulsoup pacakge lazily. (#964 )	2023-11-17 18:19:08 -08:00
Deven Patel	023a61446f	[Feature] Improve GitHub loader (#962 )	2023-11-16 22:06:36 -08:00
Deshraj Yadav	e0b73e6a5a	[Loaders] Improve web page and sitemap loader usability (#961 )	2023-11-16 16:01:43 -08:00
Deven Patel	28460f725c	[Bugfix] fix poetry lock (#960 )	2023-11-16 13:30:38 -08:00
Deshraj Yadav	c93e49d2b8	[Bug fix] Update sleep time for substack loader and version bump (#958 )	2023-11-15 19:35:30 -08:00
Deven Patel	07fb6bee54	[Features] Add Github and Youtube Channel loaders (#957 ) Co-authored-by: Deven Patel <deven298@yahoo.com> Co-authored-by: Deshraj Yadav <deshrajdry@gmail.com>	2023-11-15 19:17:42 -08:00
Deshraj Yadav	3fa7db8420	Bump version to 0.1.13 (#956 )	2023-11-15 18:42:48 -08:00
Deven Patel	c14bd7b73b	[Improvement] fix discourse loader to avoid rate limit (#953 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-15 15:33:16 -08:00
Sidharth Mohanty	5201beaab0	Bump version to `0.1.12` (#951 )	2023-11-15 09:33:26 -08:00
Sidharth Mohanty	122313d8a5	[New] Substack loader (#949 )	2023-11-14 21:52:15 -08:00
Deven Patel	82fd595306	[Improvements] improve package ux (#950 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-14 17:53:43 -08:00
Deven Patel	95c0d47236	[Feature] Discourse Loader (#948 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-13 16:39:11 -08:00
Deven Patel	919cc74e94	[Feature] Add MySQL Loader (#920 ) Co-authored-by: Deven Patel <deven298@yahoo.com> Co-authored-by: Deshraj Yadav <deshrajdry@gmail.com>	2023-11-13 13:21:36 -08:00
Deshraj Yadav	d839991acb	[Docs] Add back sitemap loader docstring (#947 )	2023-11-13 13:08:09 -08:00
Deven Patel	539286aafd	[Feature] Add Slack Loader (#932 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-13 13:06:01 -08:00
Sidharth Mohanty	23522b7b55	Add slack_bot docker image (#933 )	2023-11-13 13:04:04 -08:00
UnMonsieur	bf3fac56e4	Refactor: Make it clear what methods are private (#946 )	2023-11-13 13:00:13 -08:00
Deshraj Yadav	a5bf8e9075	[Improvement] Parallelize loading of sitemap urls	2023-11-13 12:53:34 -08:00
Deshraj Yadav	1d31b8f7e4	[Bugfix] Fix issue of "unable to open database file" (#945 )	2023-11-13 12:37:00 -08:00
Deshraj Yadav	b144c7dccc	[Bugfix] Fix hugging face command (#944 )	2023-11-13 12:17:27 -08:00
Deshraj Yadav	1364975396	[Feature] Add support for AIAssistant (#938 )	2023-11-10 16:47:34 -08:00
Deven Patel	deaa7f50f8	[Bug Fix] fix chromadb where clause for query and delete (#937 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-10 16:04:25 -08:00
Sidharth Mohanty	744ab5156f	[Bug fix] missing dir on first init for App (#934 )	2023-11-10 10:15:04 -08:00
Sidharth Mohanty	c45413969a	[refactor] Use pipeline for bots instead of App (#936 )	2023-11-10 10:13:21 -08:00
Deshraj Yadav	b314e5e080	[bug] Fix issue of missing user directory on first init (#931 )	2023-11-09 22:29:41 -08:00
Deshraj Yadav	17129e2eaa	[Improvement] Add support for reloading history for an existing app (#930 )	2023-11-09 15:17:51 -08:00
Deven Patel	654fd8d74c	[Improvement] Use SQLite for chat memory (#910 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-09 13:56:28 -08:00
Sidharth Mohanty	9d3568ef75	Update package version to 0.1.3 (#928 )	2023-11-09 11:33:18 -08:00
Sidharth Mohanty	14712cac88	Deploy remaining bots and fix schema validation (#927 )	2023-11-09 10:44:47 -08:00
Deshraj Yadav	0d568c758b	[Feat] Add anonymous telemetry to assistant (#924 )	2023-11-09 02:07:47 -08:00
Deshraj Yadav	7c6b88c7c5	[Docs] Update docs and improve assistant api (#923 )	2023-11-09 01:16:19 -08:00
Deshraj Yadav	32c93be46e	[chore] update poetry.lock file (#922 )	2023-11-09 00:59:29 -08:00
Deven Patel	7de8d85199	[Feature] Add Postgres data loader (#918 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-08 23:50:46 -08:00
Deshraj Yadav	f7dd65a3de	[Feature] Add support for OpenAI assistants and support openai version >=1.0.0 (#921 )	2023-11-08 22:49:03 -08:00
Sidharth Mohanty	d8cdbe0041	Set `check_same_thread` false so that one App can be used in parallel (#911 )	2023-11-08 10:26:31 -08:00
Sidharth Mohanty	2b8b6d3ea9	Chunker config docs (#913 )	2023-11-08 10:25:45 -08:00
Sidharth Mohanty	936c7e389f	Deploy Full stack docker image (#914 )	2023-11-08 10:25:12 -08:00
Sidharth Mohanty	6864b4207b	Dockerize discord bot and update docs to run the bot correctly (#919 )	2023-11-08 10:24:57 -08:00
Deshraj Yadav	98eb5b54be	[Docs] Update developer documentation (#916 )	2023-11-07 19:07:47 -08:00
Deshraj Yadav	3332e6e236	[Docs] Update README (#915 )	2023-11-07 18:12:27 -08:00
Deven Patel	0533da72d7	[Improvement] add delete functionality to zilliz DB (#912 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-07 02:46:39 -08:00
Sidharth Mohanty	a1de238716	Introduce chunker config in yaml config (#907 )	2023-11-06 09:43:15 -08:00