[BugFix] Skip checking thread when making sqlite connection (#978)

Co-authored-by: Deven Patel <deven298@yahoo.com>

.env.example

@@ -0,0 +1 @@
+OPENAI_API_KEY="your-openai-api-key"

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,41 @@
+name: 🐛 Bug Report
+description: Create a report to help us reproduce and fix the bug
+
+body:
+- type: markdown
+  attributes:
+    value: >
+      #### Before submitting a bug, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/gventuri/pandas-ai/issues?q=is%3Aissue+sort%3Acreated-desc+).
+- type: textarea
+  attributes:
+    label: 🐛 Describe the bug
+    description: |
+      Please provide a clear and concise description of what the bug is.
+
+      If relevant, add a minimal example so that we can reproduce the error by running the code. It is very important for the snippet to be as succinct (minimal) as possible, so please take time to trim down any irrelevant code to help us debug efficiently. We are going to copy-paste your code and we expect to get the same result as you did: avoid any external data, and include the relevant imports, etc. For example:
+
+      ```python
+      # All necessary imports at the beginning
+      import embedchain as ec
+      # Your code goes here
+
+
+      ```
+
+      Please also paste or describe the results you observe instead of the expected results. If you observe an error, please paste the error message including the **full** traceback of the exception. It may be relevant to wrap error messages in ```` ```triple quotes blocks``` ````.
+    placeholder: |
+      A clear and concise description of what the bug is.
+
+      ```python
+      Sample code to reproduce the problem
+      ```
+
+      ```
+      The error message you got, with the full traceback.
+      ````
+  validations:
+    required: true
+- type: markdown
+  attributes:
+    value: >
+      Thanks for contributing 🎉!

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: true
+contact_links:
+  - name: 1-on-1 Session
+    url: https://cal.com/taranjeetio/ec
+    about: Speak directly with Taranjeet, the founder, to discuss issues, share feedback, or explore improvements for Embedchain
+  - name: Discord
+    url: https://discord.gg/6PzXDgEjG5
+    about: General community discussions

.github/ISSUE_TEMPLATE/documentation_issue.yml

@@ -0,0 +1,11 @@
+name: Documentation
+description: Report an issue related to the Embedchain docs.
+title: "DOC: <Please write a comprehensive title after the 'DOC: ' prefix>"
+
+body:
+- type: textarea
+  attributes:
+    label: "Issue with current documentation:"
+    description: >
+      Please make sure to leave a reference to the document/code you're
+      referring to.

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,23 @@
+name: 🚀 Feature request
+description: Submit a proposal/request for a new Embedchain feature
+
+body:
+- type: textarea
+  id: feature-request
+  attributes:
+    label: 🚀 The feature
+    description: >
+      A clear and concise description of the feature proposal
+  validations:
+    required: true
+- type: textarea
+  attributes:
+    label: Motivation, pitch
+    description: >
+      Please outline the motivation for the proposal. Is your feature request related to a specific problem? e.g., *"I'm working on X and would like Y to be possible"*. If this is related to another GitHub issue, please link here too.
+  validations:
+    required: true
+- type: markdown
+  attributes:
+    value: >
+      Thanks for contributing 🎉!

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,41 @@
+## Description
+
+Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
+
+Fixes # (issue)
+
+## Type of change
+
+Please delete options that are not relevant.
+
+- [ ] Bug fix (non-breaking change which fixes an issue)
+- [ ] New feature (non-breaking change which adds functionality)
+- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
+- [ ] Refactor (does not change functionality, e.g. code style improvements, linting)
+- [ ] Documentation update
+
+## How Has This Been Tested?
+
+Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
+
+Please delete options that are not relevant.
+
+- [ ] Unit Test
+- [ ] Test Script (please provide)
+
+## Checklist:
+
+- [ ] My code follows the style guidelines of this project
+- [ ] I have performed a self-review of my own code
+- [ ] I have commented my code, particularly in hard-to-understand areas
+- [ ] I have made corresponding changes to the documentation
+- [ ] My changes generate no new warnings
+- [ ] I have added tests that prove my fix is effective or that my feature works
+- [ ] New and existing unit tests pass locally with my changes
+- [ ] Any dependent changes have been merged and published in downstream modules
+- [ ] I have checked my code and corrected any misspellings
+
+## Maintainer Checklist
+
+- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
+- [ ] Made sure Checks passed

.github/workflows/cd.yml

@@ -0,0 +1,40 @@
+name: Publish Python 🐍 distributions 📦 to PyPI and TestPyPI
+
+on:
+  release:
+    types: [published] # This will trigger the workflow when you create a new release
+
+jobs:
+  build-n-publish:
+    name: Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI
+    runs-on: ubuntu-latest
+    permissions:
+      # IMPORTANT: this permission is mandatory for trusted publishing
+      id-token: write
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: '3.11'
+
+      - name: Install Poetry
+        run: |
+          curl -sSL https://install.python-poetry.org | python3 -
+          echo "$HOME/.local/bin" >> $GITHUB_PATH
+
+      - name: Install dependencies
+        run: poetry install
+
+      - name: Build a binary wheel and a source tarball
+        run: poetry build
+
+      - name: Publish distribution 📦 to Test PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository_url: https://test.pypi.org/legacy/
+
+      - name: Publish distribution 📦 to PyPI
+        if: startsWith(github.ref, 'refs/tags')
+        uses: pypa/gh-action-pypi-publish@release/v1

.github/workflows/ci.yml

@@ -0,0 +1,53 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'embedchain/**'
+      - 'tests/**'
+      - 'examples/**'
+  pull_request:
+    paths:
+      - 'embedchain/**'
+      - 'tests/**'
+      - 'examples/**'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.10", "3.11"]
+
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install poetry
+        uses: snok/install-poetry@v1
+        with:
+          version: 1.4.2
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v2
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
+      - name: Install dependencies
+        run: poetry install --all-extras
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+      - name: Lint with ruff
+        run: make lint
+      - name: Run tests and generate coverage report
+        run: make coverage
+      - name: Upload coverage reports to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          file: coverage.xml
+        env:
+          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

.gitignore

@@ -76,7 +76,6 @@ docs/_build/
 target/
 
 # Jupyter Notebook
-.ipynb_checkpoints
 
 # IPython
 profile_default/
@@ -165,5 +164,14 @@ cython_debug/
 
 # Database
 db
+test-db
 
-.vscode
+.vscode
+.idea/
+
+.DS_Store
+
+notebooks/*.yaml
+.ipynb_checkpoints/
+
+!configs/*.yaml

.pre-commit-config.yaml

@@ -0,0 +1,20 @@
+repos:
+  - repo: https://github.com/psf/black
+    rev: 23.3.0
+    hooks:
+      - id: black
+  - repo: https://github.com/charliermarsh/ruff-pre-commit
+    rev: 'v0.0.220'
+    hooks:
+      - id: ruff
+        name: ruff
+        # Respect `exclude` and `extend-exclude` settings.
+        args: ["--force-exclude"]
+  - repo: local
+    hooks:
+      - id: pytest-check
+        name: pytest-check
+        entry: poetry run pytest
+        language: system
+        pass_filenames: false
+        always_run: true

CONTRIBUTING.md

@@ -0,0 +1,74 @@
+# Contributing to embedchain
+
+Let us make contribution easy, collaborative and fun.
+
+## Submit your Contribution through PR
+
+To make a contribution, follow these steps:
+
+1. Fork and clone this repository
+2. Do the changes on your fork with dedicated feature branch `feature/f1`
+3. If you modified the code (new feature or bug-fix), please add tests for it
+4. Include proper documentation / docstring and examples to run the feature
+5. Check the linting
+6. Ensure that all tests pass
+7. Submit a pull request
+
+For more details about pull requests, please read [GitHub's guides](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request).
+
+
+### 📦 Package manager
+
+We use `poetry` as our package manager. You can install poetry by following the instructions [here](https://python-poetry.org/docs/#installation).
+
+Please DO NOT use pip or conda to install the dependencies. Instead, use poetry:
+
+```bash
+poetry install --all-extras
+or
+poetry install --with dev
+
+#activate
+
+poetry shell
+```
+
+### 📌 Pre-commit
+
+To ensure our standards, make sure to install pre-commit before starting to contribute.
+
+```bash
+pre-commit install
+```
+
+### 🧹 Linting
+
+We use `ruff` to lint our code. You can run the linter by running the following command:
+
+```bash
+make lint
+```
+
+Make sure that the linter does not report any errors or warnings before submitting a pull request.
+
+### Code Formatting with `black`
+
+We use `black` to reformat the code by running the following command:
+
+```bash
+make format
+```
+
+### 🧪 Testing
+
+We use `pytest` to test our code. You can run the tests by running the following command:
+
+```bash
+poetry run pytest
+```
+
+Make sure that all tests pass before submitting a pull request.
+
+## 🚀 Release Process
+
+At the moment, the release process is manual. We try to make frequent releases. Usually, we release a new version when we have a new feature or bugfix. A developer with admin rights to the repository will create a new release on GitHub, and then publish the new version to PyPI.

Makefile

@@ -4,21 +4,41 @@ PIP := $(PYTHON) -m pip
 PROJECT_NAME := embedchain
 
 # Targets
-.PHONY: install format lint clean test
+.PHONY: install format lint clean test ci_lint ci_test coverage
 
 install:
-	$(PIP) install --upgrade pip
-	$(PIP) install -e .[dev]
+	poetry install
+
+install_all:
+	poetry install --all-extras
+
+install_es:
+	poetry install --extras elasticsearch
+
+install_opensearch:
+	poetry install --extras opensearch
+
+install_milvus:
+	poetry install --extras milvus
+
+shell:
+	poetry shell
+
+py_shell:
+	poetry run python
 
 format:
 	$(PYTHON) -m black .
 	$(PYTHON) -m isort .
 
-lint:
-	$(PYTHON) -m ruff .
-
 clean:
 	rm -rf dist build *.egg-info
 
+lint:
+	poetry run ruff .
+
 test:
-	$(PYTHON) -m pytest
+	poetry run pytest $(file)
+
+coverage:
+	poetry run pytest --cov=$(PROJECT_NAME) --cov-report=xml

README.md

@@ -1,623 +1,128 @@
-# embedchain
+<p align="center">
+  <img src="docs/logo/dark.svg" width="400px" alt="Embedchain Logo">
+</p>
 
-[![PyPI](https://img.shields.io/pypi/v/embedchain)](https://pypi.org/project/embedchain/)
-[![Discord](https://dcbadge.vercel.app/api/server/nhvCbCtKV?style=flat)](https://discord.gg/6PzXDgEjG5)
-[![Twitter](https://img.shields.io/twitter/follow/embedchain)](https://twitter.com/embedchain)
-[![Substack](https://img.shields.io/badge/Substack-%23006f5c.svg?logo=substack)](https://embedchain.substack.com/)
+<p align="center">
+  <a href="https://runacap.com/ross-index/q3-2023/" target="_blank" rel="noopener"><img style="width: 260px; height: 56px" src="https://runacap.com/wp-content/uploads/2023/10/ROSS_badge_black_Q3_2023.svg" alt="ROSS Index - Fastest Growing Open-Source Startups in Q3 2023 | Runa Capital" width="260" height="56"/></a>
+</p>
 
-embedchain is a framework to easily create LLM powered bots over any dataset. If you want a javascript version, check out [embedchain-js](https://github.com/embedchain/embedchainjs)
+<p align="center">
+  <a href="https://pypi.org/project/embedchain/">
+    <img src="https://img.shields.io/pypi/v/embedchain" alt="PyPI">
+  </a>
+  <a href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw">
+    <img src="https://img.shields.io/badge/slack-embedchain-brightgreen.svg?logo=slack" alt="Slack">
+  </a>
+  <a href="https://discord.gg/CUU9FPhRNt">
+    <img src="https://dcbadge.vercel.app/api/server/6PzXDgEjG5?style=flat" alt="Discord">
+  </a>
+  <a href="https://twitter.com/embedchain">
+    <img src="https://img.shields.io/twitter/follow/embedchain" alt="Twitter">
+  </a>
+  <a href="https://embedchain.substack.com/">
+    <img src="https://img.shields.io/badge/Substack-%23006f5c.svg?logo=substack" alt="Substack">
+  </a>
+  <a href="https://colab.research.google.com/drive/138lMWhENGeEu7Q1-6lNbNTHGLZXBBz_B?usp=sharing">
+    <img src="https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667" alt="Open in Colab">
+  </a>
+  <a href="https://codecov.io/gh/embedchain/embedchain">
+    <img src="https://codecov.io/gh/embedchain/embedchain/graph/badge.svg?token=EMRRHZXW1Q" alt="codecov">
+  </a>
+</p>
 
-# Table of Contents
+<hr />
 
-- [Latest Updates](#latest-updates)
-- [What is embedchain?](#what-is-embedchain)
-- [Getting Started](#getting-started)
-  - [Installation](#installation)
-  - [Usage](#usage)
-    - [App Types](#app-types)
-      - [1. App (uses OpenAI models, paid)](#1-app-uses-openai-models-paid)
-      - [2. OpenSourceApp (uses opensource models, free)](#2-opensourceapp-uses-opensource-models-free)
-      - [3. PersonApp (uses OpenAI models, paid)](#3-personapp-uses-openai-models-paid)
-    - [Add Dataset](#add-dataset)
-  - [Interface Types](#interface-types)
-    - [Query Interface](#query-interface)
-    - [Chat Interface](#chat-interface)
-  - [Format supported](#format-supported)
-    - [Youtube Video](#youtube-video)
-    - [PDF File](#pdf-file)
-    - [Web Page](#web-page)
-    - [Doc File](#doc-file)
-    - [Text](#text)
-    - [QnA Pair](#qna-pair)
-    - [Reusing a Vector DB](#reusing-a-vector-db)
-    - [More Formats coming soon](#more-formats-coming-soon)
-  - [Testing](#testing)
-- [Advanced](#advanced)
-  - [Configuration](#configuration)
-    - [Example](#example)
-    - [Configs](#configs)
-      - [InitConfig](#initconfig)
-      - [Add Config](#add-config)
-      - [Query Config](#query-config)
-      - [Chat Config](#chat-config)
-  - [Other methods](#other-methods)
-    - [Reset](#reset)
-    - [Count](#count)
-- [How does it work?](#how-does-it-work)
-- [Contribution Guidelines](#contribution-guidelines)
-- [Tech Stack](#tech-stack)
-- [Team](#team)
-  - [Author](#author)
-  - [Maintainer](#maintainer)
-  - [Citation](#citation)
+## What is Embedchain?
+Embedchain is a Data Platform for Large Language Models (LLMs). Seamlessly load, index, retrieve, and sync unstructured data to build dynamic, LLM-powered applications. Check out [embedchain-js](https://github.com/embedchain/embedchain/tree/main/embedchain-js) for a JavaScript implementation.
 
-# Latest Updates
+## 🔧 Quick install
 
-- Introduce a new interface called `chat`. It remembers the history (last 5 messages) and can be used to powerful stateful bots. You can use it by calling `.chat` on any app instance. Works for both OpenAI and OpenSourceApp.
-
-- Introduce a new app type called `OpenSourceApp`. It uses `gpt4all` as the LLM and `sentence transformers` all-MiniLM-L6-v2 as the embedding model. If you use this app, you dont have to pay for anything.
-
-# What is embedchain?
-
-Embedchain abstracts the entire process of loading a dataset, chunking it, creating embeddings and then storing in a vector database.
-
-You can add a single or multiple dataset using `.add` and `.add_local` function and then use `.query` function to find an answer from the added datasets.
-
-If you want to create a Naval Ravikant bot which has 1 youtube video, 1 book as pdf and 2 of his blog posts, as well as a question and answer pair you supply, all you need to do is add the links to the videos, pdf and blog posts and the QnA pair and embedchain will create a bot for you.
-
-```python
-
-from embedchain import App
-
-naval_chat_bot = App()
-
-# Embed Online Resources
-naval_chat_bot.add("youtube_video", "https://www.youtube.com/watch?v=3qHkcs3kG44")
-naval_chat_bot.add("pdf_file", "https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
-naval_chat_bot.add("web_page", "https://nav.al/feedback")
-naval_chat_bot.add("web_page", "https://nav.al/agi")
-
-# Embed Local Resources
-naval_chat_bot.add_local("qna_pair", ("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."))
-
-naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?")
-# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
+### Python API
+```bash
+pip install --upgrade embedchain
 ```
 
-# Getting Started
-
-## Installation
-
-First make sure that you have the package installed. If not, then install it using `pip`
+### REST API
+You can also run Embedchain as a REST API server using the following command:
 
 ```bash
-pip install embedchain
+docker run --name embedchain -p 8080:8080 embedchain/rest-api:latest
 ```
 
-## Usage
+Then, navigate to http://0.0.0.0:8080/docs to interact with the API.
 
-Creating a chatbot involves 3 steps:
+## 🔍 Usage and Demo
 
-- Import the App instance (App Types)
-- Add Dataset (Add Dataset)
-- Query or Chat on the dataset and get answers (Interface Types)
+<!-- Demo GIF or Image -->
+<p align="center">
+  <img src="docs/images/cover.gif" width="900px" alt="Embedchain Demo">
+</p>
 
-### App Types
-
-We have three types of App.
-
-#### 1. App (uses OpenAI models, paid)
-
-```python
-from embedchain import App
-
-naval_chat_bot = App()
-```
-
-- `App` uses OpenAI's model, so these are paid models. You will be charged for embedding model usage and LLM usage.
-
-- `App` uses OpenAI's embedding model to create embeddings for chunks and ChatGPT API as LLM to get answer given the relevant docs. Make sure that you have an OpenAI account and an API key. If you have don't have an API key, you can create one by visiting [this link](https://platform.openai.com/account/api-keys).
-
-- Once you have the API key, set it in an environment variable called `OPENAI_API_KEY`
+For example, you can create an Elon Musk bot using the following code:
 
 ```python
 import os
-os.environ["OPENAI_API_KEY"] = "sk-xxxx"
+from embedchain import Pipeline as App
+
+# Create a bot instance
+os.environ["OPENAI_API_KEY"] = "YOUR API KEY"
+elon_bot = App()
+
+# Embed online resources
+elon_bot.add("https://en.wikipedia.org/wiki/Elon_Musk")
+elon_bot.add("https://www.forbes.com/profile/elon-musk")
+elon_bot.add("https://www.youtube.com/watch?v=RcYjXbSJBN8")
+
+# Query the bot
+elon_bot.query("How many companies does Elon Musk run and name those?")
+# Answer: Elon Musk currently runs several companies. As of my knowledge, he is the CEO and lead designer of SpaceX, the CEO and product architect of Tesla, Inc., the CEO and founder of Neuralink, and the CEO and founder of The Boring Company. However, please note that this information may change over time, so it's always good to verify the latest updates.
+
+# (Optional): Deploy app to Embedchain Platform
+app.deploy()
+# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+# ec-xxxxxx
+
+# 🛠️ Creating pipeline on the platform...
+# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+
+# 🛠️ Adding data to your pipeline...
+# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
 ```
 
-#### 2. OpenSourceApp (uses opensource models, free)
+You can also try it in your browser with Google Colab:
 
-```python
-from embedchain import OpenSourceApp
+[![Open in Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/17ON1LPonnXAtLaZEebnOktstB_1cJJmh?usp=sharing)
 
-naval_chat_bot = OpenSourceApp()
-```
+## 📖 Documentation
+Comprehensive guides and API documentation are available to help you get the most out of Embedchain:
 
-- `OpenSourceApp` uses open source embedding and LLM model. It uses `all-MiniLM-L6-v2` from Sentence Transformers library as the embedding model and `gpt4all` as the LLM.
+- [Getting Started](https://docs.embedchain.ai/get-started/quickstart)
+- [Introduction](https://docs.embedchain.ai/get-started/introduction#what-is-embedchain)
+- [Examples](https://docs.embedchain.ai/get-started/examples)
+- [Supported data types](https://docs.embedchain.ai/data-sources/)
 
-- Here there is no need to setup any api keys. You just need to install embedchain package and these will get automatically installed.
+## 🔗 Join the Community
 
-- Once you have imported and instantiated the app, every functionality from here onwards is the same for either type of app.
+Connect with fellow developers and users by joining our [Slack Workspace](https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw). Dive into discussions, ask questions, and share your experiences.
 
-#### 3. PersonApp (uses OpenAI models, paid)
+## 🤝 Schedule a 1-on-1 Session
 
-```python
-from embedchain import PersonApp
+Book a [1-on-1 Session](https://cal.com/taranjeetio/ec) with Taranjeet, the founder, to discuss any issues, provide feedback, or explore how we can improve Embedchain for you.
 
-naval_chat_bot = PersonApp("name_of_person_or_character") #Like "Yoda"
-```
+## 🌐 Contributing
 
-- `PersonApp` uses OpenAI's model, so these are paid models. You will be charged for embedding model usage and LLM usage.
+Contributions are welcome! Please check out the issues on the repository, and feel free to open a pull request.
+For more information, please see the [contributing guidelines](CONTRIBUTING.md).
 
-- `PersonApp` uses OpenAI's embedding model to create embeddings for chunks and ChatGPT API as LLM to get answer given the relevant docs. Make sure that you have an OpenAI account and an API key. If you have don't have an API key, you can create one by visiting [this link](https://platform.openai.com/account/api-keys).
+For more reference, please go through [Development Guide](https://docs.embedchain.ai/contribution/dev) and [Documentation Guide](https://docs.embedchain.ai/contribution/docs).
 
-- Once you have the API key, set it in an environment variable called `OPENAI_API_KEY`
+<a href="https://github.com/embedchain/embedchain/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=embedchain/embedchain" />
+</a>
 
-```python
-import os
-os.environ["OPENAI_API_KEY"] = "sk-xxxx"
-```
+## Telemetry
 
-### Add Dataset
-
-- This step assumes that you have already created an `app` instance by either using `App` or `OpenSourceApp`. We are calling our app instance as `naval_chat_bot`
-
-- Now use `.add` function to add any dataset.
-
-```python
-
-# naval_chat_bot = App() or
-# naval_chat_bot = OpenSourceApp()
-
-# Embed Online Resources
-naval_chat_bot.add("youtube_video", "https://www.youtube.com/watch?v=3qHkcs3kG44")
-naval_chat_bot.add("pdf_file", "https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
-naval_chat_bot.add("web_page", "https://nav.al/feedback")
-naval_chat_bot.add("web_page", "https://nav.al/agi")
-
-# Embed Local Resources
-naval_chat_bot.add_local("qna_pair", ("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."))
-```
-
-- If there is any other app instance in your script or app, you can change the import as
-
-```python
-from embedchain import App as EmbedChainApp
-from embedchain import OpenSourceApp as EmbedChainOSApp
-from embedchain import PersonApp as EmbedChainPersonApp
-
-# or
-
-from embedchain import App as ECApp
-from embedchain import OpenSourceApp as ECOSApp
-from embedchain import PersonApp as ECPApp
-```
-
-## Interface Types
-
-### Query Interface
-
-- This interface is like a question answering bot. It takes a question and gets the answer. It does not maintain context about the previous chats.
-
-- To use this, call `.query` function to get the answer for any query.
-
-```python
-print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
-# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
-```
-
-### Chat Interface
-
-- This interface is chat interface where it remembers previous conversation. Right now it remembers 5 conversation by default.
-
-- To use this, call `.chat` function to get the answer for any query.
-
-```python
-print(naval_chat_bot.chat("How to be happy in life?"))
-# answer: The most important trick to being happy is to realize happiness is a skill you develop and a choice you make. You choose to be happy, and then you work at it. It's just like building muscles or succeeding at your job. It's about recognizing the abundance and gifts around you at all times.
-
-print(naval_chat_bot.chat("who is naval ravikant?"))
-# answer: Naval Ravikant is an Indian-American entrepreneur and investor.
-
-print(naval_chat_bot.chat("what did the author say about happiness?"))
-# answer: The author, Naval Ravikant, believes that happiness is a choice you make and a skill you develop. He compares the mind to the body, stating that just as the body can be molded and changed, so can the mind. He emphasizes the importance of being present in the moment and not getting caught up in regrets of the past or worries about the future. By being present and grateful for where you are, you can experience true happiness.
-```
-
-### Stream Response
-
-- You can add config to your query method to stream responses like ChatGPT does. You would require a downstream handler to render the chunk in your desirable format. Supports both OpenAI model and OpenSourceApp.
-
-- To use this, instantiate a `QueryConfig` or `ChatConfig` object with `stream=True`. Then pass it to the `.chat()` or `.query()` method. The following example iterates through the chunks and prints them as they appear.
-
-```python
-app = App()
-query_config = QueryConfig(stream = True)
-resp = app.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", query_config)
-
-for chunk in resp:
-    print(chunk, end="", flush=True)
-# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
-```
-
-## Format supported
-
-We support the following formats:
-
-### Youtube Video
-
-To add any youtube video to your app, use the data_type (first argument to `.add`) as `youtube_video`. Eg:
-
-```python
-app.add('youtube_video', 'a_valid_youtube_url_here')
-```
-
-### PDF File
-
-To add any pdf file, use the data_type as `pdf_file`. Eg:
-
-```python
-app.add('pdf_file', 'a_valid_url_where_pdf_file_can_be_accessed')
-```
-
-Note that we do not support password protected pdfs.
-
-### Web Page
-
-To add any web page, use the data_type as `web_page`. Eg:
-
-```python
-app.add('web_page', 'a_valid_web_page_url')
-```
-
-### Doc File
-
-To add any doc/docx file, use the data_type as `docx`. Eg:
-
-```python
-app.add('docx', 'a_local_docx_file_path')
-```
-
-### Text
-
-To supply your own text, use the data_type as `text` and enter a string. The text is not processed, this can be very versatile. Eg:
-
-```python
-app.add_local('text', 'Seek wealth, not money or status. Wealth is having assets that earn while you sleep. Money is how we transfer time and wealth. Status is your place in the social hierarchy.')
-```
-
-Note: This is not used in the examples because in most cases you will supply a whole paragraph or file, which did not fit.
-
-### QnA Pair
-
-To supply your own QnA pair, use the data_type as `qna_pair` and enter a tuple. Eg:
-
-```python
-app.add_local('qna_pair', ("Question", "Answer"))
-```
-### Sitemap
-
-To add a XML site map containing list of all urls, use the data_type as `sitemap` and enter the sitemap url. Eg:
-
-```python
-app.add('sitemap', 'a_valid_sitemap_url/sitemap.xml')
-```
-
-### Reusing a Vector DB
-
-Default behavior is to create a persistent vector DB in the directory **./db**. You can split your application into two Python scripts: one to create a local vector DB and the other to reuse this local persistent vector DB. This is useful when you want to index hundreds of documents and separately implement a chat interface.
-
-Create a local index:
-
-```python
-
-from embedchain import App
-
-naval_chat_bot = App()
-naval_chat_bot.add("youtube_video", "https://www.youtube.com/watch?v=3qHkcs3kG44")
-naval_chat_bot.add("pdf_file", "https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
-```
-
-You can reuse the local index with the same code, but without adding new documents:
-
-```python
-
-from embedchain import App
-
-naval_chat_bot = App()
-print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
-```
-
-### More Formats coming soon
-
-- If you want to add any other format, please create an [issue](https://github.com/embedchain/embedchain/issues) and we will add it to the list of supported formats.
-
-## Testing
-
-Before you consume valueable tokens, you should make sure that the embedding you have done works and that it's receiving the correct document from the database.
-
-For this you can use the `dry_run` method.
-
-Following the example above, add this to your script:
-
-```python
-print(naval_chat_bot.dry_run('Can you tell me who Naval Ravikant is?'))
-
-'''
-Use the following pieces of context to answer the query at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.
-        Q: Who is Naval Ravikant?
-A: Naval Ravikant is an Indian-American entrepreneur and investor.
-        Query: Can you tell me who Naval Ravikant is?
-        Helpful Answer:
-'''
-```
-
-_The embedding is confirmed to work as expected. It returns the right document, even if the question is asked slightly different. No prompt tokens have been consumed._
-
-**The dry run will still consume tokens to embed your query, but it is only ~1/15 of the prompt.**
-
-## Colab Notebook and Video Tutorials
-
-Chinese Colab Tutorial:https://colab.research.google.com/drive/10_7Y0x4YXWVjuhhYwVraGQLpKAatTQTm?usp=sharing
-
-Chinese Video Tutorial:https://www.bilibili.com/video/BV1YX4y1H7oN
-
-# Advanced
-
-## Configuration
-
-Embedchain is made to work out of the box. However, for advanced users we're also offering configuration options. All of these configuration options are optional and have sane defaults.
-
-### Example
-
-Here's the readme example with configuration options.
-
-```python
-import os
-from embedchain import App
-from embedchain.config import InitConfig, AddConfig, QueryConfig
-from chromadb.utils import embedding_functions
-
-# Example: use your own embedding function
-config = InitConfig(ef=embedding_functions.OpenAIEmbeddingFunction(
-                api_key=os.getenv("OPENAI_API_KEY"),
-                organization_id=os.getenv("OPENAI_ORGANIZATION"),
-                model_name="text-embedding-ada-002"
-            ))
-naval_chat_bot = App(config)
-
-# Example: define your own chunker config for `youtube_video`
-youtube_add_config = {
-        "chunker": {
-                "chunk_size": 1000,
-                "chunk_overlap": 100,
-                "length_function": len,
-        }
-}
-naval_chat_bot.add("youtube_video", "https://www.youtube.com/watch?v=3qHkcs3kG44", AddConfig(**youtube_add_config))
-
-add_config = AddConfig()
-naval_chat_bot.add("pdf_file", "https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf", add_config)
-naval_chat_bot.add("web_page", "https://nav.al/feedback", add_config)
-naval_chat_bot.add("web_page", "https://nav.al/agi", add_config)
-
-naval_chat_bot.add_local("qna_pair", ("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."), add_config)
-
-query_config = QueryConfig() # Currently no options
-print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", query_config))
-```
-
-Here's the example of using custom prompt template with `.query`
-
-```python
-from embedchain.config import QueryConfig
-from embedchain.embedchain import App
-from string import Template
-import wikipedia
-
-einstein_chat_bot = App()
-
-# Embed Wikipedia page
-page = wikipedia.page("Albert Einstein")
-einstein_chat_bot.add("text", page.content)
-
-# Example: use your own custom template with `$context` and `$query`
-einstein_chat_template = Template("""
-        You are Albert Einstein, a German-born theoretical physicist,
-        widely ranked among the greatest and most influential scientists of all time.
-
-        Use the following information about Albert Einstein to respond to 
-        the human's query acting as Albert Einstein.
-        Context: $context                                
-
-        Keep the response brief. If you don't know the answer, just say that you don't know, don't try to make up an answer.
-
-        Human: $query
-        Albert Einstein:""")
-query_config = QueryConfig(einstein_chat_template)
-queries = [
-        "Where did you complete your studies?",
-        "Why did you win nobel prize?",
-        "Why did you divorce your first wife?",
-]
-for query in queries:
-        response = einstein_chat_bot.query(query, query_config)
-        print("Query: ", query)
-        print("Response: ", response)
-
-# Output
-# Query:  Where did you complete your studies?
-# Response:  I completed my secondary education at the Argovian cantonal school in Aarau, Switzerland.
-# Query:  Why did you win nobel prize?
-# Response:  I won the Nobel Prize in Physics in 1921 for my services to Theoretical Physics, particularly for my discovery of the law of the photoelectric effect.
-# Query:  Why did you divorce your first wife?
-# Response:  We divorced due to living apart for five years.
-```
-
-**Client Mode**. By defining a (ChromaDB) server, you can run EmbedChain as a client only.
-
-```python
-from embedchain import App
-config = InitConfig(host="localhost", port="8080")
-app = App(config)
-```
-This is useful for scalability. Say you have EmbedChain behind an API with multiple workers. If you separate clients and server, all clients can connect to the server, which only has to keep one instance of the database in memory. You also don't have to worry about replication.
-
-To run a chroma db server, run `git clone https://github.com/chroma-core/chroma.git`, navigate to the directory (`cd chroma`) and then start the server with `docker-compose up -d --build`.
-
-### Configs
-
-This section describes all possible config options.
-
-#### **InitConfig**
-
-|option|description|type|default|
-|---|---|---|---|
-|log_level|log level|string|WARNING|
-|ef|embedding function|chromadb.utils.embedding_functions|{text-embedding-ada-002}|
-|db|vector database (experimental)|BaseVectorDB|ChromaDB|
-|host|hostname for (Chroma) DB server|string|None|
-|port|port number for (Chroma) DB server|string, int|None|
-
-#### **Add Config**
-
-|option|description|type|default|
-|---|---|---|---|
-|chunker|chunker config|ChunkerConfig|Default values for chunker depends on the `data_type`. Please refer [ChunkerConfig](#chunker-config)|
-|loader|loader config|LoaderConfig|None|
-
-##### **Chunker Config**
-
-|option|description|type|default|
-|---|---|---|---|
-|chunk_size|Maximum size of chunks to return|int|Default value for various `data_type` mentioned below|
-|chunk_overlap|Overlap in characters between chunks|int|Default value for various `data_type` mentioned below|
-|length_function|Function that measures the length of given chunks|typing.Callable|Default value for various `data_type` mentioned below|
-
-Default values of chunker config parameters for different `data_type`:
-
-|data_type|chunk_size|chunk_overlap|length_function|
-|---|---|---|---|
-|docx|1000|0|len|
-|text|300|0|len|
-|qna_pair|300|0|len|
-|web_page|500|0|len|
-|pdf_file|1000|0|len|
-|youtube_video|2000|0|len|
-
-##### **Loader Config**
-
-_coming soon_
-
-#### **Query Config**
-
-|option|description|type|default|
-|---|---|---|---|
-|number_documents|number of documents to be retrieved as context|int|1|
-|template|custom template for prompt|Template|Template("Use the following pieces of context to answer the query at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. \$context Query: $query Helpful Answer:")|
-|history|include conversation history from your client or database|any (recommendation: list[str])|None
-|stream|control if response is streamed back to the user|bool|False|
-|model|OpenAI model|string|gpt-3.5-turbo-0613|
-|temperature|creativity of the model (0-1)|float|0|
-|max_tokens|limit maximum tokens used|int|1000|
-|top_p|diversity of words used by the model (0-1)|float|1|
-
-#### **Chat Config**
-
-All options for query and...
-
-_coming soon_
-
-history is handled automatically, the config option is not supported.
-
-## Other methods
-
-### Reset
-
-Resets the database and deletes all embeddings. Irreversible. Requires reinitialization afterwards.
-
-```python
-app.reset()
-```
-
-### Count
-
-Counts the number of embeddings (chunks) in the database.
-
-```python
-print(app.count())
-# returns: 481
-```
-
-# How does it work?
-
-Creating a chat bot over any dataset needs the following steps to happen
-
-- load the data
-- create meaningful chunks
-- create embeddings for each chunk
-- store the chunks in vector database
-
-Whenever a user asks any query, following process happens to find the answer for the query
-
-- create the embedding for query
-- find similar documents for this query from vector database
-- pass similar documents as context to LLM to get the final answer.
-
-The process of loading the dataset and then querying involves multiple steps and each steps has nuances of it is own.
-
-- How should I chunk the data? What is a meaningful chunk size?
-- How should I create embeddings for each chunk? Which embedding model should I use?
-- How should I store the chunks in vector database? Which vector database should I use?
-- Should I store meta data along with the embeddings?
-- How should I find similar documents for a query? Which ranking model should I use?
-
-These questions may be trivial for some but for a lot of us, it needs research, experimentation and time to find out the accurate answers.
-
-embedchain is a framework which takes care of all these nuances and provides a simple interface to create bots over any dataset.
-
-In the first release, we are making it easier for anyone to get a chatbot over any dataset up and running in less than a minute. All you need to do is create an app instance, add the data sets using `.add` function and then use `.query` function to get the relevant answer.
-
-# Contribution Guidelines
-
-Thank you for your interest in contributing to the EmbedChain project! We welcome your ideas and contributions to help improve the project. Please follow the instructions below to get started:
-
-1. **Fork the repository**: Click on the "Fork" button at the top right corner of this repository page. This will create a copy of the repository in your own GitHub account.
-
-2. **Install the required dependencies**: Ensure that you have the necessary dependencies installed in your Python environment. You can do this by running the following command:
-
-```bash
-make install
-```
-
-3. **Make changes in the code**: Create a new branch in your forked repository and make your desired changes in the codebase.
-4. **Format code**: Before creating a pull request, it's important to ensure that your code follows our formatting guidelines. Run the following commands to format the code:
-
-```bash
-make lint format
-```
-
-5. **Create a pull request**: When you are ready to contribute your changes, submit a pull request to the EmbedChain repository. Provide a clear and descriptive title for your pull request, along with a detailed description of the changes you have made.
-
-# Tech Stack
-
-embedchain is built on the following stack:
-
-- [Langchain](https://github.com/hwchase17/langchain) as an LLM framework to load, chunk and index data
-- [OpenAI's Ada embedding model](https://platform.openai.com/docs/guides/embeddings) to create embeddings
-- [OpenAI's ChatGPT API](https://platform.openai.com/docs/guides/gpt/chat-completions-api) as LLM to get answers given the context
-- [Chroma](https://github.com/chroma-core/chroma) as the vector database to store embeddings
-- [gpt4all](https://github.com/nomic-ai/gpt4all) as an open source LLM
-- [sentence-transformers](https://huggingface.co/sentence-transformers) as open source embedding model
-
-# Team
-
-## Author
-
-- Taranjeet Singh ([@taranjeetio](https://twitter.com/taranjeetio))
-
-## Maintainer
-
-- [cachho](https://github.com/cachho)
+We collect anonymous usage metrics to enhance our package's quality and user experience. This includes data like feature usage frequency and system info, but never personal details. The data helps us prioritize improvements and ensure compatibility. If you wish to opt-out, set the `app.config.collect_metrics = False` in the code. We prioritize data security and don't share this data externally.
 
 ## Citation
 
@@ -625,8 +130,8 @@ If you utilize this repository, please consider citing it with:
 
 ```
 @misc{embedchain,
-  author = {Taranjeet Singh},
-  title = {Embechain: Framework to easily create LLM powered bots over any dataset},
+  author = {Taranjeet Singh, Deshraj Yadav},
+  title = {Embedchain: Data platform for LLMs - load, index, retrieve, and sync any unstructured data},
   year = {2023},
   publisher = {GitHub},
   journal = {GitHub repository},

configs/anthropic.yaml

@@ -0,0 +1,8 @@
+llm:
+  provider: anthropic
+  config:
+    model: 'claude-instant-1'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false

configs/azure_openai.yaml

@@ -0,0 +1,19 @@
+app:
+  config:
+    id: azure-openai-app
+
+llm:
+  provider: azure_openai
+  config:
+    model: gpt-35-turbo
+    deployment_name: your_llm_deployment_name
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: azure_openai
+  config:
+    model: text-embedding-ada-002
+    deployment_name: you_embedding_model_deployment_name

configs/chroma.yaml

@@ -0,0 +1,25 @@
+app:
+  config:
+    id: 'my-app'
+    collection_name: 'my-app'
+
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-app'
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-ada-002'

configs/chunker.yaml

@@ -0,0 +1,4 @@
+chunker:
+  chunk_size: 100
+  chunk_overlap: 20
+  length_function: 'len'

configs/cohere.yaml

@@ -0,0 +1,7 @@
+llm:
+  provider: cohere
+  config:
+    model: large
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1

configs/full-stack.yaml

@@ -0,0 +1,40 @@
+app:
+  config:
+    id: 'full-stack-app'
+
+chunker:
+  chunk_size: 100
+  chunk_overlap: 20
+  length_function: 'len'
+
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+    template: |
+      Use the following pieces of context to answer the query at the end.
+      If you don't know the answer, just say that you don't know, don't try to make up an answer.
+
+      $context
+
+      Query: $query
+
+      Helpful Answer:
+    system_prompt: |
+      Act as William Shakespeare. Answer the following questions in the style of William Shakespeare.
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-collection-name'
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-ada-002'

configs/gpt4.yaml

@@ -0,0 +1,8 @@
+llm:
+  provider: openai
+  config:
+    model: 'gpt-4'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false

configs/gpt4all.yaml

@@ -0,0 +1,11 @@
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: gpt4all

configs/huggingface.yaml

@@ -0,0 +1,8 @@
+llm:
+  provider: huggingface
+  config:
+    model: 'google/flan-t5-xxl'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false

configs/jina.yaml

@@ -0,0 +1,7 @@
+llm:
+  provider: jina
+  config:
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false

configs/llama2.yaml

@@ -0,0 +1,8 @@
+llm:
+  provider: llama2
+  config:
+    model: 'a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false

configs/opensearch.yaml

@@ -0,0 +1,33 @@
+app:
+  config:
+    id: 'my-app'
+    log_level: 'WARNING'
+    collect_metrics: true
+    collection_name: 'my-app'
+
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+vectordb:
+  provider: opensearch
+  config:
+    opensearch_url: 'https://localhost:9200'
+    http_auth:
+      - admin
+      - admin
+    vector_dimension: 1536
+    collection_name: 'my-app'
+    use_ssl: false
+    verify_certs: false
+
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-ada-002'
+    deployment_name: 'my-app'

configs/opensource.yaml

@@ -0,0 +1,26 @@
+app:
+  config:
+    id: 'open-source-app'
+    collection_name: 'open-source-app'
+    collect_metrics: false
+
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'open-source-app'
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: gpt4all
+  config:
+    deployment_name: 'test-deployment'

configs/pinecone.yaml

@@ -0,0 +1,6 @@
+vectordb:
+  provider: pinecone
+  config:
+    metric: cosine
+    vector_dimension: 1536
+    collection_name: my-pinecone-index

configs/pipeline.yaml

@@ -0,0 +1,26 @@
+pipeline:
+  config:
+    name: Example pipeline
+    id: pipeline-1  # Make sure that id is different every time you create a new pipeline
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: pipeline-1
+    dir: db
+    allow_reset: true
+
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedding_model:
+  provider: gpt4all
+  config:
+    model: 'all-MiniLM-L6-v2'
+    deployment_name: null

configs/vertexai.yaml

@@ -0,0 +1,6 @@
+llm:
+  provider: vertexai
+  config:
+    model: 'chat-bison'
+    temperature: 0.5
+    top_p: 0.5

configs/weaviate.yaml

@@ -0,0 +1,4 @@
+vectordb:
+  provider: weaviate
+  config:
+    collection_name: my_weaviate_index

docs/Makefile

@@ -0,0 +1,10 @@
+install:
+	npm i -g mintlify
+
+run_local:
+	mintlify dev
+
+troubleshoot:
+	mintlify install
+
+.PHONY: install run_local troubleshoot

docs/README.md

@@ -0,0 +1,25 @@
+# Contributing to embedchain docs
+
+
+### 👩‍💻 Development
+
+Install the [Mintlify CLI](https://www.npmjs.com/package/mintlify) to preview the documentation changes locally. To install, use the following command
+
+```
+npm i -g mintlify
+```
+
+Run the following command at the root of your documentation (where mint.json is)
+
+```
+mintlify dev
+```
+
+### 😎 Publishing Changes
+
+Changes will be deployed to production automatically after your PR is merged to the main branch.
+
+#### Troubleshooting
+
+- Mintlify dev isn't running - Run `mintlify install` it'll re-install dependencies.
+- Page loads as a 404 - Make sure you are running in a folder with `mint.json`

docs/_snippets/get-help.mdx

@@ -0,0 +1,11 @@
+<CardGroup cols={3}>
+  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+    Join our slack community
+  </Card>
+  <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
+    Join our discord community
+  </Card>
+  <Card title="Schedule a call" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call with Embedchain founder
+  </Card>
+</CardGroup>

docs/_snippets/missing-data-source-tip.mdx

@@ -0,0 +1,16 @@
+<p>If you can't find the specific data source, please feel free to request through one of the following channels and help us prioritize.</p>
+
+<CardGroup cols={2}>
+  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+    Let us know on our slack community
+  </Card>
+  <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
+    Let us know on discord community
+  </Card>
+  <Card title="GitHub" icon="github" href="https://github.com/embedchain/embedchain/issues/new?assignees=&labels=&projects=&template=feature_request.yml" color="#181717">
+  Open an issue on our GitHub
+  </Card>
+  <Card title="Schedule a call" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call with Embedchain founder
+  </Card>
+</CardGroup>

docs/_snippets/missing-llm-tip.mdx

@@ -0,0 +1,16 @@
+<p>If you can't find the specific LLM you need, no need to fret. We're continuously expanding our support for additional LLMs, and you can help us prioritize by opening an issue on our GitHub or simply reaching out to us on our Slack or Discord community.</p>
+
+<CardGroup cols={2}>
+  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+    Let us know on our slack community
+  </Card>
+  <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
+    Let us know on discord community
+  </Card>
+  <Card title="GitHub" icon="github" href="https://github.com/embedchain/embedchain/issues/new?assignees=&labels=&projects=&template=feature_request.yml" color="#181717">
+  Open an issue on our GitHub
+  </Card>
+  <Card title="Schedule a call" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call with Embedchain founder
+  </Card>
+</CardGroup>

docs/_snippets/missing-vector-db-tip.mdx

@@ -0,0 +1,18 @@
+
+
+<p>If you can't find the specific vector database, please feel free to request through one of the following channels and help us prioritize.</p>
+
+<CardGroup cols={2}>
+  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+    Let us know on our slack community
+  </Card>
+  <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
+    Let us know on discord community
+  </Card>
+  <Card title="GitHub" icon="github" href="https://github.com/embedchain/embedchain/issues/new?assignees=&labels=&projects=&template=feature_request.yml" color="#181717">
+  Open an issue on our GitHub
+  </Card>
+  <Card title="Schedule a call" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call with Embedchain founder
+  </Card>
+</CardGroup>

docs/advanced/configuration.mdx

@@ -0,0 +1,84 @@
+---
+title: '⚙️ Custom configurations'
+---
+
+Embedchain is made to work out of the box. However, for advanced users we're also offering configuration options. All of these configuration options are optional and have sane defaults.
+
+You can configure different components of your app (`llm`, `embedding model`, or `vector database`) through a simple yaml configuration that Embedchain offers. Here is a generic full-stack example of the yaml config:
+
+```yaml
+app:
+  config:
+    id: 'full-stack-app'
+
+chunker:
+  chunk_size: 100
+  chunk_overlap: 20
+  length_function: 'len'
+
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+    template: |
+      Use the following pieces of context to answer the query at the end.
+      If you don't know the answer, just say that you don't know, don't try to make up an answer.
+
+      $context
+
+      Query: $query
+
+      Helpful Answer:
+    system_prompt: |
+      Act as William Shakespeare. Answer the following questions in the style of William Shakespeare.
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'full-stack-app'
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-ada-002'
+```
+
+Alright, let's dive into what each key means in the yaml config above:
+
+1. `app` Section:
+    - `config`:
+        - `id` (String): The ID or name of your full-stack application.
+2. `chunker` Section:
+    - `chunk_size` (Integer): The size of each chunk of text that is sent to the language model.
+    - `chunk_overlap` (Integer): The amount of overlap between each chunk of text.
+    - `length_function` (String): The function used to calculate the length of each chunk of text. In this case, it's set to 'len'. You can also use any function import directly as a string here.
+3. `llm` Section:
+    - `provider` (String): The provider for the language model, which is set to 'openai'. You can find the full list of llm providers in [our docs](/components/llms).
+    - `model` (String): The specific model being used, 'gpt-3.5-turbo'.
+    - `config`:
+        - `temperature` (Float): Controls the randomness of the model's output. A higher value (closer to 1) makes the output more random.
+        - `max_tokens` (Integer): Controls how many tokens are used in the response.
+        - `top_p` (Float): Controls the diversity of word selection. A higher value (closer to 1) makes word selection more diverse.
+        - `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
+        - `template` (String): A custom template for the prompt that the model uses to generate responses.
+        - `system_prompt` (String): A system prompt for the model to follow when generating responses, in this case, it's set to the style of William Shakespeare.
+4. `vectordb` Section:
+    - `provider` (String): The provider for the vector database, set to 'chroma'. You can find the full list of vector database providers in [our docs](/components/vector-databases).
+    - `config`:
+        - `collection_name` (String): The initial collection name for the database, set to 'full-stack-app'.
+        - `dir` (String): The directory for the database, set to 'db'.
+        - `allow_reset` (Boolean): Indicates whether resetting the database is allowed, set to true.
+5. `embedder` Section:
+    - `provider` (String): The provider for the embedder, set to 'openai'. You can find the full list of embedding model providers in [our docs](/components/embedding-models).
+    - `config`:
+        - `model` (String): The specific model used for text embedding, 'text-embedding-ada-002'.
+
+If you have questions about the configuration above, please feel free to reach out to us using one of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/community/connect-with-us.mdx

@@ -0,0 +1,28 @@
+---
+title: 🤝 Connect with Us
+---
+
+We believe in building a vibrant and supportive community around embedchain. There are various channels through which you can connect with us, stay updated, and contribute to the ongoing discussions:
+
+<CardGroup cols={3}>
+  <Card title="Twitter" icon="twitter" href="https://twitter.com/embedchain">
+    Follow us on Twitter
+  </Card>
+  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+    Join our slack community
+  </Card>
+  <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
+    Join our discord community
+  </Card>
+  <Card title="LinkedIn" icon="linkedin" href="https://www.linkedin.com/company/embedchain/">
+  Connect with us on LinkedIn
+  </Card>
+  <Card title="Schedule a call" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call with Embedchain founder
+  </Card>
+  <Card title="Newsletter" icon="message" href="https://embedchain.substack.com/">
+  Subscribe to our newsletter
+  </Card>
+</CardGroup>
+
+We look forward to connecting with you and seeing how we can create amazing things together!

docs/community/showcase.mdx

@@ -0,0 +1,115 @@
+---
+title: '🎪 Community showcase'
+---
+
+Embedchain community has been super active in creating demos on top of Embedchain. On this page, we showcase all the apps, blogs, videos, and tutorials created by the community. ❤️
+
+## Apps
+
+### Open Source
+
+- [My GSoC23 bot- Streamlit chat](https://github.com/lucifertrj/EmbedChain_GSoC23_BOT) by Tarun Jain
+- [Discord Bot for LLM chat](https://github.com/Reidond/discord_bots_playground/tree/c8b0c36541e4b393782ee506804c4b6962426dd6/python/chat-channel-bot) by Reidond
+- [EmbedChain-Streamlit-Docker App](https://github.com/amjadraza/embedchain-streamlit-app) by amjadraza
+- [Harry Potter Philosphers Stone Bot](https://github.com/vinayak-kempawad/Harry_Potter_Philosphers_Stone_Bot/) by Vinayak Kempawad, ([LinkedIn post](https://www.linkedin.com/feed/update/urn:li:activity:7080907532155686912/))
+- [LLM bot trained on own messages](https://github.com/Harin329/harinBot) by Hao Wu
+
+### Closed Source
+
+- [Taobot.io](https://taobot.io) - chatbot & knowledgebase hybrid by [cachho](https://github.com/cachho)
+- [Create Instant ChatBot 🤖 using embedchain](https://databutton.com/v/h3e680h9) by Avra, ([Tweet](https://twitter.com/Avra_b/status/1674704745154641920/))
+- [JOBO 🤖 — The AI-driven sidekick to craft your resume](https://try-jobo.com/) by Enrico Willemse, ([LinkedIn Post](https://www.linkedin.com/posts/enrico-willemse_jobai-gptfun-embedchain-activity-7090340080879374336-ueLB/))
+- [Explore Your Knowledge Base: Interactive chats over various forms of documents](https://chatdocs.dkedar.com/) by Kedar Dabhadkar, ([LinkedIn Post](https://www.linkedin.com/posts/dkedar7_machinelearning-llmops-activity-7092524836639424513-2O3L/))
+- [Chatbot trained on 1000+ videos of Ester hicks the co-author behind the famous book Secret](https://ask-abraham.thoughtseed.repl.co) by Mohan Kumar
+
+
+## Templates
+
+### Replit
+- [Embedchain Chat Bot](https://replit.com/@taranjeet1/Embedchain-Chat-Bot) by taranjeetio
+- [Embedchain Memory Chat Bot Template](https://replit.com/@taranjeetio/Embedchain-Memory-Chat-Bot-Template) by taranjeetio
+- [Chatbot app to demonstrate question-answering using retrieved information](https://replit.com/@AllisonMorrell/EmbedChainlitPublic) by Allison Morrell, ([LinkedIn Post](https://www.linkedin.com/posts/allison-morrell-2889275a_retrievalbot-screenshots-activity-7080339991754649600-wihZ/))
+
+## Posts
+
+### Blogs
+
+- [Customer Service LINE Bot](https://www.evanlin.com/langchain-embedchain/) by Evan Lin
+- [Chatbot in Under 5 mins using Embedchain](https://medium.com/@ayush.wattal/chatbot-in-under-5-mins-using-embedchain-a4f161fcf9c5) by Ayush Wattal
+- [Understanding what the LLM framework embedchain does](https://zenn.dev/hijikix/articles/4bc8d60156a436) by Daisuke Hashimoto
+- [In bed with GPT and Node.js](https://dev.to/worldlinetech/in-bed-with-gpt-and-nodejs-4kh2) by Raphaël Semeteys, ([LinkedIn Post](https://www.linkedin.com/posts/raphaelsemeteys_in-bed-with-gpt-and-nodejs-activity-7088113552326029313-nn87/))
+- [Using Embedchain — A powerful LangChain Python wrapper to build Chat Bots even faster!⚡](https://medium.com/@avra42/using-embedchain-a-powerful-langchain-python-wrapper-to-build-chat-bots-even-faster-35c12994a360) by Avra, ([Tweet](https://twitter.com/Avra_b/status/1686767751560310784/))
+- [What is the Embedchain library?](https://jahaniwww.com/%da%a9%d8%aa%d8%a7%d8%a8%d8%ae%d8%a7%d9%86%d9%87-embedchain/) by Ali Jahani, ([LinkedIn Post](https://www.linkedin.com/posts/ajahani_aepaetaeqaexaggahyaeu-aetaexaesabraeaaeqaepaeu-activity-7097605202135904256-ppU-/))
+- [LangChain is Nice, But Have You Tried EmbedChain ?](https://medium.com/thoughts-on-machine-learning/langchain-is-nice-but-have-you-tried-embedchain-215a34421cde) by FS Ndzomga, ([Tweet](https://twitter.com/ndzfs/status/1695583640372035951/))
+- [Simplest Method to Build a Custom Chatbot with GPT-3.5 (via Embedchain)](https://www.ainewsletter.today/p/simplest-method-to-build-a-custom) by Arjun, ([Tweet](https://twitter.com/aiguy_arjun/status/1696393808467091758/))
+
+### LinkedIn
+
+- [What is embedchain](https://www.linkedin.com/posts/activity-7079393104423698432-wRyi/) by Rithesh Sreenivasan
+- [Building a chatbot with EmbedChain](https://www.linkedin.com/posts/activity-7078434598984060928-Zdso/) by Lior Sinclair
+- [Making chatbot without vs with embedchain](https://www.linkedin.com/posts/kalyanksnlp_llms-chatbots-langchain-activity-7077453416221863936-7N1L/) by Kalyan KS
+- [EmbedChain - very intuitive, first you index your data and then query!](https://www.linkedin.com/posts/shubhamsaboo_embedchain-a-framework-to-easily-create-activity-7079535460699557888-ad1X/) by Shubham Saboo
+- [EmbedChain - Harnessing power of LLM](https://www.linkedin.com/posts/uditsaini_chatbotrevolution-llmpoweredbots-embedchainframework-activity-7077520356827181056-FjTK/) by Udit S.
+- [AI assistant for ABBYY Vantage](https://www.linkedin.com/posts/maximevermeir_llm-github-abbyy-activity-7081658972071424000-fXfZ/) by Maxime V.
+- [About embedchain](https://www.linkedin.com/feed/update/urn:li:activity:7080984218914189312/) by Morris Lee
+- [How to use Embedchain](https://www.linkedin.com/posts/nehaabansal_github-embedchainembedchain-framework-activity-7085830340136595456-kbW5/) by Neha Bansal
+- [Youtube/Webpage summary for Energy Study](https://www.linkedin.com/posts/bar%C4%B1%C5%9F-sanl%C4%B1-34b82715_enerji-python-activity-7082735341563977730-Js0U/) by Barış Sanlı, ([Tweet](https://twitter.com/barissanli/status/1676968784979193857/)) 
+- [Demo: How to use Embedchain? (Contains Collab Notebook link)](https://www.linkedin.com/posts/liorsinclair_embedchain-is-getting-a-lot-of-traction-because-activity-7103044695995424768-RckT/) by Lior Sinclair
+
+### Twitter
+
+- [What is embedchain](https://twitter.com/AlphaSignalAI/status/1672668574450847745) by Lior
+- [Building a chatbot with Embedchain](https://twitter.com/Saboo_Shubham_/status/1673537044419686401) by Shubham Saboo
+- [Chatbot docker image behind an API with yaml configs with Embedchain](https://twitter.com/tricalt/status/1678411430192730113/) by Vasilije
+- [Build AI powered PDF chatbot with just five lines of Python code with Embedchain!](https://twitter.com/Saboo_Shubham_/status/1676627104866156544/) by Shubham Saboo
+- [Chatbot against a youtube video using embedchain](https://twitter.com/smaameri/status/1675201443043704834/) by Sami Maameri
+- [Highlights of EmbedChain](https://twitter.com/carl_AIwarts/status/1673542204328120321/) by carl_AIwarts
+- [Build Llama-2 chatbot in less than 5 minutes](https://twitter.com/Saboo_Shubham_/status/1682168956918833152/) by Shubham Saboo
+- [All cool features of embedchain](https://twitter.com/DhravyaShah/status/1683497882438217728/) by Dhravya Shah, ([LinkedIn Post](https://www.linkedin.com/posts/dhravyashah_what-if-i-tell-you-that-you-can-make-an-ai-activity-7089459599287726080-ZIYm/))
+- [Read paid Medium articles for Free using embedchain](https://twitter.com/kumarkaushal_/status/1688952961622585344) by Kaushal Kumar
+
+## Videos
+
+- [Embedchain in one shot](https://www.youtube.com/watch?v=vIhDh7H73Ww&t=82s) by AI with Tarun
+- [embedChain Create LLM powered bots over any dataset Python Demo Tesla Neurallink Chatbot Example](https://www.youtube.com/watch?v=bJqAn22a6Gc) by Rithesh Sreenivasan
+- [Embedchain - NEW 🔥 Langchain BABY to build LLM Bots](https://www.youtube.com/watch?v=qj_GNQ06I8o) by 1littlecoder
+- [EmbedChain -- NEW!: Build LLM-Powered Bots with Any Dataset](https://www.youtube.com/watch?v=XmaBezzGHu4) by DataInsightEdge
+- [Chat With Your PDFs in less than 10 lines of code! EMBEDCHAIN tutorial](https://www.youtube.com/watch?v=1ugkcsAcw44) by Phani Reddy
+- [How To Create A Custom Knowledge AI Powered Bot | Install + How To Use](https://www.youtube.com/watch?v=VfCrIiAst-c) by The Ai Solopreneur
+- [Build Custom Chatbot in 6 min with this Framework [Beginner Friendly]](https://www.youtube.com/watch?v=-8HxOpaFySM) by Maya Akim
+- [embedchain-streamlit-app](https://www.youtube.com/watch?v=3-9GVd-3v74) by Amjad Raza
+- [🤖CHAT with ANY ONLINE RESOURCES using EMBEDCHAIN - a LangChain wrapper, in few lines of code !](https://www.youtube.com/watch?v=Mp7zJe4TIdM) by Avra
+- [Building resource-driven LLM-powered bots with Embedchain](https://www.youtube.com/watch?v=IVfcAgxTO4I) by BugBytes
+- [embedchain-streamlit-demo](https://www.youtube.com/watch?v=yJAWB13FhYQ) by Amjad Raza
+- [Embedchain - create your own AI chatbots using open source models](https://www.youtube.com/shorts/O3rJWKwSrWE) by Dhravya Shah
+- [AI ChatBot in 5 lines Python Code](https://www.youtube.com/watch?v=zjWvLJLksv8) by Data Engineering
+- [Interview with Karl Marx](https://www.youtube.com/watch?v=5Y4Tscwj1xk) by Alexander Ray Williams
+- [Vlog where we try to build a bot based on our content on the internet](https://www.youtube.com/watch?v=I2w8CWM3bx4) by DV, ([Tweet](https://twitter.com/dvcoolster/status/1688387017544261632))
+- [CHAT with ANY ONLINE RESOURCES using EMBEDCHAIN|STREAMLIT with MEMORY |All OPENSOURCE](https://www.youtube.com/watch?v=TqQIHWoWTDQ&pp=ygUKZW1iZWRjaGFpbg%3D%3D) by DataInsightEdge
+- [Build POWERFUL LLM Bots EASILY with Your Own Data - Embedchain - Langchain 2.0? (Tutorial)](https://www.youtube.com/watch?v=jE24Y_GasE8) by WorldofAI, ([Tweet](https://twitter.com/intheworldofai/status/1696229166922780737))
+- [Embedchain: An AI knowledge base assistant for customizing enterprise private data, which can be connected to discord, whatsapp, slack, tele and other terminals (with gradio to build a request interface) in Chinese](https://www.youtube.com/watch?v=5RZzCJRk-d0) by AIGC LINK
+- [Embedchain Introduction](https://www.youtube.com/watch?v=Jet9zAqyggI) by Fahd Mirza 
+
+## Mentions
+
+### Github repos
+
+- [Awesome-LLM](https://github.com/Hannibal046/Awesome-LLM)
+- [awesome-chatgpt-api](https://github.com/reorx/awesome-chatgpt-api)
+- [awesome-langchain](https://github.com/kyrolabs/awesome-langchain)
+- [Awesome-Prompt-Engineering](https://github.com/promptslab/Awesome-Prompt-Engineering)
+- [awesome-chatgpt](https://github.com/eon01/awesome-chatgpt)
+- [Awesome-LLMOps](https://github.com/tensorchord/Awesome-LLMOps)
+- [awesome-generative-ai](https://github.com/filipecalegario/awesome-generative-ai)
+- [awesome-gpt](https://github.com/formulahendry/awesome-gpt)
+- [awesome-ChatGPT-repositories](https://github.com/taishi-i/awesome-ChatGPT-repositories)
+- [awesome-gpt-prompt-engineering](https://github.com/snwfdhmp/awesome-gpt-prompt-engineering)
+- [awesome-chatgpt](https://github.com/awesome-chatgpt/awesome-chatgpt)
+- [awesome-llm-and-aigc](https://github.com/sjinzh/awesome-llm-and-aigc)
+- [awesome-compbio-chatgpt](https://github.com/csbl-br/awesome-compbio-chatgpt)
+- [Awesome-LLM4Tool](https://github.com/OpenGVLab/Awesome-LLM4Tool)
+
+## Meetups
+
+- [Dash and ChatGPT: Future of AI-enabled apps 30/08/23](https://go.plotly.com/dash-chatgpt)
+- [Pie & AI: Bangalore - Build end-to-end LLM app using Embedchain 01/09/23](https://www.eventbrite.com/e/pie-ai-bangalore-build-end-to-end-llm-app-using-embedchain-tickets-698045722547)

docs/components/embedding-models.mdx

@@ -0,0 +1,173 @@
+---
+title: 🧩 Embedding models
+---
+
+## Overview
+
+Embedchain supports several embedding models from the following providers:
+
+<CardGroup cols={4}>
+  <Card title="OpenAI" href="#openai"></Card>
+  <Card title="Azure OpenAI" href="#azure-openai"></Card>
+  <Card title="GPT4All" href="#gpt4all"></Card>
+  <Card title="Hugging Face" href="#hugging-face"></Card>
+  <Card title="Vertex AI" href="#vertex-ai"></Card>
+</CardGroup>
+
+## OpenAI
+
+To use OpenAI embedding function, you have to set the `OPENAI_API_KEY` environment variable. You can obtain the OpenAI API key from the [OpenAI Platform](https://platform.openai.com/account/api-keys).
+
+Once you have obtained the key, you can use it like this:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ['OPENAI_API_KEY'] = 'xxx'
+
+# load embedding model configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+
+app.add("https://en.wikipedia.org/wiki/OpenAI")
+app.query("What is OpenAI?")
+```
+
+```yaml config.yaml
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-ada-002'
+```
+
+</CodeGroup>
+
+## Azure OpenAI
+
+To use Azure OpenAI embedding model, you have to set some of the azure openai related environment variables as given in the code block below:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ["OPENAI_API_TYPE"] = "azure"
+os.environ["AZURE_OPENAI_ENDPOINT"] = "https://xxx.openai.azure.com/"
+os.environ["AZURE_OPENAI_API_KEY"] = "xxx"
+os.environ["OPENAI_API_VERSION"] = "xxx"
+
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: azure_openai
+  config:
+    model: gpt-35-turbo
+    deployment_name: your_llm_deployment_name
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: azure_openai
+  config:
+    model: text-embedding-ada-002
+    deployment_name: you_embedding_model_deployment_name
+```
+</CodeGroup>
+
+You can find the list of models and deployment name on the [Azure OpenAI Platform](https://oai.azure.com/portal).
+
+## GPT4ALL
+
+GPT4All supports generating high quality embeddings of arbitrary length documents of text using a CPU optimized contrastively trained Sentence Transformer.
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load embedding model configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: gpt4all
+```
+
+</CodeGroup>
+
+## Hugging Face
+
+Hugging Face supports generating embeddings of arbitrary length documents of text using Sentence Transformer library. Example of how to generate embeddings using hugging face is given below:
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load embedding model configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: huggingface
+  config:
+    model: 'google/flan-t5-xxl'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false
+
+embedder:
+  provider: huggingface
+  config:
+    model: 'sentence-transformers/all-mpnet-base-v2'
+```
+
+</CodeGroup>
+
+## Vertex AI
+
+Embedchain supports Google's VertexAI embeddings model through a simple interface. You just have to pass the `model_name` in the config yaml and it would work out of the box.
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load embedding model configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: vertexai
+  config:
+    model: 'chat-bison'
+    temperature: 0.5
+    top_p: 0.5
+
+embedder:
+  provider: vertexai
+  config:
+    model: 'textembedding-gecko'
+```
+
+</CodeGroup>

docs/components/llms.mdx

@@ -0,0 +1,325 @@
+---
+title: 🤖 Large language models (LLMs)
+---
+
+## Overview
+
+Embedchain comes with built-in support for various popular large language models. We handle the complexity of integrating these models for you, allowing you to easily customize your language model interactions through a user-friendly interface.
+
+<CardGroup cols={4}>
+  <Card title="OpenAI" href="#openai"></Card>
+  <Card title="Azure OpenAI" href="#azure-openai"></Card>
+  <Card title="Anthropic" href="#anthropic"></Card>
+  <Card title="Cohere" href="#cohere"></Card>
+  <Card title="GPT4All" href="#gpt4all"></Card>
+  <Card title="JinaChat" href="#jinachat"></Card>
+  <Card title="Hugging Face" href="#hugging-face"></Card>
+  <Card title="Llama2" href="#llama2"></Card>
+  <Card title="Vertex AI" href="#vertex-ai"></Card>
+</CardGroup>
+
+## OpenAI
+
+To use OpenAI LLM models, you have to set the `OPENAI_API_KEY` environment variable. You can obtain the OpenAI API key from the [OpenAI Platform](https://platform.openai.com/account/api-keys).
+
+Once you have obtained the key, you can use it like this:
+
+```python
+import os
+from embedchain import Pipeline as App
+
+os.environ['OPENAI_API_KEY'] = 'xxx'
+
+app = App()
+app.add("https://en.wikipedia.org/wiki/OpenAI")
+app.query("What is OpenAI?")
+```
+
+If you are looking to configure the different parameters of the LLM, you can do so by loading the app using a [yaml config](https://github.com/embedchain/embedchain/blob/main/configs/chroma.yaml) file.
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ['OPENAI_API_KEY'] = 'xxx'
+
+# load llm configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+```
+
+</CodeGroup>
+
+
+## Azure OpenAI
+
+To use Azure OpenAI model, you have to set some of the azure openai related environment variables as given in the code block below:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ["OPENAI_API_TYPE"] = "azure"
+os.environ["OPENAI_API_BASE"] = "https://xxx.openai.azure.com/"
+os.environ["OPENAI_API_KEY"] = "xxx"
+os.environ["OPENAI_API_VERSION"] = "xxx"
+
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: azure_openai
+  config:
+    model: gpt-35-turbo
+    deployment_name: your_llm_deployment_name
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: azure_openai
+  config:
+    model: text-embedding-ada-002
+    deployment_name: you_embedding_model_deployment_name
+```
+</CodeGroup>
+
+You can find the list of models and deployment name on the [Azure OpenAI Platform](https://oai.azure.com/portal).
+
+## Anthropic
+
+To use anthropic's model, please set the `ANTHROPIC_API_KEY` which you find on their [Account Settings Page](https://console.anthropic.com/account/keys).
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ["ANTHROPIC_API_KEY"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: anthropic
+  config:
+    model: 'claude-instant-1'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+```
+
+</CodeGroup>
+
+## Cohere
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[cohere]'
+```
+
+Set the `COHERE_API_KEY` as environment variable which you can find on their [Account settings page](https://dashboard.cohere.com/api-keys).
+
+Once you have the API key, you are all set to use it with Embedchain.
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ["COHERE_API_KEY"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: cohere
+  config:
+    model: large
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+```
+
+</CodeGroup>
+
+## GPT4ALL
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[opensource]'
+```
+
+GPT4all is a free-to-use, locally running, privacy-aware chatbot. No GPU or internet required. You can use this with Embedchain using the following code:
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load llm configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: gpt4all
+```
+</CodeGroup>
+
+
+## JinaChat
+
+First, set `JINACHAT_API_KEY` in environment variable which you can obtain from [their platform](https://chat.jina.ai/api).
+
+Once you have the key, load the app using the config yaml file:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ["JINACHAT_API_KEY"] = "xxx"
+# load llm configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: jina
+  config:
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+```
+</CodeGroup>
+
+
+## Hugging Face
+
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[huggingface-hub]'
+```
+
+First, set `HUGGINGFACE_ACCESS_TOKEN` in environment variable which you can obtain from [their platform](https://huggingface.co/settings/tokens).
+
+Once you have the token, load the app using the config yaml file:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: huggingface
+  config:
+    model: 'google/flan-t5-xxl'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false
+```
+</CodeGroup>
+
+## Llama2
+
+Llama2 is integrated through [Replicate](https://replicate.com/).  Set `REPLICATE_API_TOKEN` in environment variable which you can obtain from [their platform](https://replicate.com/account/api-tokens).
+
+Once you have the token, load the app using the config yaml file:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ["REPLICATE_API_TOKEN"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: llama2
+  config:
+    model: 'a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false
+```
+</CodeGroup>
+
+## Vertex AI
+
+Setup Google Cloud Platform application credentials by following the instruction on [GCP](https://cloud.google.com/docs/authentication/external/set-up-adc). Once setup is done, use the following code to create an app using VertexAI as provider:
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load llm configuration from config.yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: vertexai
+  config:
+    model: 'chat-bison'
+    temperature: 0.5
+    top_p: 0.5
+```
+</CodeGroup>
+
+<br/ >
+<Snippet file="missing-llm-tip.mdx" />

docs/components/vector-databases.mdx

@@ -0,0 +1,224 @@
+---
+title: 🗄️ Vector databases
+---
+
+## Overview
+
+Utilizing a vector database alongside Embedchain is a seamless process. All you need to do is configure it within the YAML configuration file. We've provided examples for each supported database below:
+
+<CardGroup cols={4}>
+  <Card title="ChromaDB" href="#chromadb"></Card>
+  <Card title="Elasticsearch" href="#elasticsearch"></Card>
+  <Card title="OpenSearch" href="#opensearch"></Card>
+  <Card title="Zilliz" href="#zilliz"></Card>
+  <Card title="LanceDB" href="#lancedb"></Card>
+  <Card title="Pinecone" href="#pinecone"></Card>
+  <Card title="Qdrant" href="#qdrant"></Card>
+  <Card title="Weaviate" href="#weaviate"></Card>
+</CardGroup>
+
+## ChromaDB
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load chroma configuration from yaml file
+app = App.from_config(yaml_path="config1.yaml")
+```
+
+```yaml config1.yaml
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-collection'
+    dir: db
+    allow_reset: true
+```
+
+```yaml config2.yaml
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-collection'
+    host: localhost
+    port: 5200
+    allow_reset: true
+```
+
+</CodeGroup>
+
+
+## Elasticsearch
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[elasticsearch]'
+```
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load elasticsearch configuration from yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: elasticsearch
+  config:
+    collection_name: 'es-index'
+    es_url: http://localhost:9200
+    allow_reset: true
+    api_key: xxx
+```
+</CodeGroup>
+
+## OpenSearch
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[opensearch]'
+```
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load opensearch configuration from yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: opensearch
+  config:
+    opensearch_url: 'https://localhost:9200'
+    http_auth:
+      - admin
+      - admin
+    vector_dimension: 1536
+    collection_name: 'my-app'
+    use_ssl: false
+    verify_certs: false
+```
+
+</CodeGroup>
+
+## Zilliz
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[milvus]'
+```
+
+Set the Zilliz environment variables `ZILLIZ_CLOUD_URI` and `ZILLIZ_CLOUD_TOKEN` which you can find it on their [cloud platform](https://cloud.zilliz.com/).
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ['ZILLIZ_CLOUD_URI'] = 'https://xxx.zillizcloud.com'
+os.environ['ZILLIZ_CLOUD_TOKEN'] = 'xxx'
+
+# load zilliz configuration from yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: zilliz
+  config:
+    collection_name: 'zilliz_app'
+    uri: https://xxxx.api.gcp-region.zillizcloud.com
+    token: xxx
+    vector_dim: 1536
+    metric_type: L2
+```
+
+</CodeGroup>
+
+## LanceDB
+
+_Coming soon_
+
+## Pinecone
+
+Install pinecone related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[pinecone]'
+```
+
+In order to use Pinecone as vector database, set the environment variables `PINECONE_API_KEY` and `PINECONE_ENV` which you can find on [Pinecone dashboard](https://app.pinecone.io/).
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load pinecone configuration from yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: pinecone
+  config:
+    metric: cosine
+    vector_dimension: 1536
+    collection_name: my-pinecone-index
+```
+
+</CodeGroup>
+
+## Qdrant
+
+In order to use Qdrant as a vector database, set the environment variables `QDRANT_URL` and `QDRANT_API_KEY` which you can find on [Qdrant Dashboard](https://cloud.qdrant.io/).
+
+<CodeGroup>
+```python main.py
+from embedchain import Pipeline as App
+
+# load qdrant configuration from yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: qdrant
+  config:
+    collection_name: my_qdrant_index
+```
+</CodeGroup>
+
+## Weaviate
+
+In order to use Weaviate as a vector database, set the environment variables `WEAVIATE_ENDPOINT` and `WEAVIATE_API_KEY` which you can find on [Weaviate dashboard](https://console.weaviate.cloud/dashboard).
+
+<CodeGroup>
+```python main.py
+from embedchain import Pipeline as App
+
+# load weaviate configuration from yaml file
+app = App.from_config(yaml_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: weaviate
+  config:
+    collection_name: my_weaviate_index
+```
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />

docs/contribution/dev.mdx

@@ -0,0 +1,56 @@
+---
+title: '👨‍💻 Development'
+description: 'Contribute to Embedchain framework development'
+---
+
+Thank you for your interest in contributing to the EmbedChain project! We welcome your ideas and contributions to help improve the project. Please follow the instructions below to get started:
+
+1. **Fork the repository**: Click on the "Fork" button at the top right corner of this repository page. This will create a copy of the repository in your own GitHub account.
+
+2. **Install the required dependencies**: Ensure that you have the necessary dependencies installed in your Python environment. You can do this by running the following command:
+
+```bash
+make install
+```
+
+3. **Make changes in the code**: Create a new branch in your forked repository and make your desired changes in the codebase.
+4. **Format code**: Before creating a pull request, it's important to ensure that your code follows our formatting guidelines. Run the following commands to format the code:
+
+```bash
+make lint format
+```
+
+5. **Create a pull request**: When you are ready to contribute your changes, submit a pull request to the EmbedChain repository. Provide a clear and descriptive title for your pull request, along with a detailed description of the changes you have made.
+
+# Tech Stack
+
+embedchain is built on the following stack:
+
+- [Langchain](https://github.com/hwchase17/langchain) as an LLM framework to load, chunk and index data
+- [OpenAI's Ada embedding model](https://platform.openai.com/docs/guides/embeddings) to create embeddings
+- [OpenAI's ChatGPT API](https://platform.openai.com/docs/guides/gpt/chat-completions-api) as LLM to get answers given the context
+- [Chroma](https://github.com/chroma-core/chroma) as the vector database to store embeddings
+- [gpt4all](https://github.com/nomic-ai/gpt4all) as an open source LLM
+- [sentence-transformers](https://huggingface.co/sentence-transformers) as open source embedding model
+
+## Team
+
+### Authors
+
+- Taranjeet Singh ([@taranjeetio](https://twitter.com/taranjeetio))
+- Deshraj Yadav ([@deshrajdry](https://twitter.com/taranjeetio))
+
+### Citation
+
+If you utilize this repository, please consider citing it with:
+
+```
+@misc{embedchain,
+  author = {Taranjeet Singh, Deshraj Yadav},
+  title = {Embechain: Data platform for LLMs - Load, index, retrieve and sync any unstructured data},
+  year = {2023},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/embedchain/embedchain}},
+}
+```

docs/contribution/docs.mdx

@@ -0,0 +1,61 @@
+---
+title: '📝 Documentation'
+description: 'Contribute to Embedchain docs'
+---
+
+<Info>
+  **Prerequisite** You should have installed Node.js (version 18.10.0 or
+  higher).
+</Info>
+
+Step 1. Install Mintlify on your OS:
+
+<CodeGroup>
+
+```bash npm
+npm i -g mintlify
+```
+
+```bash yarn
+yarn global add mintlify
+```
+
+</CodeGroup>
+
+Step 2. Go to the `docs/` directory (where you can find `mint.json`) and run the following command:
+
+```bash
+mintlify dev
+```
+
+The documentation website is now available at `http://localhost:3000`.
+
+### Custom Ports
+
+Mintlify uses port 3000 by default. You can use the `--port` flag to customize the port Mintlify runs on. For example, use this command to run in port 3333:
+
+```bash
+mintlify dev --port 3333
+```
+
+You will see an error like this if you try to run Mintlify in a port that's already taken:
+
+```md
+Error: listen EADDRINUSE: address already in use :::3000
+```
+
+## Mintlify Versions
+
+Each CLI is linked to a specific version of Mintlify. Please update the CLI if your local website looks different than production.
+
+<CodeGroup>
+
+```bash npm
+npm i -g mintlify@latest
+```
+
+```bash yarn
+yarn global upgrade mintlify
+```
+
+</CodeGroup>

docs/contribution/guidelines.mdx

@@ -0,0 +1,4 @@
+---
+title: '📋 Guidelines'
+url: https://github.com/embedchain/embedchain/blob/main/CONTRIBUTING.md
+---

docs/contribution/javascript.mdx

@@ -0,0 +1,4 @@
+---
+title: ' 🟨 Javascript'
+url: https://github.com/embedchain/embedchain/tree/main/embedchain-js
+---

docs/contribution/python.mdx

@@ -0,0 +1,4 @@
+---
+title: '🐍 Python'
+url: https://github.com/embedchain/embedchain
+---

docs/data-sources/csv.mdx

@@ -0,0 +1,19 @@
+---
+title: '📊 CSV'
+---
+
+To add any csv file, use the data_type as `csv`. `csv` allows remote urls and conventional file paths. Headers are included for each line, so if you have an `age` column, `18` will be added as `age: 18`. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+app.add('https://people.sc.fsu.edu/~jburkardt/data/csv/airtravel.csv', data_type="csv")
+# Or add using the local file path
+# app.add('/path/to/file.csv', data_type="csv")
+
+app.query("Summarize the air travel data")
+# Answer: The air travel data shows the number of flights for the months of July in the years 1958, 1959, and 1960. In July 1958, there were 491 flights, in July 1959 there were 548 flights, and in July 1960 there were 622 flights.
+```
+
+Note: There is a size limit allowed for csv file beyond which it can throw error. This limit is set by the LLMs. Please consider chunking large csv files into smaller csv files.

docs/data-sources/data-type-handling.mdx

@@ -0,0 +1,52 @@
+---
+title: 'Data type handling'
+---
+
+## Automatic data type detection
+
+The add method automatically tries to detect the data_type, based on your input for the source argument. So `app.add('https://www.youtube.com/watch?v=dQw4w9WgXcQ')` is enough to embed a YouTube video.
+
+This detection is implemented for all formats. It is based on factors such as whether it's a URL, a local file, the source data type, etc.
+
+### Debugging automatic detection
+
+Set `log_level: DEBUG` in the config yaml to debug if the data type detection is done right or not. Otherwise, you will not know when, for instance, an invalid filepath is interpreted as raw text instead.
+
+### Forcing a data type
+
+To omit any issues with the data type detection, you can **force** a data_type by adding it as a `add` method argument.
+The examples below show you the keyword to force the respective `data_type`.
+
+Forcing can also be used for edge cases, such as interpreting a sitemap as a web_page, for reading its raw text instead of following links.
+
+## Remote data types
+
+<Tip>
+**Use local files in remote data types**
+
+Some data_types are meant for remote content and only work with URLs.
+You can pass local files by formatting the path using the `file:` [URI scheme](https://en.wikipedia.org/wiki/File_URI_scheme), e.g. `file:///info.pdf`.
+</Tip>
+
+## Reusing a vector database
+
+Default behavior is to create a persistent vector db in the directory **./db**. You can split your application into two Python scripts: one to create a local vector db and the other to reuse this local persistent vector db. This is useful when you want to index hundreds of documents and separately implement a chat interface.
+
+Create a local index:
+
+```python
+from embedchain import Pipeline as App
+
+naval_chat_bot = App()
+naval_chat_bot.add("https://www.youtube.com/watch?v=3qHkcs3kG44")
+naval_chat_bot.add("https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
+```
+
+You can reuse the local index with the same code, but without adding new documents:
+
+```python
+from embedchain import Pipeline as App
+
+naval_chat_bot = App()
+print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
+```

docs/data-sources/discourse.mdx

@@ -0,0 +1,44 @@
+---
+title: '🗨️ Discourse'
+---
+
+You can now easily load data from your community built with [Discourse](https://discourse.org/).
+
+## Example
+
+1. Setup the Discourse Loader with your community url.
+```Python
+from embedchain.loaders.discourse import DiscourseLoader
+
+dicourse_loader = DiscourseLoader(config={"domain": "https://community.openai.com"})
+```
+
+2. Once you setup the loader, you can create an app and load data using the above discourse loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+
+app.add("openai after:2023-10-1", data_type="discourse", loader=dicourse_loader)
+
+question = "Where can I find the OpenAI API status page?"
+app.query(question)
+# Answer: You can find the OpenAI API status page at https:/status.openai.com/.
+```
+
+NOTE: The `add` function of the app will accept any executable search query to load data. Refer [Discourse API Docs](https://docs.discourse.org/#tag/Search) to learn more about search queries.
+
+3. We automatically create a chunker to chunk your discourse data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+
+from embedchain.chunkers.discourse import DiscourseChunker
+from embedchain.config.add_config import ChunkerConfig
+
+discourse_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+discourse_chunker = DiscourseChunker(config=discourse_chunker_config)
+
+app.add("openai", data_type='discourse', loader=dicourse_loader, chunker=discourse_chunker)
+```

docs/data-sources/docs-site.mdx

@@ -0,0 +1,14 @@
+---
+title: '📚🌐 Code documentation'
+---
+
+To add any code documentation website as a loader, use the data_type as `docs_site`. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+app.add("https://docs.embedchain.ai/", data_type="docs_site")
+app.query("What is Embedchain?")
+# Answer: Embedchain is a platform that utilizes various components, including paid/proprietary ones, to provide what is believed to be the best configuration available. It uses LLM (Language Model) providers such as OpenAI, Anthpropic, Vertex_AI, GPT4ALL, Azure_OpenAI, LLAMA2, JINA, and COHERE. Embedchain allows users to import and utilize these LLM providers for their applications.'
+```

docs/data-sources/docx.mdx

@@ -0,0 +1,18 @@
+---
+title: '📄 Docx file'
+---
+
+### Docx file
+
+To add any doc/docx file, use the data_type as `docx`. `docx` allows remote urls and conventional file paths. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+app.add('https://example.com/content/intro.docx', data_type="docx")
+# Or add file using the local file path on your system
+# app.add('content/intro.docx', data_type="docx")
+
+app.query("Summarize the docx data?")
+```

docs/data-sources/gmail.mdx

@@ -0,0 +1,34 @@
+---
+title: '📬 Gmail'
+---
+
+To use GmailLoader you must install the extra dependencies with `pip install --upgrade embedchain[gmail]`.
+
+The `source` must be a valid Gmail search query, you can refer `https://support.google.com/mail/answer/7190?hl=en` to build a query.
+
+To load Gmail messages, you MUST use the data_type as `gmail`. Otherwise the source will be detected as simple `text`.
+
+To use this you need to save `credentials.json` in the directory from where you will run the loader. Follow these steps to get the credentials
+
+1. Go to the [Google Cloud Console](https://console.cloud.google.com/apis/credentials).
+2. Create a project if you don't have one already.
+3. Create an `OAuth Consent Screen` in the project. You may need to select the `external` option.
+4. Make sure the consent screen is published.
+5. Enable the [Gmail API](https://console.cloud.google.com/apis/api/gmail.googleapis.com)
+6. Create credentials from the `Credentials` tab.
+7. Select the type `OAuth Client ID`.
+8. Choose the application type `Web application`. As a name you can choose `embedchain` or any other name as per your use case.
+9. Add an authorized redirect URI for `http://localhost:8080/`.
+10. You can leave everything else at default, finish the creation.
+11. When you are done, a modal opens where you can download the details in `json` format.
+12. Put the `.json` file in your current directory and rename it to `credentials.json`
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+gmail_filter = "to: me label:inbox"
+app.add(gmail_filter, data_type="gmail")
+app.query("Summarize my email conversations")
+```

docs/data-sources/json.mdx

@@ -0,0 +1,44 @@
+---
+title: '📃 JSON'
+---
+
+To add any json file, use the data_type as `json`. Headers are included for each line, so for example if you have a json like `{"age": 18}`, then it will be added as `age: 18`.
+
+Here are the supported sources for loading `json`:
+
+```
+1. URL - valid url to json file that ends with ".json" extension.
+2. Local file - valid url to local json file that ends with ".json" extension.
+3. String - valid json string (e.g. - app.add('{"foo": "bar"}'))
+```
+
+<Tip>
+If you would like to add other data structures (e.g. list, dict etc.), convert it to a valid json first using `json.dumps()` function.
+</Tip>
+
+## Example
+
+<CodeGroup>
+
+```python python
+from embedchain import Pipeline as App
+
+app = App()
+
+# Add json file
+app.add("temp.json")
+
+app.query("What is the net worth of Elon Musk as of October 2023?")
+# As of October 2023, Elon Musk's net worth is $255.2 billion.
+```
+
+
+```json temp.json
+{
+    "question": "What is your net worth, Elon Musk?",
+    "answer": "As of October 2023, Elon Musk's net worth is $255.2 billion, making him one of the wealthiest individuals in the world."
+}
+```
+</CodeGroup>
+
+

docs/data-sources/mdx.mdx

@@ -0,0 +1,14 @@
+---
+title: '📝 Mdx file'
+---
+
+To add any `.mdx` file to your app, use the data_type (first argument to `.add()` method) as `mdx`. Note that this supports support mdx file present on machine, so this should be a file path. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+app.add('path/to/file.mdx', data_type='mdx')
+
+app.query("What are the docs about?")
+```

docs/data-sources/mysql.mdx

@@ -0,0 +1,48 @@
+---
+title: '🐬 MySQL'
+---
+
+1. Setup the MySQL loader by configuring the SQL db.
+```Python
+from embedchain.loaders.mysql import MySQLLoader
+
+config = {
+    "host": "host",
+    "port": "port",
+    "database": "database",
+    "user": "username",
+    "password": "password",
+}
+
+mysql_loader = MySQLLoader(config=config)
+```
+
+For more details on how to setup with valid config, check MySQL [documentation](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html).
+
+2. Once you setup the loader, you can create an app and load data using the above MySQL loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+app = App()
+
+app.add("SELECT * FROM table_name;", data_type='mysql', loader=mysql_loader)
+# Adds `(1, 'What is your net worth, Elon Musk?', "As of October 2023, Elon Musk's net worth is $255.2 billion.")`
+
+response = app.query(question)
+# Answer: As of October 2023, Elon Musk's net worth is $255.2 billion.
+```
+
+NOTE: The `add` function of the app will accept any executable query to load data. DO NOT pass the `CREATE`, `INSERT` queries in `add` function.
+
+3. We automatically create a chunker to chunk your SQL data, however if you wish to provide your own chunker class. Here is how you can do that:
+``Python
+
+from embedchain.chunkers.mysql import MySQLChunker
+from embedchain.config.add_config import ChunkerConfig
+
+mysql_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+mysql_chunker = MySQLChunker(config=mysql_chunker_config)
+
+app.add("SELECT * FROM table_name;", data_type='mysql', loader=mysql_loader, chunker=mysql_chunker)
+```

docs/data-sources/notion.mdx

@@ -0,0 +1,20 @@
+---
+title: '📓 Notion'
+---
+
+To use notion you must install the extra dependencies with `pip install --upgrade embedchain[community]`.
+
+To load a notion page, use the data_type as `notion`. Since it is hard to automatically detect, it is advised to specify the `data_type` when adding a notion document.
+The next argument must **end** with the `notion page id`. The id is a 32-character string. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+app.add("cfbc134ca6464fc980d0391613959196", data_type="notion")
+app.add("my-page-cfbc134ca6464fc980d0391613959196", data_type="notion")
+app.add("https://www.notion.so/my-page-cfbc134ca6464fc980d0391613959196", data_type="notion")
+
+app.query("Summarize the notion doc")
+```

docs/data-sources/openapi.mdx

@@ -0,0 +1,22 @@
+---
+title: 🙌 OpenAPI
+---
+
+To add any OpenAPI spec yaml file (currently the json file will be detected as JSON data type), use the data_type as 'openapi'. 'openapi' allows remote urls and conventional file paths.
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+app.add("https://github.com/openai/openai-openapi/blob/master/openapi.yaml", data_type="openapi")
+# Or add using the local file path
+# app.add("configs/openai_openapi.yaml", data_type="openapi")
+
+app.query("What can OpenAI API endpoint do? Can you list the things it can learn from?")
+# Answer: The OpenAI API endpoint allows users to interact with OpenAI's models and perform various tasks such as generating text, answering questions, summarizing documents, translating languages, and more. The specific capabilities and tasks that the API can learn from may vary depending on the models and features provided by OpenAI. For more detailed information, it is recommended to refer to the OpenAI API documentation at https://platform.openai.com/docs/api-reference.
+```
+
+<Note>
+The yaml file added to the App must have the required OpenAPI fields otherwise the adding OpenAPI spec will fail. Please refer to [OpenAPI Spec Doc](https://spec.openapis.org/oas/v3.1.0)
+</Note>

docs/data-sources/overview.mdx

@@ -0,0 +1,31 @@
+---
+title: Overview
+---
+
+Embedchain comes with built-in support for various data sources. We handle the complexity of loading unstructured data from these data sources, allowing you to easily customize your app through a user-friendly interface.
+
+<CardGroup cols={4}>
+  <Card title="📊 csv" href="/data-sources/csv"></Card>
+  <Card title="📃 JSON" href="/data-sources/json"></Card>
+  <Card title="📚🌐 docs site" href="/data-sources/docs-site"></Card>
+  <Card title="📄 docx" href="/data-sources/docx"></Card>
+  <Card title="📝 mdx" href="/data-sources/mdx"></Card>
+  <Card title="📓 notion" href="/data-sources/notion"></Card>
+  <Card title="📰 pdf" href="/data-sources/pdf-file"></Card>
+  <Card title="❓💬 q&a pair" href="/data-sources/qna"></Card>
+  <Card title="🗺️ sitemap" href="/data-sources/sitemap"></Card>
+  <Card title="📝 text" href="/data-sources/text"></Card>
+  <Card title="🌐📄 web page" href="/data-sources/web-page"></Card>
+  <Card title="🧾 xml" href="/data-sources/xml"></Card>
+  <Card title="🙌 OpenAPI" href="/data-sources/openapi"></Card>
+  <Card title="📺 youtube video" href="/data-sources/youtube-video"></Card>
+  <Card title="📬 Gmail" href="/data-sources/gmail"></Card>
+  <Card title="🐘 Postgres" href="/data-sources/postgres"></Card>
+  <Card title="🐬 MySQL" href="/data-sources/mysql"></Card>
+  <Card title="🤖 Slack" href="/data-sources/slack"></Card>
+  <Card title="🗨️ Discourse" href="/data-sources/discourse"></Card>
+</CardGroup>
+
+<br/ >
+
+<Snippet file="missing-data-source-tip.mdx" />

docs/data-sources/pdf-file.mdx

@@ -0,0 +1,17 @@
+---
+title: '📰 PDF file'
+---
+
+To add any pdf file, use the data_type as `pdf_file`. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+app.add('https://arxiv.org/pdf/1706.03762.pdf', data_type='pdf_file')
+app.query("What is the paper 'attention is all you need' about?")
+# Answer: The paper "Attention Is All You Need" proposes a new network architecture called the Transformer, which is based solely on attention mechanisms. It suggests moving away from complex recurrent or convolutional neural networks and instead using attention mechanisms to connect the encoder and decoder in sequence transduction models.
+```
+
+Note that we do not support password protected pdfs.

docs/data-sources/postgres.mdx

@@ -0,0 +1,64 @@
+---
+title: '🐘 Postgres'
+---
+
+1. Setup the Postgres loader by configuring the postgres db.
+```Python
+from embedchain.loaders.postgres import PostgresLoader
+
+config = {
+    "host": "host_address",
+    "port": "port_number",
+    "dbname": "database_name",
+    "user": "username",
+    "password": "password",
+}
+
+"""
+config = {
+    "url": "your_postgres_url"
+}
+"""
+
+postgres_loader = PostgresLoader(config=config)
+
+```
+
+You can either setup the loader by passing the postgresql url or by providing the config data.
+For more details on how to setup with valid url and config, check postgres [documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING:~:text=34.1.1.%C2%A0Connection%20Strings-,%23,-Several%20libpq%20functions).
+
+NOTE: if you provide the `url` field in config, all other fields will be ignored.
+
+2. Once you setup the loader, you can create an app and load data using the above postgres loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+
+question = "What is Elon Musk's networth?"
+response = app.query(question)
+# Answer: As of September 2021, Elon Musk's net worth is estimated to be around $250 billion, making him one of the wealthiest individuals in the world. However, please note that net worth can fluctuate over time due to various factors such as stock market changes and business ventures.
+
+app.add("SELECT * FROM table_name;", data_type='postgres', loader=postgres_loader)
+# Adds `(1, 'What is your net worth, Elon Musk?', "As of October 2023, Elon Musk's net worth is $255.2 billion.")`
+
+response = app.query(question)
+# Answer: As of October 2023, Elon Musk's net worth is $255.2 billion.
+```
+
+NOTE: The `add` function of the app will accept any executable query to load data. DO NOT pass the `CREATE`, `INSERT` queries in `add` function as they will result in not adding any data, so it is pointless.
+
+3. We automatically create a chunker to chunk your postgres data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+
+from embedchain.chunkers.postgres import PostgresChunker
+from embedchain.config.add_config import ChunkerConfig
+
+postgres_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+postgres_chunker = PostgresChunker(config=postgres_chunker_config)
+
+app.add("SELECT * FROM table_name;", data_type='postgres', loader=postgres_loader, chunker=postgres_chunker)
+```

docs/data-sources/qna.mdx

@@ -0,0 +1,13 @@
+---
+title: '❓💬 Queston and answer pair'
+---
+
+QnA pair is a local data type. To supply your own QnA pair, use the data_type as `qna_pair` and enter a tuple. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+app.add(("Question", "Answer"), data_type="qna_pair")
+```

docs/data-sources/sitemap.mdx

@@ -0,0 +1,13 @@
+---
+title: '🗺️ Sitemap'
+---
+
+Add all web pages from an xml-sitemap. Filters non-text files. Use the data_type as `sitemap`. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+app.add('https://example.com/sitemap.xml', data_type='sitemap')
+```

docs/data-sources/slack.mdx

@@ -0,0 +1,54 @@
+---
+title: '🤖 Slack'
+---
+
+## Pre-requisite
+- Download required packages by running `pip install --upgrade "embedchain[slack]"`.
+- Configure your slack bot token as environment variable `SLACK_USER_TOKEN`.
+    - Find your user token on your [Slack Account](https://api.slack.com/authentication/token-types)
+    - Make sure your slack user token includes [search](https://api.slack.com/scopes/search:read) scope.
+
+## Example
+1. Setup the Slack loader by configuring the Slack Webclient.
+```Python
+from embedchain.loaders.slack import SlackLoader
+
+os.environ["SLACK_USER_TOKEN"] = "xoxp-*"
+
+loader = SlackLoader()
+
+"""
+config = {
+    'base_url': slack_app_url,
+    'headers': web_headers,
+    'team_id': slack_team_id,
+}
+
+loader = SlackLoader(config)
+"""
+```
+
+NOTE: you can also pass the `config` with `base_url`, `headers`, `team_id` to setup your SlackLoader.
+
+2. Once you setup the loader, you can create an app and load data using the above slack loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+app = App()
+
+app.add("in:random", data_type="slack", loader=loader)
+question = "Which bots are available in the slack workspace's random channel?"
+# Answer: The available bot in the slack workspace's random channel is the Embedchain bot.
+```
+
+3. We automatically create a chunker to chunk your slack data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+from embedchain.chunkers.slack import SlackChunker
+from embedchain.config.add_config import ChunkerConfig
+
+slack_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+slack_chunker = SlackChunker(config=slack_chunker_config)
+
+app.add(slack_chunker, data_type="slack", loader=loader, chunker=slack_chunker)
+```

docs/data-sources/substack.mdx

@@ -0,0 +1,16 @@
+---
+title: "📝 Substack"
+---
+
+To add any Substack data sources to your app, just add the sitemap.xml of that url as the source and set the data_type to `substack`.
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+# source: for any substack just add the sitemap.xml url
+app.add('https://www.lennysnewsletter.com/sitemap.xml', data_type='substack')
+app.query("Who is Brian Chesky?")
+# Answer: Brian Chesky is the co-founder and CEO of Airbnb.
+```

docs/data-sources/text.mdx

@@ -0,0 +1,17 @@
+---
+title: '📝 Text'
+---
+
+### Text
+
+Text is a local data type. To supply your own text, use the data_type as `text` and enter a string. The text is not processed, this can be very versatile. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+app.add('Seek wealth, not money or status. Wealth is having assets that earn while you sleep. Money is how we transfer time and wealth. Status is your place in the social hierarchy.', data_type='text')
+```
+
+Note: This is not used in the examples because in most cases you will supply a whole paragraph or file, which did not fit.

docs/data-sources/web-page.mdx

@@ -0,0 +1,13 @@
+---
+title: '🌐📄 Web page'
+---
+
+To add any web page, use the data_type as `web_page`. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+app.add('a_valid_web_page_url', data_type='web_page')
+```

docs/data-sources/xml.mdx

@@ -0,0 +1,17 @@
+---
+title: '🧾 XML file'
+---
+
+### XML file
+
+To add any xml file, use the data_type as `xml`. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+app.add('content/data.xml')
+```
+
+Note: Only the text content of the xml file will be added to the app. The tags will be ignored.

docs/data-sources/youtube-video.mdx

@@ -0,0 +1,13 @@
+---
+title: '📺 Youtube video'
+---
+
+
+To add any youtube video to your app, use the data_type (first argument to `.add()` method) as `youtube_video`. Eg:
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+app.add('a_valid_youtube_url_here', data_type='youtube_video')
+```

docs/development.mdx

@@ -0,0 +1,98 @@
+---
+title: 'Development'
+description: 'Learn how to preview changes locally'
+---
+
+<Info>
+  **Prerequisite** You should have installed Node.js (version 18.10.0 or
+  higher).
+</Info>
+
+Step 1. Install Mintlify on your OS:
+
+<CodeGroup>
+
+```bash npm
+npm i -g mintlify
+```
+
+```bash yarn
+yarn global add mintlify
+```
+
+</CodeGroup>
+
+Step 2. Go to the docs are located (where you can find `mint.json`) and run the following command:
+
+```bash
+mintlify dev
+```
+
+The documentation website is now available at `http://localhost:3000`.
+
+### Custom Ports
+
+Mintlify uses port 3000 by default. You can use the `--port` flag to customize the port Mintlify runs on. For example, use this command to run in port 3333:
+
+```bash
+mintlify dev --port 3333
+```
+
+You will see an error like this if you try to run Mintlify in a port that's already taken:
+
+```md
+Error: listen EADDRINUSE: address already in use :::3000
+```
+
+## Mintlify Versions
+
+Each CLI is linked to a specific version of Mintlify. Please update the CLI if your local website looks different than production.
+
+<CodeGroup>
+
+```bash npm
+npm i -g mintlify@latest
+```
+
+```bash yarn
+yarn global upgrade mintlify
+```
+
+</CodeGroup>
+
+## Deployment
+
+<Tip>
+  Unlimited editors available under the [Startup
+  Plan](https://mintlify.com/pricing)
+</Tip>
+
+You should see the following if the deploy successfully went through:
+
+<Frame>
+  <img src="/images/checks-passed.png" style={{ borderRadius: '0.5rem' }} />
+</Frame>
+
+## Troubleshooting
+
+Here's how to solve some common problems when working with the CLI.
+
+<AccordionGroup>
+  <Accordion title="Mintlify is not loading">
+    Update to Node v18. Run `mintlify install` and try again.
+  </Accordion>
+  <Accordion title="No such file or directory on Windows">
+Go to the `C:/Users/Username/.mintlify/` directory and remove the `mint`
+folder. Then Open the Git Bash in this location and run `git clone
+https://github.com/mintlify/mint.git`.
+
+Repeat step 3.
+
+  </Accordion>
+  <Accordion title="Getting an unknown error">
+    Try navigating to the root of your device and delete the ~/.mintlify folder.
+    Then run `mintlify dev` again.
+  </Accordion>
+</AccordionGroup>
+
+Curious about what changed in a CLI version? [Check out the CLI changelog.](/changelog/command-line)

docs/examples/discord_bot.mdx

@@ -0,0 +1,70 @@
+---
+title: "🤖 Discord Bot"
+---
+
+### 🔑 Keys Setup
+
+- Set your `OPENAI_API_KEY` in your variables.env file.
+- Go to [https://discord.com/developers/applications/](https://discord.com/developers/applications/) and click on `New Application`.
+- Enter the name for your bot, accept the terms and click on `Create`. On the resulting page, enter the details of your bot as you like.
+- On the left sidebar, click on `Bot`. Under the heading `Privileged Gateway Intents`, toggle all 3 options to ON position. Save your changes.
+- Now click on `Reset Token` and copy the token value. Set it as `DISCORD_BOT_TOKEN` in .env file.
+- On the left sidebar, click on `OAuth2` and go to `General`.
+- Set `Authorization Method` to `In-app Authorization`. Under `Scopes` select `bot`.
+- Under `Bot Permissions` allow the following and then click on `Save Changes`.
+
+```text
+Send Messages (under Text Permissions)
+```
+
+- Now under `OAuth2` and go to `URL Generator`. Under `Scopes` select `bot`.
+- Under `Bot Permissions` set the same permissions as above.
+- Now scroll down and copy the `Generated URL`. Paste it in a browser window and select the Server where you want to add the bot.
+- Click on `Continue` and authorize the bot.
+- 🎉 The bot has been successfully added to your server. But it's still offline.
+
+### Take the bot online
+
+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name discord-bot -e OPENAI_API_KEY=sk-xxx -e DISCORD_BOT_TOKEN=xxx -p 8080:8080 embedchain/discord-bot:latest
+        ```
+    </Tab>
+    <Tab title="python">
+        ```bash
+        pip install --upgrade "embedchain[discord]"
+
+        python -m embedchain.bots.discord
+
+        # or if you prefer to see the question and not only the answer, run it with
+        python -m embedchain.bots.discord --include-question
+        ```
+    </Tab>
+</Tabs>
+
+### 🚀 Usage Instructions
+
+- Go to the server where you have added your bot.
+  ![Slash commands interaction with bot](https://github.com/embedchain/embedchain/assets/73601258/bf1414e3-d408-4863-b0d2-ef382a76467e)
+- You can add data sources to the bot using the slash command:
+
+```text
+/ec add <data_type> <url_or_text>
+```
+
+- You can ask your queries from the bot using the slash command:
+
+```text
+/ec query <question>
+```
+
+- You can chat with the bot using the slash command:
+
+```text
+/ec chat <question>
+```
+
+📝 Note: To use the bot privately, you can message the bot directly by right clicking the bot and selecting `Message`.
+
+🎉 Happy Chatting! 🎉

docs/examples/full_stack.mdx

@@ -0,0 +1,57 @@
+---
+title: '🌐 Full Stack'
+---
+
+The Full Stack app example can be found [here](https://github.com/embedchain/embedchain/tree/main/examples/full_stack).
+
+This guide will help you setup the full stack app on your local machine.
+
+### 🐳 Docker Setup
+
+- Create a `docker-compose.yml` file and paste the following code in it.
+
+```yaml
+version: "3.9"
+
+services:
+  backend:
+    container_name: embedchain-backend
+    restart: unless-stopped
+    build:
+      context: backend
+      dockerfile: Dockerfile
+    image: embedchain/backend
+    ports:
+      - "8000:8000"
+
+  frontend:
+    container_name: embedchain-frontend
+    restart: unless-stopped
+    build:
+      context: frontend
+      dockerfile: Dockerfile
+    image: embedchain/frontend
+    ports:
+      - "3000:3000"
+    depends_on:
+      - "backend"
+```
+
+- Run the following command,
+
+```bash
+docker-compose up
+```
+
+📝 Note: The build command might take a while to install all the packages depending on your system resources.
+
+![Fullstack App](https://github.com/embedchain/embedchain/assets/73601258/c7c04bbb-9be7-4669-a6af-039e7e972a13)
+
+### 🚀 Usage Instructions
+
+- Go to [http://localhost:3000/](http://localhost:3000/) in your browser to view the dashboard.
+- Add your `OpenAI API key` 🔑 in the Settings.
+- Create a new bot and you'll be navigated to its page.
+- Here you can add your data sources and then chat with the bot.
+
+🎉 Happy Chatting! 🎉

docs/examples/poe_bot.mdx

@@ -0,0 +1,59 @@
+---
+title: '🔮 Poe Bot'
+---
+
+### 🚀 Getting started
+
+1. Install embedchain python package:
+
+```bash
+pip install --upgrade "embedchain[poe]"
+```
+
+2. Create a free account on [Poe](https://www.poe.com?utm_source=embedchain).
+3. Click "Create Bot" button on top left.
+4. Give it a handle and an optional description.
+5. Select `Use API`.
+6. Under `API URL` enter your server or ngrok address. You can use your machine's public IP or DNS. Otherwise, employ a proxy server like [ngrok](https://ngrok.com/) to make your local bot accessible.
+7. Copy your api key and paste it in `.env` as `POE_API_KEY`.
+8. You will need to set `OPENAI_API_KEY` for generating embeddings and using LLM. Copy your OpenAI API key from [here](https://platform.openai.com/account/api-keys) and paste it in `.env` as `OPENAI_API_KEY`.
+9. Now create your bot using the following code snippet.
+
+```bash
+# make sure that you have set OPENAI_API_KEY and POE_API_KEY in .env file
+from embedchain.bots import PoeBot
+
+poe_bot = PoeBot()
+
+# add as many data sources as you want
+poe_bot.add("https://en.wikipedia.org/wiki/Adam_D%27Angelo")
+poe_bot.add("https://www.youtube.com/watch?v=pJQVAqmKua8")
+
+# start the bot
+# this start the poe bot server on port 8080 by default
+poe_bot.start()
+```
+
+10. You can paste the above in a file called `your_script.py` and then simply do
+
+```bash
+python your_script.py
+```
+
+Now your bot will start running at port `8080` by default.
+
+11. You can refer the [Supported Data formats](https://docs.embedchain.ai/advanced/data_types) section to refer the supported data types in embedchain.
+
+12. Click `Run check` to make sure your machine can be reached.
+13. Make sure your bot is private if that's what you want.
+14. Click `Create bot` at the bottom to finally create the bot
+15. Now your bot is created.
+
+### 💬 How to use
+
+- To ask the bot questions, just type your query in the Poe interface:
+```text
+<your-question-here>
+```
+
+- If you wish to add more data source to the bot, simply update your script and add as many `.add` as you like. You need to restart the server.

docs/examples/slack_bot.mdx

@@ -0,0 +1,50 @@
+---
+title: '💼 Slack Bot'
+---
+
+### 🖼️ Setup
+
+1. Create a workspace on Slack if you don't have one already by clicking [here](https://slack.com/intl/en-in/).
+2. Create a new App on your Slack account by going [here](https://api.slack.com/apps).
+3. Select `From Scratch`, then enter the Bot Name and select your workspace.
+4. On the left Sidebar, go to `OAuth and Permissions` and add the following scopes under `Bot Token Scopes`:
+```text
+app_mentions:read
+channels:history
+channels:read
+chat:write
+```
+5. Now select the option `Install to Workspace` and after it's done, copy the `Bot User OAuth Token` and set it in your secrets as `SLACK_BOT_TOKEN`.
+6. Run your bot now,
+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name slack-bot -e OPENAI_API_KEY=sk-xxx -e SLACK_BOT_TOKEN=xxx -p 8000:8000 embedchain/slack-bot
+        ```
+    </Tab>
+    <Tab title="python">
+        ```bash
+        pip install --upgrade "embedchain[slack]"
+        python3 -m embedchain.bots.slack --port 8000
+        ```
+</Tab>
+</Tabs>
+7. Expose your bot to the internet. You can use your machine's public IP or DNS. Otherwise, employ a proxy server like [ngrok](https://ngrok.com/) to make your local bot accessible.
+8. On the Slack API website go to `Event Subscriptions` on the left Sidebar and turn on `Enable Events`.
+9. In `Request URL`, enter your server or ngrok address.
+10. After it gets verified, click on `Subscribe to bot events`, add `message.channels` Bot User Event and click on `Save Changes`.
+11. Now go to your workspace, right click on the bot name in the sidebar, click `view app details`, then `add this app to a channel`.
+
+### 🚀 Usage Instructions
+
+- Go to the channel where you have added your bot.
+- To add data sources to the bot, use the command:
+```text
+add <data_type> <url_or_text>
+```
+- To ask queries from the bot, use the command:
+```text
+query <question>
+```
+
+🎉 Happy Chatting! 🎉

docs/examples/telegram_bot.mdx

@@ -0,0 +1,51 @@
+---
+title: "📱 Telegram Bot"
+---
+
+### 🖼️ Template Setup
+
+- Open the Telegram app and search for the `BotFather` user.
+- Start a chat with BotFather and use the `/newbot` command to create a new bot.
+- Follow the instructions to choose a name and username for your bot.
+- Once the bot is created, BotFather will provide you with a unique token for your bot.
+
+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name telegram-bot -e OPENAI_API_KEY=sk-xxx -e TELEGRAM_BOT_TOKEN=xxx -p 8000:8000 embedchain/telegram-bot
+        ```
+
+    <Note>
+    If you wish to use **Docker**, you would need to host your bot on a server.
+    You can use [ngrok](https://ngrok.com/) to expose your localhost to the
+    internet and then set the webhook using the ngrok URL.
+    </Note>
+
+    </Tab>
+    <Tab title="replit">
+    <Card>
+        Fork <ins>**[this](https://replit.com/@taranjeetio/EC-Telegram-Bot-Template?v=1#README.md)**</ins> replit template.
+    </Card>
+
+    - Set your `OPENAI_API_KEY` in Secrets.
+    - Set the unique token as `TELEGRAM_BOT_TOKEN` in Secrets.
+
+    </Tab>
+
+</Tabs>
+
+- Click on `Run` in the replit container and a URL will get generated for your bot.
+- Now set your webhook by running the following link in your browser:
+
+```url
+https://api.telegram.org/bot<Your_Telegram_Bot_Token>/setWebhook?url=<Replit_Generated_URL>
+```
+
+- When you get a successful response in your browser, your bot is ready to be used.
+
+### 🚀 Usage Instructions
+
+- Open your bot by searching for it using the bot name or bot username.
+- Click on `Start` or type `/start` and follow the on screen instructions.
+
+🎉 Happy Chatting! 🎉

docs/examples/whatsapp_bot.mdx

@@ -0,0 +1,55 @@
+---
+title: '💬 WhatsApp Bot'
+---
+
+### 🚀 Getting started
+
+1. Install embedchain python package:
+
+```bash
+pip install --upgrade embedchain
+```
+
+2. Launch your WhatsApp bot:
+
+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name whatsapp-bot -e OPENAI_API_KEY=sk-xxx -p 8000:8000 embedchain/whatsapp-bot
+        ```
+    </Tab>
+    <Tab title="python">
+        ```bash
+        python -m embedchain.bots.whatsapp --port 5000
+        ```
+    </Tab>
+</Tabs>
+
+
+If your bot needs to be accessible online, use your machine's public IP or DNS. Otherwise, employ a proxy server like [ngrok](https://ngrok.com/) to make your local bot accessible.
+
+3. Create a free account on [Twilio](https://www.twilio.com/try-twilio)
+    - Set up a WhatsApp Sandbox in your Twilio dashboard. Access it via the left sidebar: `Messaging > Try it out > Send a WhatsApp Message`.
+    - Follow on-screen instructions to link a phone number for chatting with your bot
+    - Copy your bot's public URL, add /chat at the end, and paste it in Twilio's WhatsApp Sandbox settings under "When a message comes in". Save the settings.
+
+- Copy your bot's public url, append `/chat` at the end and paste it under `When a message comes in` under the `Sandbox settings` for Whatsapp in Twilio. Save your settings.
+
+### 💬 How to use
+
+- To connect a new number or reconnect an old one in the Sandbox, follow Twilio's instructions.
+- To include data sources, use this command:
+```text
+add <url_or_text>
+```
+
+- To ask the bot questions, just type your query:
+```text
+<your-question-here>
+```
+
+### Example
+
+Here is an example of Elon Musk WhatsApp Bot that we created:
+
+<img src="/images/whatsapp.jpg"/>

docs/get-started/examples.mdx

@@ -0,0 +1,123 @@
+---
+title: 🔎 Examples
+description: 'Collection of Google colab notebook and Replit links for users'
+---
+
+<table>
+  <thead>
+    <tr>
+      <th>LLM</th>
+      <th>Google Colab</th>
+      <th>Replit</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td className="align-middle">OpenAI</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/openai.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/openai#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">Anthropic</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/anthropic.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/anthropic#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">Azure OpenAI</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/azure-openai.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/azureopenai#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">VertexAI</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/vertex_ai.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/vertexai#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">Cohere</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/cohere.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/cohere#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">Hugging Face</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/hugging_face_hub.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/huggingface#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">JinaChat</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/jina.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/jina#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">GPT4All</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/gpt4all.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/gpt4all#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">Llama2</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/llama2.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/llama2#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+  </tbody>
+</table>
+<table>
+  <thead>
+    <tr>
+      <th>Embedding model</th>
+      <th>Google Colab</th>
+      <th>Replit</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td className="align-middle">OpenAI</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/openai.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/openai#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">VertexAI</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/vertex_ai.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/vertexai#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">GPT4All</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/gpt4all.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/gpt4all#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">Hugging Face</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/hugging_face_hub.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/huggingface#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+  </tbody>
+</table>
+<table>
+  <thead>
+    <tr>
+      <th>Vector DB</th>
+      <th>Google Colab</th>
+      <th>Replit</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td className="align-middle">ChromaDB</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/chromadb.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/chromadb#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">Elasticsearch</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/elasticsearch.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/elasticsearchdb#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">Opensearch</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/opensearch.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/opensearchdb#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">Pinecone</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/pinecone.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+      <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/pineconedb#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
+    </tr>
+  </tbody>
+</table>

docs/get-started/faq.mdx

@@ -0,0 +1,100 @@
+---
+title: ❓ FAQs
+description: 'Collections of all the frequently asked questions'
+---
+
+#### Does Embedchain support OpenAI's Assistant APIs?
+
+Yes, it does. Please refer to the [OpenAI Assistant docs page](/get-started/openai-assistant).
+
+#### How to use `gpt-4-turbo` model released on OpenAI DevDay?
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ['OPENAI_API_KEY'] = 'xxx'
+
+# load llm configuration from gpt4_turbo.yaml file
+app = App.from_config(yaml_path="gpt4_turbo.yaml")
+```
+
+```yaml gpt4_turbo.yaml
+llm:
+  provider: openai
+  config:
+    model: 'gpt-4-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+```
+
+</CodeGroup>
+
+
+#### How to use GPT-4 as the LLM model?
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ['OPENAI_API_KEY'] = 'xxx'
+
+# load llm configuration from gpt4.yaml file
+app = App.from_config(yaml_path="gpt4.yaml")
+```
+
+```yaml gpt4.yaml
+llm:
+  provider: openai
+  config:
+    model: 'gpt-4'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+```
+
+</CodeGroup>
+
+#### I don't have OpenAI credits. How can I use some open source model?
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ['OPENAI_API_KEY'] = 'xxx'
+
+# load llm configuration from opensource.yaml file
+app = App.from_config(yaml_path="opensource.yaml")
+```
+
+```yaml opensource.yaml
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: gpt4all
+  config:
+    model: 'all-MiniLM-L6-v2'
+```
+</CodeGroup>
+
+#### How to contact support?
+
+If docs aren't sufficient, please feel free to reach out to us using one of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/get-started/introduction.mdx

@@ -0,0 +1,131 @@
+---
+title: 📚 Introduction
+description: '📝 Embedchain is a Data Platform for LLMs - load, index, retrieve, and sync any unstructured data'
+---
+
+## 🌐 What is Embedchain?
+
+Embedchain simplifies data handling by automatically processing unstructured data, breaking it into chunks, generating embeddings, and storing it in a vector database.
+
+Through various APIs, you can obtain contextual information for queries, find answers to specific questions, and engage in chat conversations using your data.
+## 🔍 Search
+
+Embedchain lets you get most relevant context by doing semantic search over your data sources for a provided query. See the example below:
+
+```python
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Get relevant context using semantic search
+context = app.search("What is the net worth of Elon?", num_documents=2)
+print(context)
+# Context:
+# [
+#     {
+#         'context': 'Elon Musk PROFILEElon MuskCEO, Tesla$221.9BReal Time Net Worthas of 10/29/23Reflects change since 5 pm ET of prior trading day. 1 in the world todayPhoto by Martin Schoeller for ForbesAbout Elon MuskElon Musk cofounded six companies, including electric car maker Tesla, rocket producer SpaceX and tunneling startup Boring Company.He owns about 21% of Tesla between stock and options, but has pledged more than half his shares as collateral for personal loans of up to $3.5 billion.SpaceX, founded in',
+#         'source': 'https://www.forbes.com/profile/elon-musk',
+#         'document_id': 'some_document_id'
+#     },
+#     {
+#         'context': 'company, which is now called X.Wealth HistoryHOVER TO REVEAL NET WORTH BY YEARForbes Lists 1Forbes 400 (2023)The Richest Person In Every State (2023) 2Billionaires (2023) 1Innovative Leaders (2019) 25Powerful People (2018) 12Richest In Tech (2017)Global Game Changers (2016)More ListsPersonal StatsAge52Source of WealthTesla, SpaceX, Self MadeSelf-Made Score8Philanthropy Score1ResidenceAustin, TexasCitizenshipUnited StatesMarital StatusSingleChildren11EducationBachelor of Arts/Science, University',
+#         'source': 'https://www.forbes.com/profile/elon-musk',
+#         'document_id': 'some_document_id'
+#     }
+# ]
+```
+
+## ❓Query
+
+Embedchain empowers developers to ask questions and receive relevant answers through a user-friendly query API. Refer to the following example to learn how to utilize the query API:
+
+```python
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Get relevant answer for your query
+answer = app.query("What is the net worth of Elon?")
+print(answer)
+# Answer: The net worth of Elon Musk is $221.9 billion.
+```
+
+## 💬 Chat
+
+Embedchain allows easy chatting over your data sources using a user-friendly chat API. Check out the example below to understand how to use the chat API:
+
+```python
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Chat on your data using `.chat()`
+answer = app.chat("How much did Elon pay for Twitter?")
+print(answer)
+# Answer: Elon Musk paid $44 billion for Twitter.
+```
+
+## 🚀 Deploy
+
+Embedchain enables developers to deploy their LLM-powered apps in production using the Embedchain platform. The platform offers free access to context on your data through its REST API. Once the pipeline is deployed, you can update your data sources anytime after deployment.
+
+See the example below on how to use the deploy API:
+
+```python
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Deploy your pipeline to Embedchain Platform
+app.deploy()
+
+# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+# ec-xxxxxx
+
+# 🛠️ Creating pipeline on the platform...
+# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+
+# 🛠️ Adding data to your pipeline...
+# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
+```
+
+## 🚀 How it works?
+
+Embedchain abstracts out the following steps from you to easily create LLM powered apps:
+
+1. Detect the data type and load data
+2. Create meaningful chunks
+3. Create embeddings for each chunk
+4. Store chunks in a vector database
+
+When a user asks a query, the following process happens to find the answer:
+
+1. Create an embedding for the query
+2. Find similar documents for the query from the vector database
+3. Pass the similar documents as context to LLM to get the final answer
+
+The process of loading the dataset and querying involves multiple steps, each with its own nuances:
+
+- How should I chunk the data? What is a meaningful chunk size?
+- How should I create embeddings for each chunk? Which embedding model should I use?
+- How should I store the chunks in a vector database? Which vector database should I use?
+- Should I store metadata along with the embeddings?
+- How should I find similar documents for a query? Which ranking model should I use?
+
+Embedchain takes care of all these nuances and provides a simple interface to create apps on any data.

docs/get-started/openai-assistant.mdx

@@ -0,0 +1,95 @@
+---
+title: '🤖 OpenAI Assistant'
+---
+
+<img src="https://blogs.swarthmore.edu/its/wp-content/uploads/2022/05/openai.jpg"  align="center" width="500" alt="OpenAI Logo"/>
+
+Embedchain now supports [OpenAI Assistants API](https://platform.openai.com/docs/assistants/overview) which allows you to build AI assistants within your own applications. An Assistant has instructions and can leverage models, tools, and knowledge to respond to user queries.
+
+At a high level, an integration of the Assistants API has the following flow:
+
+1. Create an Assistant in the API by defining custom instructions and picking a model
+2. Create a Thread when a user starts a conversation
+3. Add Messages to the Thread as the user ask questions
+4. Run the Assistant on the Thread to trigger responses. This automatically calls the relevant tools.
+
+Creating an OpenAI Assistant using Embedchain is very simple 3 step process.
+
+## Step 1: Create OpenAI Assistant
+
+Make sure that you have `OPENAI_API_KEY` set in the environment variable.
+
+```python Initialize
+from embedchain.store.assistants import OpenAIAssistant
+
+assistant = OpenAIAssistant(
+    name="OpenAI DevDay Assistant",
+    instructions="You are an organizer of OpenAI DevDay",
+)
+```
+
+If you want to use the existing assistant, you can do something like this:
+
+```python Initialize
+# Load an assistant and create a new thread
+assistant = OpenAIAssistant(assistant_id="asst_xxx")
+
+# Load a specific thread for an assistant
+assistant = OpenAIAssistant(assistant_id="asst_xxx", thread_id="thread_xxx")
+```
+
+### Arguments
+
+<ResponseField name="name" type="string">
+  Name for your AI assistant
+</ResponseField>
+
+<ResponseField name="instructions" type="string">
+  how the Assistant and model should behave or respond
+</ResponseField>
+
+<ResponseField name="assistant_id" type="string">
+  Load existing OpenAI Assistant. If you pass this, you don't have to pass other arguments.
+</ResponseField>
+
+<ResponseField name="thread_id" type="string">
+  Existing OpenAI thread id if exists
+</ResponseField>
+
+<ResponseField name="model" type="str" default="gpt-4-1106-preview">
+  OpenAI model to use
+</ResponseField>
+
+<ResponseField name="tools" type="list">
+  OpenAI tools to use. Default set to `[{"type": "retrieval"}]`
+</ResponseField>
+
+<ResponseField name="data_sources" type="list" default="[]">
+  Add data sources to your assistant. You can add in the following format: `[{"source": "https://example.com", "data_type": "web_page"}]`
+</ResponseField>
+
+<ResponseField name="telemetry" type="boolean" default="True">
+  Anonymous telemetry (doesn't collect any user information or user's files). Used to improve the Embedchain package utilization. Default is `True`.
+</ResponseField>
+
+## Step-2: Add data to thread
+
+You can add any custom data source that is supported by Embedchain. Else, you can directly pass the file path on your local system and Embedchain propagates it to OpenAI Assistant.
+```python Add data
+assistant.add("/path/to/file.pdf")
+assistant.add("https://www.youtube.com/watch?v=U9mJuUkhUzk")
+assistant.add("https://openai.com/blog/new-models-and-developer-products-announced-at-devday")
+```
+
+## Step-3: Chat with your Assistant
+```python Chat
+assistant.chat("How much OpenAI credits were offered to attendees during OpenAI DevDay?")
+# Response: 'Every attendee of OpenAI DevDay 2023 was offered $500 in OpenAI credits.'
+```
+
+You can try it out yourself using the following Google Colab notebook:
+
+<a href="https://colab.research.google.com/drive/1BKlXZYSl6AFRgiHZ5XIzXrXC_24kDYHQ?usp=sharing">
+    <img src="https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667" alt="Open in Colab" />
+</a>
+

docs/get-started/quickstart.mdx

@@ -0,0 +1,91 @@
+---
+title: '🚀 Quickstart'
+description: '💡 Start building LLM powered apps under 30 seconds'
+---
+
+Embedchain is a Data Platform for LLMs - load, index, retrieve, and sync any unstructured data. Using embedchain, you can easily create LLM powered apps over any data. 
+
+Install embedchain python package:
+
+```bash
+pip install embedchain
+```
+
+<Tip>
+Embedchain now supports OpenAI's latest `gpt-4-turbo` model. Checkout the [docs here](/get-started/faq#how-to-use-gpt-4-turbo-model-released-on-openai-devday) on how to use it.
+</Tip>
+
+Creating an app involves 3 steps:
+
+<Steps>
+  <Step title="⚙️ Import app instance">
+```python
+from embedchain import Pipeline as App
+app = App()
+```
+  </Step>
+  <Step title="🗃️ Add data sources">
+```python
+# Add different data sources
+app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+app.add("https://www.forbes.com/profile/elon-musk")
+# You can also add local data sources such as pdf, csv files etc.
+# app.add("/path/to/file.pdf")
+```
+  </Step>
+  <Step title="💬 Query or chat or search context on your data">
+```python
+app.query("What is the net worth of Elon Musk today?")
+# Answer: The net worth of Elon Musk today is $258.7 billion.
+```
+  </Step>
+  <Step title="🚀 (Optional) Deploy your pipeline to Embedchain Platform">
+```python
+app.deploy()
+# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+# ec-xxxxxx
+
+# 🛠️ Creating pipeline on the platform...
+# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+
+# 🛠️ Adding data to your pipeline...
+# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
+```
+  </Step>
+</Steps>
+
+Putting it together, you can run your first app using the following code. Make sure to set the `OPENAI_API_KEY` 🔑 environment variable in the code.
+
+```python
+import os
+from embedchain import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "xxx"
+app = App()
+
+# Add different data sources
+app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+app.add("https://www.forbes.com/profile/elon-musk")
+# You can also add local data sources such as pdf, csv files etc.
+# app.add("/path/to/file.pdf")
+
+response = app.query("What is the net worth of Elon Musk today?")
+print(response)
+# Answer: The net worth of Elon Musk today is $258.7 billion.
+
+app.deploy()
+# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+# ec-xxxxxx
+
+# 🛠️ Creating pipeline on the platform...
+# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+
+# 🛠️ Adding data to your pipeline...
+# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
+```
+
+You can try it out yourself using the following Google Colab notebook:
+
+<a href="https://colab.research.google.com/drive/17ON1LPonnXAtLaZEebnOktstB_1cJJmh?usp=sharing">
+    <img src="https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667" alt="Open in Colab" />
+</a>

docs/integration/langsmith.mdx

@@ -0,0 +1,51 @@
+---
+title: '🛠️ LangSmith'
+description: 'Integrate with Langsmith to debug and monitor your LLM app'
+---
+
+Embedchain now supports integration with [LangSmith](https://www.langchain.com/langsmith).
+
+To use langsmith, you need to do the following steps
+
+1. Have an account on langsmith and keep the environment variables in handy
+2. Set the environments variables in your app so that embedchain has context about it.
+3. Just use embedchain and everything will be logged to LangSmith, so that you can better test and monitor your application.
+
+Lets cover each step in detail.
+
+* First make sure that you a LangSmith account created and have all the necessary variables handy. LangSmith has a [good documentation](https://docs.smith.langchain.com/) on how to get started with their service.
+
+* Once you have the account setup, we will need the following environment variables
+
+```bash
+export LANGCHAIN_TRACING_V2=true
+export LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
+export LANGCHAIN_API_KEY=<your-api-key>
+export LANGCHAIN_PROJECT=<your-project>  # if not specified, defaults to "default"
+```
+
+If you are using Python, you can use the following code to set environment variables
+
+```python
+import os
+
+os.environ['LANGCHAIN_TRACING_V2'] = 'true'
+os.environ['LANGCHAIN_ENDPOINT'] = 'https://api.smith.langchain.com'
+os.environ['LANGCHAIN_API_KEY'] = <your-api-key>
+os.environ['LANGCHAIN_PROJECT] = <your-project>
+```
+
+* Now create an app using embedchain and everything will be automatically visible in the LangSmith
+
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+app.query("How many companies did Elon found?")
+```
+
+* Now the entire log for this will be visible in langsmith.
+
+<img src="/images/langsmith.png"/>

docs/logo/dark.svg

@@ -0,0 +1,24 @@
+<svg width="1161" height="212" viewBox="0 0 1161 212" fill="none" xmlns="http://www.w3.org/2000/svg">
+<path d="M1010.3 212V0H1056.64L1076.33 106V0H1116V212H1069.66L1049.98 106V212H1010.3Z" fill="white"/>
+<path d="M1001.3 212H954.958V0H1001.3V212Z" fill="white"/>
+<path d="M906.196 212L903.167 178.989H883.179L879.847 212H840.173L866.522 0H919.522L946.173 212H906.196ZM886.813 139.314H899.23L892.87 72.9886L886.813 139.314Z" fill="white"/>
+<path d="M725.385 212V0H771.722V86.3143H784.745V0H831.082V212H784.745V125.989H771.722V212H725.385Z" fill="white"/>
+<path d="M610.298 185.651V26.6514C610.298 12.1143 622.109 0 636.646 0H689.646C704.183 0 716.298 12.1143 716.298 26.6514V72.9886H669.658V39.9771C669.658 36.3429 666.932 33.3143 662.995 33.3143C659.36 33.3143 656.635 36.3429 656.635 39.9771V172.326C656.635 175.96 659.36 178.989 662.995 178.989C666.932 178.989 669.658 175.96 669.658 172.326V139.314H716.298V185.651C716.298 200.189 704.183 212 689.646 212H636.646C622.109 212 610.298 200.189 610.298 185.651Z" fill="white"/>
+<path d="M541.841 33.3143V178.989H548.201C552.138 178.989 554.864 175.96 554.864 172.326V39.9771C554.864 36.3429 552.138 33.3143 548.201 33.3143H541.841ZM601.504 26.6514V185.651C601.504 200.189 589.389 212 574.852 212H495.504V0H574.852C589.389 0 601.504 12.1143 601.504 26.6514Z" fill="white"/>
+<path d="M407.037 212V0H486.385V33.3143H453.374V86.3143H486.385V125.989H453.374V178.989H486.385V212H407.037Z" fill="white"/>
+<path d="M338.587 125.989V178.989H345.249C348.884 178.989 351.609 175.96 351.609 172.326V132.651C351.609 129.017 348.884 125.989 345.249 125.989H338.587ZM371.598 106L372.204 106.303C386.741 106.606 398.249 118.114 398.249 132.651V185.651C398.249 200.189 386.135 212 371.598 212H292.249V0H371.598C386.135 0 398.249 12.1143 398.249 26.6514V79.6514C398.249 94.1886 386.135 106 371.598 106ZM338.587 86.3143H345.249C348.884 86.3143 351.609 83.2857 351.609 79.6514V39.9771C351.609 36.3429 348.884 33.3143 345.249 33.3143H338.587V86.3143Z" fill="white"/>
+<path d="M137.532 212V0H190.23L210.218 125.989L230.207 0H283.207V212H236.87V106L216.881 212H203.555L183.87 106V212H137.532Z" fill="white"/>
+<path d="M49.0637 212V0H128.412V33.3143H95.4008V86.3143H128.412V125.989H95.4008V178.989H128.412V212H49.0637Z" fill="white"/>
+<rect x="1125.62" y="65.6279" width="25.2381" height="25.238" fill="white"/>
+<path d="M1150.86 106H1160.95V121.143H1150.86V106Z" fill="white"/>
+<rect x="1125.62" y="121.118" width="15.1428" height="15.1429" fill="white"/>
+<rect x="1095.33" y="50.5068" width="25.2381" height="25.238" fill="#010312"/>
+<rect x="1105.43" y="111.022" width="15.1428" height="15.1429" fill="#010312"/>
+<rect x="1085.23" y="126.23" width="15.1428" height="15.1429" fill="#010312"/>
+<rect x="42.9069" y="55.4932" width="10.0952" height="25.238" fill="#010312"/>
+<rect x="45.4301" y="121.118" width="10.0952" height="20.1904" fill="#010312"/>
+<rect x="22.7124" y="55.5215" width="25.2381" height="25.238" fill="white"/>
+<path d="M0 121.148H10.0952V136.291H0V121.148Z" fill="white"/>
+<rect x="15.1404" y="141.375" width="15.1428" height="15.1429" fill="white"/>
+<rect x="60.5702" y="141.375" width="15.1428" height="15.1429" fill="#010312"/>
+</svg>

docs/logo/light.svg

@@ -0,0 +1,24 @@
+<svg width="1161" height="212" viewBox="0 0 1161 212" fill="none" xmlns="http://www.w3.org/2000/svg">
+<path d="M1010.3 212V0H1056.64L1076.33 106V0H1116V212H1069.66L1049.98 106V212H1010.3Z" fill="#010312"/>
+<path d="M1001.3 212H954.958V0H1001.3V212Z" fill="#010312"/>
+<path d="M906.196 212L903.167 178.989H883.179L879.847 212H840.173L866.522 0H919.522L946.173 212H906.196ZM886.813 139.314H899.23L892.87 72.9886L886.813 139.314Z" fill="#010312"/>
+<path d="M725.385 212V0H771.722V86.3143H784.745V0H831.082V212H784.745V125.989H771.722V212H725.385Z" fill="#010312"/>
+<path d="M610.298 185.651V26.6514C610.298 12.1143 622.109 0 636.646 0H689.646C704.183 0 716.298 12.1143 716.298 26.6514V72.9886H669.658V39.9771C669.658 36.3429 666.932 33.3143 662.995 33.3143C659.36 33.3143 656.635 36.3429 656.635 39.9771V172.326C656.635 175.96 659.36 178.989 662.995 178.989C666.932 178.989 669.658 175.96 669.658 172.326V139.314H716.298V185.651C716.298 200.189 704.183 212 689.646 212H636.646C622.109 212 610.298 200.189 610.298 185.651Z" fill="#010312"/>
+<path d="M541.841 33.3143V178.989H548.201C552.138 178.989 554.864 175.96 554.864 172.326V39.9771C554.864 36.3429 552.138 33.3143 548.201 33.3143H541.841ZM601.504 26.6514V185.651C601.504 200.189 589.389 212 574.852 212H495.504V0H574.852C589.389 0 601.504 12.1143 601.504 26.6514Z" fill="#010312"/>
+<path d="M407.037 212V0H486.385V33.3143H453.374V86.3143H486.385V125.989H453.374V178.989H486.385V212H407.037Z" fill="#010312"/>
+<path d="M338.587 125.989V178.989H345.249C348.884 178.989 351.609 175.96 351.609 172.326V132.651C351.609 129.017 348.884 125.989 345.249 125.989H338.587ZM371.598 106L372.204 106.303C386.741 106.606 398.249 118.114 398.249 132.651V185.651C398.249 200.189 386.135 212 371.598 212H292.249V0H371.598C386.135 0 398.249 12.1143 398.249 26.6514V79.6514C398.249 94.1886 386.135 106 371.598 106ZM338.587 86.3143H345.249C348.884 86.3143 351.609 83.2857 351.609 79.6514V39.9771C351.609 36.3429 348.884 33.3143 345.249 33.3143H338.587V86.3143Z" fill="#010312"/>
+<path d="M137.532 212V0H190.23L210.218 125.989L230.207 0H283.207V212H236.87V106L216.881 212H203.555L183.87 106V212H137.532Z" fill="#010312"/>
+<path d="M49.0637 212V0H128.412V33.3143H95.4008V86.3143H128.412V125.989H95.4008V178.989H128.412V212H49.0637Z" fill="#010312"/>
+<rect x="1125.62" y="65.6279" width="25.2381" height="25.238" fill="#010312"/>
+<path d="M1150.86 106H1160.95V121.143H1150.86V106Z" fill="#010312"/>
+<rect x="1125.62" y="121.118" width="15.1428" height="15.1429" fill="#010312"/>
+<rect x="1095.33" y="50.5068" width="25.2381" height="25.238" fill="white"/>
+<rect x="1105.43" y="111.022" width="15.1428" height="15.1429" fill="white"/>
+<rect x="1085.23" y="126.23" width="15.1428" height="15.1429" fill="white"/>
+<rect x="42.9069" y="55.4932" width="10.0952" height="25.238" fill="white"/>
+<rect x="45.4301" y="121.118" width="10.0952" height="20.1904" fill="white"/>
+<rect x="22.7124" y="55.5215" width="25.2381" height="25.238" fill="#010312"/>
+<path d="M0 121.148H10.0952V136.291H0V121.148Z" fill="#010312"/>
+<rect x="15.1404" y="141.375" width="15.1428" height="15.1429" fill="#010312"/>
+<rect x="60.5702" y="141.375" width="15.1428" height="15.1429" fill="white"/>
+</svg>

docs/mint.json

@@ -0,0 +1,176 @@
+{
+  "$schema": "https://mintlify.com/schema.json",
+  "name": "Embedchain",
+  "logo": {
+    "dark": "/logo/dark.svg",
+    "light": "/logo/light.svg",
+    "href": "https://embedchain.ai/"
+  },
+  "favicon": "/favicon.png",
+  "colors": {
+    "primary": "#3B2FC9",
+    "light": "#6673FF",
+    "dark": "#3B2FC9",
+    "background": {
+      "dark": "#0f1117",
+      "light": "#fff"
+    }
+  },
+  "modeToggle": {
+    "default": "dark"
+  },
+  "openapi": ["/rest-api.json"],
+  "metadata": {
+    "og:image": "/images/og.png",
+    "twitter:site": "@embedchain"
+  },
+  "anchors": [
+    {
+      "name": "Embedchain Platform",
+      "icon": "tv",
+      "url": "https://app.embedchain.ai/"
+    },
+    {
+      "name": "Join our slack",
+      "icon": "slack",
+      "url": "https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw"
+    }
+  ],
+  "topbarLinks": [
+    {
+      "name": "Create account",
+      "url": "https://app.embedchain.ai/login/"
+    }
+  ],
+  "topbarCtaButton": {
+    "name": "Get started",
+    "url": "https://app.embedchain.ai"
+  },
+  "primaryTab": {
+    "name": "Docs"
+  },
+  "navigation": [
+    {
+      "group": "Get started",
+      "pages": [
+        "get-started/quickstart",
+        "get-started/introduction",
+        "get-started/openai-assistant",
+        "get-started/faq",
+        "get-started/examples"
+      ]
+    },
+    {
+      "group": "Components",
+      "pages": [
+        "components/llms",
+        "components/embedding-models",
+        "components/vector-databases"
+      ]
+    },
+    {
+      "group": "Data sources",
+      "pages": [
+        "data-sources/overview",
+        {
+          "group": "Supported data sources",
+          "pages": [
+            "data-sources/csv",
+            "data-sources/json",
+            "data-sources/docs-site",
+            "data-sources/docx",
+            "data-sources/mdx",
+            "data-sources/notion",
+            "data-sources/pdf-file",
+            "data-sources/qna",
+            "data-sources/sitemap",
+            "data-sources/text",
+            "data-sources/web-page",
+            "data-sources/openapi",
+            "data-sources/youtube-video",
+            "data-sources/discourse",
+            "data-sources/substack"
+          ]
+        },
+        "data-sources/data-type-handling"
+      ]
+    },
+    {
+      "group": "Advanced",
+      "pages": ["advanced/configuration"]
+    },
+    {
+      "group": "REST API",
+      "pages": [
+        "rest-api/getting-started",
+        "rest-api/create",
+        "rest-api/get-all-apps",
+        "rest-api/add-data",
+        "rest-api/get-data",
+        "rest-api/query",
+        "rest-api/deploy",
+        "rest-api/delete",
+        "rest-api/check-status"
+      ]
+    },
+    {
+      "group": "Use Cases",
+      "pages": [
+        "examples/full_stack",
+        "examples/discord_bot",
+        "examples/slack_bot",
+        "examples/telegram_bot",
+        "examples/whatsapp_bot",
+        "examples/poe_bot"
+      ]
+    },
+    {
+      "group": "Community",
+      "pages": ["community/connect-with-us", "community/showcase"]
+    },
+    {
+      "group": "Integrations",
+      "pages": ["integration/langsmith"]
+    },
+    {
+      "group": "Contribute",
+      "pages": [
+        "contribution/guidelines",
+        "contribution/dev",
+        "contribution/docs",
+        "contribution/python",
+        "contribution/javascript"
+      ]
+    },
+    {
+      "group": "Product",
+      "pages": ["product/release-notes"]
+    }
+  ],
+  "footerSocials": {
+    "website": "https://embedchain.ai",
+    "github": "https://github.com/embedchain/embedchain",
+    "slack": "https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw",
+    "discord": "https://discord.gg/6PzXDgEjG5",
+    "twitter": "https://twitter.com/embedchain",
+    "linkedin": "https://www.linkedin.com/company/embedchain"
+  },
+  "isWhiteLabeled": true,
+  "analytics": {
+    "posthog": {
+      "apiKey": "phc_PHQDA5KwztijnSojsxJ2c1DuJd52QCzJzT2xnSGvjN2",
+      "apiHost": "https://app.embedchain.ai/ingest"
+    }
+  },
+  "feedback": {
+    "suggestEdit": true,
+    "raiseIssue": true,
+    "thumbsRating": true
+  },
+  "search": {
+    "prompt": "✨ Search embedchain docs..."
+  },
+  "api": {
+    "baseUrl": "http://localhost:8080"
+  }
+}

docs/product/release-notes.mdx

@@ -0,0 +1,4 @@
+---
+title: ' 📜 Release Notes'
+url: https://github.com/embedchain/embedchain/releases
+---

docs/rest-api.json

@@ -0,0 +1,427 @@
+{
+    "openapi": "3.1.0",
+    "info": {
+      "title": "Embedchain REST API",
+      "description": "This is the REST API for Embedchain.",
+      "license": {
+        "name": "Apache 2.0",
+        "url": "https://github.com/embedchain/embedchain/blob/main/LICENSE"
+      },
+      "version": "0.0.1"
+    },
+    "paths": {
+      "/ping": {
+        "get": {
+          "tags": ["Utility"],
+          "summary": "Check status",
+          "description": "Endpoint to check the status of the API",
+          "operationId": "check_status_ping_get",
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": { "application/json": { "schema": {} } }
+            }
+          }
+        }
+      },
+      "/apps": {
+        "get": {
+          "tags": ["Apps"],
+          "summary": "Get all apps",
+          "description": "Get all applications",
+          "operationId": "get_all_apps_apps_get",
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": { "application/json": { "schema": {} } }
+            }
+          }
+        }
+      },
+      "/create": {
+        "post": {
+          "tags": ["Apps"],
+          "summary": "Create app",
+          "description": "Create a new app using App ID",
+          "operationId": "create_app_using_default_config_create_post",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "query",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "requestBody": {
+            "content": {
+              "multipart/form-data": {
+                "schema": {
+                  "allOf": [
+                    {
+                      "$ref": "#/components/schemas/Body_create_app_using_default_config_create_post"
+                    }
+                  ],
+                  "title": "Body"
+                }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/data": {
+        "get": {
+          "tags": ["Apps"],
+          "summary": "Get data",
+          "description": "Get all data sources for an app",
+          "operationId": "get_datasources_associated_with_app_id__app_id__data_get",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": { "application/json": { "schema": {} } }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/add": {
+        "post": {
+          "tags": ["Apps"],
+          "summary": "Add data",
+          "description": "Add a data source to an app.",
+          "operationId": "add_datasource_to_an_app__app_id__add_post",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": { "$ref": "#/components/schemas/SourceApp" }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/query": {
+        "post": {
+          "tags": ["Apps"],
+          "summary": "Query app",
+          "description": "Query an app",
+          "operationId": "query_an_app__app_id__query_post",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": { "$ref": "#/components/schemas/QueryApp" }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/chat": {
+        "post": {
+          "tags": ["Apps"],
+          "summary": "Chat",
+          "description": "Chat with an app.\n\napp_id: The ID of the app. Use \"default\" for the default app.\n\nmessage: The message that you want to send to the app.",
+          "operationId": "chat_with_an_app__app_id__chat_post",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": { "$ref": "#/components/schemas/MessageApp" }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/deploy": {
+        "post": {
+          "tags": ["Apps"],
+          "summary": "Deploy App",
+          "description": "Deploy an existing app.",
+          "operationId": "deploy_app__app_id__deploy_post",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "requestBody": {
+            "required": true,
+            "content": {
+              "application/json": {
+                "schema": { "$ref": "#/components/schemas/DeployAppRequest" }
+              }
+            }
+          },
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      },
+      "/{app_id}/delete": {
+        "delete": {
+          "tags": ["Apps"],
+          "summary": "Delete app",
+          "description": "Delete an existing app",
+          "operationId": "delete_app__app_id__delete_delete",
+          "parameters": [
+            {
+              "name": "app_id",
+              "in": "path",
+              "required": true,
+              "schema": { "type": "string", "title": "App Id" }
+            }
+          ],
+          "responses": {
+            "200": {
+              "description": "Successful Response",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/DefaultResponse" }
+                }
+              }
+            },
+            "422": {
+              "description": "Validation Error",
+              "content": {
+                "application/json": {
+                  "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "components": {
+      "schemas": {
+        "Body_create_app_using_default_config_create_post": {
+          "properties": {
+            "config": { "type": "string", "format": "binary", "title": "Config" }
+          },
+          "type": "object",
+          "title": "Body_create_app_using_default_config_create_post"
+        },
+        "DefaultResponse": {
+          "properties": { "response": { "type": "string", "title": "Response" } },
+          "type": "object",
+          "required": ["response"],
+          "title": "DefaultResponse"
+        },
+        "DeployAppRequest": {
+          "properties": {
+            "api_key": {
+              "type": "string",
+              "title": "Api Key",
+              "description": "The Embedchain API key for app deployments. You get the api key on the Embedchain platform by visiting [https://app.embedchain.ai](https://app.embedchain.ai)",
+              "default": ""
+            }
+          },
+          "type": "object",
+          "title": "DeployAppRequest",
+          "example":{
+            "api_key":"ec-xxx"
+         }
+        },
+        "HTTPValidationError": {
+          "properties": {
+            "detail": {
+              "items": { "$ref": "#/components/schemas/ValidationError" },
+              "type": "array",
+              "title": "Detail"
+            }
+          },
+          "type": "object",
+          "title": "HTTPValidationError"
+        },
+        "MessageApp": {
+          "properties": {
+            "message": {
+              "type": "string",
+              "title": "Message",
+              "description": "The message that you want to send to the App.",
+              "default": ""
+            }
+          },
+          "type": "object",
+          "title": "MessageApp"
+        },
+        "QueryApp": {
+          "properties": {
+            "query": {
+              "type": "string",
+              "title": "Query",
+              "description": "The query that you want to ask the App.",
+              "default": ""
+            }
+          },
+          "type": "object",
+          "title": "QueryApp",
+          "example":{
+            "query":"Who is Elon Musk?"
+         }
+        },
+        "SourceApp": {
+          "properties": {
+            "source": {
+              "type": "string",
+              "title": "Source",
+              "description": "The source that you want to add to the App.",
+              "default": ""
+            },
+            "data_type": {
+              "anyOf": [{ "type": "string" }, { "type": "null" }],
+              "title": "Data Type",
+              "description": "The type of data to add, remove it if you want Embedchain to detect it automatically.",
+              "default": ""
+            }
+          },
+          "type": "object",
+          "title": "SourceApp",
+          "example":{
+            "source":"https://en.wikipedia.org/wiki/Elon_Musk"
+         }
+        },
+        "ValidationError": {
+          "properties": {
+            "loc": {
+              "items": { "anyOf": [{ "type": "string" }, { "type": "integer" }] },
+              "type": "array",
+              "title": "Location"
+            },
+            "msg": { "type": "string", "title": "Message" },
+            "type": { "type": "string", "title": "Error Type" }
+          },
+          "type": "object",
+          "required": ["loc", "msg", "type"],
+          "title": "ValidationError"
+        }
+      }
+    }
+  }

docs/rest-api/add-data.mdx

@@ -0,0 +1,22 @@
+---
+openapi: post /{app_id}/add
+---
+
+<RequestExample>
+
+```bash Request
+curl --request POST \
+  --url http://localhost:8080/{app_id}/add \
+  -d "source=https://www.forbes.com/profile/elon-musk" \
+  -d "data_type=web_page"
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "response": "fec7fe91e6b2d732938a2ec2e32bfe3f" }
+```
+
+</ResponseExample>

docs/rest-api/chat.mdx

@@ -0,0 +1,3 @@
+---
+openapi: post /{app_id}/chat
+---

docs/rest-api/check-status.mdx

@@ -0,0 +1,20 @@
+---
+openapi: get /ping
+---
+
+<RequestExample>
+
+```bash Request
+  curl --request GET \
+    --url http://localhost:8080/ping
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "ping": "pong" }
+```
+
+</ResponseExample>

docs/rest-api/create.mdx

@@ -0,0 +1,95 @@
+---
+openapi: post /create
+---
+
+<RequestExample>
+
+```bash Request
+curl --request POST \
+  --url http://localhost:8080/create?app_id=app1 \
+  -F "config=@/path/to/config.yaml"
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "response": "App created successfully. App ID: app1" }
+```
+
+</ResponseExample>
+
+By default we will use the opensource **gpt4all** model to get started. You can also specify your own config by uploading a config YAML file.
+
+For example, create a `config.yaml` file (adjust according to your requirements):
+
+```yaml
+app:
+  config:
+    id: "default-app"
+
+llm:
+  provider: openai
+  config:
+    model: "gpt-3.5-turbo"
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+    template: |
+      Use the following pieces of context to answer the query at the end.
+      If you don't know the answer, just say that you don't know, don't try to make up an answer.
+
+      $context
+
+      Query: $query
+
+      Helpful Answer:
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: "rest-api-app"
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: openai
+  config:
+    model: "text-embedding-ada-002"
+```
+
+To learn more about custom configurations, check out the [custom configurations docs](https://docs.embedchain.ai/advanced/configuration). To explore more examples of config yamls for embedchain, visit [embedchain/configs](https://github.com/embedchain/embedchain/tree/main/configs).
+
+Now, you can upload this config file in the request body.
+
+For example,
+
+```bash Request
+curl --request POST \
+  --url http://localhost:8080/create?app_id=my-app \
+  -F "config=@/path/to/config.yaml"
+```
+
+**Note:** To use custom models, an **API key** might be required. Refer to the table below to determine the necessary API key for your provider.
+
+| Keys                       | Providers                      |
+| -------------------------- | ------------------------------ |
+| `OPENAI_API_KEY `          | OpenAI, Azure OpenAI, Jina etc |
+| `OPENAI_API_TYPE`          | Azure OpenAI                   |
+| `OPENAI_API_BASE`          | Azure OpenAI                   |
+| `OPENAI_API_VERSION`       | Azure OpenAI                   |
+| `COHERE_API_KEY`           | Cohere                         |
+| `ANTHROPIC_API_KEY`        | Anthropic                      |
+| `JINACHAT_API_KEY`         | Jina                           |
+| `HUGGINGFACE_ACCESS_TOKEN` | Huggingface                    |
+| `REPLICATE_API_TOKEN`      | LLAMA2                         |
+
+To add env variables, you can simply run the docker command with the `-e` flag.
+
+For example,
+
+```bash
+docker run --name embedchain -p 8080:8080 -e OPENAI_API_KEY=<YOUR_OPENAI_API_KEY> embedchain/rest-api:latest
+```

docs/rest-api/delete.mdx

@@ -0,0 +1,21 @@
+---
+openapi: delete /{app_id}/delete
+---
+
+
+<RequestExample>
+
+```bash Request
+  curl --request DELETE \
+    --url http://localhost:8080/{app_id}/delete
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "response": "App with id {app_id} deleted successfully." }
+```
+
+</ResponseExample>

docs/rest-api/deploy.mdx

@@ -0,0 +1,22 @@
+---
+openapi: post /{app_id}/deploy
+---
+
+
+<RequestExample>
+
+```bash Request
+curl --request POST \
+  --url http://localhost:8080/{app_id}/deploy \
+  -d "api_key=ec-xxxx"
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{ "response": "App deployed successfully." }
+```
+
+</ResponseExample>

docs/rest-api/get-all-apps.mdx

@@ -0,0 +1,33 @@
+---
+openapi: get /apps
+---
+
+<RequestExample>
+
+```bash Request
+curl --request GET \
+  --url http://localhost:8080/apps
+```
+
+</RequestExample>
+
+<ResponseExample>
+
+```json Response
+{
+  "results": [
+    {
+      "config": "config1.yaml",
+      "id": 1,
+      "app_id": "app1"
+    },
+    {
+      "config": "config2.yaml",
+      "id": 2,
+      "app_id": "app2"
+    }
+  ]
+}
+```
+
+</ResponseExample>