[Feature] Add support for optionally fetch all chat history for app (#1249)

Co-authored-by: Deven Patel <deven298@yahoo.com>

.env.example

@@ -1 +1 @@
-OPENAI_API_KEY=
+OPENAI_API_KEY="your-openai-api-key"

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -5,7 +5,7 @@ body:
 - type: markdown
   attributes:
     value: >
-      #### Before submitting a bug, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/gventuri/pandas-ai/issues?q=is%3Aissue+sort%3Acreated-desc+).
+      #### Before submitting a bug, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/embedchain/embedchain/issues?q=is%3Aissue+sort%3Acreated-desc+).
 - type: textarea
   attributes:
     label: 🐛 Describe the bug

.github/workflows/cd.yml

@@ -23,10 +23,10 @@ jobs:
         run: |
           curl -sSL https://install.python-poetry.org | python3 -
           echo "$HOME/.local/bin" >> $GITHUB_PATH
-  
+
       - name: Install dependencies
         run: poetry install
-  
+
       - name: Build a binary wheel and a source tarball
         run: poetry build
 
@@ -37,4 +37,4 @@ jobs:
 
       - name: Publish distribution 📦 to PyPI
         if: startsWith(github.ref, 'refs/tags')
-        uses: pypa/gh-action-pypi-publish@release/v1
+        uses: pypa/gh-action-pypi-publish@release/v1

.github/workflows/ci.yml

@@ -3,7 +3,15 @@ name: ci
 on:
   push:
     branches: [main]
+    paths:
+      - 'embedchain/**'
+      - 'tests/**'
+      - 'examples/**'
   pull_request:
+    paths:
+      - 'embedchain/**'
+      - 'tests/**'
+      - 'examples/**'
 
 jobs:
   build:
@@ -19,10 +27,27 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install poetry
-        run: pip install poetry==1.4.2
+        uses: snok/install-poetry@v1
+        with:
+          version: 1.4.2
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v2
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
       - name: Install dependencies
         run: poetry install --all-extras
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
       - name: Lint with ruff
-        run: make ci_lint
-      - name: Test with pytest
-        run: make ci_test
+        run: make lint
+      - name: Run tests and generate coverage report
+        run: make coverage
+      - name: Upload coverage reports to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          file: coverage.xml
+        env:
+          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

.gitignore

@@ -76,7 +76,6 @@ docs/_build/
 target/
 
 # Jupyter Notebook
-.ipynb_checkpoints
 
 # IPython
 profile_default/
@@ -165,9 +164,17 @@ cython_debug/
 
 # Database
 db
+test-db
 
 .vscode
-/poetry.lock
 .idea/
 
 .DS_Store
+
+notebooks/*.yaml
+.ipynb_checkpoints/
+
+!configs/*.yaml
+
+# cache db
+*.db

CONTRIBUTING.md

@@ -1,10 +1,10 @@
 # Contributing to embedchain
 
-Let us make contributing easy, collaborative and fun.
+Let us make contribution easy, collaborative and fun.
 
 ## Submit your Contribution through PR
 
-To make a contribution, follow the following steps:
+To make a contribution, follow these steps:
 
 1. Fork and clone this repository
 2. Do the changes on your fork with dedicated feature branch `feature/f1`
@@ -35,7 +35,7 @@ poetry shell
 
 ### 📌 Pre-commit
 
-To ensure our standards, make sure to install pre-commit before star to contribute.
+To ensure our standards, make sure to install pre-commit before starting to contribute.
 
 ```bash
 pre-commit install
@@ -51,7 +51,7 @@ make lint
 
 Make sure that the linter does not report any errors or warnings before submitting a pull request.
 
-### Code Format with `black`
+### Code Formatting with `black`
 
 We use `black` to reformat the code by running the following command:
 

Makefile

@@ -4,27 +4,42 @@ PIP := $(PYTHON) -m pip
 PROJECT_NAME := embedchain
 
 # Targets
-.PHONY: install format lint clean test ci_lint ci_test
+.PHONY: install format lint clean test ci_lint ci_test coverage
 
 install:
-	$(PIP) install --upgrade pip
-	$(PIP) install -e .[dev]
+	poetry install
+
+install_all:
+	poetry install --all-extras
+
+install_es:
+	poetry install --extras elasticsearch
+
+install_opensearch:
+	poetry install --extras opensearch
+
+install_milvus:
+	poetry install --extras milvus
+
+shell:
+	poetry shell
+
+py_shell:
+	poetry run python
 
 format:
 	$(PYTHON) -m black .
 	$(PYTHON) -m isort .
 
-lint:
-	$(PYTHON) -m ruff .
-
 clean:
 	rm -rf dist build *.egg-info
 
-test:
-	$(PYTHON) -m pytest
-
-ci_lint:
+lint:
 	poetry run ruff .
 
-ci_test:
-	poetry run pytest
+# for example: make test file=tests/test_factory.py
+test:
+	poetry run pytest $(file)
+
+coverage:
+	poetry run pytest --cov=$(PROJECT_NAME) --cov-report=xml

README.md

@@ -1,50 +1,63 @@
-# embedchain
+<p align="center">
+  <img src="docs/logo/dark.svg" width="400px" alt="Embedchain Logo">
+</p>
 
-[![PyPI](https://img.shields.io/pypi/v/embedchain)](https://pypi.org/project/embedchain/)
-[![Discord](https://dcbadge.vercel.app/api/server/6PzXDgEjG5?style=flat)](https://discord.gg/CUU9FPhRNt)
-[![Twitter](https://img.shields.io/twitter/follow/embedchain)](https://twitter.com/embedchain)
-[![Substack](https://img.shields.io/badge/Substack-%23006f5c.svg?logo=substack)](https://embedchain.substack.com/)
-[![Open in Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/138lMWhENGeEu7Q1-6lNbNTHGLZXBBz_B?usp=sharing)
+<p align="center">
+  <a href="https://runacap.com/ross-index/q3-2023/" target="_blank" rel="noopener"><img style="width: 260px; height: 56px" src="https://runacap.com/wp-content/uploads/2023/10/ROSS_badge_black_Q3_2023.svg" alt="ROSS Index - Fastest Growing Open-Source Startups in Q3 2023 | Runa Capital" width="260" height="56"/></a>
+</p>
 
-Embedchain is a framework to easily create LLM powered bots over any dataset. If you want a javascript version, check out [embedchain-js](https://github.com/embedchain/embedchainjs)
+<p align="center">
+  <a href="https://pypi.org/project/embedchain/">
+    <img src="https://img.shields.io/pypi/v/embedchain" alt="PyPI">
+  </a>
+  <a href="https://pepy.tech/project/embedchain">
+    <img src="https://static.pepy.tech/badge/embedchain" alt="Downloads">
+  </a>
+  <a href="https://embedchain.ai/slack">
+    <img src="https://img.shields.io/badge/slack-embedchain-brightgreen.svg?logo=slack" alt="Slack">
+  </a>
+  <a href="https://embedchain.ai/discord">
+    <img src="https://dcbadge.vercel.app/api/server/6PzXDgEjG5?style=flat" alt="Discord">
+  </a>
+  <a href="https://twitter.com/embedchain">
+    <img src="https://img.shields.io/twitter/follow/embedchain" alt="Twitter">
+  </a>
+  <a href="https://colab.research.google.com/drive/138lMWhENGeEu7Q1-6lNbNTHGLZXBBz_B?usp=sharing">
+    <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab">
+  </a>
+  <a href="https://codecov.io/gh/embedchain/embedchain">
+    <img src="https://codecov.io/gh/embedchain/embedchain/graph/badge.svg?token=EMRRHZXW1Q" alt="codecov">
+  </a>
+</p>
 
-## 🤝 Schedule a 1-on-1 Session
+<hr />
 
-Book a [1-on-1 Session](https://cal.com/taranjeetio/ec) with Taranjeet, the founder, to discuss any issues, provide feedback, or explore how we can improve Embedchain for you.
+## What is Embedchain?
+
+Embedchain is an Open Source RAG Framework that makes it easy to create and deploy AI apps. At its core, Embedchain follows the design principle of being *"Conventional but Configurable"* to serve both software engineers and machine learning engineers.
+
+Embedchain streamlines the creation of Retrieval-Augmented Generation (RAG) applications, offering a seamless process for managing various types of unstructured data. It efficiently segments data into manageable chunks, generates relevant embeddings, and stores them in a vector database for optimized retrieval. With a suite of diverse APIs, it enables users to extract contextual information, find precise answers, or engage in interactive chat conversations, all tailored to their own data.
 
 ## 🔧 Quick install
 
+### Python API
+
 ```bash
 pip install embedchain
 ```
 
-## 🔍 Demo
+## ✨ Live demo
 
-Try out embedchain in your browser:
+Checkout the [Chat with PDF](https://embedchain.ai/demo/chat-pdf) live demo we created using Embedchain. You can find the source code [here](https://github.com/embedchain/embedchain/tree/main/examples/chat-pdf).
 
-[![Open in Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/138lMWhENGeEu7Q1-6lNbNTHGLZXBBz_B?usp=sharing)
+## 🔍 Usage
 
-## 📖 Documentation
+<!-- Demo GIF or Image -->
+<p align="center">
+  <img src="docs/images/cover.gif" width="900px" alt="Embedchain Demo">
+</p>
 
-The documentation for embedchain can be found at [docs.embedchain.ai](https://docs.embedchain.ai).
-
-## 💻 Usage
-
-Embedchain empowers you to create chatbot models similar to ChatGPT, using your own evolving dataset.
-
-### Data Types Supported
-
-* Youtube video
-* PDF file
-* Web page
-* Sitemap
-* Doc file
-* Code documentation website loader
-* Notion
-
-### Queries
-
-For example, you can use Embedchain to create an Elon Musk bot using the following code:
+For example, you can create an Elon Musk bot using the following code:
 
 ```python
 import os
@@ -56,21 +69,49 @@ elon_bot = App()
 
 # Embed online resources
 elon_bot.add("https://en.wikipedia.org/wiki/Elon_Musk")
-elon_bot.add("https://tesla.com/elon-musk")
-elon_bot.add("https://www.youtube.com/watch?v=MxZpaJK74Y4")
+elon_bot.add("https://www.forbes.com/profile/elon-musk")
 
 # Query the bot
-elon_bot.query("How many companies does Elon Musk run?")
-# Answer: Elon Musk runs four companies: Tesla, SpaceX, Neuralink, and The Boring Company
+elon_bot.query("How many companies does Elon Musk run and name those?")
+# Answer: Elon Musk currently runs several companies. As of my knowledge, he is the CEO and lead designer of SpaceX, the CEO and product architect of Tesla, Inc., the CEO and founder of Neuralink, and the CEO and founder of The Boring Company. However, please note that this information may change over time, so it's always good to verify the latest updates.
 ```
 
-## 🤝 Contributing
+You can also try it in your browser with Google Colab:
+
+[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/17ON1LPonnXAtLaZEebnOktstB_1cJJmh?usp=sharing)
+
+## 📖 Documentation
+Comprehensive guides and API documentation are available to help you get the most out of Embedchain:
+
+- [Introduction](https://docs.embedchain.ai/get-started/introduction#what-is-embedchain)
+- [Getting Started](https://docs.embedchain.ai/get-started/quickstart)
+- [Examples](https://docs.embedchain.ai/examples)
+- [Supported data types](https://docs.embedchain.ai/components/data-sources/overview)
+
+## 🔗 Join the Community
+
+* Connect with fellow developers by joining our [Slack Community](https://embedchain.ai/slack) or [Discord Community](https://embedchain.ai/discord).
+
+* Dive into [GitHub Discussions](https://github.com/embedchain/embedchain/discussions), ask questions, or share your experiences.
+
+## 🤝 Schedule a 1-on-1 Session
+
+Book a [1-on-1 Session](https://cal.com/taranjeetio/ec) with the founders, to discuss any issues, provide feedback, or explore how we can improve Embedchain for you.
+
+## 🌐 Contributing
 
 Contributions are welcome! Please check out the issues on the repository, and feel free to open a pull request.
 For more information, please see the [contributing guidelines](CONTRIBUTING.md).
 
-For more refrence, please go through [Development Guide](https://docs.embedchain.ai/contribution/dev) and [Documentation Guide](https://docs.embedchain.ai/contribution/docs).
+For more reference, please go through [Development Guide](https://docs.embedchain.ai/contribution/dev) and [Documentation Guide](https://docs.embedchain.ai/contribution/docs).
 
+<a href="https://github.com/embedchain/embedchain/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=embedchain/embedchain" />
+</a>
+
+## Anonymous Telemetry
+
+We collect anonymous usage metrics to enhance our package's quality and user experience. This includes data like feature usage frequency and system info, but never personal details. The data helps us prioritize improvements and ensure compatibility. If you wish to opt-out, set the environment variable `EC_TELEMETRY=false`. We prioritize data security and don't share this data externally.
 
 ## Citation
 
@@ -78,8 +119,8 @@ If you utilize this repository, please consider citing it with:
 
 ```
 @misc{embedchain,
-  author = {Taranjeet Singh},
-  title = {Embedchain: Framework to easily create LLM powered bots over any dataset},
+  author = {Taranjeet Singh, Deshraj Yadav},
+  title = {Embedchain: The Open Source RAG Framework},
   year = {2023},
   publisher = {GitHub},
   journal = {GitHub repository},

configs/anthropic.yaml

@@ -0,0 +1,8 @@
+llm:
+  provider: anthropic
+  config:
+    model: 'claude-instant-1'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false

configs/azure_openai.yaml

@@ -0,0 +1,19 @@
+app:
+  config:
+    id: azure-openai-app
+
+llm:
+  provider: azure_openai
+  config:
+    model: gpt-35-turbo
+    deployment_name: your_llm_deployment_name
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: azure_openai
+  config:
+    model: text-embedding-ada-002
+    deployment_name: you_embedding_model_deployment_name

configs/chroma.yaml

@@ -0,0 +1,24 @@
+app:
+  config:
+    id: 'my-app'
+
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-app'
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-ada-002'

configs/chunker.yaml

@@ -0,0 +1,4 @@
+chunker:
+  chunk_size: 100
+  chunk_overlap: 20
+  length_function: 'len'

configs/cohere.yaml

@@ -0,0 +1,7 @@
+llm:
+  provider: cohere
+  config:
+    model: large
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1

configs/full-stack.yaml

@@ -0,0 +1,40 @@
+app:
+  config:
+    id: 'full-stack-app'
+
+chunker:
+  chunk_size: 100
+  chunk_overlap: 20
+  length_function: 'len'
+
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+    prompt: |
+      Use the following pieces of context to answer the query at the end.
+      If you don't know the answer, just say that you don't know, don't try to make up an answer.
+
+      $context
+
+      Query: $query
+
+      Helpful Answer:
+    system_prompt: |
+      Act as William Shakespeare. Answer the following questions in the style of William Shakespeare.
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-collection-name'
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-ada-002'

configs/google.yaml

@@ -0,0 +1,13 @@
+llm:
+  provider: google
+  config:
+    model: gemini-pro
+    max_tokens: 1000
+    temperature: 0.9
+    top_p: 1.0
+    stream: false
+
+embedder:
+  provider: google
+  config:
+    model: models/embedding-001

configs/gpt4.yaml

@@ -0,0 +1,8 @@
+llm:
+  provider: openai
+  config:
+    model: 'gpt-4'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false

configs/gpt4all.yaml

@@ -0,0 +1,11 @@
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: gpt4all

configs/huggingface.yaml

@@ -0,0 +1,8 @@
+llm:
+  provider: huggingface
+  config:
+    model: 'google/flan-t5-xxl'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false

configs/jina.yaml

@@ -0,0 +1,7 @@
+llm:
+  provider: jina
+  config:
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false

configs/llama2.yaml

@@ -0,0 +1,8 @@
+llm:
+  provider: llama2
+  config:
+    model: 'a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false

configs/ollama.yaml

@@ -0,0 +1,12 @@
+llm:
+  provider: ollama
+  config:
+    model: 'llama2'
+    temperature: 0.5
+    top_p: 1
+    stream: true
+
+embedder:
+  provider: huggingface
+  config:
+    model: 'BAAI/bge-small-en-v1.5'

configs/opensearch.yaml

@@ -0,0 +1,33 @@
+app:
+  config:
+    id: 'my-app'
+    log_level: 'WARNING'
+    collect_metrics: true
+    collection_name: 'my-app'
+
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+vectordb:
+  provider: opensearch
+  config:
+    opensearch_url: 'https://localhost:9200'
+    http_auth:
+      - admin
+      - admin
+    vector_dimension: 1536
+    collection_name: 'my-app'
+    use_ssl: false
+    verify_certs: false
+
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-ada-002'
+    deployment_name: 'my-app'

configs/opensource.yaml

@@ -0,0 +1,25 @@
+app:
+  config:
+    id: 'open-source-app'
+    collect_metrics: false
+
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'open-source-app'
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: gpt4all
+  config:
+    deployment_name: 'test-deployment'

configs/pinecone.yaml

@@ -0,0 +1,6 @@
+vectordb:
+  provider: pinecone
+  config:
+    metric: cosine
+    vector_dimension: 1536
+    collection_name: my-pinecone-index

configs/pipeline.yaml

@@ -0,0 +1,26 @@
+pipeline:
+  config:
+    name: Example pipeline
+    id: pipeline-1  # Make sure that id is different every time you create a new pipeline
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: pipeline-1
+    dir: db
+    allow_reset: true
+
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedding_model:
+  provider: gpt4all
+  config:
+    model: 'all-MiniLM-L6-v2'
+    deployment_name: null

configs/together.yaml

@@ -0,0 +1,6 @@
+llm:
+  provider: together
+  config:
+    model: mistralai/Mixtral-8x7B-Instruct-v0.1
+    temperature: 0.5
+    max_tokens: 1000

configs/vertexai.yaml

@@ -0,0 +1,6 @@
+llm:
+  provider: vertexai
+  config:
+    model: 'chat-bison'
+    temperature: 0.5
+    top_p: 0.5

configs/vllm.yaml

@@ -0,0 +1,14 @@
+llm:
+  provider: vllm
+  config:
+    model: 'meta-llama/Llama-2-70b-hf'
+    temperature: 0.5
+    top_p: 1
+    top_k: 10
+    stream: true
+    trust_remote_code: true
+
+embedder:
+  provider: huggingface
+  config:
+    model: 'BAAI/bge-small-en-v1.5'

configs/weaviate.yaml

@@ -0,0 +1,4 @@
+vectordb:
+  provider: weaviate
+  config:
+    collection_name: my_weaviate_index

docs/Makefile

@@ -0,0 +1,10 @@
+install:
+	npm i -g mintlify
+
+run_local:
+	mintlify dev
+
+troubleshoot:
+	mintlify install
+
+.PHONY: install run_local troubleshoot

docs/_snippets/get-help.mdx

@@ -0,0 +1,11 @@
+<CardGroup cols={3}>
+  <Card title="Talk to founders" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call
+  </Card>
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
+    Join our slack community
+  </Card>
+  <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
+    Join our discord community
+  </Card>
+</CardGroup>

docs/_snippets/missing-data-source-tip.mdx

@@ -0,0 +1,19 @@
+<p>If you can't find the specific data source, please feel free to request through one of the following channels and help us prioritize.</p>
+
+<CardGroup cols={2}>
+  <Card title="Google Form" icon="file" href="https://forms.gle/NDRCKsRpUHsz2Wcm8" color="#7387d0">
+    Fill out this form
+  </Card>
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
+    Let us know on our slack community
+  </Card>
+  <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
+    Let us know on discord community
+  </Card>
+  <Card title="GitHub" icon="github" href="https://github.com/embedchain/embedchain/issues/new?assignees=&labels=&projects=&template=feature_request.yml" color="#181717">
+  Open an issue on our GitHub
+  </Card>
+  <Card title="Schedule a call" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call with Embedchain founder
+  </Card>
+</CardGroup>

docs/_snippets/missing-llm-tip.mdx

@@ -0,0 +1,16 @@
+<p>If you can't find the specific LLM you need, no need to fret. We're continuously expanding our support for additional LLMs, and you can help us prioritize by opening an issue on our GitHub or simply reaching out to us on our Slack or Discord community.</p>
+
+<CardGroup cols={2}>
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
+    Let us know on our slack community
+  </Card>
+  <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
+    Let us know on discord community
+  </Card>
+  <Card title="GitHub" icon="github" href="https://github.com/embedchain/embedchain/issues/new?assignees=&labels=&projects=&template=feature_request.yml" color="#181717">
+  Open an issue on our GitHub
+  </Card>
+  <Card title="Schedule a call" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call with Embedchain founder
+  </Card>
+</CardGroup>

docs/_snippets/missing-vector-db-tip.mdx

@@ -0,0 +1,18 @@
+
+
+<p>If you can't find the specific vector database, please feel free to request through one of the following channels and help us prioritize.</p>
+
+<CardGroup cols={2}>
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
+    Let us know on our slack community
+  </Card>
+  <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
+    Let us know on discord community
+  </Card>
+  <Card title="GitHub" icon="github" href="https://github.com/embedchain/embedchain/issues/new?assignees=&labels=&projects=&template=feature_request.yml" color="#181717">
+  Open an issue on our GitHub
+  </Card>
+  <Card title="Schedule a call" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call with Embedchain founder
+  </Card>
+</CardGroup>

docs/advanced/adding_data.mdx

@@ -1,25 +0,0 @@
----
-title: '➕ Adding Data'
----
-
-## Add Dataset
-
-- This step assumes that you have already created an `app` instance by either using `App`, `OpenSourceApp` or `CustomApp`. We are calling our app instance as `naval_chat_bot` 🤖
-
-- Now use `.add` method to add any dataset.
-
-```python
-# naval_chat_bot = App() or
-# naval_chat_bot = OpenSourceApp()
-
-# Embed Online Resources
-naval_chat_bot.add("https://www.youtube.com/watch?v=3qHkcs3kG44")
-naval_chat_bot.add("https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
-naval_chat_bot.add("https://nav.al/feedback")
-naval_chat_bot.add("https://nav.al/agi")
-
-# Embed Local Resources
-naval_chat_bot.add(("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."))
-```
-
-The possible formats to add data can be found on the [Supported Data Formats](/advanced/data_types) page.

docs/advanced/app_types.mdx

@@ -1,128 +0,0 @@
----
-title: '📱 App types'
----
-
-## App Types
-
-We have three types of App.
-
-### App
-
-```python
-from embedchain import App
-app = App()
-```
-
-- `App` uses OpenAI's model, so these are paid models. 💸 You will be charged for embedding model usage and LLM usage.
-- `App` uses OpenAI's embedding model to create embeddings for chunks and ChatGPT API as LLM to get answer given the relevant docs. Make sure that you have an OpenAI account and an API key. If you have don't have an API key, you can create one by visiting [this link](https://platform.openai.com/account/api-keys).
-- `App` is opinionated. It uses the best embedding model and LLM on the market.
-- Once you have the API key, set it in an environment variable called `OPENAI_API_KEY`
-
-```python
-import os
-os.environ["OPENAI_API_KEY"] = "sk-xxxx"
-```
-
-### Llama2App
-
-```python
-import os
-
-from embedchain import Llama2App
-
-os.environ['REPLICATE_API_TOKEN'] = "REPLICATE API TOKEN"
-
-zuck_bot = Llama2App()
-
-# Embed your data
-zuck_bot.add("https://www.youtube.com/watch?v=Ff4fRgnuFgQ")
-zuck_bot.add("https://en.wikipedia.org/wiki/Mark_Zuckerberg")
-
-# Nice, your bot is ready now. Start asking questions to your bot.
-zuck_bot.query("Who is Mark Zuckerberg?")
-# Answer: Mark Zuckerberg is an American internet entrepreneur and business magnate. He is the co-founder and CEO of Facebook. Born in 1984, he dropped out of Harvard University to focus on his social media platform, which has since grown to become one of the largest and most influential technology companies in the world.
-
-# Enable web search for your bot
-zuck_bot.online = True # enable internet access for the bot
-zuck_bot.query("Who owns the new threads app and when it was founded?")
-# Answer: Based on the context provided, the new Threads app is owned by Meta, the parent company of Facebook, Instagram, and WhatsApp.
-```
-
-- `Llama2App` uses Replicate's LLM model, so these are paid models. You can get the `REPLICATE_API_TOKEN` by registering on [their website](https://replicate.com/account).
-- `Llama2App` uses OpenAI's embedding model to create embeddings for chunks. Make sure that you have an OpenAI account and an API key. If you have don't have an API key, you can create one by visiting [this link](https://platform.openai.com/account/api-keys).
-
-
-### OpenSourceApp
-
-```python
-from embedchain import OpenSourceApp
-app = OpenSourceApp()
-```
-
-- `OpenSourceApp` uses open source embedding and LLM model. It uses `all-MiniLM-L6-v2` from Sentence Transformers library as the embedding model and `gpt4all` as the LLM.
-- Here there is no need to setup any api keys. You just need to install embedchain package and these will get automatically installed. 📦
-- Once you have imported and instantiated the app, every functionality from here onwards is the same for either type of app. 📚
-- `OpenSourceApp` is opinionated. It uses the best open source embedding model and LLM on the market.
-- extra dependencies are required for this app type. Install them with `pip install embedchain[opensource]`.
-
-### CustomApp
-
-```python
-from embedchain import CustomApp
-from embedchain.config import CustomAppConfig
-from embedchain.models import Providers, EmbeddingFunctions
-
-config = CustomAppConfig(embedding_fn=EmbeddingFunctions.OPENAI, provider=Providers.OPENAI)
-app = CustomApp(config)
-```
-
-- `CustomApp` is not opinionated.
-- Configuration required. It's for advanced users who want to mix and match different embedding models and LLMs. Configuration required.
-- while it's doing that, it's still providing abstractions through `Providers`.
-- paid and free/open source providers included.
-- Once you have imported and instantiated the app, every functionality from here onwards is the same for either type of app. 📚
-- Following providers are available for an LLM
-    - OPENAI
-    - ANTHPROPIC
-    - VERTEX_AI
-    - GPT4ALL
-    - AZURE_OPENAI
-- Following embedding functions are available for an embedding function
-    - OPENAI
-    - HUGGING_FACE
-    - VERTEX_AI
-    - GPT4ALL
-    - AZURE_OPENAI
-
-
-### PersonApp
-
-```python
-from embedchain import PersonApp
-naval_chat_bot = PersonApp("name_of_person_or_character") #Like "Yoda"
-```
-
-- `PersonApp` uses OpenAI's model, so these are paid models. 💸 You will be charged for embedding model usage and LLM usage.
-- `PersonApp` uses OpenAI's embedding model to create embeddings for chunks and ChatGPT API as LLM to get answer given the relevant docs. Make sure that you have an OpenAI account and an API key. If you have don't have an API key, you can create one by visiting [this link](https://platform.openai.com/account/api-keys).
-- Once you have the API key, set it in an environment variable called `OPENAI_API_KEY`
-
-```python
-import os
-os.environ["OPENAI_API_KEY"] = "sk-xxxx"
-```
-
-#### Compatibility with other apps
-
-- If there is any other app instance in your script or app, you can change the import as
-
-```python
-from embedchain import App as EmbedChainApp
-from embedchain import OpenSourceApp as EmbedChainOSApp
-from embedchain import PersonApp as EmbedChainPersonApp
-
-# or
-
-from embedchain import App as ECApp
-from embedchain import OpenSourceApp as ECOSApp
-from embedchain import PersonApp as ECPApp
-```

docs/advanced/configuration.mdx

@@ -1,89 +0,0 @@
----
-title: '⚙️ Custom configurations'
----
-
-Embedchain is made to work out of the box. However, for advanced users we're also offering configuration options. All of these configuration options are optional and have sane defaults.
-
-## Examples
-
-### General
-
-Here's the readme example with configuration options.
-
-```python
-import os
-from embedchain import App
-from embedchain.config import AppConfig, AddConfig, QueryConfig, ChunkerConfig
-from chromadb.utils import embedding_functions
-
-# Example: set the log level for debugging
-config = AppConfig(log_level="DEBUG")
-naval_chat_bot = App(config)
-
-# Example: specify a custom collection name
-config = AppConfig(collection_name="naval_chat_bot")
-naval_chat_bot = App(config)
-
-# Example: define your own chunker config for `youtube_video`
-chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=100, length_function=len)
-naval_chat_bot.add("https://www.youtube.com/watch?v=3qHkcs3kG44", AddConfig(chunker=chunker_config))
-
-add_config = AddConfig()
-naval_chat_bot.add("https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf", config=add_config)
-naval_chat_bot.add("https://nav.al/feedback", config=add_config)
-naval_chat_bot.add("https://nav.al/agi", config=add_config)
-
-naval_chat_bot.add(("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."), config=add_config)
-
-query_config = QueryConfig()
-print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", config=query_config))
-```
-
-### Custom prompt template
-
-Here's the example of using custom prompt template with `.query`
-
-```python
-from embedchain.config import QueryConfig
-from embedchain.embedchain import App
-from string import Template
-import wikipedia
-
-einstein_chat_bot = App()
-
-# Embed Wikipedia page
-page = wikipedia.page("Albert Einstein")
-einstein_chat_bot.add(page.content)
-
-# Example: use your own custom template with `$context` and `$query`
-einstein_chat_template = Template("""
-        You are Albert Einstein, a German-born theoretical physicist,
-        widely ranked among the greatest and most influential scientists of all time.
-
-        Use the following information about Albert Einstein to respond to
-        the human's query acting as Albert Einstein.
-        Context: $context
-
-        Keep the response brief. If you don't know the answer, just say that you don't know, don't try to make up an answer.
-
-        Human: $query
-        Albert Einstein:""")
-query_config = QueryConfig(template=einstein_chat_template, system_prompt="You are Albert Einstein.")
-queries = [
-        "Where did you complete your studies?",
-        "Why did you win nobel prize?",
-        "Why did you divorce your first wife?",
-]
-for query in queries:
-        response = einstein_chat_bot.query(query, config=query_config)
-        print("Query: ", query)
-        print("Response: ", response)
-
-# Output
-# Query:  Where did you complete your studies?
-# Response:  I completed my secondary education at the Argovian cantonal school in Aarau, Switzerland.
-# Query:  Why did you win nobel prize?
-# Response:  I won the Nobel Prize in Physics in 1921 for my services to Theoretical Physics, particularly for my discovery of the law of the photoelectric effect.
-# Query:  Why did you divorce your first wife?
-# Response:  We divorced due to living apart for five years.
-```

docs/advanced/data_types.mdx

@@ -1,141 +0,0 @@
----
-title: '📋 Supported data formats'
----
-
-## Automatic data type detection
-The add method automatically tries to detect the data_type, based on your input for the source argument. So `app.add('https://www.youtube.com/watch?v=dQw4w9WgXcQ')` is enough to embed a YouTube video.
-
-This detection is implemented for all formats. It is based on factors such as whether it's a URL, a local file, the source data type, etc.
-
-### Debugging automatic detection
-
-
-Set `log_level=DEBUG` (in [AppConfig](http://localhost:3000/advanced/query_configuration#appconfig)) and make sure it's working as intended.
-
-Otherwise, you will not know when, for instance, an invalid filepath is interpreted as raw text instead.
-
-### Forcing a data type
-
-To omit any issues with the data type detection, you can **force** a data_type by adding it as a `add` method argument.
-The examples below show you the keyword to force the respective `data_type`.
-
-Forcing can also be used for edge cases, such as interpreting a sitemap as a web_page, for reading it's raw text instead of following links.
-
-## Remote Data Types
-
-<Tip>
-**Use local files in remote data types**
-
-Some data_types are meant for remote content and only work with URLs.
-You can pass local files by formatting the path using the `file:` [URI scheme](https://en.wikipedia.org/wiki/File_URI_scheme), e.g. `file:///info.pdf`.
-</Tip>
-
-### Youtube video
-
-To add any youtube video to your app, use the data_type (first argument to `.add()` method) as `youtube_video`. Eg:
-
-```python
-app.add('a_valid_youtube_url_here', data_type='youtube_video')
-```
-
-### PDF file
-
-To add any pdf file, use the data_type as `pdf_file`. Eg:
-
-```python
-app.add('a_valid_url_where_pdf_file_can_be_accessed', data_type='pdf_file')
-```
-
-Note that we do not support password protected pdfs.
-
-### Web page
-
-To add any web page, use the data_type as `web_page`. Eg:
-
-```python
-app.add('a_valid_web_page_url', data_type='web_page')
-```
-
-### Sitemap
-
-Add all web pages from an xml-sitemap. Filters non-text files. Use the data_type as `sitemap`. Eg:
-
-```python
-app.add('https://example.com/sitemap.xml', data_type='sitemap')
-```
-
-### Doc file
-
-To add any doc/docx file, use the data_type as `docx`. `docx` allows remote urls and conventional file paths. Eg:
-
-```python
-app.add('https://example.com/content/intro.docx', data_type="docx")
-app.add('content/intro.docx', data_type="docx")
-```
-
-### Code documentation website loader
-
-To add any code documentation website as a loader, use the data_type as `docs_site`. Eg:
-
-```python
-app.add("https://docs.embedchain.ai/", data_type="docs_site")
-```
-
-### Notion
-To use notion you must install the extra dependencies with `pip install embedchain[notion]`.
-
-To load a notion page, use the data_type as `notion`. Since it is hard to automatically detect, forcing this is advised.
-The next argument must **end** with the `notion page id`. The id is a 32-character string. Eg:
-
-```python
-app.add("cfbc134ca6464fc980d0391613959196", "notion")
-app.add("my-page-cfbc134ca6464fc980d0391613959196", "notion")
-app.add("https://www.notion.so/my-page-cfbc134ca6464fc980d0391613959196", "notion")
-```
-
-## Local Data Types
-
-### Text
-
-To supply your own text, use the data_type as `text` and enter a string. The text is not processed, this can be very versatile. Eg:
-
-```python
-app.add('Seek wealth, not money or status. Wealth is having assets that earn while you sleep. Money is how we transfer time and wealth. Status is your place in the social hierarchy.', data_type='text')
-```
-
-Note: This is not used in the examples because in most cases you will supply a whole paragraph or file, which did not fit.
-
-### QnA pair
-
-To supply your own QnA pair, use the data_type as `qna_pair` and enter a tuple. Eg:
-
-```python
-app.add(("Question", "Answer"), data_type="qna_pair")
-```
-
-## Reusing a vector database
-
-Default behavior is to create a persistent vector DB in the directory **./db**. You can split your application into two Python scripts: one to create a local vector DB and the other to reuse this local persistent vector DB. This is useful when you want to index hundreds of documents and separately implement a chat interface.
-
-Create a local index:
-
-```python
-from embedchain import App
-
-naval_chat_bot = App()
-naval_chat_bot.add("https://www.youtube.com/watch?v=3qHkcs3kG44")
-naval_chat_bot.add("https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
-```
-
-You can reuse the local index with the same code, but without adding new documents:
-
-```python
-from embedchain import App
-
-naval_chat_bot = App()
-print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
-```
-
-## More formats (coming soon!)
-
-- If you want to add any other format, please create an [issue](https://github.com/embedchain/embedchain/issues) and we will add it to the list of supported formats.

docs/advanced/interface_types.mdx

@@ -1,75 +0,0 @@
----
-title: '🤝 Interface types'
----
-
-## Interface Types
-
-The embedchain app exposes the following methods.
-
-### Query Interface
-
-- This interface is like a question answering bot. It takes a question and gets the answer. It does not maintain context about the previous chats.❓
-
-- To use this, call `.query()` function to get the answer for any query.
-
-```python
-print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
-# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
-```
-
-### Chat Interface
-
-- This interface is chat interface where it remembers previous conversation. Right now it remembers 5 conversation by default. 💬
-
-- To use this, call `.chat` function to get the answer for any query.
-
-```python
-print(naval_chat_bot.chat("How to be happy in life?"))
-# answer: The most important trick to being happy is to realize happiness is a skill you develop and a choice you make. You choose to be happy, and then you work at it. It's just like building muscles or succeeding at your job. It's about recognizing the abundance and gifts around you at all times.
-
-print(naval_chat_bot.chat("who is naval ravikant?"))
-# answer: Naval Ravikant is an Indian-American entrepreneur and investor.
-
-print(naval_chat_bot.chat("what did the author say about happiness?"))
-# answer: The author, Naval Ravikant, believes that happiness is a choice you make and a skill you develop. He compares the mind to the body, stating that just as the body can be molded and changed, so can the mind. He emphasizes the importance of being present in the moment and not getting caught up in regrets of the past or worries about the future. By being present and grateful for where you are, you can experience true happiness.
-```
-
-#### Dry Run
-
-Dry Run is an option in the `query` and `chat` methods that allows the user to not send their constructed prompt to the LLM, to save money. It's used for [testing](/advanced/testing#dry-run).
-
-
-### Stream Response
-
-- You can add config to your query method to stream responses like ChatGPT does. You would require a downstream handler to render the chunk in your desirable format. Supports both OpenAI model and OpenSourceApp. 📊
-
-- To use this, instantiate a `QueryConfig` or `ChatConfig` object with `stream=True`. Then pass it to the `.chat()` or `.query()` method. The following example iterates through the chunks and prints them as they appear.
-
-```python
-app = App()
-query_config = QueryConfig(stream = True)
-resp = app.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", query_config)
-
-for chunk in resp:
-    print(chunk, end="", flush=True)
-# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
-```
-
-### Other Methods
-
-#### Reset
-
-Resets the database and deletes all embeddings. Irreversible. Requires reinitialization afterwards.
-
-```python
-app.reset()
-```
-
-#### Count
-
-Counts the number of embeddings (chunks) in the database.
-
-```python
-print(app.count())
-# returns: 481
-```

docs/advanced/query_configuration.mdx

@@ -1,77 +0,0 @@
----
-title: '🔍 Query configurations'
----
-
-## AppConfig
-
-| option    | description           | type                            | default                |
-|-----------|-----------------------|---------------------------------|------------------------|
-| log_level | log level             | string                          | WARNING                |
-| embedding_fn| embedding function    | chromadb.utils.embedding_functions | \{text-embedding-ada-002\} |
-| db        | vector database (experimental) | BaseVectorDB               | ChromaDB               |
-| collection_name | initial collection name for the database | string             | embedchain_store |
-| collect_metrics | collect anonymous telemetry data to improve embedchain | boolean | true |
-
-
-## AddConfig
-
-|option|description|type|default|
-|---|---|---|---|
-|chunker|chunker config|ChunkerConfig|Default values for chunker depends on the `data_type`. Please refer [ChunkerConfig](#chunker-config)|
-|loader|loader config|LoaderConfig|None|
-
-Yes, you are passing `ChunkerConfig` to `AddConfig`, like so:
-
-```python
-chunker_config = ChunkerConfig(chunk_size=100)
-add_config = AddConfig(chunker=chunker_config)
-app.add("lorem ipsum", config=add_config)
-```
-
-### ChunkerConfig
-
-|option|description|type|default|
-|---|---|---|---|
-|chunk_size|Maximum size of chunks to return|int|Default value for various `data_type` mentioned below|
-|chunk_overlap|Overlap in characters between chunks|int|Default value for various `data_type` mentioned below|
-|length_function|Function that measures the length of given chunks|typing.Callable|Default value for various `data_type` mentioned below|
-
-Default values of chunker config parameters for different `data_type`:
-
-|data_type|chunk_size|chunk_overlap|length_function|
-|---|---|---|---|
-|docx|1000|0|len|
-|text|300|0|len|
-|qna_pair|300|0|len|
-|web_page|500|0|len|
-|pdf_file|1000|0|len|
-|youtube_video|2000|0|len|
-|docs_site|500|50|len|
-|notion|300|0|len|
-
-### LoaderConfig
-
-_coming soon_
-
-## QueryConfig
-
-|option|description|type|default|
-|---|---|---|---|
-|number_documents|Absolute number of documents to pull from the database as context.|int|1
-|template|custom template for prompt. If history is used with query, $history has to be included as well.|Template|Template("Use the following pieces of context to answer the query at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. \$context Query: \$query Helpful Answer:")|
-|model|name of the model used.|string|depends on app type|
-|temperature|Controls the randomness of the model's output. Higher values (closer to 1) make output more random, lower values make it more deterministic.|float|0|
-|max_tokens|Controls how many tokens are used. Exact implementation (whether it counts prompt and/or response) depends on the model.|int|1000|
-|top_p|Controls the diversity of words. Higher values (closer to 1) make word selection more diverse, lower values make words less diverse.|float|1|
-|history|include conversation history from your client or database.|any (recommendation: list[str])|None|
-|stream|control if response is streamed back to the user.|bool|False|
-|deployment_name|t.b.a.|str|None|
-|system_prompt|System prompt string. Unused if none.|str|None|
-
-## ChatConfig
-
-All options for query and...
-
-_coming soon_
-
-`history` is not supported, as that is handled is handled automatically, the config option is not supported.

docs/advanced/testing.mdx

@@ -1,29 +0,0 @@
----
-title: '🧪 Testing'
----
-
-## Methods for testing
-
-### Dry Run
-
-Before you consume valueable tokens, you should make sure that the embedding you have done works and that it's receiving the correct document from the database.
-
-For this you can use the `dry_run` option in your `query` or `chat` method.
-
-Following the example above, add this to your script:
-
-```python
-print(naval_chat_bot.query('Can you tell me who Naval Ravikant is?', dry_run=True))
-
-'''
-Use the following pieces of context to answer the query at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.
-        Q: Who is Naval Ravikant?
-A: Naval Ravikant is an Indian-American entrepreneur and investor.
-        Query: Can you tell me who Naval Ravikant is?
-        Helpful Answer:
-'''
-```
-
-_The embedding is confirmed to work as expected. It returns the right document, even if the question is asked slightly different. No prompt tokens have been consumed._
-
-**The dry run will still consume tokens to embed your query, but it is only ~1/15 of the prompt.**

docs/advanced/vector_database.mdx

@@ -1,34 +0,0 @@
----
-title: '💾 Vector Database'
----
-
-We support `Chroma` and `Elasticsearch` as two vector database. 
-`Chroma` is used as a default database.
-
-### Elasticsearch
-In order to use `Elasticsearch` as vector database we need to use App type `CustomApp`. 
-```python
-import os
-from embedchain import CustomApp
-from embedchain.config import CustomAppConfig, ElasticsearchDBConfig
-from embedchain.models import Providers, EmbeddingFunctions, VectorDatabases
-
-os.environ["OPENAI_API_KEY"] = 'OPENAI_API_KEY'
-
-es_config = ElasticsearchDBConfig(
-	# elasticsearch url or list of nodes url with different hosts and ports.
-	es_url='http://localhost:9200',
-	# pass named parameters supported by Python Elasticsearch client
-	ca_certs="/path/to/http_ca.crt",
-	basic_auth=("username", "password")
-)
-config = CustomAppConfig(
-	embedding_fn=EmbeddingFunctions.OPENAI, 
-	provider=Providers.OPENAI, 
-	db_type=VectorDatabases.ELASTICSEARCH, 
-	es_config=es_config,
-)
-es_app = CustomApp(config)
-```
-- Set `db_type=VectorDatabases.ELASTICSEARCH` and `es_config=ElasticsearchDBConfig(es_url='')` in `CustomAppConfig`.
-- `ElasticsearchDBConfig` accepts `es_url` as elasticsearch url or as list of nodes url with different hosts and ports. Additionally we can pass named paramaters supported by Python Elasticsearch client.

docs/api-reference/advanced/configuration.mdx

@@ -0,0 +1,241 @@
+---
+title: 'Custom configurations'
+---
+
+Embedchain offers several configuration options for your LLM, vector database, and embedding model. All of these configuration options are optional and have sane defaults.
+
+You can configure different components of your app (`llm`, `embedding model`, or `vector database`) through a simple yaml configuration that Embedchain offers. Here is a generic full-stack example of the yaml config:
+
+
+<Tip>
+Embedchain applications are configurable using YAML file, JSON file or by directly passing the config dictionary. Checkout the [docs here](/api-reference/app/overview#usage) on how to use other formats.
+</Tip>
+
+<CodeGroup>
+```yaml config.yaml
+app:
+  config:
+    name: 'full-stack-app'
+
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+    api_key: sk-xxx
+    prompt: |
+      Use the following pieces of context to answer the query at the end.
+      If you don't know the answer, just say that you don't know, don't try to make up an answer.
+
+      $context
+
+      Query: $query
+
+      Helpful Answer:
+    system_prompt: |
+      Act as William Shakespeare. Answer the following questions in the style of William Shakespeare.
+
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'full-stack-app'
+    dir: db
+    allow_reset: true
+
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-ada-002'
+    api_key: sk-xxx
+
+chunker:
+  chunk_size: 2000
+  chunk_overlap: 100
+  length_function: 'len'
+  min_chunk_size: 0
+
+cache:
+  similarity_evaluation:
+    strategy: distance
+    max_distance: 1.0
+  config:
+    similarity_threshold: 0.8
+    auto_flush: 50
+```
+
+```json config.json
+{
+  "app": {
+    "config": {
+      "name": "full-stack-app"
+    }
+  },
+  "llm": {
+    "provider": "openai",
+    "config": {
+      "model": "gpt-3.5-turbo",
+      "temperature": 0.5,
+      "max_tokens": 1000,
+      "top_p": 1,
+      "stream": false,
+      "prompt": "Use the following pieces of context to answer the query at the end.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n$context\n\nQuery: $query\n\nHelpful Answer:",
+      "system_prompt": "Act as William Shakespeare. Answer the following questions in the style of William Shakespeare.",
+      "api_key": "sk-xxx"
+    }
+  },
+  "vectordb": {
+    "provider": "chroma",
+    "config": {
+      "collection_name": "full-stack-app",
+      "dir": "db",
+      "allow_reset": true
+    }
+  },
+  "embedder": {
+    "provider": "openai",
+    "config": {
+      "model": "text-embedding-ada-002",
+      "api_key": "sk-xxx"
+    }
+  },
+  "chunker": {
+    "chunk_size": 2000,
+    "chunk_overlap": 100,
+    "length_function": "len",
+    "min_chunk_size": 0
+  },
+  "cache": {
+    "similarity_evaluation": {
+        "strategy": "distance",
+        "max_distance": 1.0,
+    },
+    "config": {
+        "similarity_threshold": 0.8,
+        "auto_flush": 50,
+    },
+  },
+}
+```
+
+```python config.py
+config = {
+    'app': {
+        'config': {
+            'name': 'full-stack-app'
+        }
+    },
+    'llm': {
+        'provider': 'openai',
+        'config': {
+            'model': 'gpt-3.5-turbo',
+            'temperature': 0.5,
+            'max_tokens': 1000,
+            'top_p': 1,
+            'stream': False,
+            'prompt': (
+                "Use the following pieces of context to answer the query at the end.\n"
+                "If you don't know the answer, just say that you don't know, don't try to make up an answer.\n"
+                "$context\n\nQuery: $query\n\nHelpful Answer:"
+            ),
+            'system_prompt': (
+                "Act as William Shakespeare. Answer the following questions in the style of William Shakespeare."
+            ),
+            'api_key': 'sk-xxx'
+        }
+    },
+    'vectordb': {
+        'provider': 'chroma',
+        'config': {
+            'collection_name': 'full-stack-app',
+            'dir': 'db',
+            'allow_reset': True
+        }
+    },
+    'embedder': {
+        'provider': 'openai',
+        'config': {
+            'model': 'text-embedding-ada-002',
+            'api_key': 'sk-xxx'
+        }
+    },
+    'chunker': {
+        'chunk_size': 2000,
+        'chunk_overlap': 100,
+        'length_function': 'len',
+        'min_chunk_size': 0
+    },
+    'cache': {
+      'similarity_evaluation': {
+          'strategy': 'distance',
+          'max_distance': 1.0,
+      },
+      'config': {
+          'similarity_threshold': 0.8,
+          'auto_flush': 50,
+      },
+    },
+}
+```
+</CodeGroup>
+
+Alright, let's dive into what each key means in the yaml config above:
+
+1. `app` Section:
+    - `config`:
+        - `name` (String): The name of your full-stack application.
+        - `id` (String): The id of your full-stack application.
+        <Note>Only use this to reload already created apps. We recommend users to not create their own ids.</Note>
+        - `collect_metrics` (Boolean): Indicates whether metrics should be collected for the app, defaults to `True`
+        - `log_level` (String): The log level for the app, defaults to `WARNING`
+2. `llm` Section:
+    - `provider` (String): The provider for the language model, which is set to 'openai'. You can find the full list of llm providers in [our docs](/components/llms).
+    - `config`:
+        - `model` (String): The specific model being used, 'gpt-3.5-turbo'.
+        - `temperature` (Float): Controls the randomness of the model's output. A higher value (closer to 1) makes the output more random.
+        - `max_tokens` (Integer): Controls how many tokens are used in the response.
+        - `top_p` (Float): Controls the diversity of word selection. A higher value (closer to 1) makes word selection more diverse.
+        - `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
+        - `prompt` (String): A prompt for the model to follow when generating responses, requires `$context` and `$query` variables.
+        - `system_prompt` (String): A system prompt for the model to follow when generating responses, in this case, it's set to the style of William Shakespeare.
+        - `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
+        - `number_documents` (Integer): Number of documents to pull from the vectordb as context, defaults to 1
+        - `api_key` (String): The API key for the language model.
+        - `model_kwargs` (Dict): Keyword arguments to pass to the language model. Used for `aws_bedrock` provider, since it requires different arguments for each model.
+3. `vectordb` Section:
+    - `provider` (String): The provider for the vector database, set to 'chroma'. You can find the full list of vector database providers in [our docs](/components/vector-databases).
+    - `config`:
+        - `collection_name` (String): The initial collection name for the vectordb, set to 'full-stack-app'.
+        - `dir` (String): The directory for the local database, set to 'db'.
+        - `allow_reset` (Boolean): Indicates whether resetting the vectordb is allowed, set to true.
+    <Note>We recommend you to checkout vectordb specific config [here](https://docs.embedchain.ai/components/vector-databases)</Note>
+4. `embedder` Section:
+    - `provider` (String): The provider for the embedder, set to 'openai'. You can find the full list of embedding model providers in [our docs](/components/embedding-models).
+    - `config`:
+        - `model` (String): The specific model used for text embedding, 'text-embedding-ada-002'.
+        - `vector_dimension` (Integer): The vector dimension of the embedding model. [Defaults](https://github.com/embedchain/embedchain/blob/e572b5a3dc1b66f1e9b3357d11a88c63b5ce06e3/embedchain/models/vector_dimensions.py)
+        - `api_key` (String): The API key for the embedding model.
+        - `deployment_name` (String): The deployment name for the embedding model.
+        - `title` (String): The title for the embedding model for Google Embedder.
+        - `task_type` (String): The task type for the embedding model for Google Embedder.
+5. `chunker` Section:
+    - `chunk_size` (Integer): The size of each chunk of text that is sent to the language model.
+    - `chunk_overlap` (Integer): The amount of overlap between each chunk of text.
+    - `length_function` (String): The function used to calculate the length of each chunk of text. In this case, it's set to 'len'. You can also use any function import directly as a string here.
+    - `min_chunk_size` (Integer): The minimum size of each chunk of text that is sent to the language model. Must be less than `chunk_size`, and greater than `chunk_overlap`.
+6. `cache` Section: (Optional)
+    - `similarity_evaluation` (Optional): The config for similarity evaluation strategy. If not provided, the default `distance` based similarity evaluation strategy is used.
+      - `strategy` (String): The strategy to use for similarity evaluation. Currently, only `distance` and `exact` based similarity evaluation is supported. Defaults to `distance`.
+      - `max_distance` (Float): The bound of maximum distance. Defaults to `1.0`.
+      - `positive` (Boolean): If the larger distance indicates more similar of two entities, set it `True`, otherwise `False`. Defaults to `False`.
+    - `config` (Optional): The config for initializing the cache. If not provided, sensible default values are used as mentioned below.
+      - `similarity_threshold` (Float): The threshold for similarity evaluation. Defaults to `0.8`.
+      - `auto_flush` (Integer): The number of queries after which the cache is flushed. Defaults to `20`.
+    <Note>
+    If you provide a cache section, the app will automatically configure and use a cache to store the results of the language model. This is useful if you want to speed up the response time and save inference cost of your app.
+    </Note>
+If you have questions about the configuration above, please feel free to reach out to us using one of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/api-reference/app/add.mdx

@@ -0,0 +1,44 @@
+---
+title: '📊 add'
+---
+
+`add()` method is used to load the data sources from different data sources to a RAG pipeline. You can find the signature below:
+
+### Parameters
+
+<ParamField path="source" type="str">
+    The data to embed, can be a URL, local file or raw content, depending on the data type.. You can find the full list of supported data sources [here](/components/data-sources/overview).
+</ParamField>
+<ParamField path="data_type" type="str" optional>
+    Type of data source. It can be automatically detected but user can force what data type to load as.
+</ParamField>
+<ParamField path="metadata" type="dict" optional>
+    Any metadata that you want to store with the data source. Metadata is generally really useful for doing metadata filtering on top of semantic search to yield faster search and better results.
+</ParamField>
+
+## Usage
+
+### Load data from webpage
+
+```python Code example
+from embedchain import App
+
+app = App()
+app.add("https://www.forbes.com/profile/elon-musk")
+# Inserting batches in chromadb: 100%|███████████████| 1/1 [00:00<00:00,  1.19it/s]
+# Successfully saved https://www.forbes.com/profile/elon-musk (DataType.WEB_PAGE). New chunks count: 4
+```
+
+### Load data from sitemap
+
+```python Code example
+from embedchain import App
+
+app = App()
+app.add("https://python.langchain.com/sitemap.xml", data_type="sitemap")
+# Loading pages: 100%|█████████████| 1108/1108 [00:47<00:00, 23.17it/s]
+# Inserting batches in chromadb: 100%|█████████| 111/111 [04:41<00:00,  2.54s/it]
+# Successfully saved https://python.langchain.com/sitemap.xml (DataType.SITEMAP). New chunks count: 11024
+```
+
+You can find complete list of supported data sources [here](/components/data-sources/overview).

docs/api-reference/app/chat.mdx

@@ -0,0 +1,146 @@
+---
+title: '💬 chat'
+---
+
+`chat()` method allows you to chat over your data sources using a user-friendly chat API. You can find the signature below:
+
+### Parameters
+
+<ParamField path="input_query" type="str">
+    Question to ask
+</ParamField>
+<ParamField path="config" type="BaseLlmConfig" optional>
+    Configure different llm settings such as prompt, temprature, number_documents etc.
+</ParamField>
+<ParamField path="dry_run" type="bool" optional>
+    The purpose is to test the prompt structure without actually running LLM inference. Defaults to `False`
+</ParamField>
+<ParamField path="where" type="dict" optional>
+    A dictionary of key-value pairs to filter the chunks from the vector database. Defaults to `None`
+</ParamField>
+<ParamField path="session_id" type="str" optional>
+    Session ID of the chat. This can be used to maintain chat history of different user sessions. Default value: `default`
+</ParamField>
+<ParamField path="citations" type="bool" optional>
+    Return citations along with the LLM answer. Defaults to `False`
+</ParamField>
+
+### Returns
+
+<ResponseField name="answer" type="str | tuple">
+  If `citations=False`, return a stringified answer to the question asked. <br />
+  If `citations=True`, returns a tuple with answer and citations respectively.
+</ResponseField>
+
+## Usage
+
+### With citations
+
+If you want to get the answer to question and return both answer and citations, use the following code snippet:
+
+```python With Citations
+from embedchain import App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Get relevant answer for your query
+answer, sources = app.chat("What is the net worth of Elon?", citations=True)
+print(answer)
+# Answer: The net worth of Elon Musk is $221.9 billion.
+
+print(sources)
+# [
+#    (
+#        'Elon Musk PROFILEElon MuskCEO, Tesla$247.1B$2.3B (0.96%)Real Time Net Worthas of 12/7/23 ...',
+#        {
+#           'url': 'https://www.forbes.com/profile/elon-musk', 
+#           'score': 0.89,
+#           ...
+#        }
+#    ),
+#    (
+#        '74% of the company, which is now called X.Wealth HistoryHOVER TO REVEAL NET WORTH BY YEARForbes ...',
+#        {
+#           'url': 'https://www.forbes.com/profile/elon-musk', 
+#           'score': 0.81,
+#           ...
+#        }
+#    ),
+#    (
+#        'founded in 2002, is worth nearly $150 billion after a $750 million tender offer in June 2023 ...',
+#        {
+#           'url': 'https://www.forbes.com/profile/elon-musk', 
+#           'score': 0.73,
+#           ...
+#        }
+#    )
+# ]
+```
+
+<Note>
+When `citations=True`, note that the returned `sources` are a list of tuples where each tuple has two elements (in the following order):
+1. source chunk
+2. dictionary with metadata about the source chunk
+    - `url`: url of the source
+    - `doc_id`: document id (used for book keeping purposes)
+    - `score`: score of the source chunk with respect to the question
+    - other metadata you might have added at the time of adding the source
+</Note>
+
+
+### Without citations
+
+If you just want to return answers and don't want to return citations, you can use the following example:
+
+```python Without Citations
+from embedchain import App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Chat on your data using `.chat()`
+answer = app.chat("What is the net worth of Elon?")
+print(answer)
+# Answer: The net worth of Elon Musk is $221.9 billion.
+```
+
+### With session id
+
+If you want to maintain chat sessions for different users, you can simply pass the `session_id` keyword argument. See the example below:
+
+```python With session id
+from embedchain import App
+
+app = App()
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Chat on your data using `.chat()`
+app.chat("What is the net worth of Elon Musk?", session_id="user1")
+# 'The net worth of Elon Musk is $250.8 billion.'
+app.chat("What is the net worth of Bill Gates?", session_id="user2")
+# "I don't know the current net worth of Bill Gates."
+app.chat("What was my last question", session_id="user1")
+# 'Your last question was "What is the net worth of Elon Musk?"'
+```
+
+### With custom context window
+
+If you want to customize the context window that you want to use during chat (default context window is 3 document chunks), you can do using the following code snippet:
+
+```python with custom chunks size
+from embedchain import App
+from embedchain.config import BaseLlmConfig
+
+app = App()
+app.add("https://www.forbes.com/profile/elon-musk")
+
+query_config = BaseLlmConfig(number_documents=5)
+app.chat("What is the net worth of Elon Musk?", config=query_config)
+```

docs/api-reference/app/delete.mdx

@@ -0,0 +1,48 @@
+---
+title: 🗑 delete
+---
+
+## Delete Document
+
+`delete()` method allows you to delete a document previously added to the app.
+
+### Usage
+
+```python
+from embedchain import App
+
+app = App()
+
+forbes_doc_id = app.add("https://www.forbes.com/profile/elon-musk")
+wiki_doc_id = app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+
+app.delete(forbes_doc_id)   # deletes the forbes document
+```
+
+<Note>
+    If you do not have the document id, you can use `app.db.get()` method to get the document and extract the `hash` key from `metadatas` dictionary object, which serves as the document id.
+</Note>
+
+
+## Delete Chat Session History
+
+`delete_session_chat_history()` method allows you to delete all previous messages in a chat history.
+
+### Usage
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add("https://www.forbes.com/profile/elon-musk")
+
+app.chat("What is the net worth of Elon Musk?")
+
+app.delete_session_chat_history()
+```
+
+<Note>
+    `delete_session_chat_history(session_id="session_1")` method also accepts `session_id` optional param for deleting chat history of a specific session.
+    It assumes the default session if no `session_id` is provided.
+</Note>

docs/api-reference/app/deploy.mdx

@@ -0,0 +1,5 @@
+---
+title: 🚀 deploy
+---
+
+The `deploy()` method is currently available on an invitation-only basis. To request access, please submit your information via the provided [Google Form](https://forms.gle/vigN11h7b4Ywat668). We will review your request and respond promptly.

docs/api-reference/app/evaluate.mdx

@@ -0,0 +1,41 @@
+---
+title: '📝 evaluate'
+---
+
+`evaluate()` method is used to evaluate the performance of a RAG app. You can find the signature below:
+
+### Parameters
+
+<ParamField path="question" type="Union[str, list[str]]">
+    A question or a list of questions to evaluate your app on.
+</ParamField>
+<ParamField path="metrics" type="Optional[list[Union[BaseMetric, str]]]" optional>
+    The metrics to evaluate your app on. Defaults to all metrics: `["context_relevancy", "answer_relevancy", "groundedness"]`
+</ParamField>
+<ParamField path="num_workers" type="int" optional>
+    Specify the number of threads to use for parallel processing.
+</ParamField>
+
+### Returns
+
+<ResponseField name="metrics" type="dict">
+    Returns the metrics you have chosen to evaluate your app on as a dictionary.
+</ResponseField>
+
+## Usage
+
+```python
+from embedchain import App
+
+app = App()
+
+# add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# run evaluation
+app.evaluate("what is the net worth of Elon Musk?")
+# {'answer_relevancy': 0.958019958036268, 'context_relevancy': 0.12903225806451613}
+
+# or
+# app.evaluate(["what is the net worth of Elon Musk?", "which companies does Elon Musk own?"])
+```

docs/api-reference/app/get.mdx

@@ -0,0 +1,33 @@
+---
+title: 📄 get
+---
+
+## Get data sources
+
+`get_data_sources()` returns a list of all the data sources added in the app.
+
+
+### Usage
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add("https://www.forbes.com/profile/elon-musk")
+app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+
+data_sources = app.get_data_sources()
+# [
+#   {
+#       'data_type': 'web_page',
+#       'data_value': 'https://en.wikipedia.org/wiki/Elon_Musk',
+#       'metadata': 'null'
+#   },
+#   {
+#       'data_type': 'web_page',
+#       'data_value': 'https://www.forbes.com/profile/elon-musk',
+#       'metadata': 'null'
+#   }
+# ]
+```

docs/api-reference/app/overview.mdx

@@ -0,0 +1,130 @@
+---
+title: "App"
+---
+
+Create a RAG app object on Embedchain. This is the main entrypoint for a developer to interact with Embedchain APIs. An app configures the llm, vector database, embedding model, and retrieval strategy of your choice.
+
+### Attributes
+
+<ParamField path="local_id" type="str">
+    App ID
+</ParamField>
+<ParamField path="name" type="str" optional>
+    Name of the app
+</ParamField>
+<ParamField path="config" type="BaseConfig">
+    Configuration of the app
+</ParamField>
+<ParamField path="llm" type="BaseLlm">
+    Configured LLM for the RAG app
+</ParamField>
+<ParamField path="db" type="BaseVectorDB">
+    Configured vector database for the RAG app
+</ParamField>
+<ParamField path="embedding_model" type="BaseEmbedder">
+    Configured embedding model for the RAG app
+</ParamField>
+<ParamField path="chunker" type="ChunkerConfig">
+    Chunker configuration
+</ParamField>
+<ParamField path="client" type="Client" optional>
+    Client object (used to deploy an app to Embedchain platform)
+</ParamField>
+<ParamField path="logger" type="logging.Logger">
+    Logger object
+</ParamField>
+
+## Usage
+
+You can create an app instance using the following methods:
+
+### Default setting
+
+```python Code Example
+from embedchain import App
+app = App()
+```
+
+
+### Python Dict
+
+```python Code Example
+from embedchain import App
+
+config_dict = {
+  'llm': {
+    'provider': 'gpt4all',
+    'config': {
+      'model': 'orca-mini-3b-gguf2-q4_0.gguf',
+      'temperature': 0.5,
+      'max_tokens': 1000,
+      'top_p': 1,
+      'stream': False
+    }
+  },
+  'embedder': {
+    'provider': 'gpt4all'
+  }
+}
+
+# load llm configuration from config dict
+app = App.from_config(config=config_dict)
+```
+
+### YAML Config
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: gpt4all
+```
+
+</CodeGroup>
+
+### JSON Config
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load llm configuration from config.json file
+app = App.from_config(config_path="config.json")
+```
+
+```json config.json
+{
+  "llm": {
+    "provider": "gpt4all",
+    "config": {
+      "model": "orca-mini-3b-gguf2-q4_0.gguf",
+      "temperature": 0.5,
+      "max_tokens": 1000,
+      "top_p": 1,
+      "stream": false
+    }
+  },
+  "embedder": {
+    "provider": "gpt4all"
+  }
+}
+```
+
+</CodeGroup>

docs/api-reference/app/query.mdx

@@ -0,0 +1,109 @@
+---
+title: '❓ query'
+---
+
+`.query()` method empowers developers to ask questions and receive relevant answers through a user-friendly query API. Function signature is given below:
+
+### Parameters
+
+<ParamField path="input_query" type="str">
+    Question to ask
+</ParamField>
+<ParamField path="config" type="BaseLlmConfig" optional>
+    Configure different llm settings such as prompt, temprature, number_documents etc.
+</ParamField>
+<ParamField path="dry_run" type="bool" optional>
+    The purpose is to test the prompt structure without actually running LLM inference. Defaults to `False`
+</ParamField>
+<ParamField path="where" type="dict" optional>
+    A dictionary of key-value pairs to filter the chunks from the vector database. Defaults to `None`
+</ParamField>
+<ParamField path="citations" type="bool" optional>
+    Return citations along with the LLM answer. Defaults to `False`
+</ParamField>
+
+### Returns
+
+<ResponseField name="answer" type="str | tuple">
+  If `citations=False`, return a stringified answer to the question asked. <br />
+  If `citations=True`, returns a tuple with answer and citations respectively.
+</ResponseField>
+
+## Usage
+
+### With citations
+
+If you want to get the answer to question and return both answer and citations, use the following code snippet:
+
+```python With Citations
+from embedchain import App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Get relevant answer for your query
+answer, sources = app.query("What is the net worth of Elon?", citations=True)
+print(answer)
+# Answer: The net worth of Elon Musk is $221.9 billion.
+
+print(sources)
+# [
+#    (
+#        'Elon Musk PROFILEElon MuskCEO, Tesla$247.1B$2.3B (0.96%)Real Time Net Worthas of 12/7/23 ...',
+#        {
+#           'url': 'https://www.forbes.com/profile/elon-musk', 
+#           'score': 0.89,
+#           ...
+#        }
+#    ),
+#    (
+#        '74% of the company, which is now called X.Wealth HistoryHOVER TO REVEAL NET WORTH BY YEARForbes ...',
+#        {
+#           'url': 'https://www.forbes.com/profile/elon-musk', 
+#           'score': 0.81,
+#           ...
+#        }
+#    ),
+#    (
+#        'founded in 2002, is worth nearly $150 billion after a $750 million tender offer in June 2023 ...',
+#        {
+#           'url': 'https://www.forbes.com/profile/elon-musk', 
+#           'score': 0.73,
+#           ...
+#        }
+#    )
+# ]
+```
+
+<Note>
+When `citations=True`, note that the returned `sources` are a list of tuples where each tuple has two elements (in the following order):
+1. source chunk
+2. dictionary with metadata about the source chunk
+    - `url`: url of the source
+    - `doc_id`: document id (used for book keeping purposes)
+    - `score`: score of the source chunk with respect to the question
+    - other metadata you might have added at the time of adding the source
+</Note>
+
+### Without citations
+
+If you just want to return answers and don't want to return citations, you can use the following example:
+
+```python Without Citations
+from embedchain import App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Get relevant answer for your query
+answer = app.query("What is the net worth of Elon?")
+print(answer)
+# Answer: The net worth of Elon Musk is $221.9 billion.
+```
+

docs/api-reference/app/reset.mdx

@@ -0,0 +1,17 @@
+---
+title: 🔄 reset
+---
+
+`reset()` method allows you to wipe the data from your RAG application and start from scratch.
+
+## Usage
+
+```python
+from embedchain import App
+
+app = App()
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Reset the app
+app.reset()
+```

docs/api-reference/app/search.mdx

@@ -0,0 +1,111 @@
+---
+title: '🔍 search'
+---
+
+`.search()` enables you to uncover the most pertinent context by performing a semantic search across your data sources based on a given query. Refer to the function signature below:
+
+### Parameters
+
+<ParamField path="query" type="str">
+    Question
+</ParamField>
+<ParamField path="num_documents" type="int" optional>
+    Number of relevant documents to fetch. Defaults to `3`
+</ParamField>
+<ParamField path="where" type="dict" optional>
+    Key value pair for metadata filtering.
+</ParamField>
+<ParamField path="raw_filter" type="dict" optional>
+    Pass raw filter query based on your vector database.
+    Currently, `raw_filter` param is only supported for Pinecone vector database.
+</ParamField>
+
+### Returns
+
+<ResponseField name="answer" type="dict">
+    Return list of dictionaries that contain the relevant chunk and their source information.
+</ResponseField>
+
+## Usage
+
+### Basic
+
+Refer to the following example on how to use the search api:
+
+```python Code example
+from embedchain import App
+
+app = App()
+app.add("https://www.forbes.com/profile/elon-musk")
+
+context = app.search("What is the net worth of Elon?", num_documents=2)
+print(context)
+```
+
+### Advanced
+
+#### Metadata filtering using `where` params
+
+Here is an advanced example of `search()` API with metadata filtering on pinecone database:
+
+```python
+import os
+
+from embedchain import App
+
+os.environ["PINECONE_API_KEY"] = "xxx"
+
+config = {
+    "vectordb": {
+        "provider": "pinecone",
+        "config": {
+            "metric": "dotproduct",
+            "vector_dimension": 1536,
+            "index_name": "ec-test",
+            "serverless_config": {"cloud": "aws", "region": "us-west-2"},
+        },
+    }
+}
+
+app = App.from_config(config=config)
+
+app.add("https://www.forbes.com/profile/bill-gates", metadata={"type": "forbes", "person": "gates"})
+app.add("https://en.wikipedia.org/wiki/Bill_Gates", metadata={"type": "wiki", "person": "gates"})
+
+results = app.search("What is the net worth of Bill Gates?", where={"person": "gates"})
+print("Num of search results: ", len(results))
+```
+
+#### Metadata filtering using `raw_filter` params
+
+Following is an example of metadata filtering by passing the raw filter query that pinecone vector database follows:
+
+```python
+import os
+
+from embedchain import App
+
+os.environ["PINECONE_API_KEY"] = "xxx"
+
+config = {
+    "vectordb": {
+        "provider": "pinecone",
+        "config": {
+            "metric": "dotproduct",
+            "vector_dimension": 1536,
+            "index_name": "ec-test",
+            "serverless_config": {"cloud": "aws", "region": "us-west-2"},
+        },
+    }
+}
+
+app = App.from_config(config=config)
+
+app.add("https://www.forbes.com/profile/bill-gates", metadata={"year": 2022, "person": "gates"})
+app.add("https://en.wikipedia.org/wiki/Bill_Gates", metadata={"year": 2024, "person": "gates"})
+
+print("Filter with person: gates and year > 2023")
+raw_filter = {"$and": [{"person": "gates"}, {"year": {"$gt": 2023}}]}
+results = app.search("What is the net worth of Bill Gates?", raw_filter=raw_filter)
+print("Num of search results: ", len(results))
+```

docs/api-reference/store/ai-assistants.mdx

@@ -0,0 +1,54 @@
+---
+title: 'AI Assistant'
+---
+
+The `AIAssistant` class, an alternative to the OpenAI Assistant API, is designed for those who prefer using large language models (LLMs) other than those provided by OpenAI. It facilitates the creation of AI Assistants with several key benefits:
+
+- **Visibility into Citations**: It offers transparent access to the sources and citations used by the AI, enhancing the understanding and trustworthiness of its responses.
+
+- **Debugging Capabilities**: Users have the ability to delve into and debug the AI's processes, allowing for a deeper understanding and fine-tuning of its performance.
+
+- **Customizable Prompts**: The class provides the flexibility to modify and tailor prompts according to specific needs, enabling more precise and relevant interactions.
+
+- **Chain of Thought Integration**: It supports the incorporation of a 'chain of thought' approach, which helps in breaking down complex queries into simpler, sequential steps, thereby improving the clarity and accuracy of responses.
+
+It is ideal for those who value customization, transparency, and detailed control over their AI Assistant's functionalities.
+
+### Arguments
+
+<ParamField path="name" type="string" optional>
+  Name for your AI assistant
+</ParamField>
+
+<ParamField path="instructions" type="string" optional>
+  How the Assistant and model should behave or respond
+</ParamField>
+
+<ParamField path="assistant_id" type="string" optional>
+  Load existing AI Assistant. If you pass this, you don't have to pass other arguments.
+</ParamField>
+
+<ParamField path="thread_id" type="string" optional>
+  Existing thread id if exists
+</ParamField>
+
+<ParamField path="yaml_path" type="str" Optional>
+    Embedchain pipeline config yaml path to use. This will define the configuration of the AI Assistant (such as configuring the LLM, vector database, and embedding model)
+</ParamField>
+
+<ParamField path="data_sources" type="list" default="[]">
+  Add data sources to your assistant. You can add in the following format: `[{"source": "https://example.com", "data_type": "web_page"}]`
+</ParamField>
+
+<ParamField path="collect_metrics" type="boolean" default="True">
+  Anonymous telemetry (doesn't collect any user information or user's files). Used to improve the Embedchain package utilization. Default is `True`.
+</ParamField>
+
+
+## Usage
+
+For detailed guidance on creating your own AI Assistant, click the link below. It provides step-by-step instructions to help you through the process:
+
+<Card title="Guide to Creating Your AI Assistant" icon="link" href="/examples/opensource-assistant">
+  Learn how to build a customized AI Assistant using the `AIAssistant` class.
+</Card>

docs/api-reference/store/openai-assistant.mdx

@@ -0,0 +1,45 @@
+---
+title: 'OpenAI Assistant'
+---
+
+### Arguments
+
+<ParamField path="name" type="string">
+  Name for your AI assistant
+</ParamField>
+
+<ParamField path="instructions" type="string">
+  how the Assistant and model should behave or respond
+</ParamField>
+
+<ParamField path="assistant_id" type="string">
+  Load existing OpenAI Assistant. If you pass this, you don't have to pass other arguments.
+</ParamField>
+
+<ParamField path="thread_id" type="string">
+  Existing OpenAI thread id if exists
+</ParamField>
+
+<ParamField path="model" type="str" default="gpt-4-1106-preview">
+  OpenAI model to use
+</ParamField>
+
+<ParamField path="tools" type="list">
+  OpenAI tools to use. Default set to `[{"type": "retrieval"}]`
+</ParamField>
+
+<ParamField path="data_sources" type="list" default="[]">
+  Add data sources to your assistant. You can add in the following format: `[{"source": "https://example.com", "data_type": "web_page"}]`
+</ParamField>
+
+<ParamField path="telemetry" type="boolean" default="True">
+  Anonymous telemetry (doesn't collect any user information or user's files). Used to improve the Embedchain package utilization. Default is `True`.
+</ParamField>
+
+## Usage
+
+For detailed guidance on creating your own OpenAI Assistant, click the link below. It provides step-by-step instructions to help you through the process:
+
+<Card title="Guide to Creating Your OpenAI Assistant" icon="link" href="/examples/openai-assistant">
+  Learn how to build an OpenAI Assistant using the `OpenAIAssistant` class.
+</Card>

docs/community/connect-with-us.mdx

@@ -0,0 +1,28 @@
+---
+title: 🤝 Connect with Us
+---
+
+We believe in building a vibrant and supportive community around embedchain. There are various channels through which you can connect with us, stay updated, and contribute to the ongoing discussions:
+
+<CardGroup cols={3}>
+  <Card title="Twitter" icon="twitter" href="https://twitter.com/embedchain">
+    Follow us on Twitter
+  </Card>
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
+    Join our slack community
+  </Card>
+  <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
+    Join our discord community
+  </Card>
+  <Card title="LinkedIn" icon="linkedin" href="https://www.linkedin.com/company/embedchain/">
+  Connect with us on LinkedIn
+  </Card>
+  <Card title="Schedule a call" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call with Embedchain founder
+  </Card>
+  <Card title="Newsletter" icon="message" href="https://embedchain.substack.com/">
+  Subscribe to our newsletter
+  </Card>
+</CardGroup>
+
+We look forward to connecting with you and seeing how we can create amazing things together!

docs/components/data-sources/beehiiv.mdx

@@ -0,0 +1,16 @@
+---
+title: "🐝 Beehiiv"
+---
+
+To add any Beehiiv data sources to your app, just add the base url as the source and set the data_type to `beehiiv`.
+
+```python
+from embedchain import App
+
+app = App()
+
+# source: just add the base url and set the data_type to 'beehiiv'
+app.add('https://aibreakfast.beehiiv.com', data_type='beehiiv')
+app.query("How much is OpenAI paying developers?")
+# Answer: OpenAI is aggressively recruiting Google's top AI researchers with offers ranging between $5 to $10 million annually, primarily in stock options.
+```

docs/components/data-sources/csv.mdx

@@ -0,0 +1,28 @@
+---
+title: '📊 CSV'
+---
+
+You can load any csv file from your local file system or through a URL. Headers are included for each line, so if you have an `age` column, `18` will be added as `age: 18`.
+
+## Usage
+
+### Load from a local file
+
+```python
+from embedchain import App
+app = App()
+app.add('/path/to/file.csv', data_type='csv')
+```
+
+### Load from URL
+
+```python
+from embedchain import App
+app = App()
+app.add('https://people.sc.fsu.edu/~jburkardt/data/csv/airtravel.csv', data_type="csv")
+```
+
+<Note>
+There is a size limit allowed for csv file beyond which it can throw error. This limit is set by the LLMs. Please consider chunking large csv files into smaller csv files.
+</Note>
+

docs/components/data-sources/custom.mdx

@@ -0,0 +1,42 @@
+---
+title: '⚙️ Custom'
+---
+
+When we say "custom", we mean that you can customize the loader and chunker to your needs. This is done by passing a custom loader and chunker to the `add` method.
+
+```python
+from embedchain import App
+import your_loader
+from my_module import CustomLoader
+from my_module import CustomChunker
+
+app = App()
+loader = CustomLoader()
+chunker = CustomChunker()
+
+app.add("source", data_type="custom", loader=loader, chunker=chunker)
+```
+
+<Note>
+    The custom loader and chunker must be a class that inherits from the [`BaseLoader`](https://github.com/embedchain/embedchain/blob/main/embedchain/loaders/base_loader.py) and [`BaseChunker`](https://github.com/embedchain/embedchain/blob/main/embedchain/chunkers/base_chunker.py) classes respectively.
+</Note>
+
+<Note>
+    If the `data_type` is not a valid data type, the `add` method will fallback to the `custom` data type and expect a custom loader and chunker to be passed by the user.
+</Note>
+
+Example:
+
+```python
+from embedchain import App
+from embedchain.loaders.github import GithubLoader
+
+app = App()
+
+loader = GithubLoader(config={"token": "ghp_xxx"})
+
+app.add("repo:embedchain/embedchain type:repo", data_type="github", loader=loader)
+
+app.query("What is Embedchain?")
+# Answer: Embedchain is a Data Platform for Large Language Models (LLMs). It allows users to seamlessly load, index, retrieve, and sync unstructured data in order to build dynamic, LLM-powered applications. There is also a JavaScript implementation called embedchain-js available on GitHub.
+```

docs/components/data-sources/data-type-handling.mdx

@@ -0,0 +1,64 @@
+---
+title: 'Data type handling'
+---
+
+## Automatic data type detection
+
+The add method automatically tries to detect the data_type, based on your input for the source argument. So `app.add('https://www.youtube.com/watch?v=dQw4w9WgXcQ')` is enough to embed a YouTube video.
+
+This detection is implemented for all formats. It is based on factors such as whether it's a URL, a local file, the source data type, etc.
+
+### Debugging automatic detection
+
+Set `log_level: DEBUG` in the config yaml to debug if the data type detection is done right or not. Otherwise, you will not know when, for instance, an invalid filepath is interpreted as raw text instead.
+
+### Forcing a data type
+
+To omit any issues with the data type detection, you can **force** a data_type by adding it as a `add` method argument.
+The examples below show you the keyword to force the respective `data_type`.
+
+Forcing can also be used for edge cases, such as interpreting a sitemap as a web_page, for reading its raw text instead of following links.
+
+## Remote data types
+
+<Tip>
+**Use local files in remote data types**
+
+Some data_types are meant for remote content and only work with URLs.
+You can pass local files by formatting the path using the `file:` [URI scheme](https://en.wikipedia.org/wiki/File_URI_scheme), e.g. `file:///info.pdf`.
+</Tip>
+
+## Reusing a vector database
+
+Default behavior is to create a persistent vector db in the directory **./db**. You can split your application into two Python scripts: one to create a local vector db and the other to reuse this local persistent vector db. This is useful when you want to index hundreds of documents and separately implement a chat interface.
+
+Create a local index:
+
+```python
+from embedchain import App
+
+naval_chat_bot = App()
+naval_chat_bot.add("https://www.youtube.com/watch?v=3qHkcs3kG44")
+naval_chat_bot.add("https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
+```
+
+You can reuse the local index with the same code, but without adding new documents:
+
+```python
+from embedchain import App
+
+naval_chat_bot = App()
+print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
+```
+
+## Resetting an app and vector database
+
+You can reset the app by simply calling the `reset` method. This will delete the vector database and all other app related files.
+
+```python
+from embedchain import App
+
+app = App()
+app.add("https://www.youtube.com/watch?v=3qHkcs3kG44")
+app.reset()
+```

docs/components/data-sources/directory.mdx

@@ -0,0 +1,41 @@
+---
+title: '📁 Directory/Folder'
+---
+
+To use an entire directory as data source, just add `data_type` as `directory` and pass in the path of the local directory.
+
+### Without customization
+
+```python
+import os
+from embedchain import App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+app.add("./elon-musk", data_type="directory")
+response = app.query("list all files")
+print(response)
+# Answer: Files are elon-musk-1.txt, elon-musk-2.pdf.
+```
+
+### Customization
+
+```python
+import os
+from embedchain import App
+from embedchain.loaders.directory_loader import DirectoryLoader
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+lconfig = {
+    "recursive": True,
+    "extensions": [".txt"]
+}
+loader = DirectoryLoader(config=lconfig)
+app = App()
+app.add("./elon-musk", loader=loader)
+response = app.query("what are all the files related to?")
+print(response)
+
+# Answer: The files are related to Elon Musk.
+```

docs/components/data-sources/discord.mdx

@@ -0,0 +1,28 @@
+---
+title: "💬 Discord"
+---
+
+To add any Discord channel messages to your app, just add the `channel_id` as the source and set the `data_type` to `discord`.
+
+<Note>
+    This loader requires a Discord bot token with read messages access.
+    To obtain the token, follow the instructions provided in this tutorial: 
+    <a href="https://www.writebots.com/discord-bot-token/">How to Get a Discord Bot Token?</a>.
+</Note>
+
+```python
+import os
+from embedchain import App
+
+# add your discord "BOT" token
+os.environ["DISCORD_TOKEN"] = "xxx"
+
+app = App()
+
+app.add("1177296711023075338", data_type="discord")
+
+response = app.query("What is Joe saying about Elon Musk?")
+
+print(response)
+# Answer: Joe is saying "Elon Musk is a genius".
+```

docs/components/data-sources/discourse.mdx

@@ -0,0 +1,44 @@
+---
+title: '🗨️ Discourse'
+---
+
+You can now easily load data from your community built with [Discourse](https://discourse.org/).
+
+## Example
+
+1. Setup the Discourse Loader with your community url.
+```Python
+from embedchain.loaders.discourse import DiscourseLoader
+
+dicourse_loader = DiscourseLoader(config={"domain": "https://community.openai.com"})
+```
+
+2. Once you setup the loader, you can create an app and load data using the above discourse loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+
+app.add("openai after:2023-10-1", data_type="discourse", loader=dicourse_loader)
+
+question = "Where can I find the OpenAI API status page?"
+app.query(question)
+# Answer: You can find the OpenAI API status page at https:/status.openai.com/.
+```
+
+NOTE: The `add` function of the app will accept any executable search query to load data. Refer [Discourse API Docs](https://docs.discourse.org/#tag/Search) to learn more about search queries.
+
+3. We automatically create a chunker to chunk your discourse data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+
+from embedchain.chunkers.discourse import DiscourseChunker
+from embedchain.config.add_config import ChunkerConfig
+
+discourse_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+discourse_chunker = DiscourseChunker(config=discourse_chunker_config)
+
+app.add("openai", data_type='discourse', loader=dicourse_loader, chunker=discourse_chunker)
+```

docs/components/data-sources/docs-site.mdx

@@ -0,0 +1,14 @@
+---
+title: '📚 Code Docs website'
+---
+
+To add any code documentation website as a loader, use the data_type as `docs_site`. Eg:
+
+```python
+from embedchain import App
+
+app = App()
+app.add("https://docs.embedchain.ai/", data_type="docs_site")
+app.query("What is Embedchain?")
+# Answer: Embedchain is a platform that utilizes various components, including paid/proprietary ones, to provide what is believed to be the best configuration available. It uses LLM (Language Model) providers such as OpenAI, Anthpropic, Vertex_AI, GPT4ALL, Azure_OpenAI, LLAMA2, JINA, Ollama, Together and COHERE. Embedchain allows users to import and utilize these LLM providers for their applications.'
+```

docs/components/data-sources/docx.mdx

@@ -0,0 +1,18 @@
+---
+title: '📄 Docx file'
+---
+
+### Docx file
+
+To add any doc/docx file, use the data_type as `docx`. `docx` allows remote urls and conventional file paths. Eg:
+
+```python
+from embedchain import App
+
+app = App()
+app.add('https://example.com/content/intro.docx', data_type="docx")
+# Or add file using the local file path on your system
+# app.add('content/intro.docx', data_type="docx")
+
+app.query("Summarize the docx data?")
+```

docs/components/data-sources/dropbox.mdx

@@ -0,0 +1,37 @@
+---
+title: '💾 Dropbox'
+---
+
+To load folders or files from your Dropbox account, configure the `data_type` parameter as `dropbox` and specify the path to the desired file or folder, starting from the root directory of your Dropbox account.
+
+For Dropbox access, an **access token** is required. Obtain this token by visiting [Dropbox Developer Apps](https://www.dropbox.com/developers/apps). There, create a new app and generate an access token for it.
+
+Ensure your app has the following settings activated:
+
+- In the Permissions section, enable `files.content.read` and `files.metadata.read`.
+
+## Usage
+
+Install the `dropbox` pypi package:
+
+```bash
+pip install dropbox
+```
+
+Following is an example of how to use the dropbox loader:
+
+```python
+import os
+from embedchain import App
+
+os.environ["DROPBOX_ACCESS_TOKEN"] = "sl.xxx"
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+
+# any path from the root of your dropbox account, you can leave it "" for the root folder
+app.add("/test", data_type="dropbox")
+
+print(app.query("Which two celebrities are mentioned here?"))
+# The two celebrities mentioned in the given context are Elon Musk and Jeff Bezos.
+```

docs/components/data-sources/github.mdx

@@ -0,0 +1,50 @@
+---
+title: 📝 Github
+---
+
+1. Setup the Github loader by configuring the Github account with username and personal access token (PAT). Check out [this](https://docs.github.com/en/enterprise-server@3.6/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token) link to learn how to create a PAT.
+```Python
+from embedchain.loaders.github import GithubLoader
+
+loader = GithubLoader(
+    config={
+        "token":"ghp_xxxx"
+        }
+    )
+```
+
+2. Once you setup the loader, you can create an app and load data using the above Github loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxxx"
+
+app = App()
+
+app.add("repo:embedchain/embedchain type:repo", data_type="github", loader=loader)
+
+response = app.query("What is Embedchain?")
+# Answer: Embedchain is a Data Platform for Large Language Models (LLMs). It allows users to seamlessly load, index, retrieve, and sync unstructured data in order to build dynamic, LLM-powered applications. There is also a JavaScript implementation called embedchain-js available on GitHub.
+```
+The `add` function of the app will accept any valid github query with qualifiers. It only supports loading github code, repository, issues and pull-requests.
+<Note>
+You must provide qualifiers `type:` and `repo:` in the query. The `type:` qualifier can be a combination of `code`, `repo`, `pr`, `issue`. The `repo:` qualifier must be a valid github repository name.
+</Note>
+
+<Card title="Valid queries" icon="lightbulb" iconType="duotone" color="#ca8b04">
+    - `repo:embedchain/embedchain type:repo` - to load the repository
+    - `repo:embedchain/embedchain type:issue,pr` - to load the issues and pull-requests of the repository
+    - `repo:embedchain/embedchain type:issue state:closed` - to load the closed issues of the repository
+</Card>
+
+3. We automatically create a chunker to chunk your GitHub data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+from embedchain.chunkers.common_chunker import CommonChunker
+from embedchain.config.add_config import ChunkerConfig
+
+github_chunker_config = ChunkerConfig(chunk_size=2000, chunk_overlap=0, length_function=len)
+github_chunker = CommonChunker(config=github_chunker_config)
+
+app.add(load_query, data_type="github", loader=loader, chunker=github_chunker)
+```

docs/components/data-sources/gmail.mdx

@@ -0,0 +1,34 @@
+---
+title: '📬 Gmail'
+---
+
+To use GmailLoader you must install the extra dependencies with `pip install --upgrade embedchain[gmail]`.
+
+The `source` must be a valid Gmail search query, you can refer `https://support.google.com/mail/answer/7190?hl=en` to build a query.
+
+To load Gmail messages, you MUST use the data_type as `gmail`. Otherwise the source will be detected as simple `text`.
+
+To use this you need to save `credentials.json` in the directory from where you will run the loader. Follow these steps to get the credentials
+
+1. Go to the [Google Cloud Console](https://console.cloud.google.com/apis/credentials).
+2. Create a project if you don't have one already.
+3. Create an `OAuth Consent Screen` in the project. You may need to select the `external` option.
+4. Make sure the consent screen is published.
+5. Enable the [Gmail API](https://console.cloud.google.com/apis/api/gmail.googleapis.com)
+6. Create credentials from the `Credentials` tab.
+7. Select the type `OAuth Client ID`.
+8. Choose the application type `Web application`. As a name you can choose `embedchain` or any other name as per your use case.
+9. Add an authorized redirect URI for `http://localhost:8080/`.
+10. You can leave everything else at default, finish the creation.
+11. When you are done, a modal opens where you can download the details in `json` format.
+12. Put the `.json` file in your current directory and rename it to `credentials.json`
+
+```python
+from embedchain import App
+
+app = App()
+
+gmail_filter = "to: me label:inbox"
+app.add(gmail_filter, data_type="gmail")
+app.query("Summarize my email conversations")
+```

docs/components/data-sources/google-drive.mdx

@@ -0,0 +1,28 @@
+---
+title: 'Google Drive'
+---
+
+To use GoogleDriveLoader you must install the extra dependencies with `pip install --upgrade embedchain[googledrive]`.
+
+The data_type must be `google_drive`. Otherwise, it will be considered a regular web page.
+
+Google Drive requires the setup of credentials. This can be done by following the steps below:
+
+1. Go to the [Google Cloud Console](https://console.cloud.google.com/apis/credentials).
+2. Create a project if you don't have one already.
+3. Enable the [Google Drive API](https://console.cloud.google.com/flows/enableapi?apiid=drive.googleapis.com)
+4. [Authorize credentials for desktop app](https://developers.google.com/drive/api/quickstart/python#authorize_credentials_for_a_desktop_application)
+5. When done, you will be able to download the credentials in `json` format. Rename the downloaded file to `credentials.json` and save it in `~/.credentials/credentials.json`
+6. Set the environment variable `GOOGLE_APPLICATION_CREDENTIALS=~/.credentials/credentials.json`
+
+The first time you use the loader, you will be prompted to enter your Google account credentials.
+
+
+```python
+from embedchain import App
+
+app = App()
+
+url = "https://drive.google.com/drive/u/0/folders/xxx-xxx"
+app.add(url, data_type="google_drive")
+```

docs/components/data-sources/image.mdx

@@ -0,0 +1,45 @@
+---
+title: "🖼️ Image"
+---
+
+
+To use an image as data source, just add `data_type` as `image` and pass in the path of the image (local or hosted).
+
+We use [GPT4 Vision](https://platform.openai.com/docs/guides/vision) to generate meaning of the image using a custom prompt, and then use the generated text as the data source.
+
+You would require an OpenAI API key with access to `gpt-4-vision-preview` model to use this feature.
+
+### Without customization
+
+```python
+import os
+from embedchain import App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+app.add("./Elon-Musk.webp", data_type="image")
+response = app.query("Describe the man in the image.")
+print(response)
+# Answer: The man in the image is dressed in formal attire, wearing a dark suit jacket and a white collared shirt. He has short hair and is standing. He appears to be gazing off to the side with a reflective expression. The background is dark with faint, warm-toned vertical lines, possibly from a lit environment behind the individual or reflections. The overall atmosphere is somewhat moody and introspective.
+```
+
+### Customization
+
+```python
+import os
+from embedchain import App
+from embedchain.loaders.image import ImageLoader
+
+image_loader = ImageLoader(
+    max_tokens=100,
+    api_key="sk-xxx",
+    prompt="Is the person looking wealthy? Structure your thoughts around what you see in the image.",
+)
+
+app = App()
+app.add("./Elon-Musk.webp", data_type="image", loader=image_loader)
+response = app.query("Describe the man in the image.")
+print(response)
+# Answer: The man in the image appears to be well-dressed in a suit and shirt, suggesting that he may be in a professional or formal setting. His composed demeanor and confident posture further indicate a sense of self-assurance. Based on these visual cues, one could infer that the man may have a certain level of economic or social status, possibly indicating wealth or professional success.
+```

docs/components/data-sources/json.mdx

@@ -0,0 +1,44 @@
+---
+title: '📃 JSON'
+---
+
+To add any json file, use the data_type as `json`. Headers are included for each line, so for example if you have a json like `{"age": 18}`, then it will be added as `age: 18`.
+
+Here are the supported sources for loading `json`:
+
+```
+1. URL - valid url to json file that ends with ".json" extension.
+2. Local file - valid url to local json file that ends with ".json" extension.
+3. String - valid json string (e.g. - app.add('{"foo": "bar"}'))
+```
+
+<Tip>
+If you would like to add other data structures (e.g. list, dict etc.), convert it to a valid json first using `json.dumps()` function.
+</Tip>
+
+## Example
+
+<CodeGroup>
+
+```python python
+from embedchain import App
+
+app = App()
+
+# Add json file
+app.add("temp.json")
+
+app.query("What is the net worth of Elon Musk as of October 2023?")
+# As of October 2023, Elon Musk's net worth is $255.2 billion.
+```
+
+
+```json temp.json
+{
+    "question": "What is your net worth, Elon Musk?",
+    "answer": "As of October 2023, Elon Musk's net worth is $255.2 billion, making him one of the wealthiest individuals in the world."
+}
+```
+</CodeGroup>
+
+

docs/components/data-sources/mdx.mdx

@@ -0,0 +1,14 @@
+---
+title: '📝 Mdx file'
+---
+
+To add any `.mdx` file to your app, use the data_type (first argument to `.add()` method) as `mdx`. Note that this supports support mdx file present on machine, so this should be a file path. Eg:
+
+```python
+from embedchain import App
+
+app = App()
+app.add('path/to/file.mdx', data_type='mdx')
+
+app.query("What are the docs about?")
+```

docs/components/data-sources/mysql.mdx

@@ -0,0 +1,47 @@
+---
+title: '🐬 MySQL'
+---
+
+1. Setup the MySQL loader by configuring the SQL db.
+```Python
+from embedchain.loaders.mysql import MySQLLoader
+
+config = {
+    "host": "host",
+    "port": "port",
+    "database": "database",
+    "user": "username",
+    "password": "password",
+}
+
+mysql_loader = MySQLLoader(config=config)
+```
+
+For more details on how to setup with valid config, check MySQL [documentation](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html).
+
+2. Once you setup the loader, you can create an app and load data using the above MySQL loader
+```Python
+from embedchain.pipeline import Pipeline as App
+
+app = App()
+
+app.add("SELECT * FROM table_name;", data_type='mysql', loader=mysql_loader)
+# Adds `(1, 'What is your net worth, Elon Musk?', "As of October 2023, Elon Musk's net worth is $255.2 billion.")`
+
+response = app.query(question)
+# Answer: As of October 2023, Elon Musk's net worth is $255.2 billion.
+```
+
+NOTE: The `add` function of the app will accept any executable query to load data. DO NOT pass the `CREATE`, `INSERT` queries in `add` function.
+
+3. We automatically create a chunker to chunk your SQL data, however if you wish to provide your own chunker class. Here is how you can do that:
+``Python
+
+from embedchain.chunkers.mysql import MySQLChunker
+from embedchain.config.add_config import ChunkerConfig
+
+mysql_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+mysql_chunker = MySQLChunker(config=mysql_chunker_config)
+
+app.add("SELECT * FROM table_name;", data_type='mysql', loader=mysql_loader, chunker=mysql_chunker)
+```

docs/components/data-sources/notion.mdx

@@ -0,0 +1,20 @@
+---
+title: '📓 Notion'
+---
+
+To use notion you must install the extra dependencies with `pip install --upgrade embedchain[community]`.
+
+To load a notion page, use the data_type as `notion`. Since it is hard to automatically detect, it is advised to specify the `data_type` when adding a notion document.
+The next argument must **end** with the `notion page id`. The id is a 32-character string. Eg:
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add("cfbc134ca6464fc980d0391613959196", data_type="notion")
+app.add("my-page-cfbc134ca6464fc980d0391613959196", data_type="notion")
+app.add("https://www.notion.so/my-page-cfbc134ca6464fc980d0391613959196", data_type="notion")
+
+app.query("Summarize the notion doc")
+```

docs/components/data-sources/openapi.mdx

@@ -0,0 +1,22 @@
+---
+title: 🙌 OpenAPI
+---
+
+To add any OpenAPI spec yaml file (currently the json file will be detected as JSON data type), use the data_type as 'openapi'. 'openapi' allows remote urls and conventional file paths.
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add("https://github.com/openai/openai-openapi/blob/master/openapi.yaml", data_type="openapi")
+# Or add using the local file path
+# app.add("configs/openai_openapi.yaml", data_type="openapi")
+
+app.query("What can OpenAI API endpoint do? Can you list the things it can learn from?")
+# Answer: The OpenAI API endpoint allows users to interact with OpenAI's models and perform various tasks such as generating text, answering questions, summarizing documents, translating languages, and more. The specific capabilities and tasks that the API can learn from may vary depending on the models and features provided by OpenAI. For more detailed information, it is recommended to refer to the OpenAI API documentation at https://platform.openai.com/docs/api-reference.
+```
+
+<Note>
+The yaml file added to the App must have the required OpenAPI fields otherwise the adding OpenAPI spec will fail. Please refer to [OpenAPI Spec Doc](https://spec.openapis.org/oas/v3.1.0)
+</Note>

docs/components/data-sources/overview.mdx

@@ -0,0 +1,40 @@
+---
+title: Overview
+---
+
+Embedchain comes with built-in support for various data sources. We handle the complexity of loading unstructured data from these data sources, allowing you to easily customize your app through a user-friendly interface.
+
+<CardGroup cols={4}>
+  <Card title="📰 PDF file" href="/components/data-sources/pdf-file"></Card>
+  <Card title="📊 CSV file" href="/components/data-sources/csv"></Card>
+  <Card title="📃 JSON file" href="/components/data-sources/json"></Card>
+  <Card title="📝 Text" href="/components/data-sources/text"></Card>
+  <Card title="📁 Directory/ Folder" href="/components/data-sources/directory"></Card>
+  <Card title="🌐 HTML Web page" href="/components/data-sources/web-page"></Card>
+  <Card title="📽️ Youtube Channel" href="/components/data-sources/youtube-channel"></Card>
+  <Card title="📺 Youtube Video" href="/components/data-sources/youtube-video"></Card>
+  <Card title="📚 Docs website" href="/components/data-sources/docs-site"></Card>
+  <Card title="📝 MDX file" href="/components/data-sources/mdx"></Card>
+  <Card title="📄 DOCX file" href="/components/data-sources/docx"></Card>
+  <Card title="📓 Notion" href="/components/data-sources/notion"></Card>
+  <Card title="🗺️ Sitemap" href="/components/data-sources/sitemap"></Card>
+  <Card title="🧾 XML file" href="/components/data-sources/xml"></Card>
+  <Card title="❓💬 Q&A pair" href="/components/data-sources/qna"></Card>
+  <Card title="🙌 OpenAPI" href="/components/data-sources/openapi"></Card>
+  <Card title="📬 Gmail" href="/components/data-sources/gmail"></Card>
+  <Card title="📝 Github" href="/components/data-sources/github"></Card>
+  <Card title="🐘 Postgres" href="/components/data-sources/postgres"></Card>
+  <Card title="🐬 MySQL" href="/components/data-sources/mysql"></Card>
+  <Card title="🤖 Slack" href="/components/data-sources/slack"></Card>
+  <Card title="💬 Discord" href="/components/data-sources/discord"></Card>
+  <Card title="🗨️ Discourse" href="/components/data-sources/discourse"></Card>
+  <Card title="📝 Substack" href="/components/data-sources/substack"></Card>
+  <Card title="🐝 Beehiiv" href="/components/data-sources/beehiiv"></Card>
+  <Card title="💾 Dropbox" href="/components/data-sources/dropbox"></Card>
+  <Card title="🖼️ Image" href="/components/data-sources/image"></Card>
+  <Card title="⚙️ Custom" href="/components/data-sources/custom"></Card>
+</CardGroup>
+
+<br/ >
+
+<Snippet file="missing-data-source-tip.mdx" />

docs/components/data-sources/pdf-file.mdx

@@ -0,0 +1,43 @@
+---
+title: '📰 PDF'
+---
+
+You can load any pdf file from your local file system or through a URL.
+
+## Usage
+
+### Load from a local file
+
+```python
+from embedchain import App
+app = App()
+app.add('/path/to/file.pdf', data_type='pdf_file')
+```
+
+### Load from URL
+
+```python
+from embedchain import App
+app = App()
+app.add('https://arxiv.org/pdf/1706.03762.pdf', data_type='pdf_file')
+app.query("What is the paper 'attention is all you need' about?", citations=True)
+# Answer: The paper "Attention Is All You Need" proposes a new network architecture called the Transformer, which is based solely on attention mechanisms. It suggests that complex recurrent or convolutional neural networks can be replaced with a simpler architecture that connects the encoder and decoder through attention. The paper discusses how this approach can improve sequence transduction models, such as neural machine translation.
+# Contexts:
+# [
+#     (
+#         'Provided proper attribution is ...',
+#         {
+#             'page': 0,
+#             'url': 'https://arxiv.org/pdf/1706.03762.pdf',
+#             'score': 0.3676220203221626,
+#             ...
+#         }
+#     ),
+# ]
+```
+
+We also store the page number under the key `page` with each chunk that helps understand where the answer is coming from. You can fetch the `page` key while during retrieval (refer to the example given above).
+
+<Note>
+Note that we do not support password protected pdf files.
+</Note>

docs/components/data-sources/postgres.mdx

@@ -0,0 +1,64 @@
+---
+title: '🐘 Postgres'
+---
+
+1. Setup the Postgres loader by configuring the postgres db.
+```Python
+from embedchain.loaders.postgres import PostgresLoader
+
+config = {
+    "host": "host_address",
+    "port": "port_number",
+    "dbname": "database_name",
+    "user": "username",
+    "password": "password",
+}
+
+"""
+config = {
+    "url": "your_postgres_url"
+}
+"""
+
+postgres_loader = PostgresLoader(config=config)
+
+```
+
+You can either setup the loader by passing the postgresql url or by providing the config data.
+For more details on how to setup with valid url and config, check postgres [documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING:~:text=34.1.1.%C2%A0Connection%20Strings-,%23,-Several%20libpq%20functions).
+
+NOTE: if you provide the `url` field in config, all other fields will be ignored.
+
+2. Once you setup the loader, you can create an app and load data using the above postgres loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+
+question = "What is Elon Musk's networth?"
+response = app.query(question)
+# Answer: As of September 2021, Elon Musk's net worth is estimated to be around $250 billion, making him one of the wealthiest individuals in the world. However, please note that net worth can fluctuate over time due to various factors such as stock market changes and business ventures.
+
+app.add("SELECT * FROM table_name;", data_type='postgres', loader=postgres_loader)
+# Adds `(1, 'What is your net worth, Elon Musk?', "As of October 2023, Elon Musk's net worth is $255.2 billion.")`
+
+response = app.query(question)
+# Answer: As of October 2023, Elon Musk's net worth is $255.2 billion.
+```
+
+NOTE: The `add` function of the app will accept any executable query to load data. DO NOT pass the `CREATE`, `INSERT` queries in `add` function as they will result in not adding any data, so it is pointless.
+
+3. We automatically create a chunker to chunk your postgres data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+
+from embedchain.chunkers.postgres import PostgresChunker
+from embedchain.config.add_config import ChunkerConfig
+
+postgres_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+postgres_chunker = PostgresChunker(config=postgres_chunker_config)
+
+app.add("SELECT * FROM table_name;", data_type='postgres', loader=postgres_loader, chunker=postgres_chunker)
+```

docs/components/data-sources/qna.mdx

@@ -0,0 +1,13 @@
+---
+title: '❓💬 Queston and answer pair'
+---
+
+QnA pair is a local data type. To supply your own QnA pair, use the data_type as `qna_pair` and enter a tuple. Eg:
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add(("Question", "Answer"), data_type="qna_pair")
+```

docs/components/data-sources/sitemap.mdx

@@ -0,0 +1,13 @@
+---
+title: '🗺️ Sitemap'
+---
+
+Add all web pages from an xml-sitemap. Filters non-text files. Use the data_type as `sitemap`. Eg:
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add('https://example.com/sitemap.xml', data_type='sitemap')
+```

docs/components/data-sources/slack.mdx

@@ -0,0 +1,71 @@
+---
+title: '🤖 Slack'
+---
+
+## Pre-requisite
+- Download required packages by running `pip install --upgrade "embedchain[slack]"`.
+- Configure your slack bot token as environment variable `SLACK_USER_TOKEN`.
+    - Find your user token on your [Slack Account](https://api.slack.com/authentication/token-types)
+    - Make sure your slack user token includes [search](https://api.slack.com/scopes/search:read) scope.
+
+## Example
+
+### Get Started
+
+This will automatically retrieve data from the workspace associated with the user's token.
+
+```python
+import os
+from embedchain import App
+
+os.environ["SLACK_USER_TOKEN"] = "xoxp-xxx"
+app = App()
+
+app.add("in:general", data_type="slack")
+
+result = app.query("what are the messages in general channel?")
+
+print(result)
+```
+
+
+### Customize your SlackLoader
+1. Setup the Slack loader by configuring the Slack Webclient.
+```Python
+from embedchain.loaders.slack import SlackLoader
+
+os.environ["SLACK_USER_TOKEN"] = "xoxp-*"
+
+config = {
+    'base_url': slack_app_url,
+    'headers': web_headers,
+    'team_id': slack_team_id,
+}
+
+loader = SlackLoader(config)
+```
+
+NOTE: you can also pass the `config` with `base_url`, `headers`, `team_id` to setup your SlackLoader.
+
+2. Once you setup the loader, you can create an app and load data using the above slack loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+app = App()
+
+app.add("in:random", data_type="slack", loader=loader)
+question = "Which bots are available in the slack workspace's random channel?"
+# Answer: The available bot in the slack workspace's random channel is the Embedchain bot.
+```
+
+3. We automatically create a chunker to chunk your slack data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+from embedchain.chunkers.slack import SlackChunker
+from embedchain.config.add_config import ChunkerConfig
+
+slack_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+slack_chunker = SlackChunker(config=slack_chunker_config)
+
+app.add(slack_chunker, data_type="slack", loader=loader, chunker=slack_chunker)
+```

docs/components/data-sources/substack.mdx

@@ -0,0 +1,16 @@
+---
+title: "📝 Substack"
+---
+
+To add any Substack data sources to your app, just add the main base url as the source and set the data_type to `substack`.
+
+```python
+from embedchain import App
+
+app = App()
+
+# source: for any substack just add the root URL
+app.add('https://www.lennysnewsletter.com', data_type='substack')
+app.query("Who is Brian Chesky?")
+# Answer: Brian Chesky is the co-founder and CEO of Airbnb.
+```

docs/components/data-sources/text.mdx

@@ -0,0 +1,17 @@
+---
+title: '📝 Text'
+---
+
+### Text
+
+Text is a local data type. To supply your own text, use the data_type as `text` and enter a string. The text is not processed, this can be very versatile. Eg:
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add('Seek wealth, not money or status. Wealth is having assets that earn while you sleep. Money is how we transfer time and wealth. Status is your place in the social hierarchy.', data_type='text')
+```
+
+Note: This is not used in the examples because in most cases you will supply a whole paragraph or file, which did not fit.

docs/components/data-sources/web-page.mdx

@@ -0,0 +1,13 @@
+---
+title: '🌐 HTML Web page'
+---
+
+To add any web page, use the data_type as `web_page`. Eg:
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add('a_valid_web_page_url', data_type='web_page')
+```

docs/components/data-sources/xml.mdx

@@ -0,0 +1,17 @@
+---
+title: '🧾 XML file'
+---
+
+### XML file
+
+To add any xml file, use the data_type as `xml`. Eg:
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add('content/data.xml')
+```
+
+Note: Only the text content of the xml file will be added to the app. The tags will be ignored.

docs/components/data-sources/youtube-channel.mdx

@@ -0,0 +1,22 @@
+---
+title: '📽️ Youtube Channel'
+---
+
+## Setup
+
+Make sure you have all the required packages installed before using this data type. You can install them by running the following command in your terminal.
+
+```bash
+pip install -u "embedchain[youtube]"
+```
+
+## Usage
+
+To add all the videos from a youtube channel to your app, use the data_type as `youtube_channel`.
+
+```python
+from embedchain import App
+
+app = App()
+app.add("@channel_name", data_type="youtube_channel")
+```

docs/components/data-sources/youtube-video.mdx

@@ -0,0 +1,22 @@
+---
+title: '📺 Youtube Video'
+---
+
+## Setup
+
+Make sure you have all the required packages installed before using this data type. You can install them by running the following command in your terminal.
+
+```bash
+pip install -u "embedchain[youtube]"
+```
+
+## Usage
+
+To add any youtube video to your app, use the data_type as `youtube_video`. Eg:
+
+```python
+from embedchain import App
+
+app = App()
+app.add('a_valid_youtube_url_here', data_type='youtube_video')
+```

docs/components/embedding-models.mdx

@@ -0,0 +1,222 @@
+---
+title: 🧩 Embedding models
+---
+
+## Overview
+
+Embedchain supports several embedding models from the following providers:
+
+<CardGroup cols={4}>
+  <Card title="OpenAI" href="#openai"></Card>
+  <Card title="GoogleAI" href="#google-ai"></Card>
+  <Card title="Azure OpenAI" href="#azure-openai"></Card>
+  <Card title="GPT4All" href="#gpt4all"></Card>
+  <Card title="Hugging Face" href="#hugging-face"></Card>
+  <Card title="Vertex AI" href="#vertex-ai"></Card>
+</CardGroup>
+
+## OpenAI
+
+To use OpenAI embedding function, you have to set the `OPENAI_API_KEY` environment variable. You can obtain the OpenAI API key from the [OpenAI Platform](https://platform.openai.com/account/api-keys).
+
+Once you have obtained the key, you can use it like this:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ['OPENAI_API_KEY'] = 'xxx'
+
+# load embedding model configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+
+app.add("https://en.wikipedia.org/wiki/OpenAI")
+app.query("What is OpenAI?")
+```
+
+```yaml config.yaml
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-3-small'
+```
+
+</CodeGroup>
+
+* OpenAI announced two new embedding models: `text-embedding-3-small` and `text-embedding-3-large`. Embedchain supports both these models. Below you can find YAML config for both:
+
+<CodeGroup>
+
+```yaml text-embedding-3-small.yaml
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-3-small'
+```
+
+```yaml text-embedding-3-large.yaml
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-3-large'
+```
+
+</CodeGroup>
+
+## Google AI
+
+To use Google AI embedding function, you have to set the `GOOGLE_API_KEY` environment variable. You can obtain the Google API key from the [Google Maker Suite](https://makersuite.google.com/app/apikey)
+
+<CodeGroup>
+```python main.py
+import os
+from embedchain import App
+
+os.environ["GOOGLE_API_KEY"] = "xxx"
+
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+embedder:
+  provider: google
+  config:
+    model: 'models/embedding-001'
+    task_type: "retrieval_document"
+    title: "Embeddings for Embedchain"
+```
+</CodeGroup>
+<br/>
+<Note>
+For more details regarding the Google AI embedding model, please refer to the [Google AI documentation](https://ai.google.dev/tutorials/python_quickstart#use_embeddings).
+</Note>
+
+## Azure OpenAI
+
+To use Azure OpenAI embedding model, you have to set some of the azure openai related environment variables as given in the code block below:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["OPENAI_API_TYPE"] = "azure"
+os.environ["AZURE_OPENAI_ENDPOINT"] = "https://xxx.openai.azure.com/"
+os.environ["AZURE_OPENAI_API_KEY"] = "xxx"
+os.environ["OPENAI_API_VERSION"] = "xxx"
+
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: azure_openai
+  config:
+    model: gpt-35-turbo
+    deployment_name: your_llm_deployment_name
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: azure_openai
+  config:
+    model: text-embedding-ada-002
+    deployment_name: you_embedding_model_deployment_name
+```
+</CodeGroup>
+
+You can find the list of models and deployment name on the [Azure OpenAI Platform](https://oai.azure.com/portal).
+
+## GPT4ALL
+
+GPT4All supports generating high quality embeddings of arbitrary length documents of text using a CPU optimized contrastively trained Sentence Transformer.
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load embedding model configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: gpt4all
+```
+
+</CodeGroup>
+
+## Hugging Face
+
+Hugging Face supports generating embeddings of arbitrary length documents of text using Sentence Transformer library. Example of how to generate embeddings using hugging face is given below:
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load embedding model configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: huggingface
+  config:
+    model: 'google/flan-t5-xxl'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false
+
+embedder:
+  provider: huggingface
+  config:
+    model: 'sentence-transformers/all-mpnet-base-v2'
+```
+
+</CodeGroup>
+
+## Vertex AI
+
+Embedchain supports Google's VertexAI embeddings model through a simple interface. You just have to pass the `model_name` in the config yaml and it would work out of the box.
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load embedding model configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: vertexai
+  config:
+    model: 'chat-bison'
+    temperature: 0.5
+    top_p: 0.5
+
+embedder:
+  provider: vertexai
+  config:
+    model: 'textembedding-gecko'
+```
+
+</CodeGroup>

docs/components/evaluation.mdx

@@ -0,0 +1,275 @@
+---
+title: 🔬 Evaluation
+---
+
+## Overview
+
+We provide out-of-the-box evaluation metrics for your RAG application. You can use them to evaluate your RAG applications and compare against different settings of your production RAG application.
+
+Currently, we provide support for following evaluation metrics:
+
+<CardGroup cols={3}>
+    <Card title="Context Relevancy" href="#context_relevancy"></Card>
+    <Card title="Answer Relevancy" href="#answer_relevancy"></Card>
+    <Card title="Groundedness" href="#groundedness"></Card>
+    <Card title="Custom Metric" href="#custom_metric"></Card>
+</CardGroup>
+
+## Quickstart
+
+Here is a basic example of running evaluation:
+
+```python example.py
+from embedchain import App
+
+app = App()
+
+# Add data sources
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Run evaluation
+app.evaluate(["What is the net worth of Elon Musk?", "How many companies Elon Musk owns?"])
+# {'answer_relevancy': 0.9987286412340826, 'groundedness': 1.0, 'context_relevancy': 0.3571428571428571}
+```
+
+Under the hood, Embedchain does the following:
+
+1. Runs semantic search in the vector database and fetches context
+2. LLM call with question, context to fetch the answer
+3. Run evaluation on following metrics: `context relevancy`, `groundedness`, and `answer relevancy` and return result
+
+## Advanced Usage
+
+We use OpenAI's `gpt-4` model as default LLM model for automatic evaluation. Hence, we require you to set `OPENAI_API_KEY` as an environment variable.
+
+### Step-1: Create dataset
+
+In order to evaluate your RAG application, you have to setup a dataset. A data point in the dataset consists of `questions`, `contexts`, `answer`. Here is an example of how to create a dataset for evaluation:
+
+```python
+from embedchain.utils.eval import EvalData
+
+data = [
+    {
+        "question": "What is the net worth of Elon Musk?",
+        "contexts": [
+            "Elon Musk PROFILEElon MuskCEO, ...",
+            "a Twitter poll on whether the journalists' ...",
+            "2016 and run by Jared Birchall.[335]...",
+        ],
+        "answer": "As of the information provided, Elon Musk's net worth is $241.6 billion.",
+    },
+    {
+        "question": "which companies does Elon Musk own?",
+        "contexts": [
+            "of December 2023[update], ...",
+            "ThielCofounderView ProfileTeslaHolds ...",
+            "Elon Musk PROFILEElon MuskCEO, ...",
+        ],
+        "answer": "Elon Musk owns several companies, including Tesla, SpaceX, Neuralink, and The Boring Company.",
+    },
+]
+
+dataset = []
+
+for d in data:
+    eval_data = EvalData(question=d["question"], contexts=d["contexts"], answer=d["answer"])
+    dataset.append(eval_data)
+```
+
+### Step-2: Run evaluation
+
+Once you have created your dataset, you can run evaluation on the dataset by picking the metric you want to run evaluation on.
+
+For example, you can run evaluation on context relevancy metric using the following code:
+
+```python
+from embedchain.evaluation.metrics import ContextRelevance
+metric = ContextRelevance()
+score = metric.evaluate(dataset)
+print(score)
+```
+
+You can choose a different metric or write your own to run evaluation on. You can check the following links:
+
+- [Context Relevancy](#context_relevancy)
+- [Answer relenvancy](#answer_relevancy)
+- [Groundedness](#groundedness)
+- [Build your own metric](#custom_metric)
+
+## Metrics
+
+### Context Relevancy <a id="context_relevancy"></a>
+
+Context relevancy is a metric to determine "how relevant the context is to the question". We use OpenAI's `gpt-4` model to determine the relevancy of the context. We achieve this by prompting the model with the question and the context and asking it to return relevant sentences from the context. We then use the following formula to determine the score:
+
+```
+context_relevance_score = num_relevant_sentences_in_context / num_of_sentences_in_context
+```
+
+#### Examples
+
+You can run the context relevancy evaluation with the following simple code:
+
+```python
+from embedchain.evaluation.metrics import ContextRelevance
+
+metric = ContextRelevance()
+score = metric.evaluate(dataset)  # 'dataset' is definted in the create dataset section
+print(score)
+# 0.27975528364849833
+```
+
+In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `ContextRelevanceConfig` class.
+
+Here is a more advanced example of how to pass a custom evaluation config for evaluating on context relevance metric:
+
+```python
+from embedchain.config.evaluation.base import ContextRelevanceConfig
+from embedchain.evaluation.metrics import ContextRelevance
+
+eval_config = ContextRelevanceConfig(model="gpt-4", api_key="sk-xxx", language="en")
+metric = ContextRelevance(config=eval_config)
+metric.evaluate(dataset)
+```
+
+#### `ContextRelevanceConfig`
+
+<ParamField path="model" type="str" optional>
+    The model to use for the evaluation. Defaults to `gpt-4`. We only support openai's models for now.
+</ParamField>
+<ParamField path="api_key" type="str" optional>
+    The openai api key to use for the evaluation. Defaults to `None`. If not provided, we will use the `OPENAI_API_KEY` environment variable.
+</ParamField>
+<ParamField path="language" type="str" optional>
+    The language of the dataset being evaluated. We need this to determine the understand the context provided in the dataset. Defaults to `en`.
+</ParamField>
+<ParamField path="prompt" type="str" optional>
+    The prompt to extract the relevant sentences from the context. Defaults to `CONTEXT_RELEVANCY_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.
+</ParamField>
+
+
+### Answer Relevancy <a id="answer_relevancy"></a>
+
+Answer relevancy is a metric to determine how relevant the answer is to the question. We prompt the model with the answer and asking it to generate questions from the answer. We then use the cosine similarity between the generated questions and the original question to determine the score.
+
+```
+answer_relevancy_score = mean(cosine_similarity(generated_questions, original_question))
+```
+
+#### Examples
+
+You can run the answer relevancy evaluation with the following simple code:
+
+```python
+from embedchain.evaluation.metrics import AnswerRelevance
+
+metric = AnswerRelevance()
+score = metric.evaluate(dataset)
+print(score)
+# 0.9505334177461916
+```
+
+In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `AnswerRelevanceConfig` class. Here is a more advanced example where you can provide your own evaluation config:
+
+```python
+from embedchain.config.evaluation.base import AnswerRelevanceConfig
+from embedchain.evaluation.metrics import AnswerRelevance
+
+eval_config = AnswerRelevanceConfig(
+    model='gpt-4',
+    embedder="text-embedding-ada-002",
+    api_key="sk-xxx",
+    num_gen_questions=2
+)
+metric = AnswerRelevance(config=eval_config)
+score = metric.evaluate(dataset)
+```
+
+#### `AnswerRelevanceConfig`
+
+<ParamField path="model" type="str" optional>
+    The model to use for the evaluation. Defaults to `gpt-4`. We only support openai's models for now.
+</ParamField>
+<ParamField path="embedder" type="str" optional>
+    The embedder to use for embedding the text. Defaults to `text-embedding-ada-002`. We only support openai's embedders for now.
+</ParamField>
+<ParamField path="api_key" type="str" optional>
+    The openai api key to use for the evaluation. Defaults to `None`. If not provided, we will use the `OPENAI_API_KEY` environment variable.
+</ParamField>
+<ParamField path="num_gen_questions" type="int" optional>
+    The number of questions to generate for each answer. We use the generated questions to compare the similarity with the original question to determine the score. Defaults to `1`.
+</ParamField>
+<ParamField path="prompt" type="str" optional>
+    The prompt to extract the `num_gen_questions` number of questions from the provided answer. Defaults to `ANSWER_RELEVANCY_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.
+</ParamField>
+
+## Groundedness <a id="groundedness"></a>
+
+Groundedness is a metric to determine how grounded the answer is to the context. We use OpenAI's `gpt-4` model to determine the groundedness of the answer. We achieve this by prompting the model with the answer and asking it to generate claims from the answer. We then again prompt the model with the context and the generated claims to determine the verdict on the claims. We then use the following formula to determine the score:
+
+```
+groundedness_score = (sum of all verdicts) / (total # of claims)
+```
+
+You can run the groundedness evaluation with the following simple code:
+
+```python
+from embedchain.evaluation.metrics import Groundedness
+metric = Groundedness()
+score = metric.evaluate(dataset)    # dataset from above
+print(score)
+# 1.0
+```
+
+In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `GroundednessConfig` class. Here is a more advanced example where you can configure the evaluation config:
+
+```python
+from embedchain.config.evaluation.base import GroundednessConfig
+from embedchain.evaluation.metrics import Groundedness
+
+eval_config = GroundednessConfig(model='gpt-4', api_key="sk-xxx")
+metric = Groundedness(config=eval_config)
+score = metric.evaluate(dataset)
+```
+
+
+#### `GroundednessConfig`
+
+<ParamField path="model" type="str" optional>
+    The model to use for the evaluation. Defaults to `gpt-4`. We only support openai's models for now.
+</ParamField>
+<ParamField path="api_key" type="str" optional>
+    The openai api key to use for the evaluation. Defaults to `None`. If not provided, we will use the `OPENAI_API_KEY` environment variable.
+</ParamField>
+<ParamField path="answer_claims_prompt" type="str" optional>
+    The prompt to extract the claims from the provided answer. Defaults to `GROUNDEDNESS_ANSWER_CLAIMS_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.
+</ParamField>
+<ParamField path="claims_inference_prompt" type="str" optional>
+    The prompt to get verdicts on the claims from the answer from the given context. Defaults to `GROUNDEDNESS_CLAIMS_INFERENCE_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.
+</ParamField>
+
+## Custom <a id="custom_metric"></a>
+
+You can also create your own evaluation metric by extending the `BaseMetric` class. You can find the source code for the existing metrics at `embedchain.evaluation.metrics` path.
+
+<Note>
+You must provide the `name` of your custom metric in the `__init__` method of your class. This name will be used to identify your metric in the evaluation report.
+</Note>
+
+```python
+from typing import Optional
+
+from embedchain.config.base_config import BaseConfig
+from embedchain.evaluation.metrics import BaseMetric
+from embedchain.utils.eval import EvalData
+
+class MyCustomMetric(BaseMetric):
+    def __init__(self, config: Optional[BaseConfig] = None):
+        super().__init__(name="my_custom_metric")
+
+    def evaluate(self, dataset: list[EvalData]):
+        score = 0.0
+        # write your evaluation logic here
+        return score
+```

docs/components/introduction.mdx

@@ -0,0 +1,12 @@
+---
+title: 🧩 Introduction
+---
+
+## Overview
+
+You can configure following components
+
+* [Data Source](/components/data-sources/overview)
+* [LLM](/components/llms)
+* [Embedding Model](/components/embedding-models)
+* [Vector Database](/components/vector-databases)

docs/components/llms.mdx

@@ -0,0 +1,707 @@
+---
+title: 🤖 Large language models (LLMs)
+---
+
+## Overview
+
+Embedchain comes with built-in support for various popular large language models. We handle the complexity of integrating these models for you, allowing you to easily customize your language model interactions through a user-friendly interface.
+
+<CardGroup cols={4}>
+  <Card title="OpenAI" href="#openai"></Card>
+  <Card title="Google AI" href="#google-ai"></Card>
+  <Card title="Azure OpenAI" href="#azure-openai"></Card>
+  <Card title="Anthropic" href="#anthropic"></Card>
+  <Card title="Cohere" href="#cohere"></Card>
+  <Card title="Together" href="#together"></Card>
+  <Card title="Ollama" href="#ollama"></Card>
+  <Card title="vLLM" href="#vllm"></Card>
+  <Card title="GPT4All" href="#gpt4all"></Card>
+  <Card title="JinaChat" href="#jinachat"></Card>
+  <Card title="Hugging Face" href="#hugging-face"></Card>
+  <Card title="Llama2" href="#llama2"></Card>
+  <Card title="Vertex AI" href="#vertex-ai"></Card>
+  <Card title="Mistral AI" href="#mistral-ai"></Card>
+  <Card title="AWS Bedrock" href="#aws-bedrock"></Card>
+</CardGroup>
+
+## OpenAI
+
+To use OpenAI LLM models, you have to set the `OPENAI_API_KEY` environment variable. You can obtain the OpenAI API key from the [OpenAI Platform](https://platform.openai.com/account/api-keys).
+
+Once you have obtained the key, you can use it like this:
+
+```python
+import os
+from embedchain import App
+
+os.environ['OPENAI_API_KEY'] = 'xxx'
+
+app = App()
+app.add("https://en.wikipedia.org/wiki/OpenAI")
+app.query("What is OpenAI?")
+```
+
+If you are looking to configure the different parameters of the LLM, you can do so by loading the app using a [yaml config](https://github.com/embedchain/embedchain/blob/main/configs/chroma.yaml) file.
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ['OPENAI_API_KEY'] = 'xxx'
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+```
+</CodeGroup>
+
+### Function Calling
+To enable [function calling](https://platform.openai.com/docs/guides/function-calling) in your application using embedchain and OpenAI, you need to pass functions into `OpenAILlm` class as an array of functions. Here are several ways in which you can achieve that:
+
+Examples:
+<Accordion title="Using Pydantic Models">
+  ```python
+import os
+from embedchain import App
+from embedchain.llm.openai import OpenAILlm
+import requests
+from pydantic import BaseModel, Field, ValidationError, field_validator
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+class QA(BaseModel):
+    """
+    A question and answer pair.
+    """
+
+    question: str = Field(
+        ..., description="The question.", example="What is a mountain?"
+    )
+    answer: str = Field(
+        ..., description="The answer.", example="A mountain is a hill."
+    )
+    person_who_is_asking: str = Field(
+        ..., description="The person who is asking the question.", example="John"
+    )
+
+    @field_validator("question")
+    def question_must_end_with_a_question_mark(cls, v):
+        """
+        Validate that the question ends with a question mark.
+        """
+        if not v.endswith("?"):
+            raise ValueError("question must end with a question mark")
+        return v
+
+    @field_validator("answer")
+    def answer_must_end_with_a_period(cls, v):
+        """
+        Validate that the answer ends with a period.
+        """
+        if not v.endswith("."):
+            raise ValueError("answer must end with a period")
+        return v
+
+llm = OpenAILlm(config=None,functions=[QA])
+app = App(llm=llm)
+
+result = app.query("Hey I am Sid. What is a mountain? A mountain is a hill.")
+
+print(result)
+  ```
+  </Accordion>
+  
+  <Accordion title="Using OpenAI JSON schema">
+```python
+import os
+from embedchain import App
+from embedchain.llm.openai import OpenAILlm
+import requests
+from pydantic import BaseModel, Field, ValidationError, field_validator
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+json_schema = {
+    "name": "get_qa",
+    "description": "A question and answer pair and the user who is asking the question.",
+    "parameters": {
+        "type": "object",
+        "properties": {
+            "question": {"type": "string", "description": "The question."},
+            "answer": {"type": "string", "description": "The answer."},
+            "person_who_is_asking": {
+                "type": "string",
+                "description": "The person who is asking the question.",
+            }
+        },
+        "required": ["question", "answer", "person_who_is_asking"],
+    },
+}
+
+llm = OpenAILlm(config=None,functions=[json_schema])
+app = App(llm=llm)
+
+result = app.query("Hey I am Sid. What is a mountain? A mountain is a hill.")
+
+print(result)
+  ```
+  </Accordion>
+  <Accordion title="Using actual python functions">
+  ```python
+import os
+from embedchain import App
+from embedchain.llm.openai import OpenAILlm
+import requests
+from pydantic import BaseModel, Field, ValidationError, field_validator
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+def find_info_of_pokemon(pokemon: str):
+    """
+    Find the information of the given pokemon.
+    Args:
+        pokemon: The pokemon.
+    """
+    req = requests.get(f"https://pokeapi.co/api/v2/pokemon/{pokemon}")
+    if req.status_code == 404:
+        raise ValueError("pokemon not found")
+    return req.json()
+
+llm = OpenAILlm(config=None,functions=[find_info_of_pokemon])
+app = App(llm=llm)
+
+result = app.query("Tell me more about the pokemon pikachu.")
+
+print(result)
+```
+</Accordion>
+
+## Google AI
+
+To use Google AI model, you have to set the `GOOGLE_API_KEY` environment variable. You can obtain the Google API key from the [Google Maker Suite](https://makersuite.google.com/app/apikey)
+
+<CodeGroup>
+```python main.py
+import os
+from embedchain import App
+
+os.environ["GOOGLE_API_KEY"] = "xxx"
+
+app = App.from_config(config_path="config.yaml")
+
+app.add("https://www.forbes.com/profile/elon-musk")
+
+response = app.query("What is the net worth of Elon Musk?")
+if app.llm.config.stream: # if stream is enabled, response is a generator
+    for chunk in response:
+        print(chunk)
+else:
+    print(response)
+```
+
+```yaml config.yaml
+llm:
+  provider: google
+  config:
+    model: gemini-pro
+    max_tokens: 1000
+    temperature: 0.5
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: google
+  config:
+    model: 'models/embedding-001'
+    task_type: "retrieval_document"
+    title: "Embeddings for Embedchain"
+```
+</CodeGroup>
+
+## Azure OpenAI
+
+To use Azure OpenAI model, you have to set some of the azure openai related environment variables as given in the code block below:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["OPENAI_API_TYPE"] = "azure"
+os.environ["OPENAI_API_BASE"] = "https://xxx.openai.azure.com/"
+os.environ["OPENAI_API_KEY"] = "xxx"
+os.environ["OPENAI_API_VERSION"] = "xxx"
+
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: azure_openai
+  config:
+    model: gpt-3.5-turbo
+    deployment_name: your_llm_deployment_name
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: azure_openai
+  config:
+    model: text-embedding-ada-002
+    deployment_name: you_embedding_model_deployment_name
+```
+</CodeGroup>
+
+You can find the list of models and deployment name on the [Azure OpenAI Platform](https://oai.azure.com/portal).
+
+## Anthropic
+
+To use anthropic's model, please set the `ANTHROPIC_API_KEY` which you find on their [Account Settings Page](https://console.anthropic.com/account/keys).
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["ANTHROPIC_API_KEY"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: anthropic
+  config:
+    model: 'claude-instant-1'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+```
+
+</CodeGroup>
+
+## Cohere
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[cohere]'
+```
+
+Set the `COHERE_API_KEY` as environment variable which you can find on their [Account settings page](https://dashboard.cohere.com/api-keys).
+
+Once you have the API key, you are all set to use it with Embedchain.
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["COHERE_API_KEY"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: cohere
+  config:
+    model: large
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+```
+
+</CodeGroup>
+
+## Together
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[together]'
+```
+
+Set the `TOGETHER_API_KEY` as environment variable which you can find on their [Account settings page](https://api.together.xyz/settings/api-keys).
+
+Once you have the API key, you are all set to use it with Embedchain.
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["TOGETHER_API_KEY"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: together
+  config:
+    model: togethercomputer/RedPajama-INCITE-7B-Base
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+```
+
+</CodeGroup>
+
+## Ollama
+
+Setup Ollama using https://github.com/jmorganca/ollama
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: ollama
+  config:
+    model: 'llama2'
+    temperature: 0.5
+    top_p: 1
+    stream: true
+```
+
+</CodeGroup>
+
+
+## vLLM
+
+Setup vLLM by following instructions given in [their docs](https://docs.vllm.ai/en/latest/getting_started/installation.html).
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: vllm
+  config:
+    model: 'meta-llama/Llama-2-70b-hf'
+    temperature: 0.5
+    top_p: 1
+    top_k: 10
+    stream: true
+    trust_remote_code: true
+```
+
+</CodeGroup>
+
+## GPT4ALL
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[opensource]'
+```
+
+GPT4all is a free-to-use, locally running, privacy-aware chatbot. No GPU or internet required. You can use this with Embedchain using the following code:
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: gpt4all
+```
+</CodeGroup>
+
+
+## JinaChat
+
+First, set `JINACHAT_API_KEY` in environment variable which you can obtain from [their platform](https://chat.jina.ai/api).
+
+Once you have the key, load the app using the config yaml file:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["JINACHAT_API_KEY"] = "xxx"
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: jina
+  config:
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+```
+</CodeGroup>
+
+
+## Hugging Face
+
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[huggingface-hub]'
+```
+
+First, set `HUGGINGFACE_ACCESS_TOKEN` in environment variable which you can obtain from [their platform](https://huggingface.co/settings/tokens).
+
+Once you have the token, load the app using the config yaml file:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: huggingface
+  config:
+    model: 'google/flan-t5-xxl'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false
+```
+</CodeGroup>
+
+### Custom Endpoints
+
+
+You can also use [Hugging Face Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index#-inference-endpoints) to access custom endpoints. First, set the `HUGGINGFACE_ACCESS_TOKEN` as above.
+
+Then, load the app using the config yaml file:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: huggingface
+  config:
+    endpoint: https://api-inference.huggingface.co/models/gpt2 # replace with your personal endpoint
+```
+</CodeGroup>
+
+If your endpoint requires additional parameters, you can pass them in the `model_kwargs` field:
+
+```
+llm:
+  provider: huggingface
+  config:
+    endpoint: <YOUR_ENDPOINT_URL_HERE>
+    model_kwargs:
+      max_new_tokens: 100
+      temperature: 0.5
+```
+
+Currently only supports `text-generation` and `text2text-generation` for now [[ref](https://api.python.langchain.com/en/latest/llms/langchain_community.llms.huggingface_endpoint.HuggingFaceEndpoint.html?highlight=huggingfaceendpoint#)].
+
+See langchain's [hugging face endpoint](https://python.langchain.com/docs/integrations/chat/huggingface#huggingfaceendpoint) for more information. 
+
+## Llama2
+
+Llama2 is integrated through [Replicate](https://replicate.com/).  Set `REPLICATE_API_TOKEN` in environment variable which you can obtain from [their platform](https://replicate.com/account/api-tokens).
+
+Once you have the token, load the app using the config yaml file:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["REPLICATE_API_TOKEN"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: llama2
+  config:
+    model: 'a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false
+```
+</CodeGroup>
+
+## Vertex AI
+
+Setup Google Cloud Platform application credentials by following the instruction on [GCP](https://cloud.google.com/docs/authentication/external/set-up-adc). Once setup is done, use the following code to create an app using VertexAI as provider:
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: vertexai
+  config:
+    model: 'chat-bison'
+    temperature: 0.5
+    top_p: 0.5
+```
+</CodeGroup>
+
+
+## Mistral AI
+
+Obtain the Mistral AI api key from their [console](https://console.mistral.ai/).
+
+<CodeGroup>
+ 
+ ```python main.py
+os.environ["MISTRAL_API_KEY"] = "xxx"
+
+app = App.from_config(config_path="config.yaml")
+
+app.add("https://www.forbes.com/profile/elon-musk")
+
+response = app.query("what is the net worth of Elon Musk?")
+# As of January 16, 2024, Elon Musk's net worth is $225.4 billion.
+
+response = app.chat("which companies does elon own?")
+# Elon Musk owns Tesla, SpaceX, Boring Company, Twitter, and X.
+
+response = app.chat("what question did I ask you already?")
+# You have asked me several times already which companies Elon Musk owns, specifically Tesla, SpaceX, Boring Company, Twitter, and X.
+```
+  
+```yaml config.yaml
+llm:
+  provider: mistralai
+  config:
+    model: mistral-tiny
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+embedder:
+  provider: mistralai
+  config:
+    model: mistral-embed
+```
+</CodeGroup>
+
+
+## AWS Bedrock
+
+### Setup
+- Before using the AWS Bedrock LLM, make sure you have the appropriate model access from [Bedrock Console](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/modelaccess).
+- You will also need to authenticate the `boto3` client by using a method in the [AWS documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials)
+- You can optionally export an `AWS_REGION`
+
+
+### Usage
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["AWS_ACCESS_KEY_ID"] = "xxx"
+os.environ["AWS_SECRET_ACCESS_KEY"] = "xxx"
+os.environ["AWS_REGION"] = "us-west-2"
+
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: aws_bedrock
+  config:
+    model: amazon.titan-text-express-v1
+    # check notes below for model_kwargs
+    model_kwargs:
+      temperature: 0.5
+      topP: 1
+      maxTokenCount: 1000
+```
+</CodeGroup>
+
+<br />
+<Note>
+  The model arguments are different for each providers. Please refer to the [AWS Bedrock Documentation](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/providers) to find the appropriate arguments for your model.
+</Note>
+
+<br/ >
+<Snippet file="missing-llm-tip.mdx" />

docs/components/vector-databases.mdx

@@ -0,0 +1,258 @@
+---
+title: 🗄️ Vector databases
+---
+
+## Overview
+
+Utilizing a vector database alongside Embedchain is a seamless process. All you need to do is configure it within the YAML configuration file. We've provided examples for each supported database below:
+
+<CardGroup cols={4}>
+  <Card title="ChromaDB" href="#chromadb"></Card>
+  <Card title="Elasticsearch" href="#elasticsearch"></Card>
+  <Card title="OpenSearch" href="#opensearch"></Card>
+  <Card title="Zilliz" href="#zilliz"></Card>
+  <Card title="LanceDB" href="#lancedb"></Card>
+  <Card title="Pinecone" href="#pinecone"></Card>
+  <Card title="Qdrant" href="#qdrant"></Card>
+  <Card title="Weaviate" href="#weaviate"></Card>
+</CardGroup>
+
+## ChromaDB
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load chroma configuration from yaml file
+app = App.from_config(config_path="config1.yaml")
+```
+
+```yaml config1.yaml
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-collection'
+    dir: db
+    allow_reset: true
+```
+
+```yaml config2.yaml
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-collection'
+    host: localhost
+    port: 5200
+    allow_reset: true
+```
+
+</CodeGroup>
+
+
+## Elasticsearch
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[elasticsearch]'
+```
+
+<Note>
+You can configure the Elasticsearch connection by providing either `es_url` or `cloud_id`. If you are using the Elasticsearch Service on Elastic Cloud, you can find the `cloud_id` on the [Elastic Cloud dashboard](https://cloud.elastic.co/deployments).
+</Note>
+
+You can authorize the connection to Elasticsearch by providing either `basic_auth`, `api_key`, or `bearer_auth`.
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load elasticsearch configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: elasticsearch
+  config:
+    collection_name: 'es-index'
+    cloud_id: 'deployment-name:xxxx'
+    basic_auth:
+      - elastic
+      - <your_password>
+    verify_certs: false
+```
+</CodeGroup>
+
+## OpenSearch
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[opensearch]'
+```
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load opensearch configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: opensearch
+  config:
+    collection_name: 'my-app'
+    opensearch_url: 'https://localhost:9200'
+    http_auth:
+      - admin
+      - admin
+    vector_dimension: 1536
+    use_ssl: false
+    verify_certs: false
+```
+
+</CodeGroup>
+
+## Zilliz
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[milvus]'
+```
+
+Set the Zilliz environment variables `ZILLIZ_CLOUD_URI` and `ZILLIZ_CLOUD_TOKEN` which you can find it on their [cloud platform](https://cloud.zilliz.com/).
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ['ZILLIZ_CLOUD_URI'] = 'https://xxx.zillizcloud.com'
+os.environ['ZILLIZ_CLOUD_TOKEN'] = 'xxx'
+
+# load zilliz configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: zilliz
+  config:
+    collection_name: 'zilliz_app'
+    uri: https://xxxx.api.gcp-region.zillizcloud.com
+    token: xxx
+    vector_dim: 1536
+    metric_type: L2
+```
+
+</CodeGroup>
+
+## LanceDB
+
+_Coming soon_
+
+## Pinecone
+
+Install pinecone related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[pinecone]'
+```
+
+In order to use Pinecone as vector database, set the environment variable `PINECONE_API_KEY` which you can find on [Pinecone dashboard](https://app.pinecone.io/).
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load pinecone configuration from yaml file
+app = App.from_config(config_path="pod_config.yaml")
+# or
+app = App.from_config(config_path="serverless_config.yaml")
+```
+
+```yaml pod_config.yaml
+vectordb:
+  provider: pinecone
+  config:
+    metric: cosine
+    vector_dimension: 1536
+    index_name: my-pinecone-index
+    pod_config:
+      environment: gcp-starter
+      metadata_config:
+        indexed:
+          - "url"
+          - "hash"
+```
+
+```yaml serverless_config.yaml
+vectordb:
+  provider: pinecone
+  config:
+    metric: cosine
+    vector_dimension: 1536
+    index_name: my-pinecone-index
+    serverless_config:
+      cloud: aws
+      region: us-west-2
+```
+
+</CodeGroup>
+
+<br />
+<Note>
+You can find more information about Pinecone configuration [here](https://docs.pinecone.io/docs/manage-indexes#create-a-pod-based-index).
+You can also optionally provide `index_name` as a config param in yaml file to specify the index name. If not provided, the index name will be `{collection_name}-{vector_dimension}`.
+</Note>
+
+## Qdrant
+
+In order to use Qdrant as a vector database, set the environment variables `QDRANT_URL` and `QDRANT_API_KEY` which you can find on [Qdrant Dashboard](https://cloud.qdrant.io/).
+
+<CodeGroup>
+```python main.py
+from embedchain import App
+
+# load qdrant configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: qdrant
+  config:
+    collection_name: my_qdrant_index
+```
+</CodeGroup>
+
+## Weaviate
+
+In order to use Weaviate as a vector database, set the environment variables `WEAVIATE_ENDPOINT` and `WEAVIATE_API_KEY` which you can find on [Weaviate dashboard](https://console.weaviate.cloud/dashboard).
+
+<CodeGroup>
+```python main.py
+from embedchain import App
+
+# load weaviate configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: weaviate
+  config:
+    collection_name: my_weaviate_index
+```
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />

docs/contribution/dev.mdx

@@ -22,27 +22,12 @@ make lint format
 
 5. **Create a pull request**: When you are ready to contribute your changes, submit a pull request to the EmbedChain repository. Provide a clear and descriptive title for your pull request, along with a detailed description of the changes you have made.
 
-# Tech Stack
-
-embedchain is built on the following stack:
-
-- [Langchain](https://github.com/hwchase17/langchain) as an LLM framework to load, chunk and index data
-- [OpenAI's Ada embedding model](https://platform.openai.com/docs/guides/embeddings) to create embeddings
-- [OpenAI's ChatGPT API](https://platform.openai.com/docs/guides/gpt/chat-completions-api) as LLM to get answers given the context
-- [Chroma](https://github.com/chroma-core/chroma) as the vector database to store embeddings
-- [gpt4all](https://github.com/nomic-ai/gpt4all) as an open source LLM
-- [sentence-transformers](https://huggingface.co/sentence-transformers) as open source embedding model
-
 ## Team
 
-### Author
+### Authors
 
 - Taranjeet Singh ([@taranjeetio](https://twitter.com/taranjeetio))
-
-### Maintainer
-
-- Deshraj Yadav ([@deshrajdry](https://twitter.com/taranjeetio))
-- [cachho](https://github.com/cachho)
+- Deshraj Yadav ([@deshrajdry](https://twitter.com/deshrajdry))
 
 ### Citation
 
@@ -50,8 +35,8 @@ If you utilize this repository, please consider citing it with:
 
 ```
 @misc{embedchain,
-  author = {Taranjeet Singh},
-  title = {Embechain: Framework to easily create LLM powered bots over any dataset},
+  author = {Taranjeet Singh, Deshraj Yadav},
+  title = {Embechain: The Open Source RAG Framework},
   year = {2023},
   publisher = {GitHub},
   journal = {GitHub repository},

docs/contribution/guidelines.mdx

@@ -0,0 +1,4 @@
+---
+title: '📋 Guidelines'
+url: https://github.com/embedchain/embedchain/blob/main/CONTRIBUTING.md
+---

docs/contribution/javascript.mdx

@@ -0,0 +1,4 @@
+---
+title: ' 🟨 Javascript'
+url: https://github.com/embedchain/embedchain/tree/main/embedchain-js
+---

docs/contribution/python.mdx

@@ -0,0 +1,4 @@
+---
+title: '🐍 Python'
+url: https://github.com/embedchain/embedchain
+---