Migrate from template to prompt arg while keeping backward compatibility (#1066)

Co-authored-by: Deven Patel <deven298@yahoo.com>

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -5,7 +5,7 @@ body:
 - type: markdown
   attributes:
     value: >
-      #### Before submitting a bug, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/gventuri/pandas-ai/issues?q=is%3Aissue+sort%3Acreated-desc+).
+      #### Before submitting a bug, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/embedchain/embedchain/issues?q=is%3Aissue+sort%3Acreated-desc+).
 - type: textarea
   attributes:
     label: 🐛 Describe the bug

Makefile

@@ -37,6 +37,7 @@ clean:
 lint:
 	poetry run ruff .
 
+# for example: make test file=tests/test_factory.py
 test:
 	poetry run pytest $(file)
 

README.md

@@ -10,6 +10,9 @@
   <a href="https://pypi.org/project/embedchain/">
     <img src="https://img.shields.io/pypi/v/embedchain" alt="PyPI">
   </a>
+  <a href="https://pepy.tech/project/embedchain">
+    <img src="https://static.pepy.tech/badge/embedchain" alt="Downloads">
+  </a>
   <a href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw">
     <img src="https://img.shields.io/badge/slack-embedchain-brightgreen.svg?logo=slack" alt="Slack">
   </a>
@@ -19,9 +22,6 @@
   <a href="https://twitter.com/embedchain">
     <img src="https://img.shields.io/twitter/follow/embedchain" alt="Twitter">
   </a>
-  <a href="https://embedchain.substack.com/">
-    <img src="https://img.shields.io/badge/Substack-%23006f5c.svg?logo=substack" alt="Substack">
-  </a>
   <a href="https://colab.research.google.com/drive/138lMWhENGeEu7Q1-6lNbNTHGLZXBBz_B?usp=sharing">
     <img src="https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667" alt="Open in Colab">
   </a>
@@ -32,26 +32,27 @@
 
 <hr />
 
+
+> ### Checkout our latest [Sadhguru AI app](https://sadhguru-ai.streamlit.app/) built using Embedchain.
+
 ## What is Embedchain?
-Embedchain is a Data Platform for Large Language Models (LLMs). Seamlessly load, index, retrieve, and sync unstructured data to build dynamic, LLM-powered applications. Check out [embedchain-js](https://github.com/embedchain/embedchain/tree/main/embedchain-js) for a JavaScript implementation.
+
+Embedchain is an Open Source RAG Framework that makes it easy to create and deploy AI apps. At its core, Embedchain follows the design principle of being *"Conventional but Configurable"* to serve both software engineers and machine learning engineers.
+
+Embedchain streamlines the creation of RAG applications, offering a seamless process for managing various types of unstructured data. It efficiently segments data into manageable chunks, generates relevant embeddings, and stores them in a vector database for optimized retrieval. With a suite of diverse APIs, it enables users to extract contextual information, find precise answers, or engage in interactive chat conversations, all tailored to their own data.
 
 ## 🔧 Quick install
 
 ### Python API
 ```bash
-pip install --upgrade embedchain
+pip install embedchain
 ```
 
-### REST API
-You can also run Embedchain as a REST API server using the following command:
+## ✨ Live demo
 
-```bash
-docker run --name embedchain -p 8080:8080 embedchain/rest-api:latest
-```
+Checkout the [Chat with PDF](https://embedchain.ai/demo/chat-pdf) live demo we created using Embedchain. You can find the source code [here](https://github.com/embedchain/embedchain/tree/main/examples/chat-pdf).
 
-Then, navigate to http://0.0.0.0:8080/docs to interact with the API.
-
-## 🔍 Usage and Demo
+## 🔍 Usage
 
 <!-- Demo GIF or Image -->
 <p align="center">
@@ -71,22 +72,10 @@ elon_bot = App()
 # Embed online resources
 elon_bot.add("https://en.wikipedia.org/wiki/Elon_Musk")
 elon_bot.add("https://www.forbes.com/profile/elon-musk")
-elon_bot.add("https://www.youtube.com/watch?v=RcYjXbSJBN8")
 
 # Query the bot
 elon_bot.query("How many companies does Elon Musk run and name those?")
 # Answer: Elon Musk currently runs several companies. As of my knowledge, he is the CEO and lead designer of SpaceX, the CEO and product architect of Tesla, Inc., the CEO and founder of Neuralink, and the CEO and founder of The Boring Company. However, please note that this information may change over time, so it's always good to verify the latest updates.
-
-# (Optional): Deploy app to Embedchain Platform
-app.deploy()
-# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
-# ec-xxxxxx
-
-# 🛠️ Creating pipeline on the platform...
-# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
-
-# 🛠️ Adding data to your pipeline...
-# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
 ```
 
 You can also try it in your browser with Google Colab:
@@ -96,18 +85,18 @@ You can also try it in your browser with Google Colab:
 ## 📖 Documentation
 Comprehensive guides and API documentation are available to help you get the most out of Embedchain:
 
-- [Getting Started](https://docs.embedchain.ai/get-started/quickstart)
 - [Introduction](https://docs.embedchain.ai/get-started/introduction#what-is-embedchain)
-- [Examples](https://docs.embedchain.ai/get-started/examples)
-- [Supported data types](https://docs.embedchain.ai/data-sources/)
+- [Getting Started](https://docs.embedchain.ai/get-started/quickstart)
+- [Examples](https://docs.embedchain.ai/examples)
+- [Supported data types](https://docs.embedchain.ai/components/data-sources/overview)
 
 ## 🔗 Join the Community
 
-Connect with fellow developers and users by joining our [Slack Workspace](https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw). Dive into discussions, ask questions, and share your experiences.
+Connect with fellow developers and users by joining our [Slack Workspace](https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw) or [Discord Community](https://discord.gg/CUU9FPhRNt). Dive into discussions, ask questions, and share your experiences.
 
 ## 🤝 Schedule a 1-on-1 Session
 
-Book a [1-on-1 Session](https://cal.com/taranjeetio/ec) with Taranjeet, the founder, to discuss any issues, provide feedback, or explore how we can improve Embedchain for you.
+Book a [1-on-1 Session](https://cal.com/taranjeetio/ec) with the founders, to discuss any issues, provide feedback, or explore how we can improve Embedchain for you.
 
 ## 🌐 Contributing
 
@@ -120,9 +109,9 @@ For more reference, please go through [Development Guide](https://docs.embedchai
   <img src="https://contrib.rocks/image?repo=embedchain/embedchain" />
 </a>
 
-## Telemetry
+## Anonymous Telemetry
 
-We collect anonymous usage metrics to enhance our package's quality and user experience. This includes data like feature usage frequency and system info, but never personal details. The data helps us prioritize improvements and ensure compatibility. If you wish to opt-out, set the `app.config.collect_metrics = False` in the code. We prioritize data security and don't share this data externally.
+We collect anonymous usage metrics to enhance our package's quality and user experience. This includes data like feature usage frequency and system info, but never personal details. The data helps us prioritize improvements and ensure compatibility. If you wish to opt-out, set the environment variable `EC_TELEMETRY=false`. We prioritize data security and don't share this data externally.
 
 ## Citation
 

configs/full-stack.yaml

@@ -15,7 +15,7 @@ llm:
     max_tokens: 1000
     top_p: 1
     stream: false
-    template: |
+    prompt: |
       Use the following pieces of context to answer the query at the end.
       If you don't know the answer, just say that you don't know, don't try to make up an answer.
 

configs/google.yaml

@@ -0,0 +1,13 @@
+llm:
+  provider: google
+  config:
+    model: gemini-pro
+    max_tokens: 1000
+    temperature: 0.9
+    top_p: 1.0
+    stream: false
+
+embedder:
+  provider: google
+  config:
+    model: models/embedding-001

configs/ollama.yaml

@@ -0,0 +1,12 @@
+llm:
+  provider: ollama
+  config:
+    model: 'llama2'
+    temperature: 0.5
+    top_p: 1
+    stream: true
+
+embedder:
+  provider: huggingface
+  config:
+    model: 'BAAI/bge-small-en-v1.5'

configs/together.yaml

@@ -0,0 +1,6 @@
+llm:
+  provider: together
+  config:
+    model: mistralai/Mixtral-8x7B-Instruct-v0.1
+    temperature: 0.5
+    max_tokens: 1000

docs/_snippets/get-help.mdx

@@ -1,11 +1,11 @@
 <CardGroup cols={3}>
+  <Card title="Talk to founders" icon="calendar" href="https://cal.com/taranjeetio/ec">
+  Schedule a call
+  </Card>
   <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
     Join our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">
     Join our discord community
   </Card>
-  <Card title="Schedule a call" icon="calendar" href="https://cal.com/taranjeetio/ec">
-  Schedule a call with Embedchain founder
-  </Card>
 </CardGroup>

docs/_snippets/missing-data-source-tip.mdx

@@ -1,6 +1,9 @@
 <p>If you can't find the specific data source, please feel free to request through one of the following channels and help us prioritize.</p>
 
 <CardGroup cols={2}>
+  <Card title="Google Form" icon="file" href="https://forms.gle/NDRCKsRpUHsz2Wcm8" color="#7387d0">
+    Fill out this form
+  </Card>
   <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
     Let us know on our slack community
   </Card>

docs/api-reference/advanced/configuration.mdx

@@ -1,14 +1,14 @@
 ---
-title: '⚙️ Custom configurations'
+title: 'Custom configurations'
 ---
 
-Embedchain is made to work out of the box. However, for advanced users we're also offering configuration options. All of these configuration options are optional and have sane defaults.
+Embedchain offers several configuration options for your LLM, vector database, and embedding model. All of these configuration options are optional and have sane defaults.
 
 You can configure different components of your app (`llm`, `embedding model`, or `vector database`) through a simple yaml configuration that Embedchain offers. Here is a generic full-stack example of the yaml config:
 
 
 <Tip>
-Embedchain applications are configurable using YAML file, JSON file or by directly passing the config dictionary.
+Embedchain applications are configurable using YAML file, JSON file or by directly passing the config dictionary. Checkout the [docs here](/api-reference/pipeline/overview#usage) on how to use other formats.
 </Tip>
 
 <CodeGroup>
@@ -25,7 +25,8 @@ llm:
     max_tokens: 1000
     top_p: 1
     stream: false
-    template: |
+    api_key: sk-xxx
+    prompt: |
       Use the following pieces of context to answer the query at the end.
       If you don't know the answer, just say that you don't know, don't try to make up an answer.
 
@@ -48,11 +49,13 @@ embedder:
   provider: openai
   config:
     model: 'text-embedding-ada-002'
+    api_key: sk-xxx
 
 chunker:
   chunk_size: 2000
   chunk_overlap: 100
   length_function: 'len'
+  min_chunk_size: 0
 ```
 
 ```json config.json
@@ -70,8 +73,9 @@ chunker:
       "max_tokens": 1000,
       "top_p": 1,
       "stream": false,
-      "template": "Use the following pieces of context to answer the query at the end.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n$context\n\nQuery: $query\n\nHelpful Answer:",
-      "system_prompt": "Act as William Shakespeare. Answer the following questions in the style of William Shakespeare."
+      "prompt": "Use the following pieces of context to answer the query at the end.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n$context\n\nQuery: $query\n\nHelpful Answer:",
+      "system_prompt": "Act as William Shakespeare. Answer the following questions in the style of William Shakespeare.",
+      "api_key": "sk-xxx"
     }
   },
   "vectordb": {
@@ -85,13 +89,15 @@ chunker:
   "embedder": {
     "provider": "openai",
     "config": {
-      "model": "text-embedding-ada-002"
+      "model": "text-embedding-ada-002",
+      "api_key": "sk-xxx"
     }
   },
   "chunker": {
     "chunk_size": 2000,
     "chunk_overlap": 100,
-    "length_function": "len"
+    "length_function": "len",
+    "min_chunk_size": 0
   }
 }
 ```
@@ -111,14 +117,15 @@ config = {
             'max_tokens': 1000,
             'top_p': 1,
             'stream': False,
-            'template': (
+            'prompt': (
                 "Use the following pieces of context to answer the query at the end.\n"
                 "If you don't know the answer, just say that you don't know, don't try to make up an answer.\n"
                 "$context\n\nQuery: $query\n\nHelpful Answer:"
             ),
             'system_prompt': (
                 "Act as William Shakespeare. Answer the following questions in the style of William Shakespeare."
-            )
+            ),
+            'api_key': 'sk-xxx'
         }
     },
     'vectordb': {
@@ -132,13 +139,15 @@ config = {
     'embedder': {
         'provider': 'openai',
         'config': {
-            'model': 'text-embedding-ada-002'
+            'model': 'text-embedding-ada-002',
+            'api_key': 'sk-xxx'
         }
     },
     'chunker': {
         'chunk_size': 2000,
         'chunk_overlap': 100,
-        'length_function': 'len'
+        'length_function': 'len',
+        'min_chunk_size': 0
     }
 }
 ```
@@ -161,10 +170,11 @@ Alright, let's dive into what each key means in the yaml config above:
         - `max_tokens` (Integer): Controls how many tokens are used in the response.
         - `top_p` (Float): Controls the diversity of word selection. A higher value (closer to 1) makes word selection more diverse.
         - `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
-        - `template` (String): A custom template for the prompt that the model uses to generate responses.
+        - `prompt` (String): A prompt for the model to follow when generating responses, requires $context and $query variables.
         - `system_prompt` (String): A system prompt for the model to follow when generating responses, in this case, it's set to the style of William Shakespeare.
         -  `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
         - `number_documents` (Integer): Number of documents to pull from the vectordb as context, defaults to 1
+        - `api_key` (String): The API key for the language model.
 3. `vectordb` Section:
     - `provider` (String): The provider for the vector database, set to 'chroma'. You can find the full list of vector database providers in [our docs](/components/vector-databases).
     - `config`:
@@ -176,10 +186,13 @@ Alright, let's dive into what each key means in the yaml config above:
     - `provider` (String): The provider for the embedder, set to 'openai'. You can find the full list of embedding model providers in [our docs](/components/embedding-models).
     - `config`:
         - `model` (String): The specific model used for text embedding, 'text-embedding-ada-002'.
+        - `api_key` (String): The API key for the embedding model.
 5. `chunker` Section:
     - `chunk_size` (Integer): The size of each chunk of text that is sent to the language model.
     - `chunk_overlap` (Integer): The amount of overlap between each chunk of text.
     - `length_function` (String): The function used to calculate the length of each chunk of text. In this case, it's set to 'len'. You can also use any function import directly as a string here.
+    - `min_chunk_size` (Integer): The minimum size of each chunk of text that is sent to the language model. Must be less than `chunk_size`, and greater than `chunk_overlap`.
+
 If you have questions about the configuration above, please feel free to reach out to us using one of the following methods:
 
 <Snippet file="get-help.mdx" />

docs/api-reference/pipeline/add.mdx

@@ -0,0 +1,44 @@
+---
+title: '📊 add'
+---
+
+`add()` method is used to load the data sources from different data sources to a RAG pipeline. You can find the signature below:
+
+### Parameters
+
+<ParamField path="source" type="str">
+    The data to embed, can be a URL, local file or raw content, depending on the data type.. You can find the full list of supported data sources [here](/components/data-sources/overview).
+</ParamField>
+<ParamField path="data_type" type="str" optional>
+    Type of data source. It can be automatically detected but user can force what data type to load as.
+</ParamField>
+<ParamField path="metadata" type="dict" optional>
+    Any metadata that you want to store with the data source. Metadata is generally really useful for doing metadata filtering on top of semantic search to yield faster search and better results.
+</ParamField>
+
+## Usage
+
+### Load data from webpage
+
+```python Code example
+from embedchain import Pipeline as App
+
+app = App()
+app.add("https://www.forbes.com/profile/elon-musk")
+# Inserting batches in chromadb: 100%|███████████████| 1/1 [00:00<00:00,  1.19it/s]
+# Successfully saved https://www.forbes.com/profile/elon-musk (DataType.WEB_PAGE). New chunks count: 4
+```
+
+### Load data from sitemap
+
+```python Code example
+from embedchain import Pipeline as App
+
+app = App()
+app.add("https://python.langchain.com/sitemap.xml", data_type="sitemap")
+# Loading pages: 100%|█████████████| 1108/1108 [00:47<00:00, 23.17it/s]
+# Inserting batches in chromadb: 100%|█████████| 111/111 [04:41<00:00,  2.54s/it]
+# Successfully saved https://python.langchain.com/sitemap.xml (DataType.SITEMAP). New chunks count: 11024
+```
+
+You can find complete list of supported data sources [here](/components/data-sources/overview).

docs/api-reference/pipeline/chat.mdx

@@ -0,0 +1,97 @@
+---
+title: '💬 chat'
+---
+
+`chat()` method allows you to chat over your data sources using a user-friendly chat API. You can find the signature below:
+
+### Parameters
+
+<ParamField path="input_query" type="str">
+    Question to ask
+</ParamField>
+<ParamField path="config" type="BaseLlmConfig" optional>
+    Configure different llm settings such as prompt, temprature, number_documents etc.
+</ParamField>
+<ParamField path="dry_run" type="bool" optional>
+    The purpose is to test the prompt structure without actually running LLM inference. Defaults to `False`
+</ParamField>
+<ParamField path="where" type="dict" optional>
+    A dictionary of key-value pairs to filter the chunks from the vector database. Defaults to `None`
+</ParamField>
+<ParamField path="citations" type="bool" optional>
+    Return citations along with the LLM answer. Defaults to `False`
+</ParamField>
+
+### Returns
+
+<ResponseField name="answer" type="str | tuple">
+  If `citations=False`, return a stringified answer to the question asked. <br />
+  If `citations=True`, returns a tuple with answer and citations respectively.
+</ResponseField>
+
+## Usage
+
+### With citations
+
+If you want to get the answer to question and return both answer and citations, use the following code snippet:
+
+```python With Citations
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Get relevant answer for your query
+answer, sources = app.chat("What is the net worth of Elon?", citations=True)
+print(answer)
+# Answer: The net worth of Elon Musk is $221.9 billion.
+
+print(sources)
+# [
+#    (
+#        'Elon Musk PROFILEElon MuskCEO, Tesla$247.1B$2.3B (0.96%)Real Time Net Worthas of 12/7/23 ...',
+#        'https://www.forbes.com/profile/elon-musk',
+#        '4651b266--4aa78839fe97'
+#    ),
+#    (
+#        '74% of the company, which is now called X.Wealth HistoryHOVER TO REVEAL NET WORTH BY YEARForbes ...',
+#        'https://www.forbes.com/profile/elon-musk',
+#        '4651b266--4aa78839fe97'
+#    ),
+#    (
+#        'founded in 2002, is worth nearly $150 billion after a $750 million tender offer in June 2023 ...',
+#        'https://www.forbes.com/profile/elon-musk',
+#        '4651b266--4aa78839fe97'
+#    )
+# ]
+```
+
+<Note>
+When `citations=True`, note that the returned `sources` are a list of tuples where each tuple has three elements (in the following order):
+1. source chunk
+2. link of the source document
+3. document id (used for book keeping purposes)
+</Note>
+
+
+### Without citations
+
+If you just want to return answers and don't want to return citations, you can use the following example:
+
+```python Without Citations
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Chat on your data using `.chat()`
+answer = app.chat("What is the net worth of Elon?")
+print(answer)
+# Answer: The net worth of Elon Musk is $221.9 billion.
+```

docs/api-reference/pipeline/delete.mdx

@@ -0,0 +1,19 @@
+---
+title: 🗑 delete
+---
+
+`delete_chat_history()` method allows you to delete all previous messages in a chat history.
+
+## Usage
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+app.add("https://www.forbes.com/profile/elon-musk")
+
+app.chat("What is the net worth of Elon Musk?")
+
+app.delete_chat_history()
+```

docs/api-reference/pipeline/deploy.mdx

@@ -0,0 +1,31 @@
+---
+title: 🚀 deploy
+---
+
+Using the `deploy()` method, Embedchain allows developers to easily launch their LLM-powered applications on the [Embedchain Platform](https://app.embedchain.ai). This platform facilitates seamless access to your data's context via a free and user-friendly REST API. Once your pipeline is deployed, you can update your data sources at any time.
+
+The `deploy()` method not only deploys your pipeline but also efficiently manages LLMs, vector databases, embedding models, and data syncing, enabling you to focus on querying, chatting, or searching without the hassle of infrastructure management.
+
+## Usage
+
+```python
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Deploy your pipeline to Embedchain Platform
+app.deploy()
+
+# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+# ec-xxxxxx
+
+# 🛠️ Creating pipeline on the platform...
+# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+
+# 🛠️ Adding data to your pipeline...
+# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
+```

docs/api-reference/pipeline/overview.mdx

@@ -0,0 +1,130 @@
+---
+title: "Pipeline"
+---
+
+Create a RAG pipeline object on Embedchain. This is the main entrypoint for a developer to interact with Embedchain APIs. A pipeline configures the llm, vector database, embedding model, and retrieval strategy of your choice.
+
+### Attributes
+
+<ParamField path="local_id" type="str">
+    Pipeline ID
+</ParamField>
+<ParamField path="name" type="str" optional>
+    Name of the pipeline
+</ParamField>
+<ParamField path="config" type="BaseConfig">
+    Configuration of the pipeline
+</ParamField>
+<ParamField path="llm" type="BaseLlm">
+    Configured LLM for the RAG pipeline
+</ParamField>
+<ParamField path="db" type="BaseVectorDB">
+    Configured vector database for the RAG pipeline
+</ParamField>
+<ParamField path="embedding_model" type="BaseEmbedder">
+    Configured embedding model for the RAG pipeline
+</ParamField>
+<ParamField path="chunker" type="ChunkerConfig">
+    Chunker configuration
+</ParamField>
+<ParamField path="client" type="Client" optional>
+    Client object (used to deploy a pipeline to Embedchain platform)
+</ParamField>
+<ParamField path="logger" type="logging.Logger">
+    Logger object
+</ParamField>
+
+## Usage
+
+You can create an embedchain pipeline instance using the following methods:
+
+### Default setting
+
+```python Code Example
+from embedchain import Pipeline as App
+app = App()
+```
+
+
+### Python Dict
+
+```python Code Example
+from embedchain import Pipeline as App
+
+config_dict = {
+  'llm': {
+    'provider': 'gpt4all',
+    'config': {
+      'model': 'orca-mini-3b-gguf2-q4_0.gguf',
+      'temperature': 0.5,
+      'max_tokens': 1000,
+      'top_p': 1,
+      'stream': False
+    }
+  },
+  'embedder': {
+    'provider': 'gpt4all'
+  }
+}
+
+# load llm configuration from config dict
+app = App.from_config(config=config_dict)
+```
+
+### YAML Config
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: gpt4all
+  config:
+    model: 'orca-mini-3b-gguf2-q4_0.gguf'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: gpt4all
+```
+
+</CodeGroup>
+
+### JSON Config
+
+<CodeGroup>
+
+```python main.py
+from embedchain import Pipeline as App
+
+# load llm configuration from config.json file
+app = App.from_config(config_path="config.json")
+```
+
+```json config.json
+{
+  "llm": {
+    "provider": "gpt4all",
+    "config": {
+      "model": "orca-mini-3b-gguf2-q4_0.gguf",
+      "temperature": 0.5,
+      "max_tokens": 1000,
+      "top_p": 1,
+      "stream": false
+    }
+  },
+  "embedder": {
+    "provider": "gpt4all"
+  }
+}
+```
+
+</CodeGroup>

docs/api-reference/pipeline/query.mdx

@@ -0,0 +1,97 @@
+---
+title: '❓ query'
+---
+
+`.query()` method empowers developers to ask questions and receive relevant answers through a user-friendly query API. Function signature is given below:
+
+### Parameters
+
+<ParamField path="input_query" type="str">
+    Question to ask
+</ParamField>
+<ParamField path="config" type="BaseLlmConfig" optional>
+    Configure different llm settings such as prompt, temprature, number_documents etc.
+</ParamField>
+<ParamField path="dry_run" type="bool" optional>
+    The purpose is to test the prompt structure without actually running LLM inference. Defaults to `False`
+</ParamField>
+<ParamField path="where" type="dict" optional>
+    A dictionary of key-value pairs to filter the chunks from the vector database. Defaults to `None`
+</ParamField>
+<ParamField path="citations" type="bool" optional>
+    Return citations along with the LLM answer. Defaults to `False`
+</ParamField>
+
+### Returns
+
+<ResponseField name="answer" type="str | tuple">
+  If `citations=False`, return a stringified answer to the question asked. <br />
+  If `citations=True`, returns a tuple with answer and citations respectively.
+</ResponseField>
+
+## Usage
+
+### With citations
+
+If you want to get the answer to question and return both answer and citations, use the following code snippet:
+
+```python With Citations
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Get relevant answer for your query
+answer, sources = app.query("What is the net worth of Elon?", citations=True)
+print(answer)
+# Answer: The net worth of Elon Musk is $221.9 billion.
+
+print(sources)
+# [
+#    (
+#        'Elon Musk PROFILEElon MuskCEO, Tesla$247.1B$2.3B (0.96%)Real Time Net Worthas of 12/7/23 ...',
+#        'https://www.forbes.com/profile/elon-musk',
+#        '4651b266--4aa78839fe97'
+#    ),
+#    (
+#        '74% of the company, which is now called X.Wealth HistoryHOVER TO REVEAL NET WORTH BY YEARForbes ...',
+#        'https://www.forbes.com/profile/elon-musk',
+#        '4651b266--4aa78839fe97'
+#    ),
+#    (
+#        'founded in 2002, is worth nearly $150 billion after a $750 million tender offer in June 2023 ...',
+#        'https://www.forbes.com/profile/elon-musk',
+#        '4651b266--4aa78839fe97'
+#    )
+# ]
+```
+
+<Note>
+When `citations=True`, note that the returned `sources` are a list of tuples where each tuple has three elements (in the following order):
+1. source chunk
+2. link of the source document
+3. document id (used for book keeping purposes)
+</Note>
+
+### Without citations
+
+If you just want to return answers and don't want to return citations, you can use the following example:
+
+```python Without Citations
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Get relevant answer for your query
+answer = app.query("What is the net worth of Elon?")
+print(answer)
+# Answer: The net worth of Elon Musk is $221.9 billion.
+```
+

docs/api-reference/pipeline/reset.mdx

@@ -0,0 +1,17 @@
+---
+title: 🔄 reset
+---
+
+`reset()` method allows you to wipe the data from your RAG application and start from scratch.
+
+## Usage
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Reset the app
+app.reset()
+```

docs/api-reference/pipeline/search.mdx

@@ -0,0 +1,51 @@
+---
+title: '🔍 search'
+---
+
+`.search()` enables you to uncover the most pertinent context by performing a semantic search across your data sources based on a given query. Refer to the function signature below:
+
+### Parameters
+
+<ParamField path="query" type="str">
+    Question
+</ParamField>
+<ParamField path="num_documents" type="int" optional>
+    Number of relevant documents to fetch. Defaults to `3`
+</ParamField>
+
+### Returns
+
+<ResponseField name="answer" type="dict">
+    Return list of dictionaries that contain the relevant chunk and their source information.
+</ResponseField>
+
+## Usage
+
+Refer to the following example on how to use the search api:
+
+```python Code example
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Get relevant context using semantic search
+context = app.search("What is the net worth of Elon?", num_documents=2)
+print(context)
+# Context:
+# [
+#     {
+#         'context': 'Elon Musk PROFILEElon MuskCEO, Tesla$221.9BReal Time Net Worthas of 10/29/23Reflects change since 5 pm ET of prior trading day. 1 in the world todayPhoto by Martin Schoeller for ForbesAbout Elon MuskElon Musk cofounded six companies, including electric car maker Tesla, rocket producer SpaceX and tunneling startup Boring Company.He owns about 21% of Tesla between stock and options, but has pledged more than half his shares as collateral for personal loans of up to $3.5 billion.SpaceX, founded in',
+#         'source': 'https://www.forbes.com/profile/elon-musk',
+#         'document_id': 'some_document_id'
+#     },
+#     {
+#         'context': 'company, which is now called X.Wealth HistoryHOVER TO REVEAL NET WORTH BY YEARForbes Lists 1Forbes 400 (2023)The Richest Person In Every State (2023) 2Billionaires (2023) 1Innovative Leaders (2019) 25Powerful People (2018) 12Richest In Tech (2017)Global Game Changers (2016)More ListsPersonal StatsAge52Source of WealthTesla, SpaceX, Self MadeSelf-Made Score8Philanthropy Score1ResidenceAustin, TexasCitizenshipUnited StatesMarital StatusSingleChildren11EducationBachelor of Arts/Science, University',
+#         'source': 'https://www.forbes.com/profile/elon-musk',
+#         'document_id': 'some_document_id'
+#     }
+# ]
+```

docs/api-reference/store/ai-assistants.mdx

@@ -0,0 +1,54 @@
+---
+title: 'AI Assistant'
+---
+
+The `AIAssistant` class, an alternative to the OpenAI Assistant API, is designed for those who prefer using large language models (LLMs) other than those provided by OpenAI. It facilitates the creation of AI Assistants with several key benefits:
+
+- **Visibility into Citations**: It offers transparent access to the sources and citations used by the AI, enhancing the understanding and trustworthiness of its responses.
+
+- **Debugging Capabilities**: Users have the ability to delve into and debug the AI's processes, allowing for a deeper understanding and fine-tuning of its performance.
+
+- **Customizable Prompts**: The class provides the flexibility to modify and tailor prompts according to specific needs, enabling more precise and relevant interactions.
+
+- **Chain of Thought Integration**: It supports the incorporation of a 'chain of thought' approach, which helps in breaking down complex queries into simpler, sequential steps, thereby improving the clarity and accuracy of responses.
+
+It is ideal for those who value customization, transparency, and detailed control over their AI Assistant's functionalities.
+
+### Arguments
+
+<ParamField path="name" type="string" optional>
+  Name for your AI assistant
+</ParamField>
+
+<ParamField path="instructions" type="string" optional>
+  How the Assistant and model should behave or respond
+</ParamField>
+
+<ParamField path="assistant_id" type="string" optional>
+  Load existing AI Assistant. If you pass this, you don't have to pass other arguments.
+</ParamField>
+
+<ParamField path="thread_id" type="string" optional>
+  Existing thread id if exists
+</ParamField>
+
+<ParamField path="yaml_path" type="str" Optional>
+    Embedchain pipeline config yaml path to use. This will define the configuration of the AI Assistant (such as configuring the LLM, vector database, and embedding model)
+</ParamField>
+
+<ParamField path="data_sources" type="list" default="[]">
+  Add data sources to your assistant. You can add in the following format: `[{"source": "https://example.com", "data_type": "web_page"}]`
+</ParamField>
+
+<ParamField path="collect_metrics" type="boolean" default="True">
+  Anonymous telemetry (doesn't collect any user information or user's files). Used to improve the Embedchain package utilization. Default is `True`.
+</ParamField>
+
+
+## Usage
+
+For detailed guidance on creating your own AI Assistant, click the link below. It provides step-by-step instructions to help you through the process:
+
+<Card title="Guide to Creating Your AI Assistant" icon="link" href="/examples/opensource-assistant">
+  Learn how to build a customized AI Assistant using the `AIAssistant` class.
+</Card>

docs/api-reference/store/openai-assistant.mdx

@@ -0,0 +1,45 @@
+---
+title: 'OpenAI Assistant'
+---
+
+### Arguments
+
+<ParamField path="name" type="string">
+  Name for your AI assistant
+</ParamField>
+
+<ParamField path="instructions" type="string">
+  how the Assistant and model should behave or respond
+</ParamField>
+
+<ParamField path="assistant_id" type="string">
+  Load existing OpenAI Assistant. If you pass this, you don't have to pass other arguments.
+</ParamField>
+
+<ParamField path="thread_id" type="string">
+  Existing OpenAI thread id if exists
+</ParamField>
+
+<ParamField path="model" type="str" default="gpt-4-1106-preview">
+  OpenAI model to use
+</ParamField>
+
+<ParamField path="tools" type="list">
+  OpenAI tools to use. Default set to `[{"type": "retrieval"}]`
+</ParamField>
+
+<ParamField path="data_sources" type="list" default="[]">
+  Add data sources to your assistant. You can add in the following format: `[{"source": "https://example.com", "data_type": "web_page"}]`
+</ParamField>
+
+<ParamField path="telemetry" type="boolean" default="True">
+  Anonymous telemetry (doesn't collect any user information or user's files). Used to improve the Embedchain package utilization. Default is `True`.
+</ParamField>
+
+## Usage
+
+For detailed guidance on creating your own OpenAI Assistant, click the link below. It provides step-by-step instructions to help you through the process:
+
+<Card title="Guide to Creating Your OpenAI Assistant" icon="link" href="/examples/openai-assistant">
+  Learn how to build an OpenAI Assistant using the `OpenAIAssistant` class.
+</Card>

docs/components/data-sources/beehiiv.mdx

@@ -0,0 +1,16 @@
+---
+title: "🐝 Beehiiv"
+---
+
+To add any Beehiiv data sources to your app, just add the base url as the source and set the data_type to `beehiiv`.
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+# source: just add the base url and set the data_type to 'beehiiv'
+app.add('https://aibreakfast.beehiiv.com', data_type='beehiiv')
+app.query("How much is OpenAI paying developers?")
+# Answer: OpenAI is aggressively recruiting Google's top AI researchers with offers ranging between $5 to $10 million annually, primarily in stock options.
+```

docs/components/data-sources/directory.mdx

@@ -0,0 +1,41 @@
+---
+title: '📁 Directory'
+---
+
+To use an entire directory as data source, just add `data_type` as `directory` and pass in the path of the local directory.
+
+### Without customization
+
+```python
+import os
+from embedchain import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+app.add("./elon-musk", data_type="directory")
+response = app.query("list all files")
+print(response)
+# Answer: Files are elon-musk-1.txt, elon-musk-2.pdf.
+```
+
+### Customization
+
+```python
+import os
+from embedchain import Pipeline as App
+from embedchain.loaders.directory_loader import DirectoryLoader
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+lconfig = {
+    "recursive": True,
+    "extensions": [".txt"]
+}
+loader = DirectoryLoader(config=lconfig)
+app = App()
+app.add("./elon-musk", loader=loader)
+response = app.query("what are all the files related to?")
+print(response)
+
+# Answer: The files are related to Elon Musk.
+```

docs/components/data-sources/docs-site.mdx

@@ -10,5 +10,5 @@ from embedchain import Pipeline as App
 app = App()
 app.add("https://docs.embedchain.ai/", data_type="docs_site")
 app.query("What is Embedchain?")
-# Answer: Embedchain is a platform that utilizes various components, including paid/proprietary ones, to provide what is believed to be the best configuration available. It uses LLM (Language Model) providers such as OpenAI, Anthpropic, Vertex_AI, GPT4ALL, Azure_OpenAI, LLAMA2, JINA, and COHERE. Embedchain allows users to import and utilize these LLM providers for their applications.'
+# Answer: Embedchain is a platform that utilizes various components, including paid/proprietary ones, to provide what is believed to be the best configuration available. It uses LLM (Language Model) providers such as OpenAI, Anthpropic, Vertex_AI, GPT4ALL, Azure_OpenAI, LLAMA2, JINA, Ollama, Together and COHERE. Embedchain allows users to import and utilize these LLM providers for their applications.'
 ```

docs/components/data-sources/overview.mdx

@@ -0,0 +1,38 @@
+---
+title: Overview
+---
+
+Embedchain comes with built-in support for various data sources. We handle the complexity of loading unstructured data from these data sources, allowing you to easily customize your app through a user-friendly interface.
+
+<CardGroup cols={4}>
+  <Card title="📰 PDF file" href="/components/data-sources/pdf-file"></Card>
+  <Card title="📊 CSV file" href="/components/data-sources/csv"></Card>
+  <Card title="📃 JSON file" href="/components/data-sources/json"></Card>
+  <Card title="📽️ Youtube Channel" href="/components/data-sources/youtube-channel"></Card>
+  <Card title="📺 Youtube" href="/components/data-sources/youtube-video"></Card>
+  <Card title="📝 Text" href="/components/data-sources/text"></Card>
+  <Card title="📚 Documentation website" href="/components/data-sources/docs-site"></Card>
+  <Card title="📄 DOCX file" href="/components/data-sources/docx"></Card>
+  <Card title="📝 MDX file" href="/components/data-sources/mdx"></Card>
+  <Card title="📓 Notion" href="/components/data-sources/notion"></Card>
+  <Card title="❓💬 Q&A pair" href="/components/data-sources/qna"></Card>
+  <Card title="🗺️ Sitemap" href="/components/data-sources/sitemap"></Card>
+  <Card title="🌐 Web page" href="/components/data-sources/web-page"></Card>
+  <Card title="🧾 XML file" href="/components/data-sources/xml"></Card>
+  <Card title="🙌 OpenAPI" href="/components/data-sources/openapi"></Card>
+  <Card title="📬 Gmail" href="/components/data-sources/gmail"></Card>
+  <Card title="🐘 Postgres" href="/components/data-sources/postgres"></Card>
+  <Card title="🐬 MySQL" href="/components/data-sources/mysql"></Card>
+  <Card title="🤖 Slack" href="/components/data-sources/slack"></Card>
+  <Card title="🗨️ Discourse" href="/components/data-sources/discourse"></Card>
+  <Card title="💬 Discord" href="/components/data-sources/discord"></Card>
+  <Card title="📝 Github" href="/components/data-sources/github"></Card>
+  <Card title="⚙️ Custom" href="/components/data-sources/custom"></Card>
+  <Card title="📝 Substack" href="/components/data-sources/substack"></Card>
+  <Card title="🐝 Beehiiv" href="/components/data-sources/beehiiv"></Card>
+  <Card title="📁 Directory" href="/components/data-sources/directory"></Card>
+</CardGroup>
+
+<br/ >
+
+<Snippet file="missing-data-source-tip.mdx" />

docs/components/data-sources/slack.mdx

@@ -9,15 +9,33 @@ title: '🤖 Slack'
     - Make sure your slack user token includes [search](https://api.slack.com/scopes/search:read) scope.
 
 ## Example
+
+### Get Started
+
+This will automatically retrieve data from the workspace associated with the user's token.
+
+```python
+import os
+from embedchain import Pipeline as App
+
+os.environ["SLACK_USER_TOKEN"] = "xoxp-xxx"
+app = App()
+
+app.add("in:general", data_type="slack")
+
+result = app.query("what are the messages in general channel?")
+
+print(result)
+```
+
+
+### Customize your SlackLoader
 1. Setup the Slack loader by configuring the Slack Webclient.
 ```Python
 from embedchain.loaders.slack import SlackLoader
 
 os.environ["SLACK_USER_TOKEN"] = "xoxp-*"
 
-loader = SlackLoader()
-
-"""
 config = {
     'base_url': slack_app_url,
     'headers': web_headers,
@@ -25,7 +43,6 @@ config = {
 }
 
 loader = SlackLoader(config)
-"""
 ```
 
 NOTE: you can also pass the `config` with `base_url`, `headers`, `team_id` to setup your SlackLoader.

docs/components/data-sources/substack.mdx

@@ -0,0 +1,16 @@
+---
+title: "📝 Substack"
+---
+
+To add any Substack data sources to your app, just add the main base url as the source and set the data_type to `substack`.
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+# source: for any substack just add the root URL
+app.add('https://www.lennysnewsletter.com', data_type='substack')
+app.query("Who is Brian Chesky?")
+# Answer: Brian Chesky is the co-founder and CEO of Airbnb.
+```

docs/components/data-sources/youtube-channel.mdx

@@ -0,0 +1,20 @@
+---
+title: '📽️ Youtube Channel'
+---
+
+To add all the videos from a youtube channel to your app, use the data_type as `youtube_channel`.
+
+<Note>
+Make sure you have all the required packages installed before using this data type. You can install them by running the following command in your terminal.
+
+```bash
+pip install -u "embedchain[youtube]"
+```
+</Note>
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+app.add("@channel_name", data_type="youtube_channel")
+```

docs/components/data-sources/youtube-video.mdx

@@ -2,8 +2,7 @@
 title: '📺 Youtube'
 ---
 
-
-To add any youtube video to your app, use the data_type (first argument to `.add()` method) as `youtube_video`. Eg:
+To add any youtube video to your app, use the data_type as `youtube_video`. Eg:
 
 ```python
 from embedchain import Pipeline as App

docs/components/embedding-models.mdx

@@ -8,6 +8,7 @@ Embedchain supports several embedding models from the following providers:
 
 <CardGroup cols={4}>
   <Card title="OpenAI" href="#openai"></Card>
+  <Card title="GoogleAI" href="#google-ai"></Card>
   <Card title="Azure OpenAI" href="#azure-openai"></Card>
   <Card title="GPT4All" href="#gpt4all"></Card>
   <Card title="Hugging Face" href="#hugging-face"></Card>
@@ -44,6 +45,34 @@ embedder:
 
 </CodeGroup>
 
+## Google AI
+
+To use Google AI embedding function, you have to set the `GOOGLE_API_KEY` environment variable. You can obtain the Google API key from the [Google Maker Suite](https://makersuite.google.com/app/apikey)
+
+<CodeGroup>
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ["GOOGLE_API_KEY"] = "xxx"
+
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+embedder:
+  provider: google
+  config:
+    model: 'models/embedding-001'
+    task_type: "retrieval_document"
+    title: "Embeddings for Embedchain"
+```
+</CodeGroup>
+<br/>
+<Note>
+For more details regarding the Google AI embedding model, please refer to the [Google AI documentation](https://ai.google.dev/tutorials/python_quickstart#use_embeddings).
+</Note>
+
 ## Azure OpenAI
 
 To use Azure OpenAI embedding model, you have to set some of the azure openai related environment variables as given in the code block below:

docs/components/llms.mdx

@@ -8,9 +8,12 @@ Embedchain comes with built-in support for various popular large language models
 
 <CardGroup cols={4}>
   <Card title="OpenAI" href="#openai"></Card>
+  <Card title="Google AI" href="#google-ai"></Card>
   <Card title="Azure OpenAI" href="#azure-openai"></Card>
   <Card title="Anthropic" href="#anthropic"></Card>
   <Card title="Cohere" href="#cohere"></Card>
+  <Card title="Together" href="#together"></Card>
+  <Card title="Ollama" href="#ollama"></Card>
   <Card title="GPT4All" href="#gpt4all"></Card>
   <Card title="JinaChat" href="#jinachat"></Card>
   <Card title="Hugging Face" href="#hugging-face"></Card>
@@ -59,9 +62,170 @@ llm:
     top_p: 1
     stream: false
 ```
-
 </CodeGroup>
 
+### Function Calling
+To enable [function calling](https://platform.openai.com/docs/guides/function-calling) in your application using embedchain and OpenAI, you need to pass functions into `OpenAILlm` class as an array of functions. Here are several ways in which you can achieve that:
+
+Examples:
+<Accordion title="Using Pydantic Models">
+  ```python
+import os
+from embedchain import Pipeline as App
+from embedchain.llm.openai import OpenAILlm
+import requests
+from pydantic import BaseModel, Field, ValidationError, field_validator
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+class QA(BaseModel):
+    """
+    A question and answer pair.
+    """
+
+    question: str = Field(
+        ..., description="The question.", example="What is a mountain?"
+    )
+    answer: str = Field(
+        ..., description="The answer.", example="A mountain is a hill."
+    )
+    person_who_is_asking: str = Field(
+        ..., description="The person who is asking the question.", example="John"
+    )
+
+    @field_validator("question")
+    def question_must_end_with_a_question_mark(cls, v):
+        """
+        Validate that the question ends with a question mark.
+        """
+        if not v.endswith("?"):
+            raise ValueError("question must end with a question mark")
+        return v
+
+    @field_validator("answer")
+    def answer_must_end_with_a_period(cls, v):
+        """
+        Validate that the answer ends with a period.
+        """
+        if not v.endswith("."):
+            raise ValueError("answer must end with a period")
+        return v
+
+llm = OpenAILlm(config=None,functions=[QA])
+app = App(llm=llm)
+
+result = app.query("Hey I am Sid. What is a mountain? A mountain is a hill.")
+
+print(result)
+  ```
+  </Accordion>
+  
+  <Accordion title="Using OpenAI JSON schema">
+```python
+import os
+from embedchain import Pipeline as App
+from embedchain.llm.openai import OpenAILlm
+import requests
+from pydantic import BaseModel, Field, ValidationError, field_validator
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+json_schema = {
+    "name": "get_qa",
+    "description": "A question and answer pair and the user who is asking the question.",
+    "parameters": {
+        "type": "object",
+        "properties": {
+            "question": {"type": "string", "description": "The question."},
+            "answer": {"type": "string", "description": "The answer."},
+            "person_who_is_asking": {
+                "type": "string",
+                "description": "The person who is asking the question.",
+            }
+        },
+        "required": ["question", "answer", "person_who_is_asking"],
+    },
+}
+
+llm = OpenAILlm(config=None,functions=[json_schema])
+app = App(llm=llm)
+
+result = app.query("Hey I am Sid. What is a mountain? A mountain is a hill.")
+
+print(result)
+  ```
+  </Accordion>
+  <Accordion title="Using actual python functions">
+  ```python
+import os
+from embedchain import Pipeline as App
+from embedchain.llm.openai import OpenAILlm
+import requests
+from pydantic import BaseModel, Field, ValidationError, field_validator
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+def find_info_of_pokemon(pokemon: str):
+    """
+    Find the information of the given pokemon.
+    Args:
+        pokemon: The pokemon.
+    """
+    req = requests.get(f"https://pokeapi.co/api/v2/pokemon/{pokemon}")
+    if req.status_code == 404:
+        raise ValueError("pokemon not found")
+    return req.json()
+
+llm = OpenAILlm(config=None,functions=[find_info_of_pokemon])
+app = App(llm=llm)
+
+result = app.query("Tell me more about the pokemon pikachu.")
+
+print(result)
+```
+</Accordion>
+
+## Google AI
+
+To use Google AI model, you have to set the `GOOGLE_API_KEY` environment variable. You can obtain the Google API key from the [Google Maker Suite](https://makersuite.google.com/app/apikey)
+
+<CodeGroup>
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ["GOOGLE_API_KEY"] = "xxx"
+
+app = App.from_config(config_path="config.yaml")
+
+app.add("https://www.forbes.com/profile/elon-musk")
+
+response = app.query("What is the net worth of Elon Musk?")
+if app.llm.config.stream: # if stream is enabled, response is a generator
+    for chunk in response:
+        print(chunk)
+else:
+    print(response)
+```
+
+```yaml config.yaml
+llm:
+  provider: google
+  config:
+    model: gemini-pro
+    max_tokens: 1000
+    temperature: 0.5
+    top_p: 1
+    stream: false
+
+embedder:
+  provider: google
+  config:
+    model: 'models/embedding-001'
+    task_type: "retrieval_document"
+    title: "Embeddings for Embedchain"
+```
+</CodeGroup>
 
 ## Azure OpenAI
 
@@ -167,6 +331,68 @@ llm:
 
 </CodeGroup>
 
+## Together
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[together]'
+```
+
+Set the `TOGETHER_API_KEY` as environment variable which you can find on their [Account settings page](https://api.together.xyz/settings/api-keys).
+
+Once you have the API key, you are all set to use it with Embedchain.
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ["TOGETHER_API_KEY"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: together
+  config:
+    model: togethercomputer/RedPajama-INCITE-7B-Base
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+```
+
+</CodeGroup>
+
+## Ollama
+
+Setup Ollama using https://github.com/jmorganca/ollama
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: ollama
+  config:
+    model: 'llama2'
+    temperature: 0.5
+    top_p: 1
+    stream: true
+```
+
+</CodeGroup>
+
 ## GPT4ALL
 
 Install related dependencies using the following command:

docs/components/vector-databases.mdx

@@ -58,6 +58,12 @@ Install related dependencies using the following command:
 pip install --upgrade 'embedchain[elasticsearch]'
 ```
 
+<Note>
+You can configure the Elasticsearch connection by providing either `es_url` or `cloud_id`. If you are using the Elasticsearch Service on Elastic Cloud, you can find the `cloud_id` on the [Elastic Cloud dashboard](https://cloud.elastic.co/deployments).
+</Note>
+
+You can authorize the connection to Elasticsearch by providing either `basic_auth`, `api_key`, or `bearer_auth`.
+
 <CodeGroup>
 
 ```python main.py
@@ -72,11 +78,10 @@ vectordb:
   provider: elasticsearch
   config:
     collection_name: 'es-index'
-    es_url: http://localhost:9200
-    http_auth:
-      - admin
-      - admin
-    api_key: xxx
+    cloud_id: 'deployment-name:xxxx'
+    basic_auth:
+      - elastic
+      - <your_password>
     verify_certs: false
 ```
 </CodeGroup>

docs/contribution/dev.mdx

@@ -22,23 +22,12 @@ make lint format
 
 5. **Create a pull request**: When you are ready to contribute your changes, submit a pull request to the EmbedChain repository. Provide a clear and descriptive title for your pull request, along with a detailed description of the changes you have made.
 
-# Tech Stack
-
-embedchain is built on the following stack:
-
-- [Langchain](https://github.com/hwchase17/langchain) as an LLM framework to load, chunk and index data
-- [OpenAI's Ada embedding model](https://platform.openai.com/docs/guides/embeddings) to create embeddings
-- [OpenAI's ChatGPT API](https://platform.openai.com/docs/guides/gpt/chat-completions-api) as LLM to get answers given the context
-- [Chroma](https://github.com/chroma-core/chroma) as the vector database to store embeddings
-- [gpt4all](https://github.com/nomic-ai/gpt4all) as an open source LLM
-- [sentence-transformers](https://huggingface.co/sentence-transformers) as open source embedding model
-
 ## Team
 
 ### Authors
 
 - Taranjeet Singh ([@taranjeetio](https://twitter.com/taranjeetio))
-- Deshraj Yadav ([@deshrajdry](https://twitter.com/taranjeetio))
+- Deshraj Yadav ([@deshrajdry](https://twitter.com/deshrajdry))
 
 ### Citation
 
@@ -47,7 +36,7 @@ If you utilize this repository, please consider citing it with:
 ```
 @misc{embedchain,
   author = {Taranjeet Singh, Deshraj Yadav},
-  title = {Embechain: Data platform for LLMs - Load, index, retrieve and sync any unstructured data},
+  title = {Embechain: The Open Source RAG Framework},
   year = {2023},
   publisher = {GitHub},
   journal = {GitHub repository},

docs/data-sources/overview.mdx

@@ -1,34 +0,0 @@
----
-title: Overview
----
-
-Embedchain comes with built-in support for various data sources. We handle the complexity of loading unstructured data from these data sources, allowing you to easily customize your app through a user-friendly interface.
-
-<CardGroup cols={4}>
-  <Card title="📊 CSV" href="/data-sources/csv"></Card>
-  <Card title="📃 JSON" href="/data-sources/json"></Card>
-  <Card title="📚 docs site" href="/data-sources/docs-site"></Card>
-  <Card title="📄 docx" href="/data-sources/docx"></Card>
-  <Card title="📝 mdx" href="/data-sources/mdx"></Card>
-  <Card title="📓 Notion" href="/data-sources/notion"></Card>
-  <Card title="📰 PDF" href="/data-sources/pdf-file"></Card>
-  <Card title="❓💬 q&a pair" href="/data-sources/qna"></Card>
-  <Card title="🗺️ sitemap" href="/data-sources/sitemap"></Card>
-  <Card title="📝 text" href="/data-sources/text"></Card>
-  <Card title="🌐 web page" href="/data-sources/web-page"></Card>
-  <Card title="🧾 xml" href="/data-sources/xml"></Card>
-  <Card title="🙌 OpenAPI" href="/data-sources/openapi"></Card>
-  <Card title="📺 Youtube" href="/data-sources/youtube-video"></Card>
-  <Card title="📬 Gmail" href="/data-sources/gmail"></Card>
-  <Card title="🐘 Postgres" href="/data-sources/postgres"></Card>
-  <Card title="🐬 MySQL" href="/data-sources/mysql"></Card>
-  <Card title="🤖 Slack" href="/data-sources/slack"></Card>
-  <Card title="🗨️ Discourse" href="/data-sources/discourse"></Card>
-  <Card title="💬 Discord" href="/data-sources/discord"></Card>
-  <Card title="📝 Github" href="/data-sources/github"></Card>
-  <Card title="⚙️ Custom" href="/data-sources/custom"></Card>
-</CardGroup>
-
-<br/ >
-
-<Snippet file="missing-data-source-tip.mdx" />

docs/data-sources/substack.mdx

@@ -1,16 +0,0 @@
----
-title: "📝 Substack"
----
-
-To add any Substack data sources to your app, just add the sitemap.xml of that url as the source and set the data_type to `substack`.
-
-```python
-from embedchain import Pipeline as App
-
-app = App()
-
-# source: for any substack just add the sitemap.xml url
-app.add('https://www.lennysnewsletter.com/sitemap.xml', data_type='substack')
-app.query("Who is Brian Chesky?")
-# Answer: Brian Chesky is the co-founder and CEO of Airbnb.
-```

docs/deployment/embedchain_ai.mdx

@@ -0,0 +1,38 @@
+---
+title: 'Embedchain.ai'
+description: 'Deploy your RAG application to embedchain.ai platform'
+---
+
+## Deploy on Embedchain Platform
+
+Embedchain enables developers to deploy their LLM-powered apps in production using the [Embedchain platform](https://app.embedchain.ai). The platform offers free access to context on your data through its REST API. Once the pipeline is deployed, you can update your data sources anytime after deployment.
+
+See the example below on how to use the deploy your app (for free):
+
+```python
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Deploy your pipeline to Embedchain Platform
+app.deploy()
+
+# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+# ec-xxxxxx
+
+# 🛠️ Creating pipeline on the platform...
+# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+
+# 🛠️ Adding data to your pipeline...
+# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
+```
+
+## Seeking help?
+
+If you run into issues with deployment, please feel free to reach out to us via any of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/deployment/fly_io.mdx

@@ -0,0 +1,101 @@
+---
+title: 'Fly.io'
+description: 'Deploy your RAG application to fly.io platform'
+---
+
+Embedchain has a nice and simple abstraction on top of the [Fly.io](https://fly.io/) tools to let developers deploy RAG application to fly.io platform seamlessly. 
+
+Follow the instructions given below to deploy your first application quickly:
+
+
+## Step-1: Install flyctl command line
+
+<CodeGroup>
+```bash OSX
+brew install flyctl
+```
+
+```bash Linux
+curl -L https://fly.io/install.sh | sh
+```
+
+```bash Windows
+pwsh -Command "iwr https://fly.io/install.ps1 -useb | iex"
+```
+</CodeGroup>
+
+Once you have installed the fly.io cli tool, signup/login to their platform using the following command:
+
+<CodeGroup>
+```bash Sign up
+fly auth signup
+```
+
+```bash Sign in
+fly auth login
+```
+</CodeGroup>
+
+In case you run into issues, refer to official [fly.io docs](https://fly.io/docs/hands-on/install-flyctl/).
+
+## Step-2: Create RAG app
+
+We provide a command line utility called `ec` in embedchain that inherits the template for `fly.io` platform and help you deploy the app. Follow the instructions to create a fly.io app using the template provided:
+
+```bash Install embedchain
+pip install embedchain
+```
+
+```bash Create application
+mkdir my-rag-app
+ec create --template=fly.io
+```
+
+This will generate a directory structure like this:
+
+```bash
+├── Dockerfile
+├── app.py
+├── fly.toml
+├── .env
+├── .env.example
+├── embedchain.json
+└── requirements.txt
+```
+
+Feel free to edit the files as required.
+- `Dockerfile`: Defines the steps to setup the application
+- `app.py`: Contains API app code
+- `fly.toml`: fly.io config file
+- `.env`: Contains environment variables for production
+- `.env.example`: Contains dummy environment variables (can ignore this file)
+- `embedchain.json`: Contains embedchain specific configuration for deployment (you don't need to configure this)
+- `requirements.txt`: Contains python dependencies for your application
+
+## Step-3: Test app locally
+
+You can run the app locally by simply doing:
+
+```bash Run locally
+pip install -r requirements.txt
+ec dev
+```
+
+## Step-4: Deploy to fly.io
+
+You can deploy to fly.io using the following command:
+```bash Deploy app
+ec deploy
+```
+
+Once this step finished, it will provide you with the deployment endpoint where you can access the app live. It will look something like this (Swagger docs):
+
+You can also check the logs, monitor app status etc on their dashboard by running command `fly dashboard`.
+
+<img src="/images/fly_io.png" />
+
+## Seeking help?
+
+If you run into issues with deployment, please feel free to reach out to us via any of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/deployment/gradio_app.mdx

@@ -0,0 +1,59 @@
+---
+title: 'Gradio.app'
+description: 'Deploy your RAG application to gradio.app platform'
+---
+
+Embedchain offers a Streamlit template to facilitate the development of RAG chatbot applications in just three easy steps.
+
+Follow the instructions given below to deploy your first application quickly:
+
+## Step-1: Create RAG app
+
+We provide a command line utility called `ec` in embedchain that inherits the template for `gradio.app` platform and help you deploy the app. Follow the instructions to create a gradio.app app using the template provided:
+
+```bash Install embedchain
+pip install embedchain
+```
+
+```bash Create application
+mkdir my-rag-app
+ec create --template=gradio.app
+```
+
+This will generate a directory structure like this:
+
+```bash
+├── app.py
+├── embedchain.json
+└── requirements.txt
+```
+
+Feel free to edit the files as required.
+- `app.py`: Contains API app code
+- `embedchain.json`: Contains embedchain specific configuration for deployment (you don't need to configure this)
+- `requirements.txt`: Contains python dependencies for your application
+
+## Step-2: Test app locally
+
+You can run the app locally by simply doing:
+
+```bash Run locally
+pip install -r requirements.txt
+ec dev
+```
+
+## Step-3: Deploy to gradio.app
+
+```bash Deploy to gradio.app
+ec deploy
+```
+
+This will run `gradio deploy` which will prompt you questions and deploy your app directly to huggingface spaces.
+
+<img src="/images/gradio_app.png" alt="gradio app" />
+
+## Seeking help?
+
+If you run into issues with deployment, please feel free to reach out to us via any of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/deployment/huggingface_spaces.mdx

@@ -0,0 +1,103 @@
+---
+title: 'Huggingface.co'
+description: 'Deploy your RAG application to huggingface.co platform'
+---
+
+With Embedchain, you can directly host your apps in just three steps to huggingface spaces where you can view and deploy your app to the world.
+
+We support two types of deployment to huggingface spaces:
+
+<CardGroup cols={2}>
+    <Card title="" href="#using-streamlit-io">
+        Streamlit.io
+    </Card>
+    <Card title="" href="#using-gradio-app">
+        Gradio.app
+    </Card>
+</CardGroup>
+
+## Using streamlit.io
+
+### Step 1: Create a new RAG app
+
+Create a new RAG app using the following command:
+
+```bash
+mkdir my-rag-app
+ec create --template=hf/streamlit.io # inside my-rag-app directory
+```
+
+When you run this for the first time, you'll be asked to login to huggingface.co. Once you login, you'll need to create a **write** token. You can create a write token by going to [huggingface.co settings](https://huggingface.co/settings/token). Once you create a token, you'll be asked to enter the token in the terminal.
+
+This will also create an `embedchain.json` file in your app directory. Add a `name` key into the `embedchain.json` file. This will be the "repo-name" of your app in huggingface spaces.
+
+```json embedchain.json
+{
+    "name": "my-rag-app",
+    "provider": "hf/streamlit.io"
+}
+```
+
+### Step-2: Test app locally
+
+You can run the app locally by simply doing:
+
+```bash Run locally
+pip install -r requirements.txt
+ec dev
+```
+
+### Step-3: Deploy to huggingface spaces
+
+```bash Deploy to huggingface spaces
+ec deploy
+```
+
+This will deploy your app to huggingface spaces. You can view your app at `https://huggingface.co/spaces/<your-username>/my-rag-app`. This will get prompted in the terminal once the app is deployed.
+
+## Using gradio.app
+
+Similar to streamlit.io, you can deploy your app to gradio.app in just three steps.
+
+### Step 1: Create a new RAG app
+
+Create a new RAG app using the following command:
+
+```bash
+mkdir my-rag-app
+ec create --template=hf/gradio.app # inside my-rag-app directory
+```
+
+When you run this for the first time, you'll be asked to login to huggingface.co. Once you login, you'll need to create a **write** token. You can create a write token by going to [huggingface.co settings](https://huggingface.co/settings/token). Once you create a token, you'll be asked to enter the token in the terminal.
+
+This will also create an `embedchain.json` file in your app directory. Add a `name` key into the `embedchain.json` file. This will be the "repo-name" of your app in huggingface spaces.
+
+```json embedchain.json
+{
+    "name": "my-rag-app",
+    "provider": "hf/gradio.app"
+}
+```
+
+### Step-2: Test app locally
+
+You can run the app locally by simply doing:
+
+```bash Run locally
+pip install -r requirements.txt
+ec dev
+```
+
+### Step-3: Deploy to huggingface spaces
+
+```bash Deploy to huggingface spaces
+ec deploy
+```
+
+This will deploy your app to huggingface spaces. You can view your app at `https://huggingface.co/spaces/<your-username>/my-rag-app`. This will get prompted in the terminal once the app is deployed.
+
+## Seeking help?
+
+If you run into issues with deployment, please feel free to reach out to us via any of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/deployment/modal_com.mdx

@@ -0,0 +1,63 @@
+---
+title: 'Modal.com'
+description: 'Deploy your RAG application to modal.com platform'
+---
+
+Embedchain has a nice and simple abstraction on top of the [Modal.com](https://modal.com/) tools to let developers deploy RAG application to modal.com platform seamlessly. 
+
+Follow the instructions given below to deploy your first application quickly:
+
+
+## Step-1 Create RAG application: 
+
+We provide a command line utility called `ec` in embedchain that inherits the template for `modal.com` platform and help you deploy the app. Follow the instructions to create a modal.com app using the template provided:
+
+
+```bash Create application
+pip install embedchain[modal]
+mkdir my-rag-app
+ec create --template=modal.com
+```
+
+This `create` command will open a browser window and ask you to login to your modal.com account and will generate a directory structure like this:
+
+```bash
+├── app.py
+├── .env
+├── .env.example
+├── embedchain.json
+└── requirements.txt
+```
+
+Feel free to edit the files as required.
+- `app.py`: Contains API app code
+- `.env`: Contains environment variables for production
+- `.env.example`: Contains dummy environment variables (can ignore this file)
+- `embedchain.json`: Contains embedchain specific configuration for deployment (you don't need to configure this)
+- `requirements.txt`: Contains python dependencies for your FastAPI application
+
+## Step-2: Test app locally
+
+You can run the app locally by simply doing:
+
+```bash Run locally
+pip install -r requirements.txt
+ec dev
+```
+
+## Step-3: Deploy to modal.com
+
+You can deploy to modal.com using the following command:
+```bash Deploy app
+ec deploy
+```
+
+Once this step finished, it will provide you with the deployment endpoint where you can access the app live. It will look something like this (Swagger docs):
+
+<img src="/images/fly_io.png" />
+
+## Seeking help?
+
+If you run into issues with deployment, please feel free to reach out to us via any of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/deployment/render_com.mdx

@@ -0,0 +1,93 @@
+---
+title: 'Render.com'
+description: 'Deploy your RAG application to render.com platform'
+---
+
+Embedchain has a nice and simple abstraction on top of the [render.com](https://render.com/) tools to let developers deploy RAG application to render.com platform seamlessly. 
+
+Follow the instructions given below to deploy your first application quickly:
+
+## Step-1: Install `render` command line
+
+<CodeGroup>
+```bash OSX
+brew tap render-oss/render
+brew install render
+```
+
+```bash Linux
+# Make sure you have deno installed -> https://docs.render.com/docs/cli#from-source-unsupported-operating-systems
+git clone https://github.com/render-oss/render-cli
+cd render-cli
+make deps
+deno task run
+deno compile
+```
+
+```bash Windows
+choco install rendercli
+```
+</CodeGroup>
+
+In case you run into issues, refer to official [render.com docs](https://docs.render.com/docs/cli).
+
+## Step-2 Create RAG application: 
+
+We provide a command line utility called `ec` in embedchain that inherits the template for `render.com` platform and help you deploy the app. Follow the instructions to create a render.com app using the template provided:
+
+
+```bash Create application
+pip install embedchain
+mkdir my-rag-app
+ec create --template=render.com
+```
+
+This `create` command will open a browser window and ask you to login to your render.com account and will generate a directory structure like this:
+
+```bash
+├── app.py
+├── .env
+├── render.yaml
+├── embedchain.json
+└── requirements.txt
+```
+
+Feel free to edit the files as required.
+- `app.py`: Contains API app code
+- `.env`: Contains environment variables for production
+- `render.yaml`: Contains render.com specific configuration for deployment (configure this according to your needs, follow [this](https://docs.render.com/docs/blueprint-spec) for more info)
+- `embedchain.json`: Contains embedchain specific configuration for deployment (you don't need to configure this)
+- `requirements.txt`: Contains python dependencies for your application
+
+## Step-3: Test app locally
+
+You can run the app locally by simply doing:
+
+```bash Run locally
+pip install -r requirements.txt
+ec dev
+```
+
+## Step-4: Deploy to render.com
+
+Before deploying to render.com, you only have to set up one thing. 
+
+In the render.yaml file, make sure to modify the repo key by inserting the URL of your Git repository where your application will be hosted. You can create a repository from [GitHub](https://github.com) or [GitLab](https://gitlab.com/users/sign_in).
+
+After that, you're ready to deploy on render.com.
+
+```bash Deploy app
+ec deploy
+```
+
+When you run this, it should open up your render dashboard and you can see the app being deployed. You can find your hosted link over there only.
+
+You can also check the logs, monitor app status etc on their dashboard by running command `render dashboard`.
+
+<img src="/images/fly_io.png" />
+
+## Seeking help?
+
+If you run into issues with deployment, please feel free to reach out to us via any of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/deployment/streamlit_io.mdx

@@ -0,0 +1,62 @@
+---
+title: 'Streamlit.io'
+description: 'Deploy your RAG application to streamlit.io platform'
+---
+
+Embedchain offers a Streamlit template to facilitate the development of RAG chatbot applications in just three easy steps.
+
+Follow the instructions given below to deploy your first application quickly:
+
+## Step-1: Create RAG app
+
+We provide a command line utility called `ec` in embedchain that inherits the template for `streamlit.io` platform and help you deploy the app. Follow the instructions to create a streamlit.io app using the template provided:
+
+```bash Install embedchain
+pip install embedchain
+```
+
+```bash Create application
+mkdir my-rag-app
+ec create --template=streamlit.io
+```
+
+This will generate a directory structure like this:
+
+```bash
+├── .streamlit
+│   └── secrets.toml
+├── app.py
+├── embedchain.json
+└── requirements.txt
+```
+
+Feel free to edit the files as required.
+- `app.py`: Contains API app code
+- `.streamlit/secrets.toml`: Contains secrets for your application
+- `embedchain.json`: Contains embedchain specific configuration for deployment (you don't need to configure this)
+- `requirements.txt`: Contains python dependencies for your application
+
+Add your `OPENAI_API_KEY` in `.streamlit/secrets.toml` file to run and deploy the app.
+
+## Step-2: Test app locally
+
+You can run the app locally by simply doing:
+
+```bash Run locally
+pip install -r requirements.txt
+ec dev
+```
+
+## Step-3: Deploy to streamlit.io
+
+![Streamlit App deploy button](https://github.com/embedchain/embedchain/assets/73601258/90658e28-29e5-4ceb-9659-37ff8b861a29)
+
+Use the deploy button from the streamlit website to deploy your app.
+
+You can refer this [guide](https://docs.streamlit.io/streamlit-community-cloud/deploy-your-app) if you run into any problems.
+
+## Seeking help?
+
+If you run into issues with deployment, please feel free to reach out to us via any of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/examples/chat-with-PDF.mdx

@@ -0,0 +1,32 @@
+### Embedchain Chat with PDF App
+
+You can easily create and deploy your own `chat-pdf` App using Embedchain.
+
+Here are few simple steps for you to create and deploy your app:
+
+1. Fork the embedchain repo from [Github](https://github.com/embedchain/embedchain).
+
+<Note>
+If you run into problems with forking, please refer to [github docs](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) for forking a repo.
+</Note>
+
+2. Navigate to `chat-pdf` example app from your forked repo:
+
+```bash
+cd <your_fork_repo>/examples/chat-pdf
+```
+
+3. Run your app in development environment with simple commands
+
+```bash
+pip install -r requirements.txt
+ec dev
+```
+
+Feel free to improve our simple `chat-pdf` streamlit app and create pull request to showcase your app [here](https://docs.embedchain.ai/examples/showcase)
+
+4. You can easily deploy your app using Streamlit interface
+
+Connect your Github account with Streamlit and refer this [guide](https://docs.streamlit.io/streamlit-community-cloud/deploy-your-app) to deploy your app.
+
+You can also use the deploy button from your streamlit website you see when running `ec dev` command.

docs/examples/full_stack.mdx

@@ -1,5 +1,5 @@
 ---
-title: '🌐 Full Stack'
+title: 'Full Stack'
 ---
 
 The Full Stack app example can be found [here](https://github.com/embedchain/embedchain/tree/main/examples/full_stack).

docs/examples/notebooks-and-replits.mdx

@@ -1,5 +1,5 @@
 ---
-title: 🔎 Examples
+title: Notebooks & Replits
 ---
 
 # Explore awesome apps
@@ -44,6 +44,14 @@ Get started with Embedchain by trying out the examples below. You can run the ex
       <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/cohere.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
       <td className="align-middle"><a target="_blank" href="https://replit.com/@taranjeetio/cohere#main.py"><img src="https://replit.com/badge?caption=Try%20with%20Replit&amp;variant=small" noZoom alt="Try with Replit Badge"/></a></td>
     </tr>
+    <tr>
+      <td className="align-middle">Together</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/together.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+    </tr>
+    <tr>
+      <td className="align-middle">Ollama</td>
+      <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/ollama.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>
+    </tr>
     <tr>
       <td className="align-middle">Hugging Face</td>
       <td className="align-middle"><a target="_blank" href="https://colab.research.google.com/github/embedchain/embedchain/blob/main/notebooks/hugging_face_hub.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" noZoom alt="Open In Colab"/></a></td>

docs/examples/openai-assistant.mdx

@@ -1,5 +1,5 @@
 ---
-title: '🤖 OpenAI Assistant'
+title: 'OpenAI Assistant'
 ---
 
 <img src="https://blogs.swarthmore.edu/its/wp-content/uploads/2022/05/openai.jpg"  align="center" width="500" alt="OpenAI Logo"/>
@@ -38,40 +38,6 @@ assistant = OpenAIAssistant(assistant_id="asst_xxx")
 assistant = OpenAIAssistant(assistant_id="asst_xxx", thread_id="thread_xxx")
 ```
 
-### Arguments
-
-<ResponseField name="name" type="string">
-  Name for your AI assistant
-</ResponseField>
-
-<ResponseField name="instructions" type="string">
-  how the Assistant and model should behave or respond
-</ResponseField>
-
-<ResponseField name="assistant_id" type="string">
-  Load existing OpenAI Assistant. If you pass this, you don't have to pass other arguments.
-</ResponseField>
-
-<ResponseField name="thread_id" type="string">
-  Existing OpenAI thread id if exists
-</ResponseField>
-
-<ResponseField name="model" type="str" default="gpt-4-1106-preview">
-  OpenAI model to use
-</ResponseField>
-
-<ResponseField name="tools" type="list">
-  OpenAI tools to use. Default set to `[{"type": "retrieval"}]`
-</ResponseField>
-
-<ResponseField name="data_sources" type="list" default="[]">
-  Add data sources to your assistant. You can add in the following format: `[{"source": "https://example.com", "data_type": "web_page"}]`
-</ResponseField>
-
-<ResponseField name="telemetry" type="boolean" default="True">
-  Anonymous telemetry (doesn't collect any user information or user's files). Used to improve the Embedchain package utilization. Default is `True`.
-</ResponseField>
-
 ## Step-2: Add data to thread
 
 You can add any custom data source that is supported by Embedchain. Else, you can directly pass the file path on your local system and Embedchain propagates it to OpenAI Assistant.
@@ -92,4 +58,3 @@ You can try it out yourself using the following Google Colab notebook:
 <a href="https://colab.research.google.com/drive/1BKlXZYSl6AFRgiHZ5XIzXrXC_24kDYHQ?usp=sharing">
     <img src="https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667" alt="Open in Colab" />
 </a>
-

docs/examples/opensource-assistant.mdx

@@ -0,0 +1,51 @@
+---
+title: 'Open-Source AI Assistant'
+---
+
+Embedchain also provides support for creating Open-Source AI Assistants (similar to [OpenAI Assistants API](https://platform.openai.com/docs/assistants/overview)) which allows you to build AI assistants within your own applications using any LLM (OpenAI or otherwise). An Assistant has instructions and can leverage models, tools, and knowledge to respond to user queries.
+
+At a high level, the Open-Source AI Assistants API has the following flow:
+
+1. Create an AI Assistant by picking a model
+2. Create a Thread when a user starts a conversation
+3. Add Messages to the Thread as the user ask questions
+4. Run the Assistant on the Thread to trigger responses. This automatically calls the relevant tools.
+
+Creating an Open-Source AI Assistant is a simple 3 step process.
+
+## Step 1: Instantiate AI Assistant
+
+```python Initialize
+from embedchain.store.assistants import AIAssistant
+
+assistant = AIAssistant(
+    name="My Assistant",
+    data_sources=[{"source": "https://www.youtube.com/watch?v=U9mJuUkhUzk"}])
+```
+
+If you want to use the existing assistant, you can do something like this:
+
+```python Initialize
+# Load an assistant and create a new thread
+assistant = AIAssistant(assistant_id="asst_xxx")
+
+# Load a specific thread for an assistant
+assistant = AIAssistant(assistant_id="asst_xxx", thread_id="thread_xxx")
+```
+
+## Step-2: Add data to thread
+
+You can add any custom data source that is supported by Embedchain. Else, you can directly pass the file path on your local system and Embedchain propagates it to OpenAI Assistant.
+
+```python Add data
+assistant.add("/path/to/file.pdf")
+assistant.add("https://www.youtube.com/watch?v=U9mJuUkhUzk")
+assistant.add("https://openai.com/blog/new-models-and-developer-products-announced-at-devday")
+```
+
+## Step-3: Chat with your AI Assistant
+
+```python Chat
+assistant.chat("How much OpenAI credits were offered to attendees during OpenAI DevDay?")
+# Response: 'Every attendee of OpenAI DevDay 2023 was offered $500 in OpenAI credits.'
+```

docs/examples/rest-api/create.mdx

@@ -37,7 +37,7 @@ llm:
     max_tokens: 1000
     top_p: 1
     stream: false
-    template: |
+    prompt: |
       Use the following pieces of context to answer the query at the end.
       If you don't know the answer, just say that you don't know, don't try to make up an answer.
 
@@ -81,6 +81,7 @@ curl --request POST \
 | `OPENAI_API_BASE`          | Azure OpenAI                   |
 | `OPENAI_API_VERSION`       | Azure OpenAI                   |
 | `COHERE_API_KEY`           | Cohere                         |
+| `TOGETHER_API_KEY`         | Together                       |
 | `ANTHROPIC_API_KEY`        | Anthropic                      |
 | `JINACHAT_API_KEY`         | Jina                           |
 | `HUGGINGFACE_ACCESS_TOKEN` | Huggingface                    |

docs/examples/showcase.mdx

@@ -0,0 +1,115 @@
+---
+title: '🎪 Community showcase'
+---
+
+Embedchain community has been super active in creating demos on top of Embedchain. On this page, we showcase all the apps, blogs, videos, and tutorials created by the community. ❤️
+
+## Apps
+
+### Open Source
+
+- [My GSoC23 bot- Streamlit chat](https://github.com/lucifertrj/EmbedChain_GSoC23_BOT) by Tarun Jain
+- [Discord Bot for LLM chat](https://github.com/Reidond/discord_bots_playground/tree/c8b0c36541e4b393782ee506804c4b6962426dd6/python/chat-channel-bot) by Reidond
+- [EmbedChain-Streamlit-Docker App](https://github.com/amjadraza/embedchain-streamlit-app) by amjadraza
+- [Harry Potter Philosphers Stone Bot](https://github.com/vinayak-kempawad/Harry_Potter_Philosphers_Stone_Bot/) by Vinayak Kempawad, ([LinkedIn post](https://www.linkedin.com/feed/update/urn:li:activity:7080907532155686912/))
+- [LLM bot trained on own messages](https://github.com/Harin329/harinBot) by Hao Wu
+
+### Closed Source
+
+- [Taobot.io](https://taobot.io) - chatbot & knowledgebase hybrid by [cachho](https://github.com/cachho)
+- [Create Instant ChatBot 🤖 using embedchain](https://databutton.com/v/h3e680h9) by Avra, ([Tweet](https://twitter.com/Avra_b/status/1674704745154641920/))
+- [JOBO 🤖 — The AI-driven sidekick to craft your resume](https://try-jobo.com/) by Enrico Willemse, ([LinkedIn Post](https://www.linkedin.com/posts/enrico-willemse_jobai-gptfun-embedchain-activity-7090340080879374336-ueLB/))
+- [Explore Your Knowledge Base: Interactive chats over various forms of documents](https://chatdocs.dkedar.com/) by Kedar Dabhadkar, ([LinkedIn Post](https://www.linkedin.com/posts/dkedar7_machinelearning-llmops-activity-7092524836639424513-2O3L/))
+- [Chatbot trained on 1000+ videos of Ester hicks the co-author behind the famous book Secret](https://ask-abraham.thoughtseed.repl.co) by Mohan Kumar
+
+
+## Templates
+
+### Replit
+- [Embedchain Chat Bot](https://replit.com/@taranjeet1/Embedchain-Chat-Bot) by taranjeetio
+- [Embedchain Memory Chat Bot Template](https://replit.com/@taranjeetio/Embedchain-Memory-Chat-Bot-Template) by taranjeetio
+- [Chatbot app to demonstrate question-answering using retrieved information](https://replit.com/@AllisonMorrell/EmbedChainlitPublic) by Allison Morrell, ([LinkedIn Post](https://www.linkedin.com/posts/allison-morrell-2889275a_retrievalbot-screenshots-activity-7080339991754649600-wihZ/))
+
+## Posts
+
+### Blogs
+
+- [Customer Service LINE Bot](https://www.evanlin.com/langchain-embedchain/) by Evan Lin
+- [Chatbot in Under 5 mins using Embedchain](https://medium.com/@ayush.wattal/chatbot-in-under-5-mins-using-embedchain-a4f161fcf9c5) by Ayush Wattal
+- [Understanding what the LLM framework embedchain does](https://zenn.dev/hijikix/articles/4bc8d60156a436) by Daisuke Hashimoto
+- [In bed with GPT and Node.js](https://dev.to/worldlinetech/in-bed-with-gpt-and-nodejs-4kh2) by Raphaël Semeteys, ([LinkedIn Post](https://www.linkedin.com/posts/raphaelsemeteys_in-bed-with-gpt-and-nodejs-activity-7088113552326029313-nn87/))
+- [Using Embedchain — A powerful LangChain Python wrapper to build Chat Bots even faster!⚡](https://medium.com/@avra42/using-embedchain-a-powerful-langchain-python-wrapper-to-build-chat-bots-even-faster-35c12994a360) by Avra, ([Tweet](https://twitter.com/Avra_b/status/1686767751560310784/))
+- [What is the Embedchain library?](https://jahaniwww.com/%da%a9%d8%aa%d8%a7%d8%a8%d8%ae%d8%a7%d9%86%d9%87-embedchain/) by Ali Jahani, ([LinkedIn Post](https://www.linkedin.com/posts/ajahani_aepaetaeqaexaggahyaeu-aetaexaesabraeaaeqaepaeu-activity-7097605202135904256-ppU-/))
+- [LangChain is Nice, But Have You Tried EmbedChain ?](https://medium.com/thoughts-on-machine-learning/langchain-is-nice-but-have-you-tried-embedchain-215a34421cde) by FS Ndzomga, ([Tweet](https://twitter.com/ndzfs/status/1695583640372035951/))
+- [Simplest Method to Build a Custom Chatbot with GPT-3.5 (via Embedchain)](https://www.ainewsletter.today/p/simplest-method-to-build-a-custom) by Arjun, ([Tweet](https://twitter.com/aiguy_arjun/status/1696393808467091758/))
+
+### LinkedIn
+
+- [What is embedchain](https://www.linkedin.com/posts/activity-7079393104423698432-wRyi/) by Rithesh Sreenivasan
+- [Building a chatbot with EmbedChain](https://www.linkedin.com/posts/activity-7078434598984060928-Zdso/) by Lior Sinclair
+- [Making chatbot without vs with embedchain](https://www.linkedin.com/posts/kalyanksnlp_llms-chatbots-langchain-activity-7077453416221863936-7N1L/) by Kalyan KS
+- [EmbedChain - very intuitive, first you index your data and then query!](https://www.linkedin.com/posts/shubhamsaboo_embedchain-a-framework-to-easily-create-activity-7079535460699557888-ad1X/) by Shubham Saboo
+- [EmbedChain - Harnessing power of LLM](https://www.linkedin.com/posts/uditsaini_chatbotrevolution-llmpoweredbots-embedchainframework-activity-7077520356827181056-FjTK/) by Udit S.
+- [AI assistant for ABBYY Vantage](https://www.linkedin.com/posts/maximevermeir_llm-github-abbyy-activity-7081658972071424000-fXfZ/) by Maxime V.
+- [About embedchain](https://www.linkedin.com/feed/update/urn:li:activity:7080984218914189312/) by Morris Lee
+- [How to use Embedchain](https://www.linkedin.com/posts/nehaabansal_github-embedchainembedchain-framework-activity-7085830340136595456-kbW5/) by Neha Bansal
+- [Youtube/Webpage summary for Energy Study](https://www.linkedin.com/posts/bar%C4%B1%C5%9F-sanl%C4%B1-34b82715_enerji-python-activity-7082735341563977730-Js0U/) by Barış Sanlı, ([Tweet](https://twitter.com/barissanli/status/1676968784979193857/)) 
+- [Demo: How to use Embedchain? (Contains Collab Notebook link)](https://www.linkedin.com/posts/liorsinclair_embedchain-is-getting-a-lot-of-traction-because-activity-7103044695995424768-RckT/) by Lior Sinclair
+
+### Twitter
+
+- [What is embedchain](https://twitter.com/AlphaSignalAI/status/1672668574450847745) by Lior
+- [Building a chatbot with Embedchain](https://twitter.com/Saboo_Shubham_/status/1673537044419686401) by Shubham Saboo
+- [Chatbot docker image behind an API with yaml configs with Embedchain](https://twitter.com/tricalt/status/1678411430192730113/) by Vasilije
+- [Build AI powered PDF chatbot with just five lines of Python code with Embedchain!](https://twitter.com/Saboo_Shubham_/status/1676627104866156544/) by Shubham Saboo
+- [Chatbot against a youtube video using embedchain](https://twitter.com/smaameri/status/1675201443043704834/) by Sami Maameri
+- [Highlights of EmbedChain](https://twitter.com/carl_AIwarts/status/1673542204328120321/) by carl_AIwarts
+- [Build Llama-2 chatbot in less than 5 minutes](https://twitter.com/Saboo_Shubham_/status/1682168956918833152/) by Shubham Saboo
+- [All cool features of embedchain](https://twitter.com/DhravyaShah/status/1683497882438217728/) by Dhravya Shah, ([LinkedIn Post](https://www.linkedin.com/posts/dhravyashah_what-if-i-tell-you-that-you-can-make-an-ai-activity-7089459599287726080-ZIYm/))
+- [Read paid Medium articles for Free using embedchain](https://twitter.com/kumarkaushal_/status/1688952961622585344) by Kaushal Kumar
+
+## Videos
+
+- [Embedchain in one shot](https://www.youtube.com/watch?v=vIhDh7H73Ww&t=82s) by AI with Tarun
+- [embedChain Create LLM powered bots over any dataset Python Demo Tesla Neurallink Chatbot Example](https://www.youtube.com/watch?v=bJqAn22a6Gc) by Rithesh Sreenivasan
+- [Embedchain - NEW 🔥 Langchain BABY to build LLM Bots](https://www.youtube.com/watch?v=qj_GNQ06I8o) by 1littlecoder
+- [EmbedChain -- NEW!: Build LLM-Powered Bots with Any Dataset](https://www.youtube.com/watch?v=XmaBezzGHu4) by DataInsightEdge
+- [Chat With Your PDFs in less than 10 lines of code! EMBEDCHAIN tutorial](https://www.youtube.com/watch?v=1ugkcsAcw44) by Phani Reddy
+- [How To Create A Custom Knowledge AI Powered Bot | Install + How To Use](https://www.youtube.com/watch?v=VfCrIiAst-c) by The Ai Solopreneur
+- [Build Custom Chatbot in 6 min with this Framework [Beginner Friendly]](https://www.youtube.com/watch?v=-8HxOpaFySM) by Maya Akim
+- [embedchain-streamlit-app](https://www.youtube.com/watch?v=3-9GVd-3v74) by Amjad Raza
+- [🤖CHAT with ANY ONLINE RESOURCES using EMBEDCHAIN - a LangChain wrapper, in few lines of code !](https://www.youtube.com/watch?v=Mp7zJe4TIdM) by Avra
+- [Building resource-driven LLM-powered bots with Embedchain](https://www.youtube.com/watch?v=IVfcAgxTO4I) by BugBytes
+- [embedchain-streamlit-demo](https://www.youtube.com/watch?v=yJAWB13FhYQ) by Amjad Raza
+- [Embedchain - create your own AI chatbots using open source models](https://www.youtube.com/shorts/O3rJWKwSrWE) by Dhravya Shah
+- [AI ChatBot in 5 lines Python Code](https://www.youtube.com/watch?v=zjWvLJLksv8) by Data Engineering
+- [Interview with Karl Marx](https://www.youtube.com/watch?v=5Y4Tscwj1xk) by Alexander Ray Williams
+- [Vlog where we try to build a bot based on our content on the internet](https://www.youtube.com/watch?v=I2w8CWM3bx4) by DV, ([Tweet](https://twitter.com/dvcoolster/status/1688387017544261632))
+- [CHAT with ANY ONLINE RESOURCES using EMBEDCHAIN|STREAMLIT with MEMORY |All OPENSOURCE](https://www.youtube.com/watch?v=TqQIHWoWTDQ&pp=ygUKZW1iZWRjaGFpbg%3D%3D) by DataInsightEdge
+- [Build POWERFUL LLM Bots EASILY with Your Own Data - Embedchain - Langchain 2.0? (Tutorial)](https://www.youtube.com/watch?v=jE24Y_GasE8) by WorldofAI, ([Tweet](https://twitter.com/intheworldofai/status/1696229166922780737))
+- [Embedchain: An AI knowledge base assistant for customizing enterprise private data, which can be connected to discord, whatsapp, slack, tele and other terminals (with gradio to build a request interface) in Chinese](https://www.youtube.com/watch?v=5RZzCJRk-d0) by AIGC LINK
+- [Embedchain Introduction](https://www.youtube.com/watch?v=Jet9zAqyggI) by Fahd Mirza 
+
+## Mentions
+
+### Github repos
+
+- [Awesome-LLM](https://github.com/Hannibal046/Awesome-LLM)
+- [awesome-chatgpt-api](https://github.com/reorx/awesome-chatgpt-api)
+- [awesome-langchain](https://github.com/kyrolabs/awesome-langchain)
+- [Awesome-Prompt-Engineering](https://github.com/promptslab/Awesome-Prompt-Engineering)
+- [awesome-chatgpt](https://github.com/eon01/awesome-chatgpt)
+- [Awesome-LLMOps](https://github.com/tensorchord/Awesome-LLMOps)
+- [awesome-generative-ai](https://github.com/filipecalegario/awesome-generative-ai)
+- [awesome-gpt](https://github.com/formulahendry/awesome-gpt)
+- [awesome-ChatGPT-repositories](https://github.com/taishi-i/awesome-ChatGPT-repositories)
+- [awesome-gpt-prompt-engineering](https://github.com/snwfdhmp/awesome-gpt-prompt-engineering)
+- [awesome-chatgpt](https://github.com/awesome-chatgpt/awesome-chatgpt)
+- [awesome-llm-and-aigc](https://github.com/sjinzh/awesome-llm-and-aigc)
+- [awesome-compbio-chatgpt](https://github.com/csbl-br/awesome-compbio-chatgpt)
+- [Awesome-LLM4Tool](https://github.com/OpenGVLab/Awesome-LLM4Tool)
+
+## Meetups
+
+- [Dash and ChatGPT: Future of AI-enabled apps 30/08/23](https://go.plotly.com/dash-chatgpt)
+- [Pie & AI: Bangalore - Build end-to-end LLM app using Embedchain 01/09/23](https://www.eventbrite.com/e/pie-ai-bangalore-build-end-to-end-llm-app-using-embedchain-tickets-698045722547)

docs/get-started/deployment.mdx

@@ -0,0 +1,22 @@
+---
+title: 'Overview'
+description: 'Deploy your RAG application to production'
+---
+
+After successfully setting up and testing your RAG app locally, the next step is to deploy it to a hosting service to make it accessible to a wider audience. Embedchain provides integration with different cloud providers so that you can seamlessly deploy your RAG applications to production without having to worry about going through the cloud provider instructions. Embedchain does all the heavy lifting for you.
+
+<CardGroup cols={4}>
+  <Card title="Fly.io" href="/deployment/fly_io"></Card>
+  <Card title="Modal.com" href="/deployment/modal_com"></Card>
+  <Card title="Render.com" href="/deployment/render_com"></Card>
+  <Card title="Streamlit.io" href="/deployment/streamlit_io"></Card>
+  <Card title="Gradio.app" href="/deployment/gradio_app"></Card>
+  <Card title="Huggingface.co" href="/deployment/huggingface_spaces"></Card>
+  <Card title="Embedchain.ai" href="/deployment/embedchain_ai"></Card>
+</CardGroup>
+
+## Seeking help?
+
+If you run into issues with deployment, please feel free to reach out to us via any of the following methods:
+
+<Snippet file="get-help.mdx" />

docs/get-started/faq.mdx

@@ -4,7 +4,7 @@ description: 'Collections of all the frequently asked questions'
 ---
 <AccordionGroup>
 <Accordion title="Does Embedchain support OpenAI's Assistant APIs?">
-Yes, it does. Please refer to the [OpenAI Assistant docs page](/get-started/openai-assistant).
+Yes, it does. Please refer to the [OpenAI Assistant docs page](/examples/openai-assistant).
 </Accordion>
 <Accordion title="How to use MistralAI language model?">
 Use the model provided on huggingface: `mistralai/Mistral-7B-v0.1`
@@ -13,7 +13,6 @@ Use the model provided on huggingface: `mistralai/Mistral-7B-v0.1`
 import os
 from embedchain import Pipeline as App
 
-os.environ["OPENAI_API_KEY"] = "sk-xxx"
 os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "hf_your_token"
 
 app = App.from_config("huggingface.yaml")
@@ -27,6 +26,11 @@ llm:
     max_tokens: 1000
     top_p: 0.5
     stream: false
+
+embedder:
+  provider: huggingface
+  config:
+    model: 'sentence-transformers/all-mpnet-base-v2'
 ```
 </CodeGroup>
 </Accordion>
@@ -86,11 +90,8 @@ llm:
 <CodeGroup>
 
 ```python main.py
-import os
 from embedchain import Pipeline as App
 
-os.environ['OPENAI_API_KEY'] = 'xxx'
-
 # load llm configuration from opensource.yaml file
 app = App.from_config(config_path="opensource.yaml")
 ```
@@ -112,10 +113,79 @@ embedder:
 ```
 </CodeGroup>
 
+</Accordion>
+<Accordion title="How to stream response while using OpenAI model in Embedchain?">
+You can achieve this by setting `stream` to `true` in the config file.
+
+<CodeGroup>
+```yaml openai.yaml
+llm:
+  provider: openai
+  config:
+    model: 'gpt-3.5-turbo'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: true
+```
+
+```python main.py
+import os
+from embedchain import Pipeline as App
+
+os.environ['OPENAI_API_KEY'] = 'sk-xxx'
+
+app = App.from_config(config_path="openai.yaml")
+
+app.add("https://www.forbes.com/profile/elon-musk")
+
+response = app.query("What is the net worth of Elon Musk?")
+# response will be streamed in stdout as it is generated.
+```
+</CodeGroup>
+</Accordion>
+
+<Accordion title="How to persist data across multiple app sessions?">
+  Set up the app by adding an `id` in the config file. This keeps the data for future use. You can include this `id` in the yaml config or input it directly in `config` dict.
+  ```python app1.py
+  import os
+  from embedchain import Pipeline as App
+
+  os.environ['OPENAI_API_KEY'] = 'sk-xxx'
+
+  app1 = App.from_config(config={
+    "app": {
+      "config": {
+        "id": "your-app-id",
+      }
+    }
+  })
+
+  app1.add("https://www.forbes.com/profile/elon-musk")
+
+  response = app1.query("What is the net worth of Elon Musk?")
+  ```
+  ```python app2.py
+  import os
+  from embedchain import Pipeline as App
+
+  os.environ['OPENAI_API_KEY'] = 'sk-xxx'
+
+  app2 = App.from_config(config={
+    "app": {
+      "config": {
+        # this will persist and load data from app1 session
+        "id": "your-app-id",
+      }
+    }
+  })
+
+  response = app2.query("What is the net worth of Elon Musk?")
+  ```
 </Accordion>
 </AccordionGroup>
 
-#### Need more help?
+#### Still have questions?
 If docs aren't sufficient, please feel free to reach out to us using one of the following methods:
 
 <Snippet file="get-help.mdx" />

docs/get-started/introduction.mdx

@@ -1,133 +1,66 @@
 ---
 title: 📚 Introduction
-description: '📝 Embedchain is a Data Platform for LLMs - load, index, retrieve, and sync any unstructured data'
 ---
 
-## 🌐 What is Embedchain?
+## What is Embedchain?
 
-Embedchain simplifies data handling by automatically processing unstructured data, breaking it into chunks, generating embeddings, and storing it in a vector database.
+Embedchain is an Open Source RAG Framework that makes it easy to create and deploy AI apps. At its core, Embedchain follows the design principle of being *"Conventional but Configurable"* to serve both software engineers and machine learning engineers.
 
-Through various APIs, you can obtain contextual information for queries, find answers to specific questions, and engage in chat conversations using your data.
-## 🔍 Search
+Embedchain streamlines the creation of RAG applications, offering a seamless process for managing various types of unstructured data. It efficiently segments data into manageable chunks, generates relevant embeddings, and stores them in a vector database for optimized retrieval. With a suite of diverse APIs, it enables users to extract contextual information, find precise answers, or engage in interactive chat conversations, all tailored to their own data.
 
-Embedchain lets you get most relevant context by doing semantic search over your data sources for a provided query. See the example below:
+## Who is Embedchain for?
 
-```python
-from embedchain import Pipeline as App
+Embedchain is designed for a diverse range of users, from AI professionals like Data Scientists and Machine Learning Engineers to those just starting their AI journey, including college students, independent developers, and hobbyists. Essentially, it's for anyone with an interest in AI, regardless of their expertise level.
 
-# Initialize app
-app = App()
+Our APIs are user-friendly yet adaptable, enabling beginners to effortlessly create LLM-powered applications with as few as 4 lines of code. At the same time, we offer extensive customization options for every aspect of the RAG pipeline. This includes the choice of LLMs, vector databases, loaders and chunkers, retrieval strategies, re-ranking, and more.
 
-# Add data source
-app.add("https://www.forbes.com/profile/elon-musk")
+Our platform's clear and well-structured abstraction layers ensure that users can tailor the system to meet their specific needs, whether they're crafting a simple project or a complex, nuanced AI application.
 
-# Get relevant context using semantic search
-context = app.search("What is the net worth of Elon?", num_documents=2)
-print(context)
-# Context:
-# [
-#     {
-#         'context': 'Elon Musk PROFILEElon MuskCEO, Tesla$221.9BReal Time Net Worthas of 10/29/23Reflects change since 5 pm ET of prior trading day. 1 in the world todayPhoto by Martin Schoeller for ForbesAbout Elon MuskElon Musk cofounded six companies, including electric car maker Tesla, rocket producer SpaceX and tunneling startup Boring Company.He owns about 21% of Tesla between stock and options, but has pledged more than half his shares as collateral for personal loans of up to $3.5 billion.SpaceX, founded in',
-#         'source': 'https://www.forbes.com/profile/elon-musk',
-#         'document_id': 'some_document_id'
-#     },
-#     {
-#         'context': 'company, which is now called X.Wealth HistoryHOVER TO REVEAL NET WORTH BY YEARForbes Lists 1Forbes 400 (2023)The Richest Person In Every State (2023) 2Billionaires (2023) 1Innovative Leaders (2019) 25Powerful People (2018) 12Richest In Tech (2017)Global Game Changers (2016)More ListsPersonal StatsAge52Source of WealthTesla, SpaceX, Self MadeSelf-Made Score8Philanthropy Score1ResidenceAustin, TexasCitizenshipUnited StatesMarital StatusSingleChildren11EducationBachelor of Arts/Science, University',
-#         'source': 'https://www.forbes.com/profile/elon-musk',
-#         'document_id': 'some_document_id'
-#     }
-# ]
-```
+## Why Use Embedchain?
 
-## ❓Query
+Developing a robust and efficient RAG (Retrieval-Augmented Generation) pipeline for production use presents numerous complexities, such as:
 
-Embedchain empowers developers to ask questions and receive relevant answers through a user-friendly query API. Refer to the following example to learn how to utilize the query API:
+- Integrating and indexing data from diverse sources.
+- Determining optimal data chunking methods for each source.
+- Synchronizing the RAG pipeline with regularly updated data sources.
+- Implementing efficient data storage in a vector store.
+- Deciding whether to include metadata with document chunks.
+- Handling permission management.
+- Configuring Large Language Models (LLMs).
+- Selecting effective prompts.
+- Choosing suitable retrieval strategies.
+- Assessing the performance of your RAG pipeline.
+- Deploying the pipeline into a production environment, among other concerns.
 
-```python
-from embedchain import Pipeline as App
+Embedchain is designed to simplify these tasks, offering conventional yet customizable APIs. Our solution handles the intricate processes of loading, chunking, indexing, and retrieving data. This enables you to concentrate on aspects that are crucial for your specific use case or business objectives, ensuring a smoother and more focused development process.
 
-# Initialize app
-app = App()
+## How it works?
 
-# Add data source
-app.add("https://www.forbes.com/profile/elon-musk")
+Embedchain makes it easy to add data to your RAG pipeline with these straightforward steps:
 
-# Get relevant answer for your query
-answer = app.query("What is the net worth of Elon?")
-print(answer)
-# Answer: The net worth of Elon Musk is $221.9 billion.
-```
+1. **Automatic Data Handling**: It automatically recognizes the data type and loads it.
+2. **Efficient Data Processing**: The system creates embeddings for key parts of your data.
+3. **Flexible Data Storage**: You get to choose where to store this processed data in a vector database.
 
-## 💬 Chat
+When a user asks a question, whether for chatting, searching, or querying, Embedchain simplifies the response process:
 
-Embedchain allows easy chatting over your data sources using a user-friendly chat API. Check out the example below to understand how to use the chat API:
+1. **Query Processing**: It turns the user's question into embeddings.
+2. **Document Retrieval**: These embeddings are then used to find related documents in the database.
+3. **Answer Generation**: The related documents are used by the LLM to craft a precise answer.
 
-```python
-from embedchain import Pipeline as App
+With Embedchain, you don’t have to worry about the complexities of building a RAG pipeline. It offers an easy-to-use interface for developing applications with any kind of data.
 
-# Initialize app
-app = App()
+## Getting started
 
-# Add data source
-app.add("https://www.forbes.com/profile/elon-musk")
+Checkout our [quickstart guide](/get-started/quickstart) to start your first RAG application.
 
-# Chat on your data using `.chat()`
-answer = app.chat("How much did Elon pay for Twitter?")
-print(answer)
-# Answer: Elon Musk paid $44 billion for Twitter.
-```
+## Support
 
-## 🚀 Deploy
+Feel free to reach out to us if you have ideas, feedback or questions that we can help out with.
 
-Embedchain enables developers to deploy their LLM-powered apps in production using the Embedchain platform. The platform offers free access to context on your data through its REST API. Once the pipeline is deployed, you can update your data sources anytime after deployment.
+<Snippet file="get-help.mdx" />
 
-See the example below on how to use the deploy API:
+## Contribute
 
-```python
-from embedchain import Pipeline as App
-
-# Initialize app
-app = App()
-
-# Add data source
-app.add("https://www.forbes.com/profile/elon-musk")
-
-# Deploy your pipeline to Embedchain Platform
-app.deploy()
-
-# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
-# ec-xxxxxx
-
-# 🛠️ Creating pipeline on the platform...
-# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
-
-# 🛠️ Adding data to your pipeline...
-# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
-```
-
-## 🛠️ How it works?
-
-Embedchain abstracts out the following steps from you to easily create LLM powered apps:
-
-1. Detect the data type and load data
-2. Create meaningful chunks
-3. Create embeddings for each chunk
-4. Store chunks in a vector database
-
-When a user asks a query, the following process happens to find the answer:
-
-1. Create an embedding for the query
-2. Find similar documents for the query from the vector database
-3. Pass the similar documents as context to LLM to get the final answer
-
-The process of loading the dataset and querying involves multiple steps, each with its own nuances:
-
-- How should I chunk the data? What is a meaningful chunk size?
-- How should I create embeddings for each chunk? Which embedding model should I use?
-- How should I store the chunks in a vector database? Which vector database should I use?
-- Should I store metadata along with the embeddings?
-- How should I find similar documents for a query? Which ranking model should I use?
-
-Embedchain takes care of all these nuances and provides a simple interface to create apps on any data.
-
-## [🚀 Get started](https://docs.embedchain.ai/get-started/quickstart)
+- [GitHub](https://github.com/embedchain/embedchain)
+- [Contribution docs](/contribution/dev)

docs/get-started/quickstart.mdx

@@ -1,20 +1,14 @@
 ---
-title: '🚀 Quickstart'
-description: '💡 Start building LLM powered apps under 30 seconds'
+title: '⚡ Quickstart'
+description: '💡 Start building ChatGPT like apps in a minute on your own data'
 ---
 
-Embedchain is a Data Platform for LLMs - load, index, retrieve, and sync any unstructured data. Using embedchain, you can easily create LLM powered apps over any data. 
-
-Install embedchain python package:
+Install python package:
 
 ```bash
 pip install embedchain
 ```
 
-<Tip>
-Embedchain now supports OpenAI's latest `gpt-4-turbo` model. Checkout the [FAQs](/get-started/faq#how-to-use-gpt-4-turbo-model-released-on-openai-devday).
-</Tip>
-
 Creating an app involves 3 steps:
 
 <Steps>
@@ -59,6 +53,7 @@ Creating an app involves 3 steps:
     app.query("What is the net worth of Elon Musk today?")
     # Answer: The net worth of Elon Musk today is $258.7 billion.
     ```
+    <hr />
     <Accordion title="Want to chat with your app?" icon="face-thinking">
       Embedchain provides a wide range of features to interact with your app. You can chat with your app, ask questions, search through your data, and much more.
       ```python
@@ -70,27 +65,4 @@ Creating an app involves 3 steps:
       To learn about other features, click [here](https://docs.embedchain.ai/get-started/introduction)
     </Accordion>
   </Step>
-  <Step title="🚀 Seamlessly launch your App on the Embedchain Platform!">
-    ```python
-    app.deploy()
-    # 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
-    # ec-xxxxxx
-
-    # 🛠️ Creating pipeline on the platform...
-    # 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
-
-    # 🛠️ Adding data to your pipeline...
-    # ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
-    ```
-    <Accordion title="Share your app with others" icon="laptop-mobile">
-      You can now share your app with others from our platform.
-      Access your app on our [platform](https://app.embedchain.ai/).
-    </Accordion>
-  </Step>
 </Steps>
-
-Putting it together, you can run your first app using the following Google Colab. Make sure to set the `OPENAI_API_KEY` 🔑 environment variable in the code.
-
-<a href="https://colab.research.google.com/drive/17ON1LPonnXAtLaZEebnOktstB_1cJJmh?usp=sharing">
-    <img src="https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667" alt="Open in Colab" />
-</a>

docs/integration/chainlit.mdx

@@ -0,0 +1,68 @@
+---
+title: '⛓️ Chainlit'
+description: 'Integrate with Chainlit to create LLM chat apps'
+---
+
+In this example, we will learn how to use Chainlit and Embedchain together.
+
+![chainlit-demo](https://github.com/embedchain/embedchain/assets/73601258/d6635624-5cdb-485b-bfbd-3b7c8f18bfff)
+
+## Setup
+
+First, install the required packages:
+
+```bash
+pip install embedchain chainlit
+```
+
+## Create a Chainlit app
+
+Create a new file called `app.py` and add the following code:
+
+```python
+import chainlit as cl
+from embedchain import Pipeline as App
+
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+@cl.on_chat_start
+async def on_chat_start():
+    app = App.from_config(config={
+        'app': {
+            'config': {
+                'name': 'chainlit-app'
+            }
+        },
+        'llm': {
+            'config': {
+                'stream': True,
+            }
+        }
+    })
+    # import your data here
+    app.add("https://www.forbes.com/profile/elon-musk/")
+    app.collect_metrics = False
+    cl.user_session.set("app", app)
+
+
+@cl.on_message
+async def on_message(message: cl.Message):
+    app = cl.user_session.get("app")
+    msg = cl.Message(content="")
+    for chunk in await cl.make_async(app.chat)(message.content):
+        await msg.stream_token(chunk)
+    
+    await msg.send()
+```
+
+## Run the app
+
+```
+chainlit run app.py
+```
+
+## Try it out
+
+Open the app in your browser and start chatting with it!

docs/integration/langsmith.mdx

@@ -48,4 +48,4 @@ app.query("How many companies did Elon found?")
 
 * Now the entire log for this will be visible in langsmith.
 
-<img src="/images/langsmith.png"/>
+<img src="/images/langsmith.png"/>

docs/integration/streamlit-mistral.mdx

@@ -0,0 +1,112 @@
+---
+title: '🚀 Streamlit'
+description: 'Integrate with Streamlit to plug and play with any LLM'
+---
+
+In this example, we will learn how to use `mistralai/Mixtral-8x7B-Instruct-v0.1` and Embedchain together with Streamlit to build a simple RAG chatbot.
+
+![Streamlit + Embedchain Demo](https://github.com/embedchain/embedchain/assets/73601258/052f7378-797c-41cf-ac81-f004d0d44dd1)
+
+## Setup
+
+Install Embedchain and Streamlit.
+```bash
+pip install embedchain streamlit
+```
+<Tabs>
+    <Tab title="app.py">
+    ```python
+    import os
+    from embedchain import Pipeline as App
+    import streamlit as st
+
+    with st.sidebar:
+        huggingface_access_token = st.text_input("Hugging face Token", key="chatbot_api_key", type="password")
+        "[Get Hugging Face Access Token](https://huggingface.co/settings/tokens)"
+        "[View the source code](https://github.com/embedchain/examples/mistral-streamlit)"
+
+
+    st.title("💬 Chatbot")
+    st.caption("🚀 An Embedchain app powered by Mistral!")
+    if "messages" not in st.session_state:
+        st.session_state.messages = [
+            {
+                "role": "assistant",
+                "content": """
+            Hi! I'm a chatbot. I can answer questions and learn new things!\n
+            Ask me anything and if you want me to learn something do `/add <source>`.\n
+            I can learn mostly everything. :)
+            """,
+            }
+        ]
+
+    for message in st.session_state.messages:
+        with st.chat_message(message["role"]):
+            st.markdown(message["content"])
+
+    if prompt := st.chat_input("Ask me anything!"):
+        if not st.session_state.chatbot_api_key:
+            st.error("Please enter your Hugging Face Access Token")
+            st.stop()
+
+        os.environ["HUGGINGFACE_ACCESS_TOKEN"] = st.session_state.chatbot_api_key
+        app = App.from_config(config_path="config.yaml")
+
+        if prompt.startswith("/add"):
+            with st.chat_message("user"):
+                st.markdown(prompt)
+                st.session_state.messages.append({"role": "user", "content": prompt})
+            prompt = prompt.replace("/add", "").strip()
+            with st.chat_message("assistant"):
+                message_placeholder = st.empty()
+                message_placeholder.markdown("Adding to knowledge base...")
+                app.add(prompt)
+                message_placeholder.markdown(f"Added {prompt} to knowledge base!")
+                st.session_state.messages.append({"role": "assistant", "content": f"Added {prompt} to knowledge base!"})
+                st.stop()
+
+        with st.chat_message("user"):
+            st.markdown(prompt)
+            st.session_state.messages.append({"role": "user", "content": prompt})
+
+        with st.chat_message("assistant"):
+            msg_placeholder = st.empty()
+            msg_placeholder.markdown("Thinking...")
+            full_response = ""
+
+            for response in app.chat(prompt):
+                msg_placeholder.empty()
+                full_response += response
+
+            msg_placeholder.markdown(full_response)
+            st.session_state.messages.append({"role": "assistant", "content": full_response})
+        ```
+    </Tab>
+    <Tab title="config.yaml">
+    ```yaml
+    app:
+        config:
+            name: 'mistral-streamlit-app'
+
+    llm:
+        provider: huggingface
+        config:
+            model: 'mistralai/Mixtral-8x7B-Instruct-v0.1'
+            temperature: 0.1
+            max_tokens: 250
+            top_p: 0.1
+            stream: true
+
+    embedder:
+        provider: huggingface
+        config:
+            model: 'sentence-transformers/all-mpnet-base-v2'
+    ```
+    </Tab>
+</Tabs>
+
+## To run it locally,
+
+```bash
+streamlit run app.py
+```

docs/mint.json

@@ -4,7 +4,7 @@
   "logo": {
     "dark": "/logo/dark.svg",
     "light": "/logo/light.svg",
-    "href": "https://embedchain.ai/"
+    "href": "https://github.com/embedchain/embedchain"
   },
   "favicon": "/favicon.png",
   "colors": {
@@ -19,105 +19,162 @@
   "modeToggle": {
     "default": "dark"
   },
-  "openapi": ["/rest-api.json"],
+  "openapi": [
+    "/rest-api.json"
+  ],
   "metadata": {
     "og:image": "/images/og.png",
     "twitter:site": "@embedchain"
   },
+  "tabs": [
+    {
+      "name": "Examples",
+      "url": "examples"
+    },
+    {
+      "name": "API Reference",
+      "url": "api-reference"
+    }
+  ],
   "anchors": [
     {
-      "name": "Embedchain Platform",
-      "icon": "tv",
-      "url": "https://app.embedchain.ai/"
+      "name": "Talk to founders",
+      "icon": "calendar",
+      "url": "https://cal.com/taranjeetio/ec"
     },
     {
       "name": "Join our slack",
       "icon": "slack",
       "url": "https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw"
+    },
+    {
+      "name": "Join our discord",
+      "icon": "discord",
+      "url": "https://discord.gg/CUU9FPhRNt"
     }
   ],
   "topbarLinks": [
     {
-      "name": "Create account",
-      "url": "https://app.embedchain.ai/login/"
+      "name": "GitHub",
+      "url": "https://github.com/embedchain/embedchain"
     }
   ],
   "topbarCtaButton": {
-    "name": "Get started",
-    "url": "https://app.embedchain.ai"
+    "name": "Join our slack",
+    "url": "https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw"
   },
   "primaryTab": {
-    "name": "Docs"
+    "name": "Documentation"
   },
   "navigation": [
     {
-      "group": "Get started",
+      "group": "Get Started",
       "pages": [
-        "get-started/quickstart",
         "get-started/introduction",
-        "get-started/openai-assistant",
-        "get-started/faq",
-        "get-started/examples"
+        "get-started/quickstart",
+        {
+        "group": "🔗 Integrations",
+          "pages": [
+            "integration/langsmith",
+            "integration/chainlit",
+            "integration/streamlit-mistral"
+          ]
+        },
+        "get-started/faq"
+      ]
+    },
+    {
+      "group": "Deployment",
+      "pages": [
+        "get-started/deployment",
+        "deployment/fly_io",
+        "deployment/modal_com",
+        "deployment/render_com",
+        "deployment/streamlit_io",
+        "deployment/gradio_app",
+        "deployment/huggingface_spaces",
+        "deployment/embedchain_ai"
+      ]
+    },
+    {
+      "group": "Use cases",
+      "pages": [
+        "use-cases/chatbots",
+        "use-cases/question-answering",
+        "use-cases/semantic-search"
       ]
     },
     {
       "group": "Components",
       "pages": [
-        "components/llms",
-        "components/embedding-models",
-        "components/vector-databases"
-      ]
-    },
-    {
-      "group": "Data sources",
-      "pages": [
-        "data-sources/overview",
         {
-          "group": "Supported data sources",
+          "group": "Data sources",
           "pages": [
-            "data-sources/csv",
-            "data-sources/json",
-            "data-sources/docs-site",
-            "data-sources/docx",
-            "data-sources/mdx",
-            "data-sources/notion",
-            "data-sources/pdf-file",
-            "data-sources/qna",
-            "data-sources/sitemap",
-            "data-sources/text",
-            "data-sources/web-page",
-            "data-sources/openapi",
-            "data-sources/youtube-video",
-            "data-sources/discourse",
-            "data-sources/substack",
-            "data-sources/discord"
+            "components/data-sources/overview",
+            {
+              "group": "Data types",
+              "pages": [
+                "components/data-sources/csv",
+                "components/data-sources/json",
+                "components/data-sources/docs-site",
+                "components/data-sources/docx",
+                "components/data-sources/mdx",
+                "components/data-sources/notion",
+                "components/data-sources/pdf-file",
+                "components/data-sources/qna",
+                "components/data-sources/sitemap",
+                "components/data-sources/text",
+                "components/data-sources/web-page",
+                "components/data-sources/openapi",
+                "components/data-sources/youtube-video",
+                "components/data-sources/discourse",
+                "components/data-sources/substack",
+                "components/data-sources/discord",
+                "components/data-sources/beehiiv",
+                "components/data-sources/directory"
+              ]
+            },
+            "components/data-sources/data-type-handling"
           ]
         },
-        "data-sources/data-type-handling"
+        "components/llms",
+        "components/vector-databases",
+        "components/embedding-models"
       ]
     },
     {
-      "group": "Advanced",
-      "pages": ["advanced/configuration"]
-    },
-    {
-      "group": "REST API",
+      "group": "Community",
       "pages": [
-        "rest-api/getting-started",
-        "rest-api/create",
-        "rest-api/get-all-apps",
-        "rest-api/add-data",
-        "rest-api/get-data",
-        "rest-api/query",
-        "rest-api/deploy",
-        "rest-api/delete",
-        "rest-api/check-status"
+        "community/connect-with-us"
       ]
     },
     {
-      "group": "Use Cases",
+      "group": "Examples",
       "pages": [
+        "examples/chat-with-PDF",
+        "examples/notebooks-and-replits",
+        {
+          "group": "REST API Service",
+          "pages": [
+            "examples/rest-api/getting-started",
+            "examples/rest-api/create",
+            "examples/rest-api/get-all-apps",
+            "examples/rest-api/add-data",
+            "examples/rest-api/get-data",
+            "examples/rest-api/query",
+            "examples/rest-api/deploy",
+            "examples/rest-api/delete",
+            "examples/rest-api/check-status"
+          ]
+        },
         "examples/full_stack",
+        "examples/openai-assistant",
+        "examples/opensource-assistant"
+      ]
+    },
+    {
+      "group": "Chatbots",
+      "pages": [
         "examples/discord_bot",
         "examples/slack_bot",
         "examples/telegram_bot",
@@ -126,15 +183,34 @@
       ]
     },
     {
-      "group": "Community",
-      "pages": ["community/connect-with-us", "community/showcase"]
+      "group": "Showcase",
+      "pages": [
+        "examples/showcase"
+      ]
     },
     {
-      "group": "Integrations",
-      "pages": ["integration/langsmith"]
+      "group": "API Reference",
+      "pages": [
+        "api-reference/pipeline/overview",
+        {
+          "group": "Pipeline methods",
+          "pages": [
+            "api-reference/pipeline/add",
+            "api-reference/pipeline/query",
+            "api-reference/pipeline/chat",
+            "api-reference/pipeline/search",
+            "api-reference/pipeline/deploy",
+            "api-reference/pipeline/reset",
+            "api-reference/pipeline/delete"
+          ]
+        },
+        "api-reference/store/openai-assistant",
+        "api-reference/store/ai-assistants",
+        "api-reference/advanced/configuration"
+      ]
     },
     {
-      "group": "Contribute",
+      "group": "Contributing",
       "pages": [
         "contribution/guidelines",
         "contribution/dev",
@@ -145,7 +221,9 @@
     },
     {
       "group": "Product",
-      "pages": ["product/release-notes"]
+      "pages": [
+        "product/release-notes"
+      ]
     }
   ],
   "footerSocials": {
@@ -173,5 +251,11 @@
   },
   "api": {
     "baseUrl": "http://localhost:8080"
-  }
-}
+  },
+  "redirects": [
+    {
+      "source": "/changelog/command-line",
+      "destination": "/get-started/introduction"
+    }
+  ]
+}

docs/platform/faq.mdx

@@ -0,0 +1,3 @@
+---
+title: 'FAQs'
+---

docs/platform/overview.mdx

@@ -0,0 +1,3 @@
+---
+title: 'Overview'
+---

docs/platform/quickstart.mdx

@@ -0,0 +1,3 @@
+---
+title: 'Quickstart'
+---