Bump version to 0.1.71 (#1223)

Co-authored-by: Deven Patel <deven298@yahoo.com>

README.md

@@ -39,7 +39,7 @@
 
 Embedchain is an Open Source RAG Framework that makes it easy to create and deploy AI apps. At its core, Embedchain follows the design principle of being *"Conventional but Configurable"* to serve both software engineers and machine learning engineers.
 
-Embedchain streamlines the creation of RAG applications, offering a seamless process for managing various types of unstructured data. It efficiently segments data into manageable chunks, generates relevant embeddings, and stores them in a vector database for optimized retrieval. With a suite of diverse APIs, it enables users to extract contextual information, find precise answers, or engage in interactive chat conversations, all tailored to their own data.
+Embedchain streamlines the creation of Retrieval-Augmented Generation (RAG) applications, offering a seamless process for managing various types of unstructured data. It efficiently segments data into manageable chunks, generates relevant embeddings, and stores them in a vector database for optimized retrieval. With a suite of diverse APIs, it enables users to extract contextual information, find precise answers, or engage in interactive chat conversations, all tailored to their own data.
 
 ## 🔧 Quick install
 

configs/vllm.yaml

@@ -0,0 +1,14 @@
+llm:
+  provider: vllm
+  config:
+    model: 'meta-llama/Llama-2-70b-hf'
+    temperature: 0.5
+    top_p: 1
+    top_k: 10
+    stream: true
+    trust_remote_code: true
+
+embedder:
+  provider: huggingface
+  config:
+    model: 'BAAI/bge-small-en-v1.5'

docs/_snippets/get-help.mdx

@@ -2,7 +2,7 @@
   <Card title="Talk to founders" icon="calendar" href="https://cal.com/taranjeetio/ec">
   Schedule a call
   </Card>
-  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
     Join our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">

docs/_snippets/missing-data-source-tip.mdx

@@ -4,7 +4,7 @@
   <Card title="Google Form" icon="file" href="https://forms.gle/NDRCKsRpUHsz2Wcm8" color="#7387d0">
     Fill out this form
   </Card>
-  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
     Let us know on our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">

docs/_snippets/missing-llm-tip.mdx

@@ -1,7 +1,7 @@
 <p>If you can't find the specific LLM you need, no need to fret. We're continuously expanding our support for additional LLMs, and you can help us prioritize by opening an issue on our GitHub or simply reaching out to us on our Slack or Discord community.</p>
 
 <CardGroup cols={2}>
-  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
     Let us know on our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">

docs/_snippets/missing-vector-db-tip.mdx

@@ -3,7 +3,7 @@
 <p>If you can't find the specific vector database, please feel free to request through one of the following channels and help us prioritize.</p>
 
 <CardGroup cols={2}>
-  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
     Let us know on our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">

docs/api-reference/advanced/configuration.mdx

@@ -8,7 +8,7 @@ You can configure different components of your app (`llm`, `embedding model`, or
 
 
 <Tip>
-Embedchain applications are configurable using YAML file, JSON file or by directly passing the config dictionary. Checkout the [docs here](/api-reference/pipeline/overview#usage) on how to use other formats.
+Embedchain applications are configurable using YAML file, JSON file or by directly passing the config dictionary. Checkout the [docs here](/api-reference/app/overview#usage) on how to use other formats.
 </Tip>
 
 <CodeGroup>
@@ -198,11 +198,12 @@ Alright, let's dive into what each key means in the yaml config above:
         - `max_tokens` (Integer): Controls how many tokens are used in the response.
         - `top_p` (Float): Controls the diversity of word selection. A higher value (closer to 1) makes word selection more diverse.
         - `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
-        - `prompt` (String): A prompt for the model to follow when generating responses, requires $context and $query variables.
+        - `prompt` (String): A prompt for the model to follow when generating responses, requires `$context` and `$query` variables.
         - `system_prompt` (String): A system prompt for the model to follow when generating responses, in this case, it's set to the style of William Shakespeare.
-        -  `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
+        - `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
         - `number_documents` (Integer): Number of documents to pull from the vectordb as context, defaults to 1
         - `api_key` (String): The API key for the language model.
+        - `model_kwargs` (Dict): Keyword arguments to pass to the language model. Used for `aws_bedrock` provider, since it requires different arguments for each model.
 3. `vectordb` Section:
     - `provider` (String): The provider for the vector database, set to 'chroma'. You can find the full list of vector database providers in [our docs](/components/vector-databases).
     - `config`:
@@ -214,7 +215,11 @@ Alright, let's dive into what each key means in the yaml config above:
     - `provider` (String): The provider for the embedder, set to 'openai'. You can find the full list of embedding model providers in [our docs](/components/embedding-models).
     - `config`:
         - `model` (String): The specific model used for text embedding, 'text-embedding-ada-002'.
+        - `vector_dimension` (Integer): The vector dimension of the embedding model. [Defaults](https://github.com/embedchain/embedchain/blob/e572b5a3dc1b66f1e9b3357d11a88c63b5ce06e3/embedchain/models/vector_dimensions.py)
         - `api_key` (String): The API key for the embedding model.
+        - `deployment_name` (String): The deployment name for the embedding model.
+        - `title` (String): The title for the embedding model for Google Embedder.
+        - `task_type` (String): The task type for the embedding model for Google Embedder.
 5. `chunker` Section:
     - `chunk_size` (Integer): The size of each chunk of text that is sent to the language model.
     - `chunk_overlap` (Integer): The amount of overlap between each chunk of text.

docs/api-reference/app/delete.mdx

@@ -0,0 +1,48 @@
+---
+title: 🗑 delete
+---
+
+## Delete Document
+
+`delete()` method allows you to delete a document previously added to the app.
+
+### Usage
+
+```python
+from embedchain import App
+
+app = App()
+
+forbes_doc_id = app.add("https://www.forbes.com/profile/elon-musk")
+wiki_doc_id = app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+
+app.delete(forbes_doc_id)   # deletes the forbes document
+```
+
+<Note>
+    If you do not have the document id, you can use `app.db.get()` method to get the document and extract the `hash` key from `metadatas` dictionary object, which serves as the document id.
+</Note>
+
+
+## Delete Chat Session History
+
+`delete_session_chat_history()` method allows you to delete all previous messages in a chat history.
+
+### Usage
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add("https://www.forbes.com/profile/elon-musk")
+
+app.chat("What is the net worth of Elon Musk?")
+
+app.delete_session_chat_history()
+```
+
+<Note>
+    `delete_session_chat_history(session_id="session_1")` method also accepts `session_id` optional param for deleting chat history of a specific session.
+    It assumes the default session if no `session_id` is provided.
+</Note>

docs/api-reference/app/evaluate.mdx

@@ -0,0 +1,41 @@
+---
+title: '📝 evaluate'
+---
+
+`evaluate()` method is used to evaluate the performance of a RAG app. You can find the signature below:
+
+### Parameters
+
+<ParamField path="question" type="Union[str, list[str]]">
+    A question or a list of questions to evaluate your app on.
+</ParamField>
+<ParamField path="metrics" type="Optional[list[Union[BaseMetric, str]]]" optional>
+    The metrics to evaluate your app on. Defaults to all metrics: `["context_relevancy", "answer_relevancy", "groundedness"]`
+</ParamField>
+<ParamField path="num_workers" type="int" optional>
+    Specify the number of threads to use for parallel processing.
+</ParamField>
+
+### Returns
+
+<ResponseField name="metrics" type="dict">
+    Returns the metrics you have chosen to evaluate your app on as a dictionary.
+</ResponseField>
+
+## Usage
+
+```python
+from embedchain import App
+
+app = App()
+
+# add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# run evaluation
+app.evaluate("what is the net worth of Elon Musk?")
+# {'answer_relevancy': 0.958019958036268, 'context_relevancy': 0.12903225806451613}
+
+# or
+# app.evaluate(["what is the net worth of Elon Musk?", "which companies does Elon Musk own?"])
+```

docs/api-reference/app/get.mdx

@@ -0,0 +1,33 @@
+---
+title: 📄 get
+---
+
+## Get data sources
+
+`get_data_sources()` returns a list of all the data sources added in the app.
+
+
+### Usage
+
+```python
+from embedchain import App
+
+app = App()
+
+app.add("https://www.forbes.com/profile/elon-musk")
+app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+
+data_sources = app.get_data_sources()
+# [
+#   {
+#       'data_type': 'web_page',
+#       'data_value': 'https://en.wikipedia.org/wiki/Elon_Musk',
+#       'metadata': 'null'
+#   },
+#   {
+#       'data_type': 'web_page',
+#       'data_value': 'https://www.forbes.com/profile/elon-musk',
+#       'metadata': 'null'
+#   }
+# ]
+```

docs/api-reference/app/overview.mdx

@@ -1,34 +1,34 @@
 ---
-title: "Pipeline"
+title: "App"
 ---
 
-Create a RAG pipeline object on Embedchain. This is the main entrypoint for a developer to interact with Embedchain APIs. A pipeline configures the llm, vector database, embedding model, and retrieval strategy of your choice.
+Create a RAG app object on Embedchain. This is the main entrypoint for a developer to interact with Embedchain APIs. An app configures the llm, vector database, embedding model, and retrieval strategy of your choice.
 
 ### Attributes
 
 <ParamField path="local_id" type="str">
-    Pipeline ID
+    App ID
 </ParamField>
 <ParamField path="name" type="str" optional>
-    Name of the pipeline
+    Name of the app
 </ParamField>
 <ParamField path="config" type="BaseConfig">
-    Configuration of the pipeline
+    Configuration of the app
 </ParamField>
 <ParamField path="llm" type="BaseLlm">
-    Configured LLM for the RAG pipeline
+    Configured LLM for the RAG app
 </ParamField>
 <ParamField path="db" type="BaseVectorDB">
-    Configured vector database for the RAG pipeline
+    Configured vector database for the RAG app
 </ParamField>
 <ParamField path="embedding_model" type="BaseEmbedder">
-    Configured embedding model for the RAG pipeline
+    Configured embedding model for the RAG app
 </ParamField>
 <ParamField path="chunker" type="ChunkerConfig">
     Chunker configuration
 </ParamField>
 <ParamField path="client" type="Client" optional>
-    Client object (used to deploy a pipeline to Embedchain platform)
+    Client object (used to deploy an app to Embedchain platform)
 </ParamField>
 <ParamField path="logger" type="logging.Logger">
     Logger object
@@ -36,7 +36,7 @@ Create a RAG pipeline object on Embedchain. This is the main entrypoint for a de
 
 ## Usage
 
-You can create an embedchain pipeline instance using the following methods:
+You can create an app instance using the following methods:
 
 ### Default setting
 
@@ -127,4 +127,4 @@ app = App.from_config(config_path="config.json")
 }
 ```
 
-</CodeGroup>
+</CodeGroup>

docs/api-reference/pipeline/delete.mdx

@@ -1,19 +0,0 @@
----
-title: 🗑 delete
----
-
-`delete_chat_history()` method allows you to delete all previous messages in a chat history.
-
-## Usage
-
-```python
-from embedchain import App
-
-app = App()
-
-app.add("https://www.forbes.com/profile/elon-musk")
-
-app.chat("What is the net worth of Elon Musk?")
-
-app.delete_chat_history()
-```

docs/community/connect-with-us.mdx

@@ -8,7 +8,7 @@ We believe in building a vibrant and supportive community around embedchain. The
   <Card title="Twitter" icon="twitter" href="https://twitter.com/embedchain">
     Follow us on Twitter
   </Card>
-  <Card title="Slack" icon="slack" href="https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw" color="#4A154B">
+  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
     Join our slack community
   </Card>
   <Card title="Discord" icon="discord" href="https://discord.gg/6PzXDgEjG5" color="#7289DA">

docs/components/data-sources/csv.mdx

@@ -2,18 +2,27 @@
 title: '📊 CSV'
 ---
 
-To add any csv file, use the data_type as `csv`. `csv` allows remote urls and conventional file paths. Headers are included for each line, so if you have an `age` column, `18` will be added as `age: 18`. Eg:
+You can load any csv file from your local file system or through a URL. Headers are included for each line, so if you have an `age` column, `18` will be added as `age: 18`.
+
+## Usage
+
+### Load from a local file
 
 ```python
 from embedchain import App
-
 app = App()
-app.add('https://people.sc.fsu.edu/~jburkardt/data/csv/airtravel.csv', data_type="csv")
-# Or add using the local file path
-# app.add('/path/to/file.csv', data_type="csv")
-
-app.query("Summarize the air travel data")
-# Answer: The air travel data shows the number of flights for the months of July in the years 1958, 1959, and 1960. In July 1958, there were 491 flights, in July 1959 there were 548 flights, and in July 1960 there were 622 flights.
+app.add('/path/to/file.csv', data_type='csv')
 ```
 
-Note: There is a size limit allowed for csv file beyond which it can throw error. This limit is set by the LLMs. Please consider chunking large csv files into smaller csv files.
+### Load from URL
+
+```python
+from embedchain import App
+app = App()
+app.add('https://people.sc.fsu.edu/~jburkardt/data/csv/airtravel.csv', data_type="csv")
+```
+
+<Note>
+There is a size limit allowed for csv file beyond which it can throw error. This limit is set by the LLMs. Please consider chunking large csv files into smaller csv files.
+</Note>
+

docs/components/data-sources/dropbox.mdx

@@ -10,6 +10,16 @@ Ensure your app has the following settings activated:
 
 - In the Permissions section, enable `files.content.read` and `files.metadata.read`.
 
+## Usage
+
+Install the `dropbox` pypi package:
+
+```bash
+pip install dropbox
+```
+
+Following is an example of how to use the dropbox loader:
+
 ```python
 import os
 from embedchain import Pipeline as App

docs/components/data-sources/pdf-file.mdx

@@ -1,18 +1,28 @@
 ---
-title: '📰 PDF file'
+title: '📰 PDF'
 ---
 
-To add any pdf file, use the data_type as `pdf_file`. Eg:
+You can load any pdf file from your local file system or through a URL.
+
+## Usage
+
+### Load from a local file
 
 ```python
 from embedchain import App
-
 app = App()
+app.add('/path/to/file.pdf', data_type='pdf_file')
+```
 
+### Load from URL
+
+```python
+from embedchain import App
+app = App()
 app.add('https://arxiv.org/pdf/1706.03762.pdf', data_type='pdf_file')
 app.query("What is the paper 'attention is all you need' about?", citations=True)
 # Answer: The paper "Attention Is All You Need" proposes a new network architecture called the Transformer, which is based solely on attention mechanisms. It suggests that complex recurrent or convolutional neural networks can be replaced with a simpler architecture that connects the encoder and decoder through attention. The paper discusses how this approach can improve sequence transduction models, such as neural machine translation.
-# Contexts: 
+# Contexts:
 # [
 #     (
 #         'Provided proper attribution is ...',
@@ -23,25 +33,11 @@ app.query("What is the paper 'attention is all you need' about?", citations=True
 #             ...
 #         }
 #     ),
-#     (
-#         'Attention Visualizations Input ...',
-#         {
-#             'page': 12,
-#             'url': 'https://arxiv.org/pdf/1706.03762.pdf',
-#             'score': 0.41679039679873736,
-#             ...
-#         }
-#     ),
-#     (
-#         'sequence learning ...',
-#         {
-#             'page': 10,
-#             'url': 'https://arxiv.org/pdf/1706.03762.pdf',
-#             'score': 0.4188303600897153,
-#             ...
-#         }
-#     )
 # ]
 ```
 
-Note that we do not support password protected pdfs.
+We also store the page number under the key `page` with each chunk that helps understand where the answer is coming from. You can fetch the `page` key while during retrieval (refer to the example given above).
+
+<Note>
+Note that we do not support password protected pdf files.
+</Note>

docs/components/data-sources/youtube-channel.mdx

@@ -2,15 +2,17 @@
 title: '📽️ Youtube Channel'
 ---
 
-To add all the videos from a youtube channel to your app, use the data_type as `youtube_channel`.
+## Setup
 
-<Note>
 Make sure you have all the required packages installed before using this data type. You can install them by running the following command in your terminal.
 
 ```bash
 pip install -u "embedchain[youtube]"
 ```
-</Note>
+
+## Usage
+
+To add all the videos from a youtube channel to your app, use the data_type as `youtube_channel`.
 
 ```python
 from embedchain import App

docs/components/data-sources/youtube-video.mdx

@@ -2,6 +2,16 @@
 title: '📺 Youtube Video'
 ---
 
+## Setup
+
+Make sure you have all the required packages installed before using this data type. You can install them by running the following command in your terminal.
+
+```bash
+pip install -u "embedchain[youtube]"
+```
+
+## Usage
+
 To add any youtube video to your app, use the data_type as `youtube_video`. Eg:
 
 ```python

docs/components/embedding-models.mdx

@@ -40,7 +40,27 @@ app.query("What is OpenAI?")
 embedder:
   provider: openai
   config:
-    model: 'text-embedding-ada-002'
+    model: 'text-embedding-3-small'
+```
+
+</CodeGroup>
+
+* OpenAI announced two new embedding models: `text-embedding-3-small` and `text-embedding-3-large`. Embedchain supports both these models. Below you can find YAML config for both:
+
+<CodeGroup>
+
+```yaml text-embedding-3-small.yaml
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-3-small'
+```
+
+```yaml text-embedding-3-large.yaml
+embedder:
+  provider: openai
+  config:
+    model: 'text-embedding-3-large'
 ```
 
 </CodeGroup>

docs/components/evaluation.mdx

@@ -0,0 +1,275 @@
+---
+title: 🔬 Evaluation
+---
+
+## Overview
+
+We provide out-of-the-box evaluation metrics for your RAG application. You can use them to evaluate your RAG applications and compare against different settings of your production RAG application.
+
+Currently, we provide support for following evaluation metrics:
+
+<CardGroup cols={3}>
+    <Card title="Context Relevancy" href="#context_relevancy"></Card>
+    <Card title="Answer Relevancy" href="#answer_relevancy"></Card>
+    <Card title="Groundedness" href="#groundedness"></Card>
+    <Card title="Custom Metric" href="#custom_metric"></Card>
+</CardGroup>
+
+## Quickstart
+
+Here is a basic example of running evaluation:
+
+```python example.py
+from embedchain import App
+
+app = App()
+
+# Add data sources
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Run evaluation
+app.evaluate(["What is the net worth of Elon Musk?", "How many companies Elon Musk owns?"])
+# {'answer_relevancy': 0.9987286412340826, 'groundedness': 1.0, 'context_relevancy': 0.3571428571428571}
+```
+
+Under the hood, Embedchain does the following:
+
+1. Runs semantic search in the vector database and fetches context
+2. LLM call with question, context to fetch the answer
+3. Run evaluation on following metrics: `context relevancy`, `groundedness`, and `answer relevancy` and return result
+
+## Advanced Usage
+
+We use OpenAI's `gpt-4` model as default LLM model for automatic evaluation. Hence, we require you to set `OPENAI_API_KEY` as an environment variable.
+
+### Step-1: Create dataset
+
+In order to evaluate your RAG application, you have to setup a dataset. A data point in the dataset consists of `questions`, `contexts`, `answer`. Here is an example of how to create a dataset for evaluation:
+
+```python
+from embedchain.utils.eval import EvalData
+
+data = [
+    {
+        "question": "What is the net worth of Elon Musk?",
+        "contexts": [
+            "Elon Musk PROFILEElon MuskCEO, ...",
+            "a Twitter poll on whether the journalists' ...",
+            "2016 and run by Jared Birchall.[335]...",
+        ],
+        "answer": "As of the information provided, Elon Musk's net worth is $241.6 billion.",
+    },
+    {
+        "question": "which companies does Elon Musk own?",
+        "contexts": [
+            "of December 2023[update], ...",
+            "ThielCofounderView ProfileTeslaHolds ...",
+            "Elon Musk PROFILEElon MuskCEO, ...",
+        ],
+        "answer": "Elon Musk owns several companies, including Tesla, SpaceX, Neuralink, and The Boring Company.",
+    },
+]
+
+dataset = []
+
+for d in data:
+    eval_data = EvalData(question=d["question"], contexts=d["contexts"], answer=d["answer"])
+    dataset.append(eval_data)
+```
+
+### Step-2: Run evaluation
+
+Once you have created your dataset, you can run evaluation on the dataset by picking the metric you want to run evaluation on.
+
+For example, you can run evaluation on context relevancy metric using the following code:
+
+```python
+from embedchain.evaluation.metrics import ContextRelevance
+metric = ContextRelevance()
+score = metric.evaluate(dataset)
+print(score)
+```
+
+You can choose a different metric or write your own to run evaluation on. You can check the following links:
+
+- [Context Relevancy](#context_relevancy)
+- [Answer relenvancy](#answer_relevancy)
+- [Groundedness](#groundedness)
+- [Build your own metric](#custom_metric)
+
+## Metrics
+
+### Context Relevancy <a id="context_relevancy"></a>
+
+Context relevancy is a metric to determine "how relevant the context is to the question". We use OpenAI's `gpt-4` model to determine the relevancy of the context. We achieve this by prompting the model with the question and the context and asking it to return relevant sentences from the context. We then use the following formula to determine the score:
+
+```
+context_relevance_score = num_relevant_sentences_in_context / num_of_sentences_in_context
+```
+
+#### Examples
+
+You can run the context relevancy evaluation with the following simple code:
+
+```python
+from embedchain.evaluation.metrics import ContextRelevance
+
+metric = ContextRelevance()
+score = metric.evaluate(dataset)  # 'dataset' is definted in the create dataset section
+print(score)
+# 0.27975528364849833
+```
+
+In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `ContextRelevanceConfig` class.
+
+Here is a more advanced example of how to pass a custom evaluation config for evaluating on context relevance metric:
+
+```python
+from embedchain.config.evaluation.base import ContextRelevanceConfig
+from embedchain.evaluation.metrics import ContextRelevance
+
+eval_config = ContextRelevanceConfig(model="gpt-4", api_key="sk-xxx", language="en")
+metric = ContextRelevance(config=eval_config)
+metric.evaluate(dataset)
+```
+
+#### `ContextRelevanceConfig`
+
+<ParamField path="model" type="str" optional>
+    The model to use for the evaluation. Defaults to `gpt-4`. We only support openai's models for now.
+</ParamField>
+<ParamField path="api_key" type="str" optional>
+    The openai api key to use for the evaluation. Defaults to `None`. If not provided, we will use the `OPENAI_API_KEY` environment variable.
+</ParamField>
+<ParamField path="language" type="str" optional>
+    The language of the dataset being evaluated. We need this to determine the understand the context provided in the dataset. Defaults to `en`.
+</ParamField>
+<ParamField path="prompt" type="str" optional>
+    The prompt to extract the relevant sentences from the context. Defaults to `CONTEXT_RELEVANCY_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.
+</ParamField>
+
+
+### Answer Relevancy <a id="answer_relevancy"></a>
+
+Answer relevancy is a metric to determine how relevant the answer is to the question. We prompt the model with the answer and asking it to generate questions from the answer. We then use the cosine similarity between the generated questions and the original question to determine the score.
+
+```
+answer_relevancy_score = mean(cosine_similarity(generated_questions, original_question))
+```
+
+#### Examples
+
+You can run the answer relevancy evaluation with the following simple code:
+
+```python
+from embedchain.evaluation.metrics import AnswerRelevance
+
+metric = AnswerRelevance()
+score = metric.evaluate(dataset)
+print(score)
+# 0.9505334177461916
+```
+
+In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `AnswerRelevanceConfig` class. Here is a more advanced example where you can provide your own evaluation config:
+
+```python
+from embedchain.config.evaluation.base import AnswerRelevanceConfig
+from embedchain.evaluation.metrics import AnswerRelevance
+
+eval_config = AnswerRelevanceConfig(
+    model='gpt-4',
+    embedder="text-embedding-ada-002",
+    api_key="sk-xxx",
+    num_gen_questions=2
+)
+metric = AnswerRelevance(config=eval_config)
+score = metric.evaluate(dataset)
+```
+
+#### `AnswerRelevanceConfig`
+
+<ParamField path="model" type="str" optional>
+    The model to use for the evaluation. Defaults to `gpt-4`. We only support openai's models for now.
+</ParamField>
+<ParamField path="embedder" type="str" optional>
+    The embedder to use for embedding the text. Defaults to `text-embedding-ada-002`. We only support openai's embedders for now.
+</ParamField>
+<ParamField path="api_key" type="str" optional>
+    The openai api key to use for the evaluation. Defaults to `None`. If not provided, we will use the `OPENAI_API_KEY` environment variable.
+</ParamField>
+<ParamField path="num_gen_questions" type="int" optional>
+    The number of questions to generate for each answer. We use the generated questions to compare the similarity with the original question to determine the score. Defaults to `1`.
+</ParamField>
+<ParamField path="prompt" type="str" optional>
+    The prompt to extract the `num_gen_questions` number of questions from the provided answer. Defaults to `ANSWER_RELEVANCY_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.
+</ParamField>
+
+## Groundedness <a id="groundedness"></a>
+
+Groundedness is a metric to determine how grounded the answer is to the context. We use OpenAI's `gpt-4` model to determine the groundedness of the answer. We achieve this by prompting the model with the answer and asking it to generate claims from the answer. We then again prompt the model with the context and the generated claims to determine the verdict on the claims. We then use the following formula to determine the score:
+
+```
+groundedness_score = (sum of all verdicts) / (total # of claims)
+```
+
+You can run the groundedness evaluation with the following simple code:
+
+```python
+from embedchain.evaluation.metrics import Groundedness
+metric = Groundedness()
+score = metric.evaluate(dataset)    # dataset from above
+print(score)
+# 1.0
+```
+
+In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `GroundednessConfig` class. Here is a more advanced example where you can configure the evaluation config:
+
+```python
+from embedchain.config.evaluation.base import GroundednessConfig
+from embedchain.evaluation.metrics import Groundedness
+
+eval_config = GroundednessConfig(model='gpt-4', api_key="sk-xxx")
+metric = Groundedness(config=eval_config)
+score = metric.evaluate(dataset)
+```
+
+
+#### `GroundednessConfig`
+
+<ParamField path="model" type="str" optional>
+    The model to use for the evaluation. Defaults to `gpt-4`. We only support openai's models for now.
+</ParamField>
+<ParamField path="api_key" type="str" optional>
+    The openai api key to use for the evaluation. Defaults to `None`. If not provided, we will use the `OPENAI_API_KEY` environment variable.
+</ParamField>
+<ParamField path="answer_claims_prompt" type="str" optional>
+    The prompt to extract the claims from the provided answer. Defaults to `GROUNDEDNESS_ANSWER_CLAIMS_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.
+</ParamField>
+<ParamField path="claims_inference_prompt" type="str" optional>
+    The prompt to get verdicts on the claims from the answer from the given context. Defaults to `GROUNDEDNESS_CLAIMS_INFERENCE_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.
+</ParamField>
+
+## Custom <a id="custom_metric"></a>
+
+You can also create your own evaluation metric by extending the `BaseMetric` class. You can find the source code for the existing metrics at `embedchain.evaluation.metrics` path.
+
+<Note>
+You must provide the `name` of your custom metric in the `__init__` method of your class. This name will be used to identify your metric in the evaluation report.
+</Note>
+
+```python
+from typing import Optional
+
+from embedchain.config.base_config import BaseConfig
+from embedchain.evaluation.metrics import BaseMetric
+from embedchain.utils.eval import EvalData
+
+class MyCustomMetric(BaseMetric):
+    def __init__(self, config: Optional[BaseConfig] = None):
+        super().__init__(name="my_custom_metric")
+
+    def evaluate(self, dataset: list[EvalData]):
+        score = 0.0
+        # write your evaluation logic here
+        return score
+```

docs/components/llms.mdx

@@ -14,11 +14,14 @@ Embedchain comes with built-in support for various popular large language models
   <Card title="Cohere" href="#cohere"></Card>
   <Card title="Together" href="#together"></Card>
   <Card title="Ollama" href="#ollama"></Card>
+  <Card title="vLLM" href="#vllm"></Card>
   <Card title="GPT4All" href="#gpt4all"></Card>
   <Card title="JinaChat" href="#jinachat"></Card>
   <Card title="Hugging Face" href="#hugging-face"></Card>
   <Card title="Llama2" href="#llama2"></Card>
   <Card title="Vertex AI" href="#vertex-ai"></Card>
+  <Card title="Mistral AI" href="#mistral-ai"></Card>
+  <Card title="AWS Bedrock" href="#aws-bedrock"></Card>
 </CardGroup>
 
 ## OpenAI
@@ -249,7 +252,7 @@ app = App.from_config(config_path="config.yaml")
 llm:
   provider: azure_openai
   config:
-    model: gpt-35-turbo
+    model: gpt-3.5-turbo
     deployment_name: your_llm_deployment_name
     temperature: 0.5
     max_tokens: 1000
@@ -393,6 +396,35 @@ llm:
 
 </CodeGroup>
 
+
+## vLLM
+
+Setup vLLM by following instructions given in [their docs](https://docs.vllm.ai/en/latest/getting_started/installation.html).
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: vllm
+  config:
+    model: 'meta-llama/Llama-2-70b-hf'
+    temperature: 0.5
+    top_p: 1
+    top_k: 10
+    stream: true
+    trust_remote_code: true
+```
+
+</CodeGroup>
+
 ## GPT4ALL
 
 Install related dependencies using the following command:
@@ -494,6 +526,49 @@ llm:
 ```
 </CodeGroup>
 
+### Custom Endpoints
+
+
+You can also use [Hugging Face Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index#-inference-endpoints) to access custom endpoints. First, set the `HUGGINGFACE_ACCESS_TOKEN` as above.
+
+Then, load the app using the config yaml file:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "xxx"
+
+# load llm configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: huggingface
+  config:
+    endpoint: https://api-inference.huggingface.co/models/gpt2 # replace with your personal endpoint
+```
+</CodeGroup>
+
+If your endpoint requires additional parameters, you can pass them in the `model_kwargs` field:
+
+```
+llm:
+  provider: huggingface
+  config:
+    endpoint: <YOUR_ENDPOINT_URL_HERE>
+    model_kwargs:
+      max_new_tokens: 100
+      temperature: 0.5
+```
+
+Currently only supports `text-generation` and `text2text-generation` for now [[ref](https://api.python.langchain.com/en/latest/llms/langchain_community.llms.huggingface_endpoint.HuggingFaceEndpoint.html?highlight=huggingfaceendpoint#)].
+
+See langchain's [hugging face endpoint](https://python.langchain.com/docs/integrations/chat/huggingface#huggingfaceendpoint) for more information. 
+
 ## Llama2
 
 Llama2 is integrated through [Replicate](https://replicate.com/).  Set `REPLICATE_API_TOKEN` in environment variable which you can obtain from [their platform](https://replicate.com/account/api-tokens).
@@ -547,5 +622,84 @@ llm:
 ```
 </CodeGroup>
 
+
+## Mistral AI
+
+Obtain the Mistral AI api key from their [console](https://console.mistral.ai/).
+
+<CodeGroup>
+ 
+ ```python main.py
+os.environ["MISTRAL_API_KEY"] = "xxx"
+
+app = App.from_config(config_path="config.yaml")
+
+app.add("https://www.forbes.com/profile/elon-musk")
+
+response = app.query("what is the net worth of Elon Musk?")
+# As of January 16, 2024, Elon Musk's net worth is $225.4 billion.
+
+response = app.chat("which companies does elon own?")
+# Elon Musk owns Tesla, SpaceX, Boring Company, Twitter, and X.
+
+response = app.chat("what question did I ask you already?")
+# You have asked me several times already which companies Elon Musk owns, specifically Tesla, SpaceX, Boring Company, Twitter, and X.
+```
+  
+```yaml config.yaml
+llm:
+  provider: mistralai
+  config:
+    model: mistral-tiny
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+embedder:
+  provider: mistralai
+  config:
+    model: mistral-embed
+```
+</CodeGroup>
+
+
+## AWS Bedrock
+
+### Setup
+- Before using the AWS Bedrock LLM, make sure you have the appropriate model access from [Bedrock Console](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/modelaccess).
+- You will also need `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to authenticate the API with AWS. You can find these in your [AWS Console](https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/users).
+
+
+### Usage
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ["AWS_ACCESS_KEY_ID"] = "xxx"
+os.environ["AWS_SECRET_ACCESS_KEY"] = "xxx"
+
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+llm:
+  provider: aws_bedrock
+  config:
+    model: amazon.titan-text-express-v1
+    # check notes below for model_kwargs
+    model_kwargs:
+      temperature: 0.5
+      topP: 1
+      maxTokenCount: 1000
+```
+</CodeGroup>
+
+<br />
+<Note>
+  The model arguments are different for each providers. Please refer to the [AWS Bedrock Documentation](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/providers) to find the appropriate arguments for your model.
+</Note>
+
 <br/ >
 <Snippet file="missing-llm-tip.mdx" />

docs/components/vector-databases.mdx

@@ -167,7 +167,7 @@ Install pinecone related dependencies using the following command:
 pip install --upgrade 'embedchain[pinecone]'
 ```
 
-In order to use Pinecone as vector database, set the environment variables `PINECONE_API_KEY` and `PINECONE_ENV` which you can find on [Pinecone dashboard](https://app.pinecone.io/).
+In order to use Pinecone as vector database, set the environment variable `PINECONE_API_KEY` which you can find on [Pinecone dashboard](https://app.pinecone.io/).
 
 <CodeGroup>
 
@@ -175,20 +175,46 @@ In order to use Pinecone as vector database, set the environment variables `PINE
 from embedchain import App
 
 # load pinecone configuration from yaml file
-app = App.from_config(config_path="config.yaml")
+app = App.from_config(config_path="pod_config.yaml")
+# or
+app = App.from_config(config_path="serverless_config.yaml")
 ```
 
-```yaml config.yaml
+```yaml pod_config.yaml
 vectordb:
   provider: pinecone
   config:
     metric: cosine
     vector_dimension: 1536
     collection_name: my-pinecone-index
+    pod_config:
+      environment: gcp-starter
+      metadata_config:
+        indexed:
+          - "url"
+          - "hash"
+```
+
+```yaml serverless_config.yaml
+vectordb:
+  provider: pinecone
+  config:
+    metric: cosine
+    vector_dimension: 1536
+    collection_name: my-pinecone-index
+    serverless_config:
+      cloud: aws
+      region: us-west-2
 ```
 
 </CodeGroup>
 
+<br />
+<Note>
+You can find more information about Pinecone configuration [here](https://docs.pinecone.io/docs/manage-indexes#create-a-pod-based-index).
+You can also optionally provide `index_name` as a config param in yaml file to specify the index name. If not provided, the index name will be `{collection_name}-{vector_dimension}`.
+</Note>
+
 ## Qdrant
 
 In order to use Qdrant as a vector database, set the environment variables `QDRANT_URL` and `QDRANT_API_KEY` which you can find on [Qdrant Dashboard](https://cloud.qdrant.io/).

docs/examples/nextjs-assistant.mdx

@@ -0,0 +1,124 @@
+Fork the Embedchain repo on [Github](https://github.com/embedchain/embedchain) to create your own NextJS discord and slack bot powered by Embedchain.
+
+If you run into problems with forking, please refer to [github docs](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) for forking a repo.
+
+We will work from the `examples/nextjs` folder so change your current working directory by running the command - `cd <your_forked_repo>/examples/nextjs`
+
+# Installation
+
+First, lets start by install all the required packages and dependencies.
+
+- Install all the required python packages by running ```pip install -r requirements.txt```
+
+- We will use [Fly.io](https://fly.io/) to deploy our embedchain app, discord and slack bot. Follow the step one to install [Fly.io CLI](https://docs.embedchain.ai/deployment/fly_io#step-1-install-flyctl-command-line)
+
+# Developement
+
+## Embedchain App
+
+First, we need an Embedchain app powered with the knowledge of NextJS. We have already created an embedchain app using FastAPI in `ec_app` folder for you. Feel free to ingest data of your choice to power the App.
+
+<Note>
+Navigate to `ec_app` folder and create `.env` file in this folder and set your OpenAI API key as shown in `.env.example` file. If you want to use other open-source models, feel free to use the app config in `app.py`. More details for using custom configuration for Embedchain app is [available here](https://docs.embedchain.ai/api-reference/advanced/configuration).
+</Note>
+
+Before running the ec commands to develope the app, open `fly.toml` file and update the `name` variable to something unique. This is important as `fly.io` requires users to provide a globally unique deployment app names.
+
+Now, we need to launch this application with fly.io. You can see your app on [fly.io dashboard](https://fly.io/dashboard). Run the following command to launch your app on fly.io:
+```bash
+fly launch --no-deploy
+```
+
+To run the app in development, run the following command:
+
+```bash
+ec dev
+```
+
+Run `ec deploy` to deploy your app on Fly.io. Once you deploy your app, save the endpoint on which our discord and slack bot will send requests.
+
+
+## Discord bot
+
+For discord bot, you will need to create the bot on discord developer portal and get the discord bot token and your discord bot name.
+
+While keeping in mind the following note, create the discord bot by following the instructions from our [discord bot docs](https://docs.embedchain.ai/examples/discord_bot) and get discord bot token.
+
+<Note>
+You do not need to set `OPENAI_API_KEY` to run this discord bot. Follow the remaining instructions to create a discord bot app. We recommend you to give the following sets of bot permissions to run the discord bot without errors:
+
+```
+(General Permissions)
+Read Message/View Channels
+
+(Text Permissions)
+Send Messages
+Create Public Thread
+Create Private Thread
+Send Messages in Thread
+Manage Threads
+Embed Links
+Read Message History
+```
+</Note>
+
+Once you have your discord bot token and discord app name. Navigate to `nextjs_discord` folder and create `.env` file and define your discord bot token, discord bot name and endpoint of your embedchain app as shown in `.env.example` file.
+
+To run the app in development:
+
+```bash
+python app.py
+```
+
+Before deploying the app, open `fly.toml` file and update the `name` variable to something unique. This is important as `fly.io` requires users to provide a globally unique deployment app names.
+
+Now, we need to launch this application with fly.io. You can see your app on [fly.io dashboard](https://fly.io/dashboard). Run the following command to launch your app on fly.io:
+```bash
+fly launch --no-deploy
+```
+
+Run `ec deploy` to deploy your app on Fly.io. Once you deploy your app, your discord bot will be live!
+
+
+## Slack bot
+
+For Slack bot, you will need to create the bot on slack developer portal and get the slack bot token and slack app token.
+
+### Setup
+
+- Create a workspace on Slack if you don't have one already by clicking [here](https://slack.com/intl/en-in/).
+- Create a new App on your Slack account by going [here](https://api.slack.com/apps).
+- Select `From Scratch`, then enter the Bot Name and select your workspace.
+- Go to `App Credentials` section on the `Basic Information` tab from the left sidebar, create your app token and save it in your `.env` file as `SLACK_APP_TOKEN`.
+- Go to `Socket Mode` tab from the left sidebar and enable the socket mode to listen to slack message from your workspace.
+- (Optional) Under the `App Home` tab you can change your App display name and default name.
+- Navigate to `Event Subscription` tab, and enable the event subscription so that we can listen to slack events.
+- Once you enable the event subscription, you will need to subscribe to bot events to authorize the bot to listen to app mention events of the bot. Do that by tapping on `Add Bot User Event` button and select `app_mention`.
+- On the left Sidebar, go to `OAuth and Permissions` and add the following scopes under `Bot Token Scopes`:
+```text
+app_mentions:read
+channels:history
+channels:read
+chat:write
+emoji:read
+reactions:write
+reactions:read
+```
+- Now select the option `Install to Workspace` and after it's done, copy the `Bot User OAuth Token` and set it in your `.env` file as `SLACK_BOT_TOKEN`.
+
+Once you have your slack bot token and slack app token. Navigate to `nextjs_slack` folder and create `.env` file and define your slack bot token, slack app token and endpoint of your embedchain app as shown in `.env.example` file.
+
+To run the app in development:
+
+```bash
+python app.py
+```
+
+Before deploying the app, open `fly.toml` file and update the `name` variable to something unique. This is important as `fly.io` requires users to provide a globally unique deployment app names.
+
+Now, we need to launch this application with fly.io. You can see your app on [fly.io dashboard](https://fly.io/dashboard). Run the following command to launch your app on fly.io:
+```bash
+fly launch --no-deploy
+```
+
+Run `ec deploy` to deploy your app on Fly.io. Once you deploy your app, your slack bot will be live!

docs/examples/showcase.mdx

@@ -20,7 +20,7 @@ Embedchain community has been super active in creating demos on top of Embedchai
 - [Create Instant ChatBot 🤖 using embedchain](https://databutton.com/v/h3e680h9) by Avra, ([Tweet](https://twitter.com/Avra_b/status/1674704745154641920/))
 - [JOBO 🤖 — The AI-driven sidekick to craft your resume](https://try-jobo.com/) by Enrico Willemse, ([LinkedIn Post](https://www.linkedin.com/posts/enrico-willemse_jobai-gptfun-embedchain-activity-7090340080879374336-ueLB/))
 - [Explore Your Knowledge Base: Interactive chats over various forms of documents](https://chatdocs.dkedar.com/) by Kedar Dabhadkar, ([LinkedIn Post](https://www.linkedin.com/posts/dkedar7_machinelearning-llmops-activity-7092524836639424513-2O3L/))
-- [Chatbot trained on 1000+ videos of Ester hicks the co-author behind the famous book Secret](https://ask-abraham.thoughtseed.repl.co) by Mohan Kumar
+- [Chatbot trained on 1000+ videos of Ester hicks the co-author behind the famous book Secret](https://askabraham.tokenofme.io/) by Mohan Kumar
 
 
 ## Templates

docs/examples/slack-AI.mdx

@@ -0,0 +1,67 @@
+[Embedchain Examples Repo](https://github.com/embedchain/examples) contains code on how to build your own Slack AI to chat with the unstructured data lying in your slack channels.
+
+![Slack AI Demo](/images/slack-ai.png)
+
+## Getting started
+
+Create a Slack AI involves 3 steps
+
+* Create slack user
+* Set environment variables
+* Run the app locally
+
+### Step 1: Create Slack user token
+
+Follow the steps given below to fetch your slack user token to get data through Slack APIs:
+
+1. Create a workspace on Slack if you don’t have one already by clicking [here](https://slack.com/intl/en-in/).
+2. Create a new App on your Slack account by going [here](https://api.slack.com/apps).
+3. Select `From Scratch`, then enter the App Name and select your workspace.
+4. Navigate to `OAuth & Permissions` tab from the left sidebar and go to the `scopes` section. Add the following scopes under `User Token Scopes`:
+
+    ```
+    # Following scopes are needed for reading channel history
+    channels:history
+    channels:read
+
+    # Following scopes are needed to fetch list of channels from slack
+    groups:read
+    mpim:read
+    im:read
+    ```
+
+5. Click on the `Install to Workspace` button under `OAuth Tokens for Your Workspace` section in the same page and install the app in your slack workspace.
+6. After installing the app you will see the `User OAuth Token`, save that token as you will need to configure it as `SLACK_USER_TOKEN` for this demo.
+
+### Step 2: Set environment variables
+
+Navigate to `api` folder and set your `HUGGINGFACE_ACCESS_TOKEN` and `SLACK_USER_TOKEN` in `.env.example` file. Then rename the `.env.example` file to `.env`.
+
+
+<Note>
+By default, we use `Mixtral` model from Hugging Face. However, if you prefer to use OpenAI model, then set `OPENAI_API_KEY` instead of `HUGGINGFACE_ACCESS_TOKEN` along with `SLACK_USER_TOKEN` in `.env` file, and update the code in `api/utils/app.py` file to use OpenAI model instead of Hugging Face model.
+</Note>
+
+### Step 3: Run app locally
+
+Follow the instructions given below to run app locally based on your development setup (with docker or without docker):
+
+#### With docker
+
+```bash
+docker-compose build
+ec start --docker
+```
+
+#### Without docker
+
+```bash
+ec install-reqs
+ec start
+```
+
+Finally, you will have the Slack AI frontend running on http://localhost:3000. You can also access the REST APIs on http://localhost:8000.
+
+## Credits
+
+This demo was built using the Embedchain's [full stack demo template](https://docs.embedchain.ai/get-started/full-stack). Follow the instructions [given here](https://docs.embedchain.ai/get-started/full-stack) to create your own full stack RAG application.

docs/get-started/full-stack.mdx

@@ -2,34 +2,61 @@
 title: '💻 Full stack'
 ---
 
-Embedchain provides a clean and simple cli utility that lets you create full-stack RAG applications locally with a single command. 
+Get started with full-stack RAG applications using Embedchain's easy-to-use CLI tool. Set up everything with just a few commands, whether you prefer Docker or not.
 
-## Prerequisite
+## Prerequisites
+
+Choose your setup method:
+
+### Without Docker
+
+Ensure these are installed:
 
-Make sure that you have installed the following:
 - Embedchain python package (`pip install embedchain`)
 - [Node.js](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) and [Yarn](https://classic.yarnpkg.com/lang/en/docs/install/)
 
-## Get started
+### With Docker
+
+Install Docker from [Docker's official website](https://docs.docker.com/engine/install/).
+
+## Quick Start Guide
+
+### Setting Up
 
 For the purpose of the demo, you have to set `OPENAI_API_KEY` to start with but you can choose any llm by changing the configuration easily.
 
-Now run the following command:
+### Installation Commands
 
-```bash
-ec runserver
+<CodeGroup>
+
+```bash without docker
+ec create-app my-app
+cd my-app
+ec start
 ```
 
-Once you run this command, Embedchain does the following:
+```bash with docker
+ec create-app my-app --docker
+cd my-app
+ec start --docker
+```
 
-1. Fetch full stack template that uses FastAPI for backend, and Next.JS template for frontend
-2. Install necessary requirements
-3. Launch the frontend and backend server for you to interact with.
+</CodeGroup>
 
-Once you are done, visit `http://localhost:3000` and you will see a chat UI as shown below.
+### What Happens Next?
+
+1. Embedchain fetches a full stack template (FastAPI backend, Next.JS frontend).
+2. Installs required components.
+3. Launches both frontend and backend servers.
+
+### See It In Action
+
+Open http://localhost:3000 to view the chat UI.
 
 ![full stack example](/images/fullstack.png)
 
-You can navigate to [Embedchain admin panel] where  you can see the chunks created for your documents that you ingested for your RAG application. Below is a screenshot for the same:
+### Admin Panel
+
+Check out the Embedchain admin panel to see the document chunks for your RAG application.
 
 ![full stack chunks](/images/fullstack-chunks.png)

docs/get-started/quickstart.mdx

@@ -5,7 +5,7 @@ description: '💡 Create a RAG app on your own data in a minute'
 
 ## Installation
 
-First install the python package.
+First install the Python package:
 
 ```bash
 pip install embedchain
@@ -47,7 +47,7 @@ app.query("What is the net worth of Elon Musk today?")
 llm:
   provider: huggingface
   config:
-    model: 'mistralai/Mistral-7B-v0.1'
+    model: 'mistralai/Mistral-7B-Instruct-v0.2'
     top_p: 0.5
 embedder:
   provider: huggingface
@@ -80,4 +80,4 @@ Now that you have created your first app, you can follow any of the links:
 * [Introduction](/get-started/introduction)
 * [Customization](/components/introduction)
 * [Use cases](/use-cases/introduction)
-* [Deployment](/get-started/deployment)
+* [Deployment](/get-started/deployment)

docs/mint.json

@@ -131,7 +131,8 @@
         },
         "components/llms",
         "components/vector-databases",
-        "components/embedding-models"
+        "components/embedding-models",
+        "components/evaluation"
       ]
     },
     {
@@ -174,7 +175,9 @@
         },
         "examples/full_stack",
         "examples/openai-assistant",
-        "examples/opensource-assistant"
+        "examples/opensource-assistant",
+        "examples/nextjs-assistant",
+        "examples/slack-AI"
       ]
     },
     {
@@ -196,17 +199,19 @@
     {
       "group": "API Reference",
       "pages": [
-        "api-reference/pipeline/overview",
+        "api-reference/app/overview",
         {
-          "group": "Pipeline methods",
+          "group": "App methods",
           "pages": [
-            "api-reference/pipeline/add",
-            "api-reference/pipeline/query",
-            "api-reference/pipeline/chat",
-            "api-reference/pipeline/search",
-            "api-reference/pipeline/deploy",
-            "api-reference/pipeline/reset",
-            "api-reference/pipeline/delete"
+            "api-reference/app/add",
+            "api-reference/app/query",
+            "api-reference/app/chat",
+            "api-reference/app/search",
+            "api-reference/app/get",
+            "api-reference/app/evaluate",
+            "api-reference/app/deploy",
+            "api-reference/app/reset",
+            "api-reference/app/delete"
           ]
         },
         "api-reference/store/openai-assistant",
@@ -234,7 +239,7 @@
   "footerSocials": {
     "website": "https://embedchain.ai",
     "github": "https://github.com/embedchain/embedchain",
-    "slack": "https://join.slack.com/t/embedchain/shared_invite/zt-22uwz3c46-Zg7cIh5rOBteT_xe1jwLDw",
+    "slack": "https://embedchain.ai/slack",
     "discord": "https://discord.gg/6PzXDgEjG5",
     "twitter": "https://twitter.com/embedchain",
     "linkedin": "https://www.linkedin.com/company/embedchain"

embedchain-js/README.md

@@ -178,7 +178,7 @@ await app.addLocal("qna_pair", ["Question", "Answer"]);
 
 ## Testing
 
-Before you consume valueable tokens, you should make sure that the embedding you have done works and that it's receiving the correct document from the database.
+Before you consume valuable tokens, you should make sure that the embedding you have done works and that it's receiving the correct document from the database.
 
 For this you can use the `dryRun` method.
 

embedchain/app.py

@@ -1,13 +1,15 @@
 import ast
+import concurrent.futures
 import json
 import logging
 import os
 import sqlite3
 import uuid
-from typing import Any, Dict, Optional
+from typing import Any, Optional, Union
 
 import requests
 import yaml
+from tqdm import tqdm
 
 from embedchain.cache import (Config, ExactMatchEvaluation,
                               SearchDistanceEvaluation, cache,
@@ -18,16 +20,20 @@ from embedchain.constants import SQLITE_PATH
 from embedchain.embedchain import EmbedChain
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.embedder.openai import OpenAIEmbedder
+from embedchain.evaluation.base import BaseMetric
+from embedchain.evaluation.metrics import (AnswerRelevance, ContextRelevance,
+                                           Groundedness)
 from embedchain.factory import EmbedderFactory, LlmFactory, VectorDBFactory
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm
 from embedchain.llm.openai import OpenAILlm
 from embedchain.telemetry.posthog import AnonymousTelemetry
+from embedchain.utils.evaluation import EvalData, EvalMetric
 from embedchain.utils.misc import validate_config
 from embedchain.vectordb.base import BaseVectorDB
 from embedchain.vectordb.chroma import ChromaDB
 
-# Setup the user directory if doesn't exist already
+# Set up the user directory if it doesn't exist already
 Client.setup_dir()
 
 
@@ -364,7 +370,7 @@ class App(EmbedChain):
     def from_config(
         cls,
         config_path: Optional[str] = None,
-        config: Optional[Dict[str, Any]] = None,
+        config: Optional[dict[str, Any]] = None,
         auto_deploy: bool = False,
         yaml_path: Optional[str] = None,
     ):
@@ -374,7 +380,7 @@ class App(EmbedChain):
         :param config_path: Path to the YAML or JSON configuration file.
         :type config_path: Optional[str]
         :param config: A dictionary containing the configuration.
-        :type config: Optional[Dict[str, Any]]
+        :type config: Optional[dict[str, Any]]
         :param auto_deploy: Whether to deploy the pipeline automatically, defaults to False
         :type auto_deploy: bool, optional
         :param yaml_path: (Deprecated) Path to the YAML configuration file. Use config_path instead.
@@ -393,7 +399,7 @@ class App(EmbedChain):
 
         if config_path:
             file_extension = os.path.splitext(config_path)[1]
-            with open(config_path, "r") as file:
+            with open(config_path, "r", encoding="UTF-8") as file:
                 if file_extension in [".yaml", ".yml"]:
                     config_data = yaml.safe_load(file)
                 elif file_extension == ".json":
@@ -455,3 +461,103 @@ class App(EmbedChain):
             chunker=chunker_config_data,
             cache_config=cache_config,
         )
+
+    def _eval(self, dataset: list[EvalData], metric: Union[BaseMetric, str]):
+        """
+        Evaluate the app on a dataset for a given metric.
+        """
+        metric_str = metric.name if isinstance(metric, BaseMetric) else metric
+        eval_class_map = {
+            EvalMetric.CONTEXT_RELEVANCY.value: ContextRelevance,
+            EvalMetric.ANSWER_RELEVANCY.value: AnswerRelevance,
+            EvalMetric.GROUNDEDNESS.value: Groundedness,
+        }
+
+        if metric_str in eval_class_map:
+            return eval_class_map[metric_str]().evaluate(dataset)
+
+        # Handle the case for custom metrics
+        if isinstance(metric, BaseMetric):
+            return metric.evaluate(dataset)
+        else:
+            raise ValueError(f"Invalid metric: {metric}")
+
+    def evaluate(
+        self,
+        questions: Union[str, list[str]],
+        metrics: Optional[list[Union[BaseMetric, str]]] = None,
+        num_workers: int = 4,
+    ):
+        """
+        Evaluate the app on a question.
+
+        param: questions: A question or a list of questions to evaluate.
+        type: questions: Union[str, list[str]]
+        param: metrics: A list of metrics to evaluate. Defaults to all metrics.
+        type: metrics: Optional[list[Union[BaseMetric, str]]]
+        param: num_workers: Number of workers to use for parallel processing.
+        type: num_workers: int
+        return: A dictionary containing the evaluation results.
+        rtype: dict
+        """
+        if "OPENAI_API_KEY" not in os.environ:
+            raise ValueError("Please set the OPENAI_API_KEY environment variable with permission to use `gpt4` model.")
+
+        queries, answers, contexts = [], [], []
+        if isinstance(questions, list):
+            with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor:
+                future_to_data = {executor.submit(self.query, q, citations=True): q for q in questions}
+                for future in tqdm(
+                    concurrent.futures.as_completed(future_to_data),
+                    total=len(future_to_data),
+                    desc="Getting answer and contexts for questions",
+                ):
+                    question = future_to_data[future]
+                    queries.append(question)
+                    answer, context = future.result()
+                    answers.append(answer)
+                    contexts.append(list(map(lambda x: x[0], context)))
+        else:
+            answer, context = self.query(questions, citations=True)
+            queries = [questions]
+            answers = [answer]
+            contexts = [list(map(lambda x: x[0], context))]
+
+        metrics = metrics or [
+            EvalMetric.CONTEXT_RELEVANCY.value,
+            EvalMetric.ANSWER_RELEVANCY.value,
+            EvalMetric.GROUNDEDNESS.value,
+        ]
+
+        logging.info(f"Collecting data from {len(queries)} questions for evaluation...")
+        dataset = []
+        for q, a, c in zip(queries, answers, contexts):
+            dataset.append(EvalData(question=q, answer=a, contexts=c))
+
+        logging.info(f"Evaluating {len(dataset)} data points...")
+        result = {}
+        with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor:
+            future_to_metric = {executor.submit(self._eval, dataset, metric): metric for metric in metrics}
+            for future in tqdm(
+                concurrent.futures.as_completed(future_to_metric),
+                total=len(future_to_metric),
+                desc="Evaluating metrics",
+            ):
+                metric = future_to_metric[future]
+                if isinstance(metric, BaseMetric):
+                    result[metric.name] = future.result()
+                else:
+                    result[metric] = future.result()
+
+        if self.config.collect_metrics:
+            telemetry_props = self._telemetry_props
+            metrics_names = []
+            for metric in metrics:
+                if isinstance(metric, BaseMetric):
+                    metrics_names.append(metric.name)
+                else:
+                    metrics_names.append(metric)
+            telemetry_props["metrics"] = metrics_names
+            self.telemetry.capture(event_name="evaluate", properties=telemetry_props)
+
+        return result

embedchain/bots/poe.py

@@ -1,7 +1,7 @@
 import argparse
 import logging
 import os
-from typing import List, Optional
+from typing import Optional
 
 from embedchain.helpers.json_serializable import register_deserializable
 
@@ -53,7 +53,7 @@ class PoeBot(BaseBot, PoeBot):
         answer = self.handle_message(last_message, history)
         yield self.text_event(answer)
 
-    def handle_message(self, message, history: Optional[List[str]] = None):
+    def handle_message(self, message, history: Optional[list[str]] = None):
         if message.startswith("/add "):
             response = self.add_data(message)
         else:
@@ -70,7 +70,7 @@ class PoeBot(BaseBot, PoeBot):
     #         response = "Some error occurred while adding data."
     #     return response
 
-    def ask_bot(self, message, history: List[str]):
+    def ask_bot(self, message, history: list[str]):
         try:
             self.app.llm.set_history(history=history)
             response = self.query(message)

embedchain/cache.py

@@ -1,6 +1,6 @@
 import logging
 import os  # noqa: F401
-from typing import Any, Dict
+from typing import Any
 
 from gptcache import cache  # noqa: F401
 from gptcache.adapter.adapter import adapt  # noqa: F401
@@ -15,7 +15,7 @@ from gptcache.similarity_evaluation.exact_match import \
     ExactMatchEvaluation  # noqa: F401
 
 
-def gptcache_pre_function(data: Dict[str, Any], **params: Dict[str, Any]):
+def gptcache_pre_function(data: dict[str, Any], **params: dict[str, Any]):
     return data["input_query"]
 
 

embedchain/chunkers/base_chunker.py

@@ -17,7 +17,7 @@ class BaseChunker(JSONSerializable):
         """
         Loads data and chunks it.
 
-        :param loader: The loader which's `load_data` method is used to create
+        :param loader: The loader whose `load_data` method is used to create
         the raw data.
         :param src: The data to be handled by the loader. Can be a URL for
         remote sources or local content for local loaders.
@@ -25,9 +25,9 @@ class BaseChunker(JSONSerializable):
         """
         documents = []
         chunk_ids = []
-        idMap = {}
+        id_map = {}
         min_chunk_size = config.min_chunk_size if config is not None else 1
-        logging.info(f"[INFO] Skipping chunks smaller than {min_chunk_size} characters")
+        logging.info(f"Skipping chunks smaller than {min_chunk_size} characters")
         data_result = loader.load_data(src)
         data_records = data_result["data"]
         doc_id = data_result["doc_id"]
@@ -49,8 +49,8 @@ class BaseChunker(JSONSerializable):
             for chunk in chunks:
                 chunk_id = hashlib.sha256((chunk + url).encode()).hexdigest()
                 chunk_id = f"{app_id}--{chunk_id}" if app_id is not None else chunk_id
-                if idMap.get(chunk_id) is None and len(chunk) >= min_chunk_size:
-                    idMap[chunk_id] = True
+                if id_map.get(chunk_id) is None and len(chunk) >= min_chunk_size:
+                    id_map[chunk_id] = True
                     chunk_ids.append(chunk_id)
                     documents.append(chunk)
                     metadatas.append(meta_data)
@@ -77,5 +77,6 @@ class BaseChunker(JSONSerializable):
 
         # TODO: This should be done during initialization. This means it has to be done in the child classes.
 
-    def get_word_count(self, documents):
+    @staticmethod
+    def get_word_count(documents) -> int:
         return sum([len(document.split(" ")) for document in documents])

embedchain/cli.py

@@ -1,35 +1,33 @@
 import json
 import os
-import re
 import shutil
 import signal
 import subprocess
 import sys
 import tempfile
+import time
 import zipfile
 from pathlib import Path
 
 import click
-import pkg_resources
 import requests
 from rich.console import Console
 
 from embedchain.telemetry.posthog import AnonymousTelemetry
+from embedchain.utils.cli import (deploy_fly, deploy_gradio_app,
+                                  deploy_hf_spaces, deploy_modal,
+                                  deploy_render, deploy_streamlit,
+                                  get_pkg_path_from_name, setup_fly_io_app,
+                                  setup_gradio_app, setup_hf_app,
+                                  setup_modal_com_app, setup_render_com_app,
+                                  setup_streamlit_io_app)
 
 console = Console()
-
-
-@click.group()
-def cli():
-    pass
-
-
-anonymous_telemetry = AnonymousTelemetry()
-
-
 api_process = None
 ui_process = None
 
+anonymous_telemetry = AnonymousTelemetry()
+
 
 def signal_handler(sig, frame):
     """Signal handler to catch termination signals and kill server processes."""
@@ -44,110 +42,135 @@ def signal_handler(sig, frame):
     sys.exit(0)
 
 
-def get_pkg_path_from_name(template: str):
-    try:
-        # Determine the installation location of the embedchain package
-        package_path = pkg_resources.resource_filename("embedchain", "")
-    except ImportError:
-        console.print("❌ [bold red]Failed to locate the 'embedchain' package. Is it installed?[/bold red]")
+@click.group()
+def cli():
+    pass
+
+
+@cli.command()
+@click.argument("app_name")
+@click.option("--docker", is_flag=True, help="Use docker to create the app.")
+@click.pass_context
+def create_app(ctx, app_name, docker):
+    if Path(app_name).exists():
+        console.print(
+            f"❌ [red]Directory '{app_name}' already exists. Try using a new directory name, or remove it.[/red]"
+        )
         return
 
-    # Construct the source path from the embedchain package
-    src_path = os.path.join(package_path, "deployment", template)
+    os.makedirs(app_name)
+    os.chdir(app_name)
 
-    if not os.path.exists(src_path):
-        console.print(f"❌ [bold red]Template '{template}' not found.[/bold red]")
+    # Step 1: Download the zip file
+    zip_url = "http://github.com/embedchain/ec-admin/archive/main.zip"
+    console.print(f"Creating a new embedchain app in [green]{Path().resolve()}[/green]\n")
+    try:
+        response = requests.get(zip_url)
+        response.raise_for_status()
+        with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
+            tmp_file.write(response.content)
+            zip_file_path = tmp_file.name
+        console.print("✅ [bold green]Fetched template successfully.[/bold green]")
+    except requests.RequestException as e:
+        console.print(f"❌ [bold red]Failed to download zip file: {e}[/bold red]")
+        anonymous_telemetry.capture(event_name="ec_create_app", properties={"success": False})
         return
 
-    return src_path
-
-
-def setup_fly_io_app(extra_args):
-    fly_launch_command = ["fly", "launch", "--region", "sjc", "--no-deploy"] + list(extra_args)
+    # Step 2: Extract the zip file
     try:
-        console.print(f"🚀 [bold cyan]Running: {' '.join(fly_launch_command)}[/bold cyan]")
-        shutil.move(".env.example", ".env")
-        subprocess.run(fly_launch_command, check=True)
-        console.print("✅ [bold green]'fly launch' executed successfully.[/bold green]")
-    except subprocess.CalledProcessError as e:
-        console.print(f"❌ [bold red]An error occurred: {e}[/bold red]")
-    except FileNotFoundError:
-        console.print(
-            "❌ [bold red]'fly' command not found. Please ensure Fly CLI is installed and in your PATH.[/bold red]"
-        )
+        with zipfile.ZipFile(zip_file_path, "r") as zip_ref:
+            # Get the name of the root directory inside the zip file
+            root_dir = Path(zip_ref.namelist()[0])
+            for member in zip_ref.infolist():
+                # Build the path to extract the file to, skipping the root directory
+                target_file = Path(member.filename).relative_to(root_dir)
+                source_file = zip_ref.open(member, "r")
+                if member.is_dir():
+                    # Create directory if it doesn't exist
+                    os.makedirs(target_file, exist_ok=True)
+                else:
+                    with open(target_file, "wb") as file:
+                        # Write the file
+                        shutil.copyfileobj(source_file, file)
+            console.print("✅ [bold green]Extracted zip file successfully.[/bold green]")
+            anonymous_telemetry.capture(event_name="ec_create_app", properties={"success": True})
+    except zipfile.BadZipFile:
+        console.print("❌ [bold red]Error in extracting zip file. The file might be corrupted.[/bold red]")
+        anonymous_telemetry.capture(event_name="ec_create_app", properties={"success": False})
+        return
 
-
-def setup_modal_com_app(extra_args):
-    modal_setup_file = os.path.join(os.path.expanduser("~"), ".modal.toml")
-    if os.path.exists(modal_setup_file):
-        console.print(
-            """✅ [bold green]Modal setup already done. You can now install the dependencies by doing \n
-            `pip install -r requirements.txt`[/bold green]"""
-        )
+    if docker:
+        subprocess.run(["docker-compose", "build"], check=True)
     else:
-        modal_setup_cmd = ["modal", "setup"] + list(extra_args)
-        console.print(f"🚀 [bold cyan]Running: {' '.join(modal_setup_cmd)}[/bold cyan]")
-        subprocess.run(modal_setup_cmd, check=True)
-    shutil.move(".env.example", ".env")
-    console.print(
-        """Great! Now you can install the dependencies by doing: \n
-                  `pip install -r requirements.txt`\n
-                  \n
-                  To run your app locally:\n
-                  `ec dev`
-                  """
-    )
+        ctx.invoke(install_reqs)
 
 
-def setup_render_com_app():
-    render_setup_file = os.path.join(os.path.expanduser("~"), ".render/config.yaml")
-    if os.path.exists(render_setup_file):
-        console.print(
-            """✅ [bold green]Render setup already done. You can now install the dependencies by doing \n
-            `pip install -r requirements.txt`[/bold green]"""
-        )
-    else:
-        render_setup_cmd = ["render", "config", "init"]
-        console.print(f"🚀 [bold cyan]Running: {' '.join(render_setup_cmd)}[/bold cyan]")
-        subprocess.run(render_setup_cmd, check=True)
-    shutil.move(".env.example", ".env")
-    console.print(
-        """Great! Now you can install the dependencies by doing: \n
-                  `pip install -r requirements.txt`\n
-                  \n
-                  To run your app locally:\n
-                  `ec dev`
-                  """
-    )
+@cli.command()
+def install_reqs():
+    try:
+        console.print("Installing python requirements...\n")
+        time.sleep(2)
+        os.chdir("api")
+        subprocess.run(["pip", "install", "-r", "requirements.txt"], check=True)
+        os.chdir("..")
+        console.print("\n ✅ [bold green]Installed API requirements successfully.[/bold green]\n")
+    except Exception as e:
+        console.print(f"❌ [bold red]Failed to install API requirements: {e}[/bold red]")
+        anonymous_telemetry.capture(event_name="ec_install_reqs", properties={"success": False})
+        return
+
+    try:
+        os.chdir("ui")
+        subprocess.run(["yarn"], check=True)
+        console.print("\n✅ [bold green]Successfully installed frontend requirements.[/bold green]")
+        anonymous_telemetry.capture(event_name="ec_install_reqs", properties={"success": True})
+    except Exception as e:
+        console.print(f"❌ [bold red]Failed to install frontend requirements. Error: {e}[/bold red]")
+        anonymous_telemetry.capture(event_name="ec_install_reqs", properties={"success": False})
 
 
-def setup_streamlit_io_app():
-    # nothing needs to be done here
-    console.print("Great! Now you can install the dependencies by doing `pip install -r requirements.txt`")
+@cli.command()
+@click.option("--docker", is_flag=True, help="Run inside docker.")
+def start(docker):
+    if docker:
+        subprocess.run(["docker-compose", "up"], check=True)
+        return
 
+    # Set up signal handling
+    signal.signal(signal.SIGINT, signal_handler)
+    signal.signal(signal.SIGTERM, signal_handler)
 
-def setup_gradio_app():
-    # nothing needs to be done here
-    console.print("Great! Now you can install the dependencies by doing `pip install -r requirements.txt`")
+    # Step 1: Start the API server
+    try:
+        os.chdir("api")
+        api_process = subprocess.Popen(["python", "-m", "main"], stdout=None, stderr=None)
+        os.chdir("..")
+        console.print("✅ [bold green]API server started successfully.[/bold green]")
+    except Exception as e:
+        console.print(f"❌ [bold red]Failed to start the API server: {e}[/bold red]")
+        anonymous_telemetry.capture(event_name="ec_start", properties={"success": False})
+        return
 
+    # Sleep for 2 seconds to give the user time to read the message
+    time.sleep(2)
 
-def setup_hf_app():
-    subprocess.run(["pip", "install", "huggingface_hub[cli]"], check=True)
-    hf_setup_file = os.path.join(os.path.expanduser("~"), ".cache/huggingface/token")
-    if os.path.exists(hf_setup_file):
-        console.print(
-            """✅ [bold green]HuggingFace setup already done. You can now install the dependencies by doing \n
-            `pip install -r requirements.txt`[/bold green]"""
-        )
-    else:
-        console.print(
-            """🚀 [cyan]Running: huggingface-cli login \n
-                Please provide a [bold]WRITE[/bold] token so that we can directly deploy\n
-                your apps from the terminal.[/cyan]
-                """
-        )
-        subprocess.run(["huggingface-cli", "login"], check=True)
-    console.print("Great! Now you can install the dependencies by doing `pip install -r requirements.txt`")
+    # Step 2: Install UI requirements and start the UI server
+    try:
+        os.chdir("ui")
+        subprocess.run(["yarn"], check=True)
+        ui_process = subprocess.Popen(["yarn", "dev"])
+        console.print("✅ [bold green]UI server started successfully.[/bold green]")
+        anonymous_telemetry.capture(event_name="ec_start", properties={"success": True})
+    except Exception as e:
+        console.print(f"❌ [bold red]Failed to start the UI server: {e}[/bold red]")
+        anonymous_telemetry.capture(event_name="ec_start", properties={"success": False})
+
+    # Keep the script running until it receives a kill signal
+    try:
+        api_process.wait()
+        ui_process.wait()
+    except KeyboardInterrupt:
+        console.print("\n🛑 [bold yellow]Stopping server...[/bold yellow]")
 
 
 @cli.command()
@@ -172,7 +195,7 @@ def create(template, extra_args):
         setup_streamlit_io_app()
     elif template == "gradio.app":
         setup_gradio_app()
-    elif template == "hf/gradio.app" or template == "hf/streamlit.app":
+    elif template == "hf/gradio.app" or template == "hf/streamlit.io":
         setup_hf_app()
     else:
         raise ValueError(f"Unknown template '{template}'.")
@@ -269,7 +292,7 @@ def dev(debug, host, port):
         run_dev_modal_com()
     elif template == "render.com":
         run_dev_render_com(debug, host, port)
-    elif template == "streamlit.io" or template == "hf/streamlit.app":
+    elif template == "streamlit.io" or template == "hf/streamlit.io":
         run_dev_streamlit_io()
     elif template == "gradio.app" or template == "hf/gradio.app":
         run_dev_gradio()
@@ -277,141 +300,6 @@ def dev(debug, host, port):
         raise ValueError(f"Unknown template '{template}'.")
 
 
-def read_env_file(env_file_path):
-    """
-    Reads an environment file and returns a dictionary of key-value pairs.
-
-    Args:
-    env_file_path (str): The path to the .env file.
-
-    Returns:
-    dict: Dictionary of environment variables.
-    """
-    env_vars = {}
-    with open(env_file_path, "r") as file:
-        for line in file:
-            # Ignore comments and empty lines
-            if line.strip() and not line.strip().startswith("#"):
-                # Assume each line is in the format KEY=VALUE
-                key_value_match = re.match(r"(\w+)=(.*)", line.strip())
-                if key_value_match:
-                    key, value = key_value_match.groups()
-                    env_vars[key] = value
-    return env_vars
-
-
-def deploy_fly():
-    app_name = ""
-    with open("fly.toml", "r") as file:
-        for line in file:
-            if line.strip().startswith("app ="):
-                app_name = line.split("=")[1].strip().strip('"')
-
-    if not app_name:
-        console.print("❌ [bold red]App name not found in fly.toml[/bold red]")
-        return
-
-    env_vars = read_env_file(".env")
-    secrets_command = ["flyctl", "secrets", "set", "-a", app_name] + [f"{k}={v}" for k, v in env_vars.items()]
-
-    deploy_command = ["fly", "deploy"]
-    try:
-        # Set secrets
-        console.print(f"🔐 [bold cyan]Setting secrets for {app_name}[/bold cyan]")
-        subprocess.run(secrets_command, check=True)
-
-        # Deploy application
-        console.print(f"🚀 [bold cyan]Running: {' '.join(deploy_command)}[/bold cyan]")
-        subprocess.run(deploy_command, check=True)
-        console.print("✅ [bold green]'fly deploy' executed successfully.[/bold green]")
-
-    except subprocess.CalledProcessError as e:
-        console.print(f"❌ [bold red]An error occurred: {e}[/bold red]")
-    except FileNotFoundError:
-        console.print(
-            "❌ [bold red]'fly' command not found. Please ensure Fly CLI is installed and in your PATH.[/bold red]"
-        )
-
-
-def deploy_modal():
-    modal_deploy_cmd = ["modal", "deploy", "app"]
-    try:
-        console.print(f"🚀 [bold cyan]Running: {' '.join(modal_deploy_cmd)}[/bold cyan]")
-        subprocess.run(modal_deploy_cmd, check=True)
-        console.print("✅ [bold green]'modal deploy' executed successfully.[/bold green]")
-    except subprocess.CalledProcessError as e:
-        console.print(f"❌ [bold red]An error occurred: {e}[/bold red]")
-    except FileNotFoundError:
-        console.print(
-            "❌ [bold red]'modal' command not found. Please ensure Modal CLI is installed and in your PATH.[/bold red]"
-        )
-
-
-def deploy_streamlit():
-    streamlit_deploy_cmd = ["streamlit", "run", "app.py"]
-    try:
-        console.print(f"🚀 [bold cyan]Running: {' '.join(streamlit_deploy_cmd)}[/bold cyan]")
-        console.print(
-            """\n\n✅ [bold yellow]To deploy a streamlit app, you can directly it from the UI.\n
-        Click on the 'Deploy' button on the top right corner of the app.\n
-        For more information, please refer to https://docs.embedchain.ai/deployment/streamlit_io
-        [/bold yellow]
-                      \n\n"""
-        )
-        subprocess.run(streamlit_deploy_cmd, check=True)
-    except subprocess.CalledProcessError as e:
-        console.print(f"❌ [bold red]An error occurred: {e}[/bold red]")
-    except FileNotFoundError:
-        console.print(
-            """❌ [bold red]'streamlit' command not found.\n
-            Please ensure Streamlit CLI is installed and in your PATH.[/bold red]"""
-        )
-
-
-def deploy_render():
-    render_deploy_cmd = ["render", "blueprint", "launch"]
-
-    try:
-        console.print(f"🚀 [bold cyan]Running: {' '.join(render_deploy_cmd)}[/bold cyan]")
-        subprocess.run(render_deploy_cmd, check=True)
-        console.print("✅ [bold green]'render blueprint launch' executed successfully.[/bold green]")
-    except subprocess.CalledProcessError as e:
-        console.print(f"❌ [bold red]An error occurred: {e}[/bold red]")
-    except FileNotFoundError:
-        console.print(
-            "❌ [bold red]'render' command not found. Please ensure Render CLI is installed and in your PATH.[/bold red]"  # noqa:E501
-        )
-
-
-def deploy_gradio_app():
-    gradio_deploy_cmd = ["gradio", "deploy"]
-
-    try:
-        console.print(f"🚀 [bold cyan]Running: {' '.join(gradio_deploy_cmd)}[/bold cyan]")
-        subprocess.run(gradio_deploy_cmd, check=True)
-        console.print("✅ [bold green]'gradio deploy' executed successfully.[/bold green]")
-    except subprocess.CalledProcessError as e:
-        console.print(f"❌ [bold red]An error occurred: {e}[/bold red]")
-    except FileNotFoundError:
-        console.print(
-            "❌ [bold red]'gradio' command not found. Please ensure Gradio CLI is installed and in your PATH.[/bold red]"  # noqa:E501
-        )
-
-
-def deploy_hf_spaces(ec_app_name):
-    if not ec_app_name:
-        console.print("❌ [bold red]'name' not found in embedchain.json[/bold red]")
-        return
-    hf_spaces_deploy_cmd = ["huggingface-cli", "upload", ec_app_name, ".", ".", "--repo-type=space"]
-
-    try:
-        console.print(f"🚀 [bold cyan]Running: {' '.join(hf_spaces_deploy_cmd)}[/bold cyan]")
-        subprocess.run(hf_spaces_deploy_cmd, check=True)
-        console.print("✅ [bold green]'huggingface-cli upload' executed successfully.[/bold green]")
-    except subprocess.CalledProcessError as e:
-        console.print(f"❌ [bold red]An error occurred: {e}[/bold red]")
-
-
 @cli.command()
 def deploy():
     # Check for platform-specific files
@@ -437,86 +325,3 @@ def deploy():
         deploy_hf_spaces(ec_app_name)
     else:
         console.print("❌ [bold red]No recognized deployment platform found.[/bold red]")
-
-
-@cli.command()
-def runserver():
-    # Set up signal handling
-    signal.signal(signal.SIGINT, signal_handler)
-    signal.signal(signal.SIGTERM, signal_handler)
-
-    # Check if 'api' and 'ui' directories exist
-    if os.path.exists("api") and os.path.exists("ui"):
-        pass
-    else:
-        # Step 1: Download the zip file
-        zip_url = "http://github.com/embedchain/ec-admin/archive/main.zip"
-        try:
-            response = requests.get(zip_url)
-            response.raise_for_status()
-            with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
-                tmp_file.write(response.content)
-                zip_file_path = tmp_file.name
-            console.print("✅ [bold green]Downloaded zip file successfully.[/bold green]")
-        except requests.RequestException as e:
-            console.print(f"❌ [bold red]Failed to download zip file: {e}[/bold red]")
-            return
-
-        # Step 2: Extract the zip file
-        try:
-            with zipfile.ZipFile(zip_file_path, "r") as zip_ref:
-                # Get the name of the root directory inside the zip file
-                root_dir = Path(zip_ref.namelist()[0])
-                for member in zip_ref.infolist():
-                    # Build the path to extract the file to, skipping the root directory
-                    target_file = Path(member.filename).relative_to(root_dir)
-                    source_file = zip_ref.open(member, "r")
-                    if member.is_dir():
-                        # Create directory if it doesn't exist
-                        os.makedirs(target_file, exist_ok=True)
-                    else:
-                        with open(target_file, "wb") as file:
-                            # Write the file
-                            shutil.copyfileobj(source_file, file)
-                console.print("✅ [bold green]Extracted zip file successfully.[/bold green]")
-        except zipfile.BadZipFile:
-            console.print("❌ [bold red]Error in extracting zip file. The file might be corrupted.[/bold red]")
-            return
-
-        # Step 3: Install API requirements
-        try:
-            os.chdir("api")
-            subprocess.run(["pip", "install", "-r", "requirements.txt"], check=True)
-            os.chdir("..")
-            console.print("✅ [bold green]Installed API requirements successfully.[/bold green]")
-        except Exception as e:
-            console.print(f"❌ [bold red]Failed to install API requirements: {e}[/bold red]")
-            return
-
-    # Step 4: Start the API server
-    try:
-        os.chdir("api")
-        api_process = subprocess.Popen(
-            ["uvicorn", "main:app", "--reload", "--host", "127.0.0.1", "--port", "8000"], stdout=None, stderr=None
-        )
-        os.chdir("..")
-        console.print("✅ [bold green]API server started successfully.[/bold green]")
-    except Exception as e:
-        console.print(f"❌ [bold red]Failed to start the API server: {e}[/bold red]")
-        return
-
-    # Step 5: Install UI requirements and start the UI server
-    try:
-        os.chdir("ui")
-        subprocess.run(["yarn"], check=True)
-        subprocess.Popen(["yarn", "dev"])
-        console.print("✅ [bold green]UI server started successfully.[/bold green]")
-    except Exception as e:
-        console.print(f"❌ [bold red]Failed to start the UI server: {e}[/bold red]")
-
-    # Keep the script running until it receives a kill signal
-    try:
-        api_process.wait()
-        ui_process.wait()
-    except KeyboardInterrupt:
-        console.print("\n🛑 [bold yellow]Stopping server...[/bold yellow]")

embedchain/client.py

@@ -31,7 +31,7 @@ class Client:
                 )
 
     @classmethod
-    def setup_dir(self):
+    def setup_dir(cls):
         """
         Loads the user id from the config file if it exists, otherwise generates a new
         one and saves it to the config file.

embedchain/config/add_config.py

@@ -1,7 +1,8 @@
 import builtins
 import logging
+from collections.abc import Callable
 from importlib import import_module
-from typing import Callable, Optional
+from typing import Optional
 
 from embedchain.config.base_config import BaseConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -26,7 +27,7 @@ class ChunkerConfig(BaseConfig):
         if self.min_chunk_size >= self.chunk_size:
             raise ValueError(f"min_chunk_size {min_chunk_size} should be less than chunk_size {chunk_size}")
         if self.min_chunk_size < self.chunk_overlap:
-            logging.warn(
+            logging.warning(
                 f"min_chunk_size {min_chunk_size} should be greater than chunk_overlap {chunk_overlap}, otherwise it is redundant."  # noqa:E501
             )
 
@@ -35,7 +36,8 @@ class ChunkerConfig(BaseConfig):
         else:
             self.length_function = length_function if length_function else len
 
-    def load_func(self, dotpath: str):
+    @staticmethod
+    def load_func(dotpath: str):
         if "." not in dotpath:
             return getattr(builtins, dotpath)
         else:

embedchain/config/base_config.py

@@ -1,4 +1,4 @@
-from typing import Any, Dict
+from typing import Any
 
 from embedchain.helpers.json_serializable import JSONSerializable
 
@@ -12,10 +12,10 @@ class BaseConfig(JSONSerializable):
         """Initializes a configuration class for a class."""
         pass
 
-    def as_dict(self) -> Dict[str, Any]:
+    def as_dict(self) -> dict[str, Any]:
         """Return config object as a dict
 
         :return: config object as dict
-        :rtype: Dict[str, Any]
+        :rtype: dict[str, Any]
         """
         return vars(self)

embedchain/config/cache_config.py

@@ -1,4 +1,4 @@
-from typing import Any, Dict, Optional
+from typing import Any, Optional
 
 from embedchain.config.base_config import BaseConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -10,12 +10,12 @@ class CacheSimilarityEvalConfig(BaseConfig):
     This is the evaluator to compare two embeddings according to their distance computed in embedding retrieval stage.
     In the retrieval stage, `search_result` is the distance used for approximate nearest neighbor search and have been
     put into `cache_dict`. `max_distance` is used to bound this distance to make it between [0-`max_distance`].
-    `positive` is used to indicate this distance is directly proportional to the similarity of two entites.
-    If `positive` is set `False`, `max_distance` will be used to substract this distance to get the final score.
+    `positive` is used to indicate this distance is directly proportional to the similarity of two entities.
+    If `positive` is set `False`, `max_distance` will be used to subtract this distance to get the final score.
 
     :param max_distance: the bound of maximum distance.
     :type max_distance: float
-    :param positive: if the larger distance indicates more similar of two entities, It is True. Otherwise it is False.
+    :param positive: if the larger distance indicates more similar of two entities, It is True. Otherwise, it is False.
     :type positive: bool
     """
 
@@ -29,7 +29,8 @@ class CacheSimilarityEvalConfig(BaseConfig):
         self.max_distance = max_distance
         self.positive = positive
 
-    def from_config(config: Optional[Dict[str, Any]]):
+    @staticmethod
+    def from_config(config: Optional[dict[str, Any]]):
         if config is None:
             return CacheSimilarityEvalConfig()
         else:
@@ -63,7 +64,8 @@ class CacheInitConfig(BaseConfig):
         self.similarity_threshold = similarity_threshold
         self.auto_flush = auto_flush
 
-    def from_config(config: Optional[Dict[str, Any]]):
+    @staticmethod
+    def from_config(config: Optional[dict[str, Any]]):
         if config is None:
             return CacheInitConfig()
         else:
@@ -83,7 +85,8 @@ class CacheConfig(BaseConfig):
         self.similarity_eval_config = similarity_eval_config
         self.init_config = init_config
 
-    def from_config(config: Optional[Dict[str, Any]]):
+    @staticmethod
+    def from_config(config: Optional[dict[str, Any]]):
         if config is None:
             return CacheConfig()
         else:

embedchain/config/embedder/base.py

@@ -6,7 +6,11 @@ from embedchain.helpers.json_serializable import register_deserializable
 @register_deserializable
 class BaseEmbedderConfig:
     def __init__(
-        self, model: Optional[str] = None, deployment_name: Optional[str] = None, api_key: Optional[str] = None
+        self,
+        model: Optional[str] = None,
+        deployment_name: Optional[str] = None,
+        vector_dimension: Optional[int] = None,
+        api_key: Optional[str] = None,
     ):
         """
         Initialize a new instance of an embedder config class.
@@ -18,4 +22,5 @@ class BaseEmbedderConfig:
         """
         self.model = model
         self.deployment_name = deployment_name
+        self.vector_dimension = vector_dimension
         self.api_key = api_key

embedchain/config/evaluation/__init__.py

@@ -0,0 +1,2 @@
+from .base import (AnswerRelevanceConfig, ContextRelevanceConfig,  # noqa: F401
+                   GroundednessConfig)

embedchain/config/evaluation/base.py

@@ -0,0 +1,92 @@
+from typing import Optional
+
+from embedchain.config.base_config import BaseConfig
+
+ANSWER_RELEVANCY_PROMPT = """
+Please provide $num_gen_questions questions from the provided answer.
+You must provide the complete question, if are not able to provide the complete question, return empty string ("").
+Please only provide one question per line without numbers or bullets to distinguish them.
+You must only provide the questions and no other text.
+
+$answer
+"""  # noqa:E501
+
+
+CONTEXT_RELEVANCY_PROMPT = """
+Please extract relevant sentences from the provided context that is required to answer the given question.
+If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the empty string ("").
+While extracting candidate sentences you're not allowed to make any changes to sentences from given context or make up any sentences.
+You must only provide sentences from the given context and nothing else.
+
+Context: $context
+Question: $question
+"""  # noqa:E501
+
+GROUNDEDNESS_ANSWER_CLAIMS_PROMPT = """
+Please provide one or more statements from each sentence of the provided answer.
+You must provide the symantically equivalent statements for each sentence of the answer.
+You must provide the complete statement, if are not able to provide the complete statement, return empty string ("").
+Please only provide one statement per line WITHOUT numbers or bullets.
+If the question provided is not being answered in the provided answer, return empty string ("").
+You must only provide the statements and no other text.
+
+$question
+$answer
+"""  # noqa:E501
+
+GROUNDEDNESS_CLAIMS_INFERENCE_PROMPT = """
+Given the context and the provided claim statements, please provide a verdict for each claim statement whether it can be completely infered from the given context or not.
+Use only "1" (yes), "0" (no) and "-1" (null) for "yes", "no" or "null" respectively.
+You must provide one verdict per line, ONLY WITH "1", "0" or "-1" as per your verdict to the given statement and nothing else.
+You must provide the verdicts in the same order as the claim statements.
+
+Contexts: 
+$context
+
+Claim statements: 
+$claim_statements
+"""  # noqa:E501
+
+
+class GroundednessConfig(BaseConfig):
+    def __init__(
+        self,
+        model: str = "gpt-4",
+        api_key: Optional[str] = None,
+        answer_claims_prompt: str = GROUNDEDNESS_ANSWER_CLAIMS_PROMPT,
+        claims_inference_prompt: str = GROUNDEDNESS_CLAIMS_INFERENCE_PROMPT,
+    ):
+        self.model = model
+        self.api_key = api_key
+        self.answer_claims_prompt = answer_claims_prompt
+        self.claims_inference_prompt = claims_inference_prompt
+
+
+class AnswerRelevanceConfig(BaseConfig):
+    def __init__(
+        self,
+        model: str = "gpt-4",
+        embedder: str = "text-embedding-ada-002",
+        api_key: Optional[str] = None,
+        num_gen_questions: int = 1,
+        prompt: str = ANSWER_RELEVANCY_PROMPT,
+    ):
+        self.model = model
+        self.embedder = embedder
+        self.api_key = api_key
+        self.num_gen_questions = num_gen_questions
+        self.prompt = prompt
+
+
+class ContextRelevanceConfig(BaseConfig):
+    def __init__(
+        self,
+        model: str = "gpt-4",
+        api_key: Optional[str] = None,
+        language: str = "en",
+        prompt: str = CONTEXT_RELEVANCY_PROMPT,
+    ):
+        self.model = model
+        self.api_key = api_key
+        self.language = language
+        self.prompt = prompt

embedchain/config/llm/base.py

@@ -1,7 +1,7 @@
 import logging
 import re
 from string import Template
-from typing import Any, Dict, List, Optional
+from typing import Any, Optional
 
 from embedchain.config.base_config import BaseConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -68,10 +68,12 @@ class BaseLlmConfig(BaseConfig):
         stream: bool = False,
         deployment_name: Optional[str] = None,
         system_prompt: Optional[str] = None,
-        where: Dict[str, Any] = None,
+        where: dict[str, Any] = None,
         query_type: Optional[str] = None,
-        callbacks: Optional[List] = None,
+        callbacks: Optional[list] = None,
         api_key: Optional[str] = None,
+        endpoint: Optional[str] = None,
+        model_kwargs: Optional[dict[str, Any]] = None,
     ):
         """
         Initializes a configuration class instance for the LLM.
@@ -104,9 +106,17 @@ class BaseLlmConfig(BaseConfig):
         :param system_prompt: System prompt string, defaults to None
         :type system_prompt: Optional[str], optional
         :param where: A dictionary of key-value pairs to filter the database results., defaults to None
-        :type where: Dict[str, Any], optional
+        :type where: dict[str, Any], optional
+        :param api_key: The api key of the custom endpoint, defaults to None
+        :type api_key: Optional[str], optional
+        :param endpoint: The api url of the custom endpoint, defaults to None
+        :type endpoint: Optional[str], optional
+        :param model_kwargs: A dictionary of key-value pairs to pass to the model, defaults to None
+        :type model_kwargs: Optional[Dict[str, Any]], optional
         :param callbacks: Langchain callback functions to use, defaults to None
-        :type callbacks: Optional[List], optional
+        :type callbacks: Optional[list], optional
+        :param query_type: The type of query to use, defaults to None
+        :type query_type: Optional[str], optional
         :raises ValueError: If the template is not valid as template should
         contain $context and $query (and optionally $history)
         :raises ValueError: Stream is not boolean
@@ -132,8 +142,10 @@ class BaseLlmConfig(BaseConfig):
         self.query_type = query_type
         self.callbacks = callbacks
         self.api_key = api_key
+        self.endpoint = endpoint
+        self.model_kwargs = model_kwargs
 
-        if type(prompt) is str:
+        if isinstance(prompt, str):
             prompt = Template(prompt)
 
         if self.validate_prompt(prompt):
@@ -146,24 +158,26 @@ class BaseLlmConfig(BaseConfig):
         self.stream = stream
         self.where = where
 
-    def validate_prompt(self, prompt: Template) -> bool:
+    @staticmethod
+    def validate_prompt(prompt: Template) -> Optional[re.Match[str]]:
         """
         validate the prompt
 
         :param prompt: the prompt to validate
         :type prompt: Template
         :return: valid (true) or invalid (false)
-        :rtype: bool
+        :rtype: Optional[re.Match[str]]
         """
         return re.search(query_re, prompt.template) and re.search(context_re, prompt.template)
 
-    def _validate_prompt_history(self, prompt: Template) -> bool:
+    @staticmethod
+    def _validate_prompt_history(prompt: Template) -> Optional[re.Match[str]]:
         """
         validate the prompt with history
 
         :param prompt: the prompt to validate
         :type prompt: Template
         :return: valid (true) or invalid (false)
-        :rtype: bool
+        :rtype: Optional[re.Match[str]]
         """
         return re.search(history_re, prompt.template)

embedchain/config/vectordb/elasticsearch.py

@@ -1,5 +1,5 @@
 import os
-from typing import Dict, List, Optional, Union
+from typing import Optional, Union
 
 from embedchain.config.vectordb.base import BaseVectorDbConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -11,9 +11,9 @@ class ElasticsearchDBConfig(BaseVectorDbConfig):
         self,
         collection_name: Optional[str] = None,
         dir: Optional[str] = None,
-        es_url: Union[str, List[str]] = None,
+        es_url: Union[str, list[str]] = None,
         cloud_id: Optional[str] = None,
-        **ES_EXTRA_PARAMS: Dict[str, any],
+        **ES_EXTRA_PARAMS: dict[str, any],
     ):
         """
         Initializes a configuration class instance for an Elasticsearch client.
@@ -23,13 +23,13 @@ class ElasticsearchDBConfig(BaseVectorDbConfig):
         :param dir: Path to the database directory, where the database is stored, defaults to None
         :type dir: Optional[str], optional
         :param es_url: elasticsearch url or list of nodes url to be used for connection, defaults to None
-        :type es_url: Union[str, List[str]], optional
+        :type es_url: Union[str, list[str]], optional
         :param ES_EXTRA_PARAMS: extra params dict that can be passed to elasticsearch.
-        :type ES_EXTRA_PARAMS: Dict[str, Any], optional
+        :type ES_EXTRA_PARAMS: dict[str, Any], optional
         """
         if es_url and cloud_id:
             raise ValueError("Only one of `es_url` and `cloud_id` can be set.")
-        # self, es_url: Union[str, List[str]] = None, **ES_EXTRA_PARAMS: Dict[str, any]):
+        # self, es_url: Union[str, list[str]] = None, **ES_EXTRA_PARAMS: dict[str, any]):
         self.ES_URL = es_url or os.environ.get("ELASTICSEARCH_URL")
         self.CLOUD_ID = cloud_id or os.environ.get("ELASTICSEARCH_CLOUD_ID")
         if not self.ES_URL and not self.CLOUD_ID:

embedchain/config/vectordb/opensearch.py

@@ -1,4 +1,4 @@
-from typing import Dict, Optional, Tuple
+from typing import Optional
 
 from embedchain.config.vectordb.base import BaseVectorDbConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -9,11 +9,11 @@ class OpenSearchDBConfig(BaseVectorDbConfig):
     def __init__(
         self,
         opensearch_url: str,
-        http_auth: Tuple[str, str],
+        http_auth: tuple[str, str],
         vector_dimension: int = 1536,
         collection_name: Optional[str] = None,
         dir: Optional[str] = None,
-        **extra_params: Dict[str, any],
+        **extra_params: dict[str, any],
     ):
         """
         Initializes a configuration class instance for an OpenSearch client.
@@ -23,7 +23,7 @@ class OpenSearchDBConfig(BaseVectorDbConfig):
         :param opensearch_url: URL of the OpenSearch domain
         :type opensearch_url: str, Eg, "http://localhost:9200"
         :param http_auth: Tuple of username and password
-        :type http_auth: Tuple[str, str], Eg, ("username", "password")
+        :type http_auth: tuple[str, str], Eg, ("username", "password")
         :param vector_dimension: Dimension of  the vector, defaults to 1536 (openai embedding model)
         :type vector_dimension: int, optional
         :param dir: Path to the database directory, where the database is stored, defaults to None

embedchain/config/vectordb/pinecone.py

@@ -1,4 +1,5 @@
-from typing import Dict, Optional
+import os
+from typing import Optional
 
 from embedchain.config.vectordb.base import BaseVectorDbConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -9,12 +10,29 @@ class PineconeDBConfig(BaseVectorDbConfig):
     def __init__(
         self,
         collection_name: Optional[str] = None,
+        api_key: Optional[str] = None,
+        index_name: Optional[str] = None,
         dir: Optional[str] = None,
         vector_dimension: int = 1536,
         metric: Optional[str] = "cosine",
-        **extra_params: Dict[str, any],
+        pod_config: Optional[dict[str, any]] = None,
+        serverless_config: Optional[dict[str, any]] = None,
+        **extra_params: dict[str, any],
     ):
         self.metric = metric
+        self.api_key = api_key
         self.vector_dimension = vector_dimension
         self.extra_params = extra_params
-        super().__init__(collection_name=collection_name, dir=dir)
+        self.index_name = index_name or f"{collection_name}-{vector_dimension}".lower().replace("_", "-")
+        if pod_config is None and serverless_config is None:
+            # If no config is provided, use the default pod spec config
+            pod_environment = os.environ.get("PINECONE_ENV", "gcp-starter")
+            self.pod_config = {"environment": pod_environment, "metadata_config": {"indexed": ["*"]}}
+        else:
+            self.pod_config = pod_config
+        self.serverless_config = serverless_config
+
+        if self.pod_config and self.serverless_config:
+            raise ValueError("Only one of pod_config or serverless_config can be provided.")
+
+        super().__init__(collection_name=collection_name, dir=None)

embedchain/config/vectordb/qdrant.py

@@ -1,4 +1,4 @@
-from typing import Dict, Optional
+from typing import Optional
 
 from embedchain.config.vectordb.base import BaseVectorDbConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -7,18 +7,18 @@ from embedchain.helpers.json_serializable import register_deserializable
 @register_deserializable
 class QdrantDBConfig(BaseVectorDbConfig):
     """
-    Config to initialize an qdrant client.
-    :param url. qdrant url or list of nodes url to be used for connection
+    Config to initialize a qdrant client.
+    :param: url. qdrant url or list of nodes url to be used for connection
     """
 
     def __init__(
         self,
         collection_name: Optional[str] = None,
         dir: Optional[str] = None,
-        hnsw_config: Optional[Dict[str, any]] = None,
-        quantization_config: Optional[Dict[str, any]] = None,
+        hnsw_config: Optional[dict[str, any]] = None,
+        quantization_config: Optional[dict[str, any]] = None,
         on_disk: Optional[bool] = None,
-        **extra_params: Dict[str, any],
+        **extra_params: dict[str, any],
     ):
         """
         Initializes a configuration class instance for a qdrant client.
@@ -28,9 +28,9 @@ class QdrantDBConfig(BaseVectorDbConfig):
         :param dir: Path to the database directory, where the database is stored, defaults to None
         :type dir: Optional[str], optional
         :param hnsw_config: Params for HNSW index
-        :type hnsw_config: Optional[Dict[str, any]], defaults to None
+        :type hnsw_config: Optional[dict[str, any]], defaults to None
         :param quantization_config: Params for quantization, if None - quantization will be disabled
-        :type quantization_config: Optional[Dict[str, any]], defaults to None
+        :type quantization_config: Optional[dict[str, any]], defaults to None
         :param on_disk: If true - point`s payload will not be stored in memory.
                 It will be read from the disk every time it is requested.
                 This setting saves RAM by (slightly) increasing the response time.

embedchain/config/vectordb/weaviate.py

@@ -1,4 +1,4 @@
-from typing import Dict, Optional
+from typing import Optional
 
 from embedchain.config.vectordb.base import BaseVectorDbConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -10,7 +10,7 @@ class WeaviateDBConfig(BaseVectorDbConfig):
         self,
         collection_name: Optional[str] = None,
         dir: Optional[str] = None,
-        **extra_params: Dict[str, any],
+        **extra_params: dict[str, any],
     ):
         self.extra_params = extra_params
         super().__init__(collection_name=collection_name, dir=dir)

embedchain/config/vectordb/zilliz.py

@@ -26,7 +26,7 @@ class ZillizDBConfig(BaseVectorDbConfig):
         :param uri: Cluster endpoint obtained from the Zilliz Console, defaults to None
         :type uri: Optional[str], optional
         :param token: API Key, if a Serverless Cluster, username:password, if a Dedicated Cluster, defaults to None
-        :type port: Optional[str], optional
+        :type token: Optional[str], optional
         """
         self.uri = uri or os.environ.get("ZILLIZ_CLOUD_URI")
         if not self.uri:

embedchain/data_formatter/data_formatter.py

@@ -34,7 +34,8 @@ class DataFormatter(JSONSerializable):
         self.loader = self._get_loader(data_type=data_type, config=config.loader, loader=loader)
         self.chunker = self._get_chunker(data_type=data_type, config=config.chunker, chunker=chunker)
 
-    def _lazy_load(self, module_path: str):
+    @staticmethod
+    def _lazy_load(module_path: str):
         module_path, class_name = module_path.rsplit(".", 1)
         module = import_module(module_path)
         return getattr(module, class_name)

embedchain/embedchain.py

@@ -2,12 +2,14 @@ import hashlib
 import json
 import logging
 import sqlite3
-from typing import Any, Dict, List, Optional, Tuple, Union
+from typing import Any, Optional, Union
 
 from dotenv import load_dotenv
 from langchain.docstore.document import Document
 
-from embedchain.cache import adapt, get_gptcache_session, gptcache_data_convert, gptcache_update_cache_callback
+from embedchain.cache import (adapt, get_gptcache_session,
+                              gptcache_data_convert,
+                              gptcache_update_cache_callback)
 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config import AddConfig, BaseLlmConfig, ChunkerConfig
 from embedchain.config.base_app_config import BaseAppConfig
@@ -17,7 +19,8 @@ from embedchain.embedder.base import BaseEmbedder
 from embedchain.helpers.json_serializable import JSONSerializable
 from embedchain.llm.base import BaseLlm
 from embedchain.loaders.base_loader import BaseLoader
-from embedchain.models.data_type import DataType, DirectDataType, IndirectDataType, SpecialDataType
+from embedchain.models.data_type import (DataType, DirectDataType,
+                                         IndirectDataType, SpecialDataType)
 from embedchain.telemetry.posthog import AnonymousTelemetry
 from embedchain.utils.misc import detect_datatype, is_valid_json_string
 from embedchain.vectordb.base import BaseVectorDB
@@ -81,7 +84,7 @@ class EmbedChain(JSONSerializable):
         # Attributes that aren't subclass related.
         self.user_asks = []
 
-        self.chunker: ChunkerConfig = None
+        self.chunker: Optional[ChunkerConfig] = None
         # Send anonymous telemetry
         self._telemetry_props = {"class": self.__class__.__name__}
         self.telemetry = AnonymousTelemetry(enabled=self.config.collect_metrics)
@@ -131,12 +134,12 @@ class EmbedChain(JSONSerializable):
         self,
         source: Any,
         data_type: Optional[DataType] = None,
-        metadata: Optional[Dict[str, Any]] = None,
+        metadata: Optional[dict[str, Any]] = None,
         config: Optional[AddConfig] = None,
         dry_run=False,
         loader: Optional[BaseLoader] = None,
         chunker: Optional[BaseChunker] = None,
-        **kwargs: Optional[Dict[str, Any]],
+        **kwargs: Optional[dict[str, Any]],
     ):
         """
         Adds the data from the given URL to the vector db.
@@ -149,7 +152,7 @@ class EmbedChain(JSONSerializable):
         defaults to None
         :type data_type: Optional[DataType], optional
         :param metadata: Metadata associated with the data source., defaults to None
-        :type metadata: Optional[Dict[str, Any]], optional
+        :type metadata: Optional[dict[str, Any]], optional
         :param config: The `AddConfig` instance to use as configuration options., defaults to None
         :type config: Optional[AddConfig], optional
         :raises ValueError: Invalid data type
@@ -238,9 +241,9 @@ class EmbedChain(JSONSerializable):
         self,
         source: Any,
         data_type: Optional[DataType] = None,
-        metadata: Optional[Dict[str, Any]] = None,
+        metadata: Optional[dict[str, Any]] = None,
         config: Optional[AddConfig] = None,
-        **kwargs: Optional[Dict[str, Any]],
+        **kwargs: Optional[dict[str, Any]],
     ):
         """
         Adds the data from the given URL to the vector db.
@@ -256,7 +259,7 @@ class EmbedChain(JSONSerializable):
         defaults to None
         :type data_type: Optional[DataType], optional
         :param metadata: Metadata associated with the data source., defaults to None
-        :type metadata: Optional[Dict[str, Any]], optional
+        :type metadata: Optional[dict[str, Any]], optional
         :param config: The `AddConfig` instance to use as configuration options., defaults to None
         :type config: Optional[AddConfig], optional
         :raises ValueError: Invalid data type
@@ -287,7 +290,7 @@ class EmbedChain(JSONSerializable):
             #   Or it's different, then it will be added as a new text.
             return None
         elif chunker.data_type.value in [item.value for item in IndirectDataType]:
-            # These types have a indirect source reference
+            # These types have an indirect source reference
             # As long as the reference is the same, they can be updated.
             where = {"url": src}
             if chunker.data_type == DataType.JSON and is_valid_json_string(src):
@@ -337,11 +340,11 @@ class EmbedChain(JSONSerializable):
         loader: BaseLoader,
         chunker: BaseChunker,
         src: Any,
-        metadata: Optional[Dict[str, Any]] = None,
+        metadata: Optional[dict[str, Any]] = None,
         source_hash: Optional[str] = None,
         add_config: Optional[AddConfig] = None,
         dry_run=False,
-        **kwargs: Optional[Dict[str, Any]],
+        **kwargs: Optional[dict[str, Any]],
     ):
         """
         Loads the data from the given URL, chunks it, and adds it to database.
@@ -354,7 +357,7 @@ class EmbedChain(JSONSerializable):
         :param source_hash: Hexadecimal hash of the source.
         :param dry_run: Optional. A dry run returns chunks and doesn't update DB.
         :type dry_run: bool, defaults to False
-        :return: (List) documents (embedded text), (List) metadata, (list) ids, (int) number of chunks
+        :return: (list) documents (embedded text), (list) metadata, (list) ids, (int) number of chunks
         """
         existing_doc_id = self._get_existing_doc_id(chunker=chunker, src=src)
         app_id = self.config.id if self.config is not None else None
@@ -366,7 +369,7 @@ class EmbedChain(JSONSerializable):
         metadatas = embeddings_data["metadatas"]
         ids = embeddings_data["ids"]
         new_doc_id = embeddings_data["doc_id"]
-        embeddings = embeddings_data.get("embeddings")
+
         if existing_doc_id and existing_doc_id == new_doc_id:
             print("Doc content has not changed. Skipping creating chunks and embeddings")
             return [], [], [], 0
@@ -430,19 +433,14 @@ class EmbedChain(JSONSerializable):
         # Count before, to calculate a delta in the end.
         chunks_before_addition = self.db.count()
 
-        self.db.add(
-            embeddings=embeddings,
-            documents=documents,
-            metadatas=metadatas,
-            ids=ids,
-            **kwargs,
-        )
+        self.db.add(documents=documents, metadatas=metadatas, ids=ids, **kwargs)
         count_new_chunks = self.db.count() - chunks_before_addition
 
-        print((f"Successfully saved {src} ({chunker.data_type}). New chunks count: {count_new_chunks}"))
+        print(f"Successfully saved {src} ({chunker.data_type}). New chunks count: {count_new_chunks}")
         return list(documents), metadatas, ids, count_new_chunks
 
-    def _format_result(self, results):
+    @staticmethod
+    def _format_result(results):
         return [
             (Document(page_content=result[0], metadata=result[1] or {}), result[2])
             for result in zip(
@@ -458,8 +456,8 @@ class EmbedChain(JSONSerializable):
         config: Optional[BaseLlmConfig] = None,
         where=None,
         citations: bool = False,
-        **kwargs: Optional[Dict[str, Any]],
-    ) -> Union[List[Tuple[str, str, str]], List[str]]:
+        **kwargs: Optional[dict[str, Any]],
+    ) -> Union[list[tuple[str, str, str]], list[str]]:
         """
         Queries the vector database based on the given input query.
         Gets relevant doc based on the query
@@ -473,7 +471,7 @@ class EmbedChain(JSONSerializable):
         :param citations: A boolean to indicate if db should fetch citation source
         :type citations: bool
         :return: List of contents of the document that matched your query
-        :rtype: List[str]
+        :rtype: list[str]
         """
         query_config = config or self.llm.config
         if where is not None:
@@ -501,10 +499,10 @@ class EmbedChain(JSONSerializable):
         input_query: str,
         config: BaseLlmConfig = None,
         dry_run=False,
-        where: Optional[Dict] = None,
+        where: Optional[dict] = None,
         citations: bool = False,
-        **kwargs: Dict[str, Any],
-    ) -> Union[Tuple[str, List[Tuple[str, Dict]]], str]:
+        **kwargs: dict[str, Any],
+    ) -> Union[tuple[str, list[tuple[str, dict]]], str]:
         """
         Queries the vector database based on the given input query.
         Gets relevant doc based on the query and then passes it to an
@@ -519,13 +517,13 @@ class EmbedChain(JSONSerializable):
         the LLM. The purpose is to test the prompt, not the response., defaults to False
         :type dry_run: bool, optional
         :param where: A dictionary of key-value pairs to filter the database results., defaults to None
-        :type where: Optional[Dict[str, str]], optional
+        :type where: Optional[dict[str, str]], optional
         :param kwargs: To read more params for the query function. Ex. we use citations boolean
         param to return context along with the answer
-        :type kwargs: Dict[str, Any]
+        :type kwargs: dict[str, Any]
         :return: The answer to the query, with citations if the citation flag is True
         or the dry run result
-        :rtype: str, if citations is False, otherwise Tuple[str,List[Tuple[str,str,str]]]
+        :rtype: str, if citations is False, otherwise tuple[str, list[tuple[str,str,str]]]
         """
         contexts = self._retrieve_from_database(
             input_query=input_query, config=config, where=where, citations=citations, **kwargs
@@ -566,10 +564,10 @@ class EmbedChain(JSONSerializable):
         config: Optional[BaseLlmConfig] = None,
         dry_run=False,
         session_id: str = "default",
-        where: Optional[Dict[str, str]] = None,
+        where: Optional[dict[str, str]] = None,
         citations: bool = False,
-        **kwargs: Dict[str, Any],
-    ) -> Union[Tuple[str, List[Tuple[str, Dict]]], str]:
+        **kwargs: dict[str, Any],
+    ) -> Union[tuple[str, list[tuple[str, dict]]], str]:
         """
         Queries the vector database on the given input query.
         Gets relevant doc based on the query and then passes it to an
@@ -588,13 +586,13 @@ class EmbedChain(JSONSerializable):
         :param session_id: The session id to use for chat history, defaults to 'default'.
         :type session_id: Optional[str], optional
         :param where: A dictionary of key-value pairs to filter the database results., defaults to None
-        :type where: Optional[Dict[str, str]], optional
+        :type where: Optional[dict[str, str]], optional
         :param kwargs: To read more params for the query function. Ex. we use citations boolean
         param to return context along with the answer
-        :type kwargs: Dict[str, Any]
+        :type kwargs: dict[str, Any]
         :return: The answer to the query, with citations if the citation flag is True
         or the dry run result
-        :rtype: str, if citations is False, otherwise Tuple[str,List[Tuple[str,str,str]]]
+        :rtype: str, if citations is False, otherwise tuple[str, list[tuple[str,str,str]]]
         """
         contexts = self._retrieve_from_database(
             input_query=input_query, config=config, where=where, citations=citations, **kwargs
@@ -659,13 +657,32 @@ class EmbedChain(JSONSerializable):
         self.db.reset()
         self.cursor.execute("DELETE FROM data_sources WHERE pipeline_id = ?", (self.config.id,))
         self.connection.commit()
-        self.delete_chat_history()
+        self.delete_all_chat_history(app_id=self.config.id)
         # Send anonymous telemetry
         self.telemetry.capture(event_name="reset", properties=self._telemetry_props)
 
-    def get_history(self, num_rounds: int = 10, display_format: bool = True):
-        return self.llm.memory.get(app_id=self.config.id, num_rounds=num_rounds, display_format=display_format)
+    def get_history(self, num_rounds: int = 10, display_format: bool = True, session_id: Optional[str] = "default"):
+        history = self.llm.memory.get(
+            app_id=self.config.id, session_id=session_id, num_rounds=num_rounds, display_format=display_format
+        )
+        return history
 
-    def delete_chat_history(self, session_id: str = "default"):
+    def delete_session_chat_history(self, session_id: str = "default"):
         self.llm.memory.delete(app_id=self.config.id, session_id=session_id)
         self.llm.update_history(app_id=self.config.id)
+
+    def delete_all_chat_history(self, app_id: str):
+        self.llm.memory.delete(app_id=app_id)
+        self.llm.update_history(app_id=app_id)
+
+    def delete(self, source_id: str):
+        """
+        Deletes the data from the database.
+        :param source_hash: The hash of the source.
+        :type source_hash: str
+        """
+        self.db.delete(where={"hash": source_id})
+        logging.info(f"Successfully deleted {source_id}")
+        # Send anonymous telemetry
+        if self.config.collect_metrics:
+            self.telemetry.capture(event_name="delete", properties=self._telemetry_props)

embedchain/embedder/base.py

@@ -1,4 +1,5 @@
-from typing import Any, Callable, Optional
+from collections.abc import Callable
+from typing import Any, Optional
 
 from embedchain.config.embedder.base import BaseEmbedderConfig
 
@@ -29,7 +30,7 @@ class BaseEmbedder:
 
     def __init__(self, config: Optional[BaseEmbedderConfig] = None):
         """
-        Intialize the embedder class.
+        Initialize the embedder class.
 
         :param config: embedder configuration option class, defaults to None
         :type config: Optional[BaseEmbedderConfig], optional

embedchain/embedder/google.py

@@ -1,4 +1,4 @@
-from typing import Optional
+from typing import Optional, Union
 
 import google.generativeai as genai
 from chromadb import EmbeddingFunction, Embeddings
@@ -13,12 +13,19 @@ class GoogleAIEmbeddingFunction(EmbeddingFunction):
         super().__init__()
         self.config = config or GoogleAIEmbedderConfig()
 
-    def __call__(self, input: str) -> Embeddings:
+    def __call__(self, input: Union[list[str], str]) -> Embeddings:
         model = self.config.model
         title = self.config.title
         task_type = self.config.task_type
-        embeddings = genai.embed_content(model=model, content=input, task_type=task_type, title=title)
-        return embeddings["embedding"]
+        if isinstance(input, str):
+            input_ = [input]
+        else:
+            input_ = input
+        data = genai.embed_content(model=model, content=input_, task_type=task_type, title=title)
+        embeddings = data["embedding"]
+        if isinstance(input_, str):
+            embeddings = [embeddings]
+        return embeddings
 
 
 class GoogleAIEmbedder(BaseEmbedder):
@@ -27,5 +34,5 @@ class GoogleAIEmbedder(BaseEmbedder):
         embedding_fn = GoogleAIEmbeddingFunction(config=config)
         self.set_embedding_fn(embedding_fn=embedding_fn)
 
-        vector_dimension = VectorDimensions.GOOGLE_AI.value
+        vector_dimension = self.config.vector_dimension or VectorDimensions.GOOGLE_AI.value
         self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/embedder/gpt4all.py

@@ -16,5 +16,5 @@ class GPT4AllEmbedder(BaseEmbedder):
         embedding_fn = BaseEmbedder._langchain_default_concept(embeddings)
         self.set_embedding_fn(embedding_fn=embedding_fn)
 
-        vector_dimension = VectorDimensions.GPT4ALL.value
+        vector_dimension = self.config.vector_dimension or VectorDimensions.GPT4ALL.value
         self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/embedder/huggingface.py

@@ -15,5 +15,5 @@ class HuggingFaceEmbedder(BaseEmbedder):
         embedding_fn = BaseEmbedder._langchain_default_concept(embeddings)
         self.set_embedding_fn(embedding_fn=embedding_fn)
 
-        vector_dimension = VectorDimensions.HUGGING_FACE.value
+        vector_dimension = self.config.vector_dimension or VectorDimensions.HUGGING_FACE.value
         self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/embedder/mistralai.py

@@ -0,0 +1,46 @@
+import os
+from typing import Optional, Union
+
+from chromadb import EmbeddingFunction, Embeddings
+
+from embedchain.config import BaseEmbedderConfig
+from embedchain.embedder.base import BaseEmbedder
+from embedchain.models import VectorDimensions
+
+
+class MistralAIEmbeddingFunction(EmbeddingFunction):
+    def __init__(self, config: BaseEmbedderConfig) -> None:
+        super().__init__()
+        try:
+            from langchain_mistralai import MistralAIEmbeddings
+        except ModuleNotFoundError:
+            raise ModuleNotFoundError(
+                "The required dependencies for MistralAI are not installed."
+                'Please install with `pip install --upgrade "embedchain[mistralai]"`'
+            ) from None
+        self.config = config
+        api_key = self.config.api_key or os.getenv("MISTRAL_API_KEY")
+        self.client = MistralAIEmbeddings(mistral_api_key=api_key)
+        self.client.model = self.config.model
+
+    def __call__(self, input: Union[list[str], str]) -> Embeddings:
+        if isinstance(input, str):
+            input_ = [input]
+        else:
+            input_ = input
+        response = self.client.embed_documents(input_)
+        return response
+
+
+class MistralAIEmbedder(BaseEmbedder):
+    def __init__(self, config: Optional[BaseEmbedderConfig] = None):
+        super().__init__(config)
+
+        if self.config.model is None:
+            self.config.model = "mistral-embed"
+
+        embedding_fn = MistralAIEmbeddingFunction(config=self.config)
+        self.set_embedding_fn(embedding_fn=embedding_fn)
+
+        vector_dimension = self.config.vector_dimension or VectorDimensions.MISTRAL_AI.value
+        self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/embedder/openai.py

@@ -32,4 +32,5 @@ class OpenAIEmbedder(BaseEmbedder):
                 model_name=self.config.model,
             )
         self.set_embedding_fn(embedding_fn=embedding_fn)
-        self.set_vector_dimension(vector_dimension=VectorDimensions.OPENAI.value)
+        vector_dimension = self.config.vector_dimension or VectorDimensions.OPENAI.value
+        self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/embedder/vertexai.py

@@ -15,5 +15,5 @@ class VertexAIEmbedder(BaseEmbedder):
         embedding_fn = BaseEmbedder._langchain_default_concept(embeddings)
         self.set_embedding_fn(embedding_fn=embedding_fn)
 
-        vector_dimension = VectorDimensions.VERTEX_AI.value
+        vector_dimension = self.config.vector_dimension or VectorDimensions.VERTEX_AI.value
         self.set_vector_dimension(vector_dimension=vector_dimension)

embedchain/evaluation/base.py

@@ -0,0 +1,29 @@
+from abc import ABC, abstractmethod
+
+from embedchain.utils.evaluation import EvalData
+
+
+class BaseMetric(ABC):
+    """Base class for a metric.
+
+    This class provides a common interface for all metrics.
+    """
+
+    def __init__(self, name: str = "base_metric"):
+        """
+        Initialize the BaseMetric.
+        """
+        self.name = name
+
+    @abstractmethod
+    def evaluate(self, dataset: list[EvalData]):
+        """
+        Abstract method to evaluate the dataset.
+
+        This method should be implemented by subclasses to perform the actual
+        evaluation on the dataset.
+
+        :param dataset: dataset to evaluate
+        :type dataset: list[EvalData]
+        """
+        raise NotImplementedError()

embedchain/evaluation/metrics/__init__.py

@@ -0,0 +1,3 @@
+from .answer_relevancy import AnswerRelevance  # noqa: F401
+from .context_relevancy import ContextRelevance  # noqa: F401
+from .groundedness import Groundedness  # noqa: F401

embedchain/evaluation/metrics/answer_relevancy.py

@@ -0,0 +1,93 @@
+import concurrent.futures
+import logging
+import os
+from string import Template
+from typing import Optional
+
+import numpy as np
+from openai import OpenAI
+from tqdm import tqdm
+
+from embedchain.config.evaluation.base import AnswerRelevanceConfig
+from embedchain.evaluation.base import BaseMetric
+from embedchain.utils.evaluation import EvalData, EvalMetric
+
+
+class AnswerRelevance(BaseMetric):
+    """
+    Metric for evaluating the relevance of answers.
+    """
+
+    def __init__(self, config: Optional[AnswerRelevanceConfig] = AnswerRelevanceConfig()):
+        super().__init__(name=EvalMetric.ANSWER_RELEVANCY.value)
+        self.config = config
+        api_key = self.config.api_key or os.getenv("OPENAI_API_KEY")
+        if not api_key:
+            raise ValueError("API key not found. Set 'OPENAI_API_KEY' or pass it in the config.")
+        self.client = OpenAI(api_key=api_key)
+
+    def _generate_prompt(self, data: EvalData) -> str:
+        """
+        Generates a prompt based on the provided data.
+        """
+        return Template(self.config.prompt).substitute(
+            num_gen_questions=self.config.num_gen_questions, answer=data.answer
+        )
+
+    def _generate_questions(self, prompt: str) -> list[str]:
+        """
+        Generates questions from the prompt.
+        """
+        response = self.client.chat.completions.create(
+            model=self.config.model,
+            messages=[{"role": "user", "content": prompt}],
+        )
+        return response.choices[0].message.content.strip().split("\n")
+
+    def _generate_embedding(self, question: str) -> np.ndarray:
+        """
+        Generates the embedding for a question.
+        """
+        response = self.client.embeddings.create(
+            input=question,
+            model=self.config.embedder,
+        )
+        return np.array(response.data[0].embedding)
+
+    def _compute_similarity(self, original: np.ndarray, generated: np.ndarray) -> float:
+        """
+        Computes the cosine similarity between two embeddings.
+        """
+        original = original.reshape(1, -1)
+        norm = np.linalg.norm(original) * np.linalg.norm(generated, axis=1)
+        return np.dot(generated, original.T).flatten() / norm
+
+    def _compute_score(self, data: EvalData) -> float:
+        """
+        Computes the relevance score for a given data item.
+        """
+        prompt = self._generate_prompt(data)
+        generated_questions = self._generate_questions(prompt)
+        original_embedding = self._generate_embedding(data.question)
+        generated_embeddings = np.array([self._generate_embedding(q) for q in generated_questions])
+        similarities = self._compute_similarity(original_embedding, generated_embeddings)
+        return np.mean(similarities)
+
+    def evaluate(self, dataset: list[EvalData]) -> float:
+        """
+        Evaluates the dataset and returns the average answer relevance score.
+        """
+        results = []
+
+        with concurrent.futures.ThreadPoolExecutor() as executor:
+            future_to_data = {executor.submit(self._compute_score, data): data for data in dataset}
+            for future in tqdm(
+                concurrent.futures.as_completed(future_to_data), total=len(dataset), desc="Evaluating Answer Relevancy"
+            ):
+                data = future_to_data[future]
+                try:
+                    results.append(future.result())
+                except Exception as e:
+                    logging.error(f"Error evaluating answer relevancy for {data}: {e}")
+
+        return np.mean(results) if results else 0.0

embedchain/evaluation/metrics/context_relevancy.py

@@ -0,0 +1,69 @@
+import concurrent.futures
+import os
+from string import Template
+from typing import Optional
+
+import numpy as np
+import pysbd
+from openai import OpenAI
+from tqdm import tqdm
+
+from embedchain.config.evaluation.base import ContextRelevanceConfig
+from embedchain.evaluation.base import BaseMetric
+from embedchain.utils.evaluation import EvalData, EvalMetric
+
+
+class ContextRelevance(BaseMetric):
+    """
+    Metric for evaluating the relevance of context in a dataset.
+    """
+
+    def __init__(self, config: Optional[ContextRelevanceConfig] = ContextRelevanceConfig()):
+        super().__init__(name=EvalMetric.CONTEXT_RELEVANCY.value)
+        self.config = config
+        api_key = self.config.api_key or os.getenv("OPENAI_API_KEY")
+        if not api_key:
+            raise ValueError("API key not found. Set 'OPENAI_API_KEY' or pass it in the config.")
+        self.client = OpenAI(api_key=api_key)
+        self._sbd = pysbd.Segmenter(language=self.config.language, clean=False)
+
+    def _sentence_segmenter(self, text: str) -> list[str]:
+        """
+        Segments the given text into sentences.
+        """
+        return self._sbd.segment(text)
+
+    def _compute_score(self, data: EvalData) -> float:
+        """
+        Computes the context relevance score for a given data item.
+        """
+        original_context = "\n".join(data.contexts)
+        prompt = Template(self.config.prompt).substitute(context=original_context, question=data.question)
+        response = self.client.chat.completions.create(
+            model=self.config.model, messages=[{"role": "user", "content": prompt}]
+        )
+        useful_context = response.choices[0].message.content.strip()
+        useful_context_sentences = self._sentence_segmenter(useful_context)
+        original_context_sentences = self._sentence_segmenter(original_context)
+
+        if not original_context_sentences:
+            return 0.0
+        return len(useful_context_sentences) / len(original_context_sentences)
+
+    def evaluate(self, dataset: list[EvalData]) -> float:
+        """
+        Evaluates the dataset and returns the average context relevance score.
+        """
+        scores = []
+
+        with concurrent.futures.ThreadPoolExecutor() as executor:
+            futures = [executor.submit(self._compute_score, data) for data in dataset]
+            for future in tqdm(
+                concurrent.futures.as_completed(futures), total=len(dataset), desc="Evaluating Context Relevancy"
+            ):
+                try:
+                    scores.append(future.result())
+                except Exception as e:
+                    print(f"Error during evaluation: {e}")
+
+        return np.mean(scores) if scores else 0.0

embedchain/evaluation/metrics/groundedness.py

@@ -0,0 +1,102 @@
+import concurrent.futures
+import logging
+import os
+from string import Template
+from typing import Optional
+
+import numpy as np
+from openai import OpenAI
+from tqdm import tqdm
+
+from embedchain.config.evaluation.base import GroundednessConfig
+from embedchain.evaluation.base import BaseMetric
+from embedchain.utils.evaluation import EvalData, EvalMetric
+
+
+class Groundedness(BaseMetric):
+    """
+    Metric for groundedness of answer from the given contexts.
+    """
+
+    def __init__(self, config: Optional[GroundednessConfig] = None):
+        super().__init__(name=EvalMetric.GROUNDEDNESS.value)
+        self.config = config or GroundednessConfig()
+        api_key = self.config.api_key or os.getenv("OPENAI_API_KEY")
+        if not api_key:
+            raise ValueError("Please set the OPENAI_API_KEY environment variable or pass the `api_key` in config.")
+        self.client = OpenAI(api_key=api_key)
+
+    def _generate_answer_claim_prompt(self, data: EvalData) -> str:
+        """
+        Generate the prompt for the given data.
+        """
+        prompt = Template(self.config.answer_claims_prompt).substitute(question=data.question, answer=data.answer)
+        return prompt
+
+    def _get_claim_statements(self, prompt: str) -> np.ndarray:
+        """
+        Get claim statements from the answer.
+        """
+        response = self.client.chat.completions.create(
+            model=self.config.model,
+            messages=[{"role": "user", "content": f"{prompt}"}],
+        )
+        result = response.choices[0].message.content.strip()
+        claim_statements = np.array([statement for statement in result.split("\n") if statement])
+        return claim_statements
+
+    def _generate_claim_inference_prompt(self, data: EvalData, claim_statements: list[str]) -> str:
+        """
+        Generate the claim inference prompt for the given data and claim statements.
+        """
+        prompt = Template(self.config.claims_inference_prompt).substitute(
+            context="\n".join(data.contexts), claim_statements="\n".join(claim_statements)
+        )
+        return prompt
+
+    def _get_claim_verdict_scores(self, prompt: str) -> np.ndarray:
+        """
+        Get verdicts for claim statements.
+        """
+        response = self.client.chat.completions.create(
+            model=self.config.model,
+            messages=[{"role": "user", "content": f"{prompt}"}],
+        )
+        result = response.choices[0].message.content.strip()
+        claim_verdicts = result.split("\n")
+        verdict_score_map = {"1": 1, "0": 0, "-1": np.nan}
+        verdict_scores = np.array([verdict_score_map[verdict] for verdict in claim_verdicts])
+        return verdict_scores
+
+    def _compute_score(self, data: EvalData) -> float:
+        """
+        Compute the groundedness score for a single data point.
+        """
+        answer_claims_prompt = self._generate_answer_claim_prompt(data)
+        claim_statements = self._get_claim_statements(answer_claims_prompt)
+
+        claim_inference_prompt = self._generate_claim_inference_prompt(data, claim_statements)
+        verdict_scores = self._get_claim_verdict_scores(claim_inference_prompt)
+        return np.sum(verdict_scores) / claim_statements.size
+
+    def evaluate(self, dataset: list[EvalData]):
+        """
+        Evaluate the dataset and returns the average groundedness score.
+        """
+        results = []
+
+        with concurrent.futures.ThreadPoolExecutor() as executor:
+            future_to_data = {executor.submit(self._compute_score, data): data for data in dataset}
+            for future in tqdm(
+                concurrent.futures.as_completed(future_to_data),
+                total=len(future_to_data),
+                desc="Evaluating Groundedness",
+            ):
+                data = future_to_data[future]
+                try:
+                    score = future.result()
+                    results.append(score)
+                except Exception as e:
+                    logging.error(f"Error while evaluating groundedness for data point {data}: {e}")
+
+        return np.mean(results) if results else 0.0

embedchain/factory.py

@@ -21,6 +21,8 @@ class LlmFactory:
         "openai": "embedchain.llm.openai.OpenAILlm",
         "vertexai": "embedchain.llm.vertex_ai.VertexAILlm",
         "google": "embedchain.llm.google.GoogleLlm",
+        "aws_bedrock": "embedchain.llm.aws_bedrock.AWSBedrockLlm",
+        "mistralai": "embedchain.llm.mistralai.MistralAILlm",
     }
     provider_to_config_class = {
         "embedchain": "embedchain.config.llm.base.BaseLlmConfig",
@@ -50,6 +52,7 @@ class EmbedderFactory:
         "openai": "embedchain.embedder.openai.OpenAIEmbedder",
         "vertexai": "embedchain.embedder.vertexai.VertexAIEmbedder",
         "google": "embedchain.embedder.google.GoogleAIEmbedder",
+        "mistralai": "embedchain.embedder.mistralai.MistralAIEmbedder",
     }
     provider_to_config_class = {
         "azure_openai": "embedchain.config.embedder.base.BaseEmbedderConfig",

embedchain/helpers/callbacks.py

@@ -1,5 +1,5 @@
 import queue
-from typing import Any, Dict, List, Union
+from typing import Any, Union
 
 from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
 from langchain.schema import LLMResult
@@ -29,7 +29,7 @@ class StreamingStdOutCallbackHandlerYield(StreamingStdOutCallbackHandler):
         super().__init__()
         self.q = q
 
-    def on_llm_start(self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) -> None:
+    def on_llm_start(self, serialized: dict[str, Any], prompts: list[str], **kwargs: Any) -> None:
         """Run when LLM starts running."""
         with self.q.mutex:
             self.q.queue.clear()

embedchain/helpers/json_serializable.py

@@ -1,7 +1,7 @@
 import json
 import logging
 from string import Template
-from typing import Any, Dict, Type, TypeVar, Union
+from typing import Any, Type, TypeVar, Union
 
 T = TypeVar("T", bound="JSONSerializable")
 
@@ -42,7 +42,7 @@ class JSONSerializable:
     A class to represent a JSON serializable object.
 
     This class provides methods to serialize and deserialize objects,
-    as well as save serialized objects to a file and load them back.
+    as well as to save serialized objects to a file and load them back.
     """
 
     _deserializable_classes = set()  # Contains classes that are whitelisted for deserialization.
@@ -84,7 +84,7 @@ class JSONSerializable:
             return cls()
 
     @staticmethod
-    def _auto_encoder(obj: Any) -> Union[Dict[str, Any], None]:
+    def _auto_encoder(obj: Any) -> Union[dict[str, Any], None]:
         """
         Automatically encode an object for JSON serialization.
 
@@ -126,7 +126,7 @@ class JSONSerializable:
         raise TypeError(f"Object of type {type(obj)} is not JSON serializable")
 
     @classmethod
-    def _auto_decoder(cls, dct: Dict[str, Any]) -> Any:
+    def _auto_decoder(cls, dct: dict[str, Any]) -> Any:
         """
         Automatically decode a dictionary to an object during JSON deserialization.
 

embedchain/llm/aws_bedrock.py

@@ -0,0 +1,48 @@
+from typing import Optional
+
+from langchain.llms import Bedrock
+
+from embedchain.config import BaseLlmConfig
+from embedchain.helpers.json_serializable import register_deserializable
+from embedchain.llm.base import BaseLlm
+
+
+@register_deserializable
+class AWSBedrockLlm(BaseLlm):
+    def __init__(self, config: Optional[BaseLlmConfig] = None):
+        super().__init__(config)
+
+    def get_llm_model_answer(self, prompt) -> str:
+        response = self._get_answer(prompt, self.config)
+        return response
+
+    def _get_answer(self, prompt: str, config: BaseLlmConfig) -> str:
+        try:
+            import boto3
+        except ModuleNotFoundError:
+            raise ModuleNotFoundError(
+                "The required dependencies for AWSBedrock are not installed."
+                'Please install with `pip install --upgrade "embedchain[aws-bedrock]"`'
+            ) from None
+
+        self.boto_client = boto3.client("bedrock-runtime", "us-west-2")
+
+        kwargs = {
+            "model_id": config.model or "amazon.titan-text-express-v1",
+            "client": self.boto_client,
+            "model_kwargs": config.model_kwargs
+            or {
+                "temperature": config.temperature,
+            },
+        }
+
+        if config.stream:
+            from langchain.callbacks.streaming_stdout import \
+                StreamingStdOutCallbackHandler
+
+            callbacks = [StreamingStdOutCallbackHandler()]
+            llm = Bedrock(**kwargs, streaming=config.stream, callbacks=callbacks)
+        else:
+            llm = Bedrock(**kwargs)
+
+        return llm(prompt)

embedchain/llm/base.py

@@ -1,5 +1,6 @@
 import logging
-from typing import Any, Dict, Generator, List, Optional
+from collections.abc import Generator
+from typing import Any, Optional
 
 from langchain.schema import BaseMessage as LCBaseMessage
 
@@ -55,7 +56,7 @@ class BaseLlm(JSONSerializable):
         app_id: str,
         question: str,
         answer: str,
-        metadata: Optional[Dict[str, Any]] = None,
+        metadata: Optional[dict[str, Any]] = None,
         session_id: str = "default",
     ):
         chat_message = ChatMessage()
@@ -64,7 +65,7 @@ class BaseLlm(JSONSerializable):
         self.memory.add(app_id=app_id, chat_message=chat_message, session_id=session_id)
         self.update_history(app_id=app_id, session_id=session_id)
 
-    def generate_prompt(self, input_query: str, contexts: List[str], **kwargs: Dict[str, Any]) -> str:
+    def generate_prompt(self, input_query: str, contexts: list[str], **kwargs: dict[str, Any]) -> str:
         """
         Generates a prompt based on the given query and context, ready to be
         passed to an LLM
@@ -72,11 +73,11 @@ class BaseLlm(JSONSerializable):
         :param input_query: The query to use.
         :type input_query: str
         :param contexts: List of similar documents to the query used as context.
-        :type contexts: List[str]
+        :type contexts: list[str]
         :return: The prompt
         :rtype: str
         """
-        context_string = (" | ").join(contexts)
+        context_string = " | ".join(contexts)
         web_search_result = kwargs.get("web_search_result", "")
         if web_search_result:
             context_string = self._append_search_and_context(context_string, web_search_result)
@@ -110,7 +111,8 @@ class BaseLlm(JSONSerializable):
             prompt = self.config.prompt.substitute(context=context_string, query=input_query)
         return prompt
 
-    def _append_search_and_context(self, context: str, web_search_result: str) -> str:
+    @staticmethod
+    def _append_search_and_context(context: str, web_search_result: str) -> str:
         """Append web search context to existing context
 
         :param context: Existing context
@@ -134,7 +136,8 @@ class BaseLlm(JSONSerializable):
         """
         return self.get_llm_model_answer(prompt)
 
-    def access_search_and_get_results(self, input_query: str):
+    @staticmethod
+    def access_search_and_get_results(input_query: str):
         """
         Search the internet for additional context
 
@@ -153,7 +156,8 @@ class BaseLlm(JSONSerializable):
         logging.info(f"Access search to get answers for {input_query}")
         return search.run(input_query)
 
-    def _stream_response(self, answer: Any) -> Generator[Any, Any, None]:
+    @staticmethod
+    def _stream_response(answer: Any) -> Generator[Any, Any, None]:
         """Generator to be used as streaming response
 
         :param answer: Answer chunk from llm
@@ -167,7 +171,7 @@ class BaseLlm(JSONSerializable):
             yield chunk
         logging.info(f"Answer: {streamed_answer}")
 
-    def query(self, input_query: str, contexts: List[str], config: BaseLlmConfig = None, dry_run=False):
+    def query(self, input_query: str, contexts: list[str], config: BaseLlmConfig = None, dry_run=False):
         """
         Queries the vector database based on the given input query.
         Gets relevant doc based on the query and then passes it to an
@@ -176,7 +180,7 @@ class BaseLlm(JSONSerializable):
         :param input_query: The query to use.
         :type input_query: str
         :param contexts: Embeddings retrieved from the database to be used as context.
-        :type contexts: List[str]
+        :type contexts: list[str]
         :param config: The `BaseLlmConfig` instance to use as configuration options. This is used for one method call.
         To persistently use a config, declare it during app init., defaults to None
         :type config: Optional[BaseLlmConfig], optional
@@ -220,7 +224,7 @@ class BaseLlm(JSONSerializable):
                 self.config: BaseLlmConfig = BaseLlmConfig.deserialize(prev_config)
 
     def chat(
-        self, input_query: str, contexts: List[str], config: BaseLlmConfig = None, dry_run=False, session_id: str = None
+        self, input_query: str, contexts: list[str], config: BaseLlmConfig = None, dry_run=False, session_id: str = None
     ):
         """
         Queries the vector database on the given input query.
@@ -232,7 +236,7 @@ class BaseLlm(JSONSerializable):
         :param input_query: The query to use.
         :type input_query: str
         :param contexts: Embeddings retrieved from the database to be used as context.
-        :type contexts: List[str]
+        :type contexts: list[str]
         :param config: The `BaseLlmConfig` instance to use as configuration options. This is used for one method call.
         To persistently use a config, declare it during app init., defaults to None
         :type config: Optional[BaseLlmConfig], optional
@@ -278,7 +282,7 @@ class BaseLlm(JSONSerializable):
                 self.config: BaseLlmConfig = BaseLlmConfig.deserialize(prev_config)
 
     @staticmethod
-    def _get_messages(prompt: str, system_prompt: Optional[str] = None) -> List[LCBaseMessage]:
+    def _get_messages(prompt: str, system_prompt: Optional[str] = None) -> list[LCBaseMessage]:
         """
         Construct a list of langchain messages
 
@@ -287,7 +291,7 @@ class BaseLlm(JSONSerializable):
         :param system_prompt: System prompt, defaults to None
         :type system_prompt: Optional[str], optional
         :return: List of messages
-        :rtype: List[BaseMessage]
+        :rtype: list[BaseMessage]
         """
         from langchain.schema import HumanMessage, SystemMessage
 

embedchain/llm/google.py

@@ -1,7 +1,8 @@
 import importlib
 import logging
 import os
-from typing import Any, Generator, Optional, Union
+from collections.abc import Generator
+from typing import Any, Optional, Union
 
 import google.generativeai as genai
 
@@ -44,7 +45,7 @@ class GoogleLlm(BaseLlm):
             "temperature": self.config.temperature or 0.5,
         }
 
-        if self.config.top_p >= 0.0 and self.config.top_p <= 1.0:
+        if 0.0 <= self.config.top_p <= 1.0:
             generation_config_params["top_p"] = self.config.top_p
         else:
             raise ValueError("`top_p` must be > 0.0 and < 1.0")

embedchain/llm/gpt4all.py

@@ -1,6 +1,7 @@
 import os
+from collections.abc import Iterable
 from pathlib import Path
-from typing import Iterable, Optional, Union
+from typing import Optional, Union
 
 from langchain.callbacks.stdout import StdOutCallbackHandler
 from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

embedchain/llm/huggingface.py

@@ -3,6 +3,7 @@ import logging
 import os
 from typing import Optional
 
+from langchain.llms.huggingface_endpoint import HuggingFaceEndpoint
 from langchain.llms.huggingface_hub import HuggingFaceHub
 
 from embedchain.config import BaseLlmConfig
@@ -33,12 +34,21 @@ class HuggingFaceLlm(BaseLlm):
 
     @staticmethod
     def _get_answer(prompt: str, config: BaseLlmConfig) -> str:
+        if config.model:
+            return HuggingFaceLlm._from_model(prompt=prompt, config=config)
+        elif config.endpoint:
+            return HuggingFaceLlm._from_endpoint(prompt=prompt, config=config)
+        else:
+            raise ValueError("Either `model` or `endpoint` must be set")
+
+    @staticmethod
+    def _from_model(prompt: str, config: BaseLlmConfig) -> str:
         model_kwargs = {
             "temperature": config.temperature or 0.1,
             "max_new_tokens": config.max_tokens,
         }
 
-        if config.top_p > 0.0 and config.top_p < 1.0:
+        if 0.0 < config.top_p < 1.0:
             model_kwargs["top_p"] = config.top_p
         else:
             raise ValueError("`top_p` must be > 0.0 and < 1.0")
@@ -52,3 +62,13 @@ class HuggingFaceLlm(BaseLlm):
         )
 
         return llm(prompt)
+
+    @staticmethod
+    def _from_endpoint(prompt: str, config: BaseLlmConfig) -> str:
+        llm = HuggingFaceEndpoint(
+            huggingfacehub_api_token=os.environ["HUGGINGFACE_ACCESS_TOKEN"],
+            endpoint_url=config.endpoint,
+            task="text-generation",
+            model_kwargs=config.model_kwargs,
+        )
+        return llm(prompt)

embedchain/llm/mistralai.py

@@ -0,0 +1,52 @@
+import os
+from typing import Optional
+
+from embedchain.config import BaseLlmConfig
+from embedchain.helpers.json_serializable import register_deserializable
+from embedchain.llm.base import BaseLlm
+
+
+@register_deserializable
+class MistralAILlm(BaseLlm):
+    def __init__(self, config: Optional[BaseLlmConfig] = None):
+        super().__init__(config)
+        if not self.config.api_key and "MISTRAL_API_KEY" not in os.environ:
+            raise ValueError("Please set the MISTRAL_API_KEY environment variable or pass it in the config.")
+
+    def get_llm_model_answer(self, prompt):
+        return MistralAILlm._get_answer(prompt=prompt, config=self.config)
+
+    @staticmethod
+    def _get_answer(prompt: str, config: BaseLlmConfig):
+        try:
+            from langchain_core.messages import HumanMessage, SystemMessage
+            from langchain_mistralai.chat_models import ChatMistralAI
+        except ModuleNotFoundError:
+            raise ModuleNotFoundError(
+                "The required dependencies for MistralAI are not installed."
+                'Please install with `pip install --upgrade "embedchain[mistralai]"`'
+            ) from None
+
+        api_key = config.api_key or os.getenv("MISTRAL_API_KEY")
+        client = ChatMistralAI(mistral_api_key=api_key)
+        messages = []
+        if config.system_prompt:
+            messages.append(SystemMessage(content=config.system_prompt))
+        messages.append(HumanMessage(content=prompt))
+        kwargs = {
+            "model": config.model or "mistral-tiny",
+            "temperature": config.temperature,
+            "max_tokens": config.max_tokens,
+            "top_p": config.top_p,
+        }
+
+        # TODO: Add support for streaming
+        if config.stream:
+            answer = ""
+            for chunk in client.stream(**kwargs, input=messages):
+                answer += chunk.content
+            return answer
+        else:
+            response = client.invoke(**kwargs, input=messages)
+            answer = response.content
+            return answer

embedchain/llm/ollama.py

@@ -1,4 +1,5 @@
-from typing import Iterable, Optional, Union
+from collections.abc import Iterable
+from typing import Optional, Union
 
 from langchain.callbacks.manager import CallbackManager
 from langchain.callbacks.stdout import StdOutCallbackHandler
@@ -20,7 +21,8 @@ class OllamaLlm(BaseLlm):
     def get_llm_model_answer(self, prompt):
         return self._get_answer(prompt=prompt, config=self.config)
 
-    def _get_answer(self, prompt: str, config: BaseLlmConfig) -> Union[str, Iterable]:
+    @staticmethod
+    def _get_answer(prompt: str, config: BaseLlmConfig) -> Union[str, Iterable]:
         callback_manager = [StreamingStdOutCallbackHandler()] if config.stream else [StdOutCallbackHandler()]
 
         llm = Ollama(

embedchain/llm/openai.py

@@ -1,6 +1,6 @@
 import json
 import os
-from typing import Any, Dict, Optional
+from typing import Any, Optional
 
 from langchain.chat_models import ChatOpenAI
 from langchain.schema import AIMessage, HumanMessage, SystemMessage
@@ -12,7 +12,7 @@ from embedchain.llm.base import BaseLlm
 
 @register_deserializable
 class OpenAILlm(BaseLlm):
-    def __init__(self, config: Optional[BaseLlmConfig] = None, functions: Optional[Dict[str, Any]] = None):
+    def __init__(self, config: Optional[BaseLlmConfig] = None, functions: Optional[dict[str, Any]] = None):
         self.functions = functions
         super().__init__(config=config)
 

embedchain/llm/vllm.py

@@ -0,0 +1,40 @@
+from typing import Iterable, Optional, Union
+
+from langchain.callbacks.manager import CallbackManager
+from langchain.callbacks.stdout import StdOutCallbackHandler
+from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
+from langchain_community.llms import VLLM as BaseVLLM
+
+from embedchain.config import BaseLlmConfig
+from embedchain.helpers.json_serializable import register_deserializable
+from embedchain.llm.base import BaseLlm
+
+
+@register_deserializable
+class VLLM(BaseLlm):
+    def __init__(self, config: Optional[BaseLlmConfig] = None):
+        super().__init__(config=config)
+        if self.config.model is None:
+            self.config.model = "mosaicml/mpt-7b"
+
+    def get_llm_model_answer(self, prompt):
+        return self._get_answer(prompt=prompt, config=self.config)
+
+    @staticmethod
+    def _get_answer(prompt: str, config: BaseLlmConfig) -> Union[str, Iterable]:
+        callback_manager = [StreamingStdOutCallbackHandler()] if config.stream else [StdOutCallbackHandler()]
+
+        # Prepare the arguments for BaseVLLM
+        llm_args = {
+            "model": config.model,
+            "temperature": config.temperature,
+            "top_p": config.top_p,
+            "callback_manager": CallbackManager(callback_manager),
+        }
+
+        # Add model_kwargs if they are not None
+        if config.model_kwargs is not None:
+            llm_args.update(config.model_kwargs)
+
+        llm = BaseVLLM(**llm_args)
+        return llm(prompt)

embedchain/loaders/base_loader.py

@@ -5,7 +5,7 @@ class BaseLoader(JSONSerializable):
     def __init__(self):
         pass
 
-    def load_data():
+    def load_data(self, url):
         """
         Implemented by child classes
         """

embedchain/loaders/directory_loader.py

@@ -1,7 +1,7 @@
 import hashlib
 import logging
 from pathlib import Path
-from typing import Any, Dict, Optional
+from typing import Any, Optional
 
 from embedchain.config import AddConfig
 from embedchain.data_formatter.data_formatter import DataFormatter
@@ -15,7 +15,7 @@ from embedchain.utils.misc import detect_datatype
 class DirectoryLoader(BaseLoader):
     """Load data from a directory."""
 
-    def __init__(self, config: Optional[Dict[str, Any]] = None):
+    def __init__(self, config: Optional[dict[str, Any]] = None):
         super().__init__()
         config = config or {}
         self.recursive = config.get("recursive", True)
@@ -32,7 +32,7 @@ class DirectoryLoader(BaseLoader):
         doc_id = hashlib.sha256((str(data_list) + str(directory_path)).encode()).hexdigest()
 
         for error in self.errors:
-            logging.warn(error)
+            logging.warning(error)
 
         return {"doc_id": doc_id, "data": data_list}
 

embedchain/loaders/discourse.py

@@ -1,7 +1,7 @@
 import hashlib
 import logging
 import time
-from typing import Any, Dict, Optional
+from typing import Any, Optional
 
 import requests
 
@@ -10,7 +10,7 @@ from embedchain.utils.misc import clean_string
 
 
 class DiscourseLoader(BaseLoader):
-    def __init__(self, config: Optional[Dict[str, Any]] = None):
+    def __init__(self, config: Optional[dict[str, Any]] = None):
         super().__init__()
         if not config:
             raise ValueError(

embedchain/loaders/docs_site_loader.py

@@ -49,7 +49,8 @@ class DocsSiteLoader(BaseLoader):
         urls = [link for link in self.visited_links if urlparse(link).netloc == urlparse(url).netloc]
         return urls
 
-    def _load_data_from_url(self, url):
+    @staticmethod
+    def _load_data_from_url(url: str) -> list:
         response = requests.get(url)
         if response.status_code != 200:
             logging.info(f"Failed to fetch the website: {response.status_code}")

embedchain/loaders/dropbox.py

@@ -1,6 +1,5 @@
 import hashlib
 import os
-from typing import List
 
 from dropbox.files import FileMetadata
 
@@ -29,7 +28,7 @@ class DropboxLoader(BaseLoader):
         except exceptions.AuthError as ex:
             raise ValueError("Invalid Dropbox access token. Please verify your token and try again.") from ex
 
-    def _download_folder(self, path: str, local_root: str) -> List[FileMetadata]:
+    def _download_folder(self, path: str, local_root: str) -> list[FileMetadata]:
         """Download a folder from Dropbox and save it preserving the directory structure."""
         entries = self.dbx.files_list_folder(path).entries
         for entry in entries:

embedchain/loaders/github.py

@@ -4,7 +4,7 @@ import logging
 import os
 import re
 import shlex
-from typing import Any, Dict, Optional
+from typing import Any, Optional
 
 from tqdm import tqdm
 
@@ -18,9 +18,9 @@ VALID_SEARCH_TYPES = set(["code", "repo", "pr", "issue", "discussion"])
 
 
 class GithubLoader(BaseLoader):
-    """Load data from github search query."""
+    """Load data from GitHub search query."""
 
-    def __init__(self, config: Optional[Dict[str, Any]] = None):
+    def __init__(self, config: Optional[dict[str, Any]] = None):
         super().__init__()
         if not config:
             raise ValueError(
@@ -48,7 +48,7 @@ class GithubLoader(BaseLoader):
             self.client = None
 
     def _github_search_code(self, query: str):
-        """Search github code."""
+        """Search GitHub code."""
         data = []
         results = self.client.search_code(query)
         for result in tqdm(results, total=results.totalCount, desc="Loading code files from github"):
@@ -66,7 +66,8 @@ class GithubLoader(BaseLoader):
             )
         return data
 
-    def _get_github_repo_data(self, repo_url: str):
+    @staticmethod
+    def _get_github_repo_data(repo_url: str):
         local_hash = hashlib.sha256(repo_url.encode()).hexdigest()
         local_path = f"/tmp/{local_hash}"
         data = []
@@ -121,14 +122,14 @@ class GithubLoader(BaseLoader):
 
         return data
 
-    def _github_search_repo(self, query: str):
-        """Search github repo."""
+    def _github_search_repo(self, query: str) -> list[dict]:
+        """Search GitHub repo."""
         data = []
         logging.info(f"Searching github repos with query: {query}")
         results = self.client.search_repositories(query)
         # Add repo urls and descriptions
         urls = list(map(lambda x: x.html_url, results))
-        discriptions = list(map(lambda x: x.description, results))
+        descriptions = list(map(lambda x: x.description, results))
         data.append(
             {
                 "content": clean_string(desc),
@@ -136,7 +137,7 @@ class GithubLoader(BaseLoader):
                     "url": url,
                 },
             }
-            for url, desc in zip(urls, discriptions)
+            for url, desc in zip(urls, descriptions)
         )
 
         # Add repo contents
@@ -146,8 +147,8 @@ class GithubLoader(BaseLoader):
             data = self._get_github_repo_data(clone_url)
         return data
 
-    def _github_search_issues_and_pr(self, query: str, type: str):
-        """Search github issues and PRs."""
+    def _github_search_issues_and_pr(self, query: str, type: str) -> list[dict]:
+        """Search GitHub issues and PRs."""
         data = []
 
         query = f"{query} is:{type}"
@@ -161,7 +162,7 @@ class GithubLoader(BaseLoader):
             title = result.title
             body = result.body
             if not body:
-                logging.warn(f"Skipping issue because empty content for: {url}")
+                logging.warning(f"Skipping issue because empty content for: {url}")
                 continue
             labels = " ".join([label.name for label in result.labels])
             issue_comments = result.get_comments()
@@ -186,7 +187,7 @@ class GithubLoader(BaseLoader):
 
     # need to test more for discussion
     def _github_search_discussions(self, query: str):
-        """Search github discussions."""
+        """Search GitHub discussions."""
         data = []
 
         query = f"{query} is:discussion"
@@ -202,7 +203,7 @@ class GithubLoader(BaseLoader):
                     title = discussion.title
                     body = discussion.body
                     if not body:
-                        logging.warn(f"Skipping discussion because empty content for: {url}")
+                        logging.warning(f"Skipping discussion because empty content for: {url}")
                         continue
                     comments = []
                     comments_created_at = []
@@ -233,11 +234,14 @@ class GithubLoader(BaseLoader):
             data = self._github_search_issues_and_pr(query, search_type)
         elif search_type == "discussion":
             raise ValueError("GithubLoader does not support searching discussions yet.")
+        else:
+            raise NotImplementedError(f"{search_type} not supported")
 
         return data
 
-    def _get_valid_github_query(self, query: str):
-        """Check if query is valid and return search types and valid github query."""
+    @staticmethod
+    def _get_valid_github_query(query: str):
+        """Check if query is valid and return search types and valid GitHub query."""
         query_terms = shlex.split(query)
         # query must provide repo to load data from
         if len(query_terms) < 1 or "repo:" not in query:
@@ -273,7 +277,7 @@ class GithubLoader(BaseLoader):
         return types, query
 
     def load_data(self, search_query: str, max_results: int = 1000):
-        """Load data from github search query."""
+        """Load data from GitHub search query."""
 
         if not self.client:
             raise ValueError(

embedchain/loaders/gmail.py

@@ -5,7 +5,7 @@ import os
 from email import message_from_bytes
 from email.utils import parsedate_to_datetime
 from textwrap import dedent
-from typing import Dict, List, Optional
+from typing import Optional
 
 from bs4 import BeautifulSoup
 
@@ -57,7 +57,7 @@ class GmailReader:
                 token.write(creds.to_json())
         return creds
 
-    def load_emails(self) -> List[Dict]:
+    def load_emails(self) -> list[dict]:
         response = self.service.users().messages().list(userId="me", q=self.query).execute()
         messages = response.get("messages", [])
 
@@ -67,7 +67,7 @@ class GmailReader:
         raw_message = self.service.users().messages().get(userId="me", id=message_id, format="raw").execute()
         return base64.urlsafe_b64decode(raw_message["raw"])
 
-    def _parse_email(self, raw_email) -> Dict:
+    def _parse_email(self, raw_email) -> dict:
         mime_msg = message_from_bytes(raw_email)
         return {
             "subject": self._get_header(mime_msg, "Subject"),
@@ -124,7 +124,7 @@ class GmailLoader(BaseLoader):
         return {"doc_id": self._generate_doc_id(query, data), "data": data}
 
     @staticmethod
-    def _process_email(email: Dict) -> str:
+    def _process_email(email: dict) -> str:
         content = BeautifulSoup(email["body"], "html.parser").get_text()
         content = clean_string(content)
         return dedent(
@@ -137,6 +137,6 @@ class GmailLoader(BaseLoader):
         )
 
     @staticmethod
-    def _generate_doc_id(query: str, data: List[Dict]) -> str:
+    def _generate_doc_id(query: str, data: list[dict]) -> str:
         content_strings = [email["content"] for email in data]
         return hashlib.sha256((query + ", ".join(content_strings)).encode()).hexdigest()

embedchain/loaders/image.py

@@ -20,7 +20,8 @@ class ImageLoader(BaseLoader):
         self.api_key = api_key or os.environ["OPENAI_API_KEY"]
         self.client = OpenAI(api_key=self.api_key)
 
-    def _encode_image(self, image_path: str):
+    @staticmethod
+    def _encode_image(image_path: str):
         with open(image_path, "rb") as image_file:
             return base64.b64encode(image_file.read()).decode("utf-8")
 

embedchain/loaders/json.py

@@ -2,7 +2,7 @@ import hashlib
 import json
 import os
 import re
-from typing import Dict, List, Union
+from typing import Union
 
 import requests
 
@@ -15,14 +15,15 @@ class JSONReader:
         """Initialize the JSONReader."""
         pass
 
-    def load_data(self, json_data: Union[Dict, str]) -> List[str]:
+    @staticmethod
+    def load_data(json_data: Union[dict, str]) -> list[str]:
         """Load data from a JSON structure.
 
         Args:
-            json_data (Union[Dict, str]): The JSON data to load.
+            json_data (Union[dict, str]): The JSON data to load.
 
         Returns:
-            List[str]: A list of strings representing the leaf nodes of the JSON.
+            list[str]: A list of strings representing the leaf nodes of the JSON.
         """
         if isinstance(json_data, str):
             json_data = json.loads(json_data)
@@ -35,7 +36,7 @@ class JSONReader:
         return ["\n".join(useful_lines)]
 
 
-VALID_URL_PATTERN = "^https:\/\/[0-9A-z.]+.[0-9A-z.]+.[a-z]+\/.*\.json$"
+VALID_URL_PATTERN = "^https:\/\/[0-9A-Za-z]+(\.[0-9A-Za-z]+)*\/[0-9A-Za-z_\/]*\.json$"
 
 
 class JSONLoader(BaseLoader):

embedchain/loaders/mysql.py

@@ -1,13 +1,13 @@
 import hashlib
 import logging
-from typing import Any, Dict, Optional
+from typing import Any, Optional
 
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils.misc import clean_string
 
 
 class MySQLLoader(BaseLoader):
-    def __init__(self, config: Optional[Dict[str, Any]]):
+    def __init__(self, config: Optional[dict[str, Any]]):
         super().__init__()
         if not config:
             raise ValueError(
@@ -20,7 +20,7 @@ class MySQLLoader(BaseLoader):
         self.cursor = None
         self._setup_loader(config=config)
 
-    def _setup_loader(self, config: Dict[str, Any]):
+    def _setup_loader(self, config: dict[str, Any]):
         try:
             import mysql.connector as sqlconnector
         except ImportError as e:
@@ -39,7 +39,8 @@ class MySQLLoader(BaseLoader):
                     Refer `https://docs.embedchain.ai/data-sources/mysql`.",
             )
 
-    def _check_query(self, query):
+    @staticmethod
+    def _check_query(query):
         if not isinstance(query, str):
             raise ValueError(
                 f"Invalid mysql query: {query}",

embedchain/loaders/notion.py

@@ -1,7 +1,7 @@
 import hashlib
 import logging
 import os
-from typing import Any, Dict, List, Optional
+from typing import Any, Optional
 
 import requests
 
@@ -15,7 +15,7 @@ class NotionDocument:
     A simple Document class to hold the text and additional information of a page.
     """
 
-    def __init__(self, text: str, extra_info: Dict[str, Any]):
+    def __init__(self, text: str, extra_info: dict[str, Any]):
         self.text = text
         self.extra_info = extra_info
 
@@ -82,7 +82,7 @@ class NotionPageLoader:
         result_lines = "\n".join(result_lines_arr)
         return result_lines
 
-    def load_data(self, page_ids: List[str]) -> List[NotionDocument]:
+    def load_data(self, page_ids: list[str]) -> list[NotionDocument]:
         """Load data from the given list of page IDs."""
         docs = []
         for page_id in page_ids:

embedchain/loaders/openapi.py

@@ -32,7 +32,7 @@ class OpenAPILoader(BaseLoader):
         file_path = content
         data_content = []
         with OpenAPILoader._get_file_content(content=content) as file:
-            yaml_data = yaml.load(file, Loader=yaml.Loader)
+            yaml_data = yaml.load(file, Loader=yaml.SafeLoader)
             for i, (key, value) in enumerate(yaml_data.items()):
                 string_data = f"{key}: {value}"
                 meta_data = {"url": file_path, "row": i + 1}

embedchain/loaders/pdf_file.py

@@ -15,7 +15,10 @@ from embedchain.utils.misc import clean_string
 class PdfFileLoader(BaseLoader):
     def load_data(self, url):
         """Load data from a PDF file."""
-        loader = PyPDFLoader(url)
+        headers = {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36",  # noqa:E501
+        }
+        loader = PyPDFLoader(url, headers=headers)
         data = []
         all_content = []
         pages = loader.load_and_split()

embedchain/loaders/postgres.py

@@ -1,12 +1,12 @@
 import hashlib
 import logging
-from typing import Any, Dict, Optional
+from typing import Any, Optional
 
 from embedchain.loaders.base_loader import BaseLoader
 
 
 class PostgresLoader(BaseLoader):
-    def __init__(self, config: Optional[Dict[str, Any]] = None):
+    def __init__(self, config: Optional[dict[str, Any]] = None):
         super().__init__()
         if not config:
             raise ValueError(f"Must provide the valid config. Received: {config}")
@@ -15,7 +15,7 @@ class PostgresLoader(BaseLoader):
         self.cursor = None
         self._setup_loader(config=config)
 
-    def _setup_loader(self, config: Dict[str, Any]):
+    def _setup_loader(self, config: dict[str, Any]):
         try:
             import psycopg
         except ImportError as e:
@@ -24,7 +24,6 @@ class PostgresLoader(BaseLoader):
                     Run `pip install --upgrade 'embedchain[postgres]'`"
             ) from e
 
-        config_info = ""
         if "url" in config:
             config_info = config.get("url")
         else:
@@ -37,7 +36,8 @@ class PostgresLoader(BaseLoader):
         self.connection = psycopg.connect(conninfo=config_info)
         self.cursor = self.connection.cursor()
 
-    def _check_query(self, query):
+    @staticmethod
+    def _check_query(query):
         if not isinstance(query, str):
             raise ValueError(
                 f"Invalid postgres query: {query}. Provide the valid source to add from postgres, make sure you are following `https://docs.embedchain.ai/data-sources/postgres`",  # noqa:E501

embedchain/loaders/sitemap.py

@@ -31,10 +31,13 @@ class SitemapLoader(BaseLoader):
     def load_data(self, sitemap_source):
         output = []
         web_page_loader = WebPageLoader()
+        headers = {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36",  # noqa:E501
+        }
 
         if urlparse(sitemap_source).scheme in ("http", "https"):
             try:
-                response = requests.get(sitemap_source)
+                response = requests.get(sitemap_source, headers=headers)
                 response.raise_for_status()
                 soup = BeautifulSoup(response.text, "xml")
             except requests.RequestException as e:

embedchain/loaders/slack.py

@@ -2,7 +2,7 @@ import hashlib
 import logging
 import os
 import ssl
-from typing import Any, Dict, Optional
+from typing import Any, Optional
 
 import certifi
 
@@ -13,18 +13,18 @@ SLACK_API_BASE_URL = "https://www.slack.com/api/"
 
 
 class SlackLoader(BaseLoader):
-    def __init__(self, config: Optional[Dict[str, Any]] = None):
+    def __init__(self, config: Optional[dict[str, Any]] = None):
         super().__init__()
 
-        if config is not None:
-            self.config = config
-        else:
-            self.config = {"base_url": SLACK_API_BASE_URL}
+        self.config = config if config else {}
+
+        if "base_url" not in self.config:
+            self.config["base_url"] = SLACK_API_BASE_URL
 
         self.client = None
         self._setup_loader(self.config)
 
-    def _setup_loader(self, config: Dict[str, Any]):
+    def _setup_loader(self, config: dict[str, Any]):
         try:
             from slack_sdk import WebClient
         except ImportError as e:
@@ -56,7 +56,8 @@ class SlackLoader(BaseLoader):
         )
         logging.info("Slack Loader setup successful!")
 
-    def _check_query(self, query):
+    @staticmethod
+    def _check_query(query):
         if not isinstance(query, str):
             raise ValueError(
                 f"Invalid query passed to Slack loader, found: {query}. Check `https://docs.embedchain.ai/data-sources/slack` to learn more."  # noqa:E501
@@ -73,11 +74,11 @@ class SlackLoader(BaseLoader):
                 query=query,
                 sort="timestamp",
                 sort_dir="desc",
-                count=1000,
+                count=self.config.get("count", 100),
             )
 
             messages = results.get("messages")
-            num_message = results.get("total")
+            num_message = len(messages)
             logging.info(f"Found {num_message} messages for query: {query}")
 
             matches = messages.get("matches", [])
@@ -86,9 +87,13 @@ class SlackLoader(BaseLoader):
                 text = message.get("text")
                 content = clean_string(text)
 
-                message_meta_data_keys = ["channel", "iid", "team", "ts", "type", "user", "username"]
-                meta_data = message.fromkeys(message_meta_data_keys, "")
+                message_meta_data_keys = ["iid", "team", "ts", "type", "user", "username"]
+                meta_data = {}
+                for key in message.keys():
+                    if key in message_meta_data_keys:
+                        meta_data[key] = message.get(key)
                 meta_data.update({"url": url})
+
                 data.append(
                     {
                         "content": content,

embedchain/loaders/unstructured_file.py

@@ -8,7 +8,7 @@ from embedchain.utils.misc import clean_string
 @register_deserializable
 class UnstructuredLoader(BaseLoader):
     def load_data(self, url):
-        """Load data from a Unstructured file."""
+        """Load data from an Unstructured file."""
         try:
             from langchain.document_loaders import UnstructuredFileLoader
         except ImportError:

embedchain/loaders/web_page.py

@@ -21,8 +21,11 @@ class WebPageLoader(BaseLoader):
     _session = requests.Session()
 
     def load_data(self, url):
-        """Load data from a web page using a shared requests session."""
-        response = self._session.get(url, timeout=30)
+        """Load data from a web page using a shared requests' session."""
+        headers = {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36",  # noqa:E501
+        }
+        response = self._session.get(url, headers=headers, timeout=30)
         response.raise_for_status()
         data = response.content
         content = self._get_clean_content(data, url)
@@ -40,7 +43,8 @@ class WebPageLoader(BaseLoader):
             ],
         }
 
-    def _get_clean_content(self, html, url) -> str:
+    @staticmethod
+    def _get_clean_content(html, url) -> str:
         soup = BeautifulSoup(html, "html.parser")
         original_size = len(str(soup.get_text()))
 
@@ -60,8 +64,8 @@ class WebPageLoader(BaseLoader):
             tag.decompose()
 
         ids_to_exclude = ["sidebar", "main-navigation", "menu-main-menu"]
-        for id in ids_to_exclude:
-            tags = soup.find_all(id=id)
+        for id_ in ids_to_exclude:
+            tags = soup.find_all(id=id_)
             for tag in tags:
                 tag.decompose()