[Bug fix] Fix issues related to logging configuration (#1318 )

[Version] Update version to 0.1.96 (#1317 )
[Fix] Added missing provider for 'vllm' (#1316 )
2024-03-14 00:45:37 -07:00 · 2024-03-14 00:02:51 -07:00 · 2024-03-14 00:01:30 -07:00 · 2024-03-13 17:13:30 -07:00 · 2024-03-13 11:01:46 -07:00 · 2024-03-12 12:15:37 -07:00
150 changed files with 2655 additions and 1525 deletions
@@ -39,7 +39,7 @@ jobs:
          path: .venv
          key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
      - name: Install dependencies
-        run: poetry install --all-extras
+        run: make install_all
        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
      - name: Lint with ruff
        run: make lint
@@ -165,6 +165,7 @@ cython_debug/
 # Database
 db
 test-db
+!embedchain/core/db/

 .vscode
 .idea/
@@ -11,6 +11,7 @@ install:

 install_all:
 	poetry install --all-extras
+	poetry run pip install pinecone-text pinecone-client

 install_es:
 	poetry install --extras elasticsearch
@@ -37,6 +38,12 @@ clean:
 lint:
 	poetry run ruff .

+build:
+	poetry build
+
+publish:
+	poetry publish
+
 # for example: make test file=tests/test_factory.py
 test:
 	poetry run pytest $(file)
@@ -2,10 +2,6 @@
  <img src="docs/logo/dark.svg" width="400px" alt="Embedchain Logo">
 </p>

-<p align="center">
-  <a href="https://runacap.com/ross-index/q3-2023/" target="_blank" rel="noopener"><img style="width: 260px; height: 56px" src="https://runacap.com/wp-content/uploads/2023/10/ROSS_badge_black_Q3_2023.svg" alt="ROSS Index - Fastest Growing Open-Source Startups in Q3 2023 | Runa Capital" width="260" height="56"/></a>
-</p>
-
 <p align="center">
  <a href="https://pypi.org/project/embedchain/">
    <img src="https://img.shields.io/pypi/v/embedchain" alt="PyPI">
@@ -34,9 +30,9 @@

 ## What is Embedchain?

-Embedchain is an Open Source RAG Framework that makes it easy to create and deploy AI apps. At its core, Embedchain follows the design principle of being *"Conventional but Configurable"* to serve both software engineers and machine learning engineers.
+Embedchain is an Open Source Framework for personalizing LLM responses. It makes it easy to create and deploy personalized AI apps. At its core, Embedchain follows the design principle of being *"Conventional but Configurable"* to serve both software engineers and machine learning engineers.

-Embedchain streamlines the creation of Retrieval-Augmented Generation (RAG) applications, offering a seamless process for managing various types of unstructured data. It efficiently segments data into manageable chunks, generates relevant embeddings, and stores them in a vector database for optimized retrieval. With a suite of diverse APIs, it enables users to extract contextual information, find precise answers, or engage in interactive chat conversations, all tailored to their own data.
+Embedchain streamlines the creation of personalized LLM applications, offering a seamless process for managing various types of unstructured data. It efficiently segments data into manageable chunks, generates relevant embeddings, and stores them in a vector database for optimized retrieval. With a suite of diverse APIs, it enables users to extract contextual information, find precise answers, or engage in interactive chat conversations, all tailored to their own data.

 ## 🔧 Quick install

@@ -64,15 +60,15 @@ import os
 from embedchain import App

 # Create a bot instance
-os.environ["OPENAI_API_KEY"] = "YOUR API KEY"
-elon_bot = App()
+os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"
+app = App()

 # Embed online resources
-elon_bot.add("https://en.wikipedia.org/wiki/Elon_Musk")
-elon_bot.add("https://www.forbes.com/profile/elon-musk")
+app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+app.add("https://www.forbes.com/profile/elon-musk")

-# Query the bot
-elon_bot.query("How many companies does Elon Musk run and name those?")
+# Query the app
+app.query("How many companies does Elon Musk run and name those?")
 # Answer: Elon Musk currently runs several companies. As of my knowledge, he is the CEO and lead designer of SpaceX, the CEO and product architect of Tesla, Inc., the CEO and founder of Neuralink, and the CEO and founder of The Boring Company. However, please note that this information may change over time, so it's always good to verify the latest updates.
 ```

@@ -5,6 +5,7 @@ llm:
    temperature: 0.5
    top_p: 1
    stream: true
+    base_url: http://localhost:11434

 embedder:
  provider: huggingface
@@ -1,6 +1,6 @@


-<p>If you can't find the specific vector database, please feel free to request through one of the following channels and help us prioritize.</p>
+<p>If you can't find specific feature or run into issues, please feel free to reach out through one of the following channels.</p>

 <CardGroup cols={2}>
  <Card title="Slack" icon="slack" href="https://embedchain.ai/slack" color="#4A154B">
@@ -37,7 +37,14 @@ Create a local index:
 ```python
 from embedchain import App

-naval_chat_bot = App()
+config = {
+    "app": {
+        "config": {
+            "id": "app-1"
+        }
+    }
+}
+naval_chat_bot = App.from_config(config=config)
 naval_chat_bot.add("https://www.youtube.com/watch?v=3qHkcs3kG44")
 naval_chat_bot.add("https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
 ```
@@ -47,7 +54,14 @@ You can reuse the local index with the same code, but without adding new documen
 ```python
 from embedchain import App

-naval_chat_bot = App()
+config = {
+    "app": {
+        "config": {
+            "id": "app-1"
+        }
+    }
+}
+naval_chat_bot = App.from_config(config=config)
 print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
 ```

@@ -58,7 +72,14 @@ You can reset the app by simply calling the `reset` method. This will delete the
 ```python
 from embedchain import App

-app = App()
+app = App()config = {
+    "app": {
+        "config": {
+            "id": "app-1"
+        }
+    }
+}
+naval_chat_bot = App.from_config(config=config)
 app.add("https://www.youtube.com/watch?v=3qHkcs3kG44")
 app.reset()
 ```
@@ -29,11 +29,13 @@ response = app.query("What is Embedchain?")
 ```
 The `add` function of the app will accept any valid github query with qualifiers. It only supports loading github code, repository, issues and pull-requests.
 <Note>
-You must provide qualifiers `type:` and `repo:` in the query. The `type:` qualifier can be a combination of `code`, `repo`, `pr`, `issue`. The `repo:` qualifier must be a valid github repository name.
+You must provide qualifiers `type:` and `repo:` in the query. The `type:` qualifier can be a combination of `code`, `repo`, `pr`, `issue`, `branch`, `file`. The `repo:` qualifier must be a valid github repository name.
 </Note>

 <Card title="Valid queries" icon="lightbulb" iconType="duotone" color="#ca8b04">
    - `repo:embedchain/embedchain type:repo` - to load the repository
+    - `repo:embedchain/embedchain type:branch name:feature_test` - to load the branch of the repository
+    - `repo:embedchain/embedchain type:file path:README.md` - to load the specific file of the repository
    - `repo:embedchain/embedchain type:issue,pr` - to load the issues and pull-requests of the repository
    - `repo:embedchain/embedchain type:issue state:closed` - to load the closed issues of the repository
 </Card>
@@ -25,4 +25,4 @@ app = App()

 url = "https://drive.google.com/drive/u/0/folders/xxx-xxx"
 app.add(url, data_type="google_drive")
-```
+```
@@ -5,34 +5,35 @@ title: Overview
 Embedchain comes with built-in support for various data sources. We handle the complexity of loading unstructured data from these data sources, allowing you to easily customize your app through a user-friendly interface.

 <CardGroup cols={4}>
-  <Card title="📰 PDF file" href="/components/data-sources/pdf-file"></Card>
-  <Card title="📊 CSV file" href="/components/data-sources/csv"></Card>
-  <Card title="📃 JSON file" href="/components/data-sources/json"></Card>
-  <Card title="📝 Text" href="/components/data-sources/text"></Card>
-  <Card title="📁 Directory/ Folder" href="/components/data-sources/directory"></Card>
-  <Card title="🌐 HTML Web page" href="/components/data-sources/web-page"></Card>
-  <Card title="📽️ Youtube Channel" href="/components/data-sources/youtube-channel"></Card>
-  <Card title="📺 Youtube Video" href="/components/data-sources/youtube-video"></Card>
-  <Card title="📚 Docs website" href="/components/data-sources/docs-site"></Card>
-  <Card title="📝 MDX file" href="/components/data-sources/mdx"></Card>
-  <Card title="📄 DOCX file" href="/components/data-sources/docx"></Card>
-  <Card title="📓 Notion" href="/components/data-sources/notion"></Card>
-  <Card title="🗺️ Sitemap" href="/components/data-sources/sitemap"></Card>
-  <Card title="🧾 XML file" href="/components/data-sources/xml"></Card>
-  <Card title="❓💬 Q&A pair" href="/components/data-sources/qna"></Card>
-  <Card title="🙌 OpenAPI" href="/components/data-sources/openapi"></Card>
-  <Card title="📬 Gmail" href="/components/data-sources/gmail"></Card>
-  <Card title="📝 Github" href="/components/data-sources/github"></Card>
-  <Card title="🐘 Postgres" href="/components/data-sources/postgres"></Card>
-  <Card title="🐬 MySQL" href="/components/data-sources/mysql"></Card>
-  <Card title="🤖 Slack" href="/components/data-sources/slack"></Card>
-  <Card title="💬 Discord" href="/components/data-sources/discord"></Card>
-  <Card title="🗨️ Discourse" href="/components/data-sources/discourse"></Card>
-  <Card title="📝 Substack" href="/components/data-sources/substack"></Card>
-  <Card title="🐝 Beehiiv" href="/components/data-sources/beehiiv"></Card>
-  <Card title="💾 Dropbox" href="/components/data-sources/dropbox"></Card>
-  <Card title="🖼️ Image" href="/components/data-sources/image"></Card>
-  <Card title="⚙️ Custom" href="/components/data-sources/custom"></Card>
+  <Card title="PDF file" href="/components/data-sources/pdf-file"></Card>
+  <Card title="CSV file" href="/components/data-sources/csv"></Card>
+  <Card title="JSON file" href="/components/data-sources/json"></Card>
+  <Card title="Text" href="/components/data-sources/text"></Card>
+  <Card title="Directory" href="/components/data-sources/directory"></Card>
+  <Card title="Web page" href="/components/data-sources/web-page"></Card>
+  <Card title="Youtube Channel" href="/components/data-sources/youtube-channel"></Card>
+  <Card title="Youtube Video" href="/components/data-sources/youtube-video"></Card>
+  <Card title="Docs website" href="/components/data-sources/docs-site"></Card>
+  <Card title="MDX file" href="/components/data-sources/mdx"></Card>
+  <Card title="DOCX file" href="/components/data-sources/docx"></Card>
+  <Card title="Notion" href="/components/data-sources/notion"></Card>
+  <Card title="Sitemap" href="/components/data-sources/sitemap"></Card>
+  <Card title="XML file" href="/components/data-sources/xml"></Card>
+  <Card title="Q&A pair" href="/components/data-sources/qna"></Card>
+  <Card title="OpenAPI" href="/components/data-sources/openapi"></Card>
+  <Card title="Gmail" href="/components/data-sources/gmail"></Card>
+  <Card title="Google Drive" href="/components/data-sources/google-drive"></Card>
+  <Card title="GitHub" href="/components/data-sources/github"></Card>
+  <Card title="Postgres" href="/components/data-sources/postgres"></Card>
+  <Card title="MySQL" href="/components/data-sources/mysql"></Card>
+  <Card title="Slack" href="/components/data-sources/slack"></Card>
+  <Card title="Discord" href="/components/data-sources/discord"></Card>
+  <Card title="Discourse" href="/components/data-sources/discourse"></Card>
+  <Card title="Substack" href="/components/data-sources/substack"></Card>
+  <Card title="Beehiiv" href="/components/data-sources/beehiiv"></Card>
+  <Card title="Dropbox" href="/components/data-sources/dropbox"></Card>
+  <Card title="Image" href="/components/data-sources/image"></Card>
+  <Card title="Custom" href="/components/data-sources/custom"></Card>
 </CardGroup>

 <br/ >
@@ -7,7 +7,7 @@ title: '📽️ Youtube Channel'
 Make sure you have all the required packages installed before using this data type. You can install them by running the following command in your terminal.

 ```bash
-pip install -u "embedchain[youtube]"
+pip install -U "embedchain[youtube]"
 ```

 ## Usage
@@ -7,7 +7,7 @@ title: '📺 Youtube Video'
 Make sure you have all the required packages installed before using this data type. You can install them by running the following command in your terminal.

 ```bash
-pip install -u "embedchain[youtube]"
+pip install -U "embedchain[youtube]"
 ```

 ## Usage
@@ -13,6 +13,8 @@ Embedchain supports several embedding models from the following providers:
  <Card title="GPT4All" href="#gpt4all"></Card>
  <Card title="Hugging Face" href="#hugging-face"></Card>
  <Card title="Vertex AI" href="#vertex-ai"></Card>
+  <Card title="NVIDIA AI" href="#nvidia-ai"></Card>
+  <Card title="Cohere" href="#cohere"></Card>
 </CardGroup>

 ## OpenAI
@@ -220,3 +222,139 @@ embedder:
 ```

 </CodeGroup>
+
+## NVIDIA AI
+
+[NVIDIA AI Foundation Endpoints](https://www.nvidia.com/en-us/ai-data-science/foundation-models/) let you quickly use NVIDIA's AI models, such as Mixtral 8x7B, Llama 2 etc, through our API. These models are available in the [NVIDIA NGC catalog](https://catalog.ngc.nvidia.com/ai-foundation-models), fully optimized and ready to use on NVIDIA's AI platform. They are designed for high speed and easy customization, ensuring smooth performance on any accelerated setup.
+
+
+### Usage
+
+In order to use embedding models and LLMs from NVIDIA AI, create an account on [NVIDIA NGC Service](https://catalog.ngc.nvidia.com/).
+
+Generate an API key from their dashboard. Set the API key as `NVIDIA_API_KEY` environment variable. Note that the `NVIDIA_API_KEY` will start with `nvapi-`.
+
+Below is an example of how to use LLM model and embedding model from NVIDIA AI:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ['NVIDIA_API_KEY'] = 'nvapi-xxxx'
+
+config = {
+    "app": {
+        "config": {
+            "id": "my-app",
+        },
+    },
+    "llm": {
+        "provider": "nvidia",
+        "config": {
+            "model": "nemotron_steerlm_8b",
+        },
+    },
+    "embedder": {
+        "provider": "nvidia",
+        "config": {
+            "model": "nvolveqa_40k",
+            "vector_dimension": 1024,
+        },
+    },
+}
+
+app = App.from_config(config=config)
+
+app.add("https://www.forbes.com/profile/elon-musk")
+answer = app.query("What is the net worth of Elon Musk today?")
+# Answer: The net worth of Elon Musk is subject to fluctuations based on the market value of his holdings in various companies.
+# As of March 1, 2024, his net worth is estimated to be approximately $210 billion. However, this figure can change rapidly due to stock market fluctuations and other factors.
+# Additionally, his net worth may include other assets such as real estate and art, which are not reflected in his stock portfolio.
+```
+</CodeGroup>
+
+
+## Cohere
+
+To use embedding models and LLMs from COHERE, create an account on [COHERE](https://dashboard.cohere.com/welcome/login?redirect_uri=%2Fapi-keys).
+
+Generate an API key from their dashboard. Set the API key as `COHERE_API_KEY` environment variable.
+
+Once you have obtained the key, you can use it like this:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ['COHERE_API_KEY'] = 'xxx'
+
+# load embedding model configuration from config.yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+embedder:
+  provider: cohere
+  config:
+    model: 'embed-english-light-v3.0'
+```
+
+</CodeGroup>
+
+* Cohere has few embedding models: `embed-english-v3.0`, `embed-multilingual-v3.0`, `embed-multilingual-light-v3.0`, `embed-english-v2.0`, `embed-english-light-v2.0` and `embed-multilingual-v2.0`. Embedchain supports all these models. Below you can find YAML config for all:
+
+<CodeGroup>
+
+```yaml embed-english-v3.0.yaml
+embedder:
+  provider: cohere
+  config:
+    model: 'embed-english-v3.0'
+    vector_dimension: 1024
+```
+
+```yaml embed-multilingual-v3.0.yaml
+embedder:
+  provider: cohere
+  config:
+    model: 'embed-multilingual-v3.0'
+    vector_dimension: 1024
+```
+
+```yaml embed-multilingual-light-v3.0.yaml
+embedder:
+  provider: cohere
+  config:
+    model: 'embed-multilingual-light-v3.0'
+    vector_dimension: 384
+```
+
+```yaml embed-english-v2.0.yaml
+embedder:
+  provider: cohere
+  config:
+    model: 'embed-english-v2.0'
+    vector_dimension: 4096
+```
+
+```yaml embed-english-light-v2.0.yaml
+embedder:
+  provider: cohere
+  config:
+    model: 'embed-english-light-v2.0'
+    vector_dimension: 1024
+```
+
+```yaml embed-multilingual-v2.0.yaml
+embedder:
+  provider: cohere
+  config:
+    model: 'embed-multilingual-v2.0'
+    vector_dimension: 768
+```
+
+</CodeGroup>
@@ -9,4 +9,5 @@ You can configure following components
 * [Data Source](/components/data-sources/overview)
 * [LLM](/components/llms)
 * [Embedding Model](/components/embedding-models)
-* [Vector Database](/components/vector-databases)
+* [Vector Database](/components/vector-databases)
+* [Evaluation](/components/evaluation)
@@ -22,6 +22,8 @@ Embedchain comes with built-in support for various popular large language models
  <Card title="Vertex AI" href="#vertex-ai"></Card>
  <Card title="Mistral AI" href="#mistral-ai"></Card>
  <Card title="AWS Bedrock" href="#aws-bedrock"></Card>
+  <Card title="Groq" href="#groq"></Card>
+  <Card title="NVIDIA AI" href="#nvidia-ai"></Card>
 </CardGroup>

 ## OpenAI
@@ -68,125 +70,75 @@ llm:
 </CodeGroup>

 ### Function Calling
-To enable [function calling](https://platform.openai.com/docs/guides/function-calling) in your application using embedchain and OpenAI, you need to pass functions into `OpenAILlm` class as an array of functions. Here are several ways in which you can achieve that:
+Embedchain supports OpenAI [Function calling](https://platform.openai.com/docs/guides/function-calling) with a single function. It accepts inputs in accordance with the [Langchain interface](https://python.langchain.com/docs/modules/model_io/chat/function_calling#legacy-args-functions-and-function_call).

-Examples:
-<Accordion title="Using Pydantic Models">
+<Accordion title="Pydantic Model">
  ```python
-import os
-from embedchain import App
-from embedchain.llm.openai import OpenAILlm
-import requests
-from pydantic import BaseModel, Field, ValidationError, field_validator
+  from pydantic import BaseModel

-os.environ["OPENAI_API_KEY"] = "sk-xxx"
+  class multiply(BaseModel):
+      """Multiply two integers together."""

-class QA(BaseModel):
-    """
-    A question and answer pair.
-    """
-
-    question: str = Field(
-        ..., description="The question.", example="What is a mountain?"
-    )
-    answer: str = Field(
-        ..., description="The answer.", example="A mountain is a hill."
-    )
-    person_who_is_asking: str = Field(
-        ..., description="The person who is asking the question.", example="John"
-    )
-
-    @field_validator("question")
-    def question_must_end_with_a_question_mark(cls, v):
-        """
-        Validate that the question ends with a question mark.
-        """
-        if not v.endswith("?"):
-            raise ValueError("question must end with a question mark")
-        return v
-
-    @field_validator("answer")
-    def answer_must_end_with_a_period(cls, v):
-        """
-        Validate that the answer ends with a period.
-        """
-        if not v.endswith("."):
-            raise ValueError("answer must end with a period")
-        return v
-
-llm = OpenAILlm(config=None,functions=[QA])
-app = App(llm=llm)
-
-result = app.query("Hey I am Sid. What is a mountain? A mountain is a hill.")
-
-print(result)
+      a: int = Field(..., description="First integer")
+      b: int = Field(..., description="Second integer")
  ```
-  </Accordion>
-  
-  <Accordion title="Using OpenAI JSON schema">
+</Accordion>
+
+<Accordion title="Python function">
+  ```python
+  def multiply(a: int, b: int) -> int:
+      """Multiply two integers together.
+
+      Args:
+          a: First integer
+          b: Second integer
+      """
+      return a * b
+  ```
+</Accordion>
+<Accordion title="OpenAI tool dictionary">
+  ```python
+  multiply = {
+    "type": "function",
+    "function": {
+      "name": "multiply",
+      "description": "Multiply two integers together.",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "a": {
+            "description": "First integer",
+            "type": "integer"
+          },
+          "b": {
+            "description": "Second integer",
+            "type": "integer"
+          }
+        },
+        "required": [
+          "a",
+          "b"
+        ]
+      }
+    }
+  }
+  ```
+</Accordion>
+
+With any of the previous inputs, the OpenAI LLM can be queried to provide the appropriate arguments for the function.
+
 ```python
 import os
 from embedchain import App
 from embedchain.llm.openai import OpenAILlm
-import requests
-from pydantic import BaseModel, Field, ValidationError, field_validator

 os.environ["OPENAI_API_KEY"] = "sk-xxx"

-json_schema = {
-    "name": "get_qa",
-    "description": "A question and answer pair and the user who is asking the question.",
-    "parameters": {
-        "type": "object",
-        "properties": {
-            "question": {"type": "string", "description": "The question."},
-            "answer": {"type": "string", "description": "The answer."},
-            "person_who_is_asking": {
-                "type": "string",
-                "description": "The person who is asking the question.",
-            }
-        },
-        "required": ["question", "answer", "person_who_is_asking"],
-    },
-}
-
-llm = OpenAILlm(config=None,functions=[json_schema])
+llm = OpenAILlm(tools=multiply)
 app = App(llm=llm)

-result = app.query("Hey I am Sid. What is a mountain? A mountain is a hill.")
-
-print(result)
-  ```
-  </Accordion>
-  <Accordion title="Using actual python functions">
-  ```python
-import os
-from embedchain import App
-from embedchain.llm.openai import OpenAILlm
-import requests
-from pydantic import BaseModel, Field, ValidationError, field_validator
-
-os.environ["OPENAI_API_KEY"] = "sk-xxx"
-
-def find_info_of_pokemon(pokemon: str):
-    """
-    Find the information of the given pokemon.
-    Args:
-        pokemon: The pokemon.
-    """
-    req = requests.get(f"https://pokeapi.co/api/v2/pokemon/{pokemon}")
-    if req.status_code == 404:
-        raise ValueError("pokemon not found")
-    return req.json()
-
-llm = OpenAILlm(config=None,functions=[find_info_of_pokemon])
-app = App(llm=llm)
-
-result = app.query("Tell me more about the pokemon pikachu.")
-
-print(result)
+result = app.query("What is the result of 125 multiplied by fifteen?")
 ```
-</Accordion>

 ## Google AI

@@ -500,7 +452,15 @@ pip install --upgrade 'embedchain[huggingface-hub]'

 First, set `HUGGINGFACE_ACCESS_TOKEN` in environment variable which you can obtain from [their platform](https://huggingface.co/settings/tokens).

-Once you have the token, load the app using the config yaml file:
+You can load the LLMs from Hugging Face using three ways:
+
+- [Hugging Face Hub](#hugging-face-hub)
+- [Hugging Face Local Pipelines](#hugging-face-local-pipelines)
+- [Hugging Face Inference Endpoint](#hugging-face-inference-endpoint)
+
+### Hugging Face Hub
+
+To load the model from Hugging Face Hub, use the following code:

 <CodeGroup>

@@ -510,24 +470,49 @@ from embedchain import App

 os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "xxx"

-# load llm configuration from config.yaml file
-app = App.from_config(config_path="config.yaml")
-```
+config = {
+  "app": {"config": {"id": "my-app"}},
+  "llm": {
+      "provider": "huggingface",
+      "config": {
+          "model": "bigscience/bloom-1b7",
+          "top_p": 0.5,
+          "max_length": 200,
+          "temperature": 0.1,
+      },
+  },
+}

-```yaml config.yaml
-llm:
-  provider: huggingface
-  config:
-    model: 'google/flan-t5-xxl'
-    temperature: 0.5
-    max_tokens: 1000
-    top_p: 0.5
-    stream: false
+app = App.from_config(config=config)
 ```
 </CodeGroup>

-### Custom Endpoints
+### Hugging Face Local Pipelines

+If you want to load the locally downloaded model from Hugging Face, you can do so by following the code provided below:
+
+<CodeGroup>
+```python main.py
+from embedchain import App
+
+config = {
+  "app": {"config": {"id": "my-app"}},
+  "llm": {
+      "provider": "huggingface",
+      "config": {
+          "model": "Trendyol/Trendyol-LLM-7b-chat-v0.1",
+          "local": True,  # Necessary if you want to run model locally
+          "top_p": 0.5,
+          "max_tokens": 1000,
+          "temperature": 0.1,
+      },
+  }
+}
+app = App.from_config(config=config)
+```
+</CodeGroup>
+
+### Hugging Face Inference Endpoint

 You can also use [Hugging Face Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index#-inference-endpoints) to access custom endpoints. First, set the `HUGGINGFACE_ACCESS_TOKEN` as above.

@@ -536,35 +521,23 @@ Then, load the app using the config yaml file:
 <CodeGroup>

 ```python main.py
-import os
 from embedchain import App

-os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "xxx"
+config = {
+  "app": {"config": {"id": "my-app"}},
+  "llm": {
+      "provider": "huggingface",
+      "config": {
+        "endpoint": "https://api-inference.huggingface.co/models/gpt2",
+        "model_params": {"temprature": 0.1, "max_new_tokens": 100}
+      },
+  },
+}
+app = App.from_config(config=config)

-# load llm configuration from config.yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-llm:
-  provider: huggingface
-  config:
-    endpoint: https://api-inference.huggingface.co/models/gpt2 # replace with your personal endpoint
 ```
 </CodeGroup>

-If your endpoint requires additional parameters, you can pass them in the `model_kwargs` field:
-
-```
-llm:
-  provider: huggingface
-  config:
-    endpoint: <YOUR_ENDPOINT_URL_HERE>
-    model_kwargs:
-      max_new_tokens: 100
-      temperature: 0.5
-```
-
 Currently only supports `text-generation` and `text2text-generation` for now [[ref](https://api.python.langchain.com/en/latest/llms/langchain_community.llms.huggingface_endpoint.HuggingFaceEndpoint.html?highlight=huggingfaceendpoint#)].

 See langchain's [hugging face endpoint](https://python.langchain.com/docs/integrations/chat/huggingface#huggingfaceendpoint) for more information. 
@@ -704,4 +677,112 @@ llm:
 </Note>

 <br/ >
+
+## Groq
+
+[Groq](https://groq.com/) is the creator of the world's first Language Processing Unit (LPU), providing exceptional speed performance for AI workloads running on their LPU Inference Engine.
+
+
+### Usage
+
+In order to use LLMs from Groq, go to their [platform](https://console.groq.com/keys) and get the API key.
+
+Set the API key as `GROQ_API_KEY` environment variable or pass in your app configuration to use the model as given below in the example.
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+# Set your API key here or pass as the environment variable
+groq_api_key = "gsk_xxxx"
+
+config = {
+    "llm": {
+        "provider": "groq",
+        "config": {
+            "model": "mixtral-8x7b-32768",
+            "api_key": groq_api_key,
+            "stream": True
+        }
+    }
+}
+
+app = App.from_config(config=config)
+# Add your data source here
+app.add("https://docs.embedchain.ai/sitemap.xml", data_type="sitemap")
+app.query("Write a poem about Embedchain")
+
+# In the realm of data, vast and wide,
+# Embedchain stands with knowledge as its guide.
+# A platform open, for all to try,
+# Building bots that can truly fly.
+
+# With REST API, data in reach,
+# Deployment a breeze, as easy as a speech.
+# Updating data sources, anytime, anyday,
+# Embedchain's power, never sway.
+
+# A knowledge base, an assistant so grand,
+# Connecting to platforms, near and far.
+# Discord, WhatsApp, Slack, and more,
+# Embedchain's potential, never a bore.
+```
+</CodeGroup>
+
+## NVIDIA AI
+
+[NVIDIA AI Foundation Endpoints](https://www.nvidia.com/en-us/ai-data-science/foundation-models/) let you quickly use NVIDIA's AI models, such as Mixtral 8x7B, Llama 2 etc, through our API. These models are available in the [NVIDIA NGC catalog](https://catalog.ngc.nvidia.com/ai-foundation-models), fully optimized and ready to use on NVIDIA's AI platform. They are designed for high speed and easy customization, ensuring smooth performance on any accelerated setup.
+
+
+### Usage
+
+In order to use LLMs from NVIDIA AI, create an account on [NVIDIA NGC Service](https://catalog.ngc.nvidia.com/).
+
+Generate an API key from their dashboard. Set the API key as `NVIDIA_API_KEY` environment variable. Note that the `NVIDIA_API_KEY` will start with `nvapi-`.
+
+Below is an example of how to use LLM model and embedding model from NVIDIA AI:
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ['NVIDIA_API_KEY'] = 'nvapi-xxxx'
+
+config = {
+    "app": {
+        "config": {
+            "id": "my-app",
+        },
+    },
+    "llm": {
+        "provider": "nvidia",
+        "config": {
+            "model": "nemotron_steerlm_8b",
+        },
+    },
+    "embedder": {
+        "provider": "nvidia",
+        "config": {
+            "model": "nvolveqa_40k",
+            "vector_dimension": 1024,
+        },
+    },
+}
+
+app = App.from_config(config=config)
+
+app.add("https://www.forbes.com/profile/elon-musk")
+answer = app.query("What is the net worth of Elon Musk today?")
+# Answer: The net worth of Elon Musk is subject to fluctuations based on the market value of his holdings in various companies.
+# As of March 1, 2024, his net worth is estimated to be approximately $210 billion. However, this figure can change rapidly due to stock market fluctuations and other factors.
+# Additionally, his net worth may include other assets such as real estate and art, which are not reflected in his stock portfolio.
+```
+</CodeGroup>
+
+<br/ >
+
 <Snippet file="missing-llm-tip.mdx" />
@@ -17,242 +17,4 @@ Utilizing a vector database alongside Embedchain is a seamless process. All you
  <Card title="Weaviate" href="#weaviate"></Card>
 </CardGroup>

-## ChromaDB
-
-<CodeGroup>
-
-```python main.py
-from embedchain import App
-
-# load chroma configuration from yaml file
-app = App.from_config(config_path="config1.yaml")
-```
-
-```yaml config1.yaml
-vectordb:
-  provider: chroma
-  config:
-    collection_name: 'my-collection'
-    dir: db
-    allow_reset: true
-```
-
-```yaml config2.yaml
-vectordb:
-  provider: chroma
-  config:
-    collection_name: 'my-collection'
-    host: localhost
-    port: 5200
-    allow_reset: true
-```
-
-</CodeGroup>
-
-
-## Elasticsearch
-
-Install related dependencies using the following command:
-
-```bash
-pip install --upgrade 'embedchain[elasticsearch]'
-```
-
-<Note>
-You can configure the Elasticsearch connection by providing either `es_url` or `cloud_id`. If you are using the Elasticsearch Service on Elastic Cloud, you can find the `cloud_id` on the [Elastic Cloud dashboard](https://cloud.elastic.co/deployments).
-</Note>
-
-You can authorize the connection to Elasticsearch by providing either `basic_auth`, `api_key`, or `bearer_auth`.
-
-<CodeGroup>
-
-```python main.py
-from embedchain import App
-
-# load elasticsearch configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: elasticsearch
-  config:
-    collection_name: 'es-index'
-    cloud_id: 'deployment-name:xxxx'
-    basic_auth:
-      - elastic
-      - <your_password>
-    verify_certs: false
-```
-</CodeGroup>
-
-## OpenSearch
-
-Install related dependencies using the following command:
-
-```bash
-pip install --upgrade 'embedchain[opensearch]'
-```
-
-<CodeGroup>
-
-```python main.py
-from embedchain import App
-
-# load opensearch configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: opensearch
-  config:
-    collection_name: 'my-app'
-    opensearch_url: 'https://localhost:9200'
-    http_auth:
-      - admin
-      - admin
-    vector_dimension: 1536
-    use_ssl: false
-    verify_certs: false
-```
-
-</CodeGroup>
-
-## Zilliz
-
-Install related dependencies using the following command:
-
-```bash
-pip install --upgrade 'embedchain[milvus]'
-```
-
-Set the Zilliz environment variables `ZILLIZ_CLOUD_URI` and `ZILLIZ_CLOUD_TOKEN` which you can find it on their [cloud platform](https://cloud.zilliz.com/).
-
-<CodeGroup>
-
-```python main.py
-import os
-from embedchain import App
-
-os.environ['ZILLIZ_CLOUD_URI'] = 'https://xxx.zillizcloud.com'
-os.environ['ZILLIZ_CLOUD_TOKEN'] = 'xxx'
-
-# load zilliz configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: zilliz
-  config:
-    collection_name: 'zilliz_app'
-    uri: https://xxxx.api.gcp-region.zillizcloud.com
-    token: xxx
-    vector_dim: 1536
-    metric_type: L2
-```
-
-</CodeGroup>
-
-## LanceDB
-
-_Coming soon_
-
-## Pinecone
-
-Install pinecone related dependencies using the following command:
-
-```bash
-pip install --upgrade 'embedchain[pinecone]'
-```
-
-In order to use Pinecone as vector database, set the environment variable `PINECONE_API_KEY` which you can find on [Pinecone dashboard](https://app.pinecone.io/).
-
-<CodeGroup>
-
-```python main.py
-from embedchain import App
-
-# load pinecone configuration from yaml file
-app = App.from_config(config_path="pod_config.yaml")
-# or
-app = App.from_config(config_path="serverless_config.yaml")
-```
-
-```yaml pod_config.yaml
-vectordb:
-  provider: pinecone
-  config:
-    metric: cosine
-    vector_dimension: 1536
-    index_name: my-pinecone-index
-    pod_config:
-      environment: gcp-starter
-      metadata_config:
-        indexed:
-          - "url"
-          - "hash"
-```
-
-```yaml serverless_config.yaml
-vectordb:
-  provider: pinecone
-  config:
-    metric: cosine
-    vector_dimension: 1536
-    index_name: my-pinecone-index
-    serverless_config:
-      cloud: aws
-      region: us-west-2
-```
-
-</CodeGroup>
-
-<br />
-<Note>
-You can find more information about Pinecone configuration [here](https://docs.pinecone.io/docs/manage-indexes#create-a-pod-based-index).
-You can also optionally provide `index_name` as a config param in yaml file to specify the index name. If not provided, the index name will be `{collection_name}-{vector_dimension}`.
-</Note>
-
-## Qdrant
-
-In order to use Qdrant as a vector database, set the environment variables `QDRANT_URL` and `QDRANT_API_KEY` which you can find on [Qdrant Dashboard](https://cloud.qdrant.io/).
-
-<CodeGroup>
-```python main.py
-from embedchain import App
-
-# load qdrant configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: qdrant
-  config:
-    collection_name: my_qdrant_index
-```
-</CodeGroup>
-
-## Weaviate
-
-In order to use Weaviate as a vector database, set the environment variables `WEAVIATE_ENDPOINT` and `WEAVIATE_API_KEY` which you can find on [Weaviate dashboard](https://console.weaviate.cloud/dashboard).
-
-<CodeGroup>
-```python main.py
-from embedchain import App
-
-# load weaviate configuration from yaml file
-app = App.from_config(config_path="config.yaml")
-```
-
-```yaml config.yaml
-vectordb:
-  provider: weaviate
-  config:
-    collection_name: my_weaviate_index
-```
-</CodeGroup>
-
 <Snippet file="missing-vector-db-tip.mdx" />
@@ -0,0 +1,35 @@
+---
+title: ChromaDB
+---
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load chroma configuration from yaml file
+app = App.from_config(config_path="config1.yaml")
+```
+
+```yaml config1.yaml
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-collection'
+    dir: db
+    allow_reset: true
+```
+
+```yaml config2.yaml
+vectordb:
+  provider: chroma
+  config:
+    collection_name: 'my-collection'
+    host: localhost
+    port: 5200
+    allow_reset: true
+```
+
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />
@@ -0,0 +1,39 @@
+---
+title: Elasticsearch
+---
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[elasticsearch]'
+```
+
+<Note>
+You can configure the Elasticsearch connection by providing either `es_url` or `cloud_id`. If you are using the Elasticsearch Service on Elastic Cloud, you can find the `cloud_id` on the [Elastic Cloud dashboard](https://cloud.elastic.co/deployments).
+</Note>
+
+You can authorize the connection to Elasticsearch by providing either `basic_auth`, `api_key`, or `bearer_auth`.
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load elasticsearch configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: elasticsearch
+  config:
+    collection_name: 'es-index'
+    cloud_id: 'deployment-name:xxxx'
+    basic_auth:
+      - elastic
+      - <your_password>
+    verify_certs: false
+```
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />
@@ -0,0 +1,36 @@
+---
+title: OpenSearch
+---
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[opensearch]'
+```
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# load opensearch configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: opensearch
+  config:
+    collection_name: 'my-app'
+    opensearch_url: 'https://localhost:9200'
+    http_auth:
+      - admin
+      - admin
+    vector_dimension: 1536
+    use_ssl: false
+    verify_certs: false
+```
+
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />
@@ -0,0 +1,109 @@
+---
+title: Pinecone
+---
+
+## Overview
+
+Install pinecone related dependencies using the following command:
+
+```bash
+pip install --upgrade 'pinecone-client pinecone-text'
+```
+
+In order to use Pinecone as vector database, set the environment variable `PINECONE_API_KEY` which you can find on [Pinecone dashboard](https://app.pinecone.io/).
+
+<CodeGroup>
+
+```python main.py
+from embedchain import App
+
+# Load pinecone configuration from yaml file
+app = App.from_config(config_path="pod_config.yaml")
+# Or
+app = App.from_config(config_path="serverless_config.yaml")
+```
+
+```yaml pod_config.yaml
+vectordb:
+  provider: pinecone
+  config:
+    metric: cosine
+    vector_dimension: 1536
+    index_name: my-pinecone-index
+    pod_config:
+      environment: gcp-starter
+      metadata_config:
+        indexed:
+          - "url"
+          - "hash"
+```
+
+```yaml serverless_config.yaml
+vectordb:
+  provider: pinecone
+  config:
+    metric: cosine
+    vector_dimension: 1536
+    index_name: my-pinecone-index
+    serverless_config:
+      cloud: aws
+      region: us-west-2
+```
+
+</CodeGroup>
+
+<br />
+<Note>
+You can find more information about Pinecone configuration [here](https://docs.pinecone.io/docs/manage-indexes#create-a-pod-based-index).
+You can also optionally provide `index_name` as a config param in yaml file to specify the index name. If not provided, the index name will be `{collection_name}-{vector_dimension}`.
+</Note>
+
+## Usage
+
+### Hybrid search
+
+Here is an example of how you can do hybrid search using Pinecone as a vector database through Embedchain.
+
+```python
+import os
+
+from embedchain import App
+
+config = {
+    'app': {
+        "config": {
+            "id": "ec-docs-hybrid-search"
+        }
+    },
+    'vectordb': {
+        'provider': 'pinecone',
+        'config': {
+            'metric': 'dotproduct',
+            'vector_dimension': 1536,
+            'index_name': 'my-index',
+            'serverless_config': {
+                'cloud': 'aws',
+                'region': 'us-west-2'
+            },
+            'hybrid_search': True, # Remember to set this for hybrid search
+        }
+    }
+}
+
+# Initialize app
+app = App.from_config(config=config)
+
+# Add documents
+app.add("/path/to/file.pdf", data_type="pdf_file", namespace="my-namespace")
+
+# Query
+app.query("<YOUR QUESTION HERE>", namespace="my-namespace")
+
+# Chat
+app.chat("<YOUR QUESTION HERE>", namespace="my-namespace")
+```
+
+Under the hood, Embedchain fetches the relevant chunks from the documents you added by doing hybrid search on the pinecone index.
+If you have questions on how pinecone hybrid search works, please refer to their [offical documentation here](https://docs.pinecone.io/docs/hybrid-search).
+
+<Snippet file="missing-vector-db-tip.mdx" />
@@ -0,0 +1,23 @@
+---
+title: Qdrant
+---
+
+In order to use Qdrant as a vector database, set the environment variables `QDRANT_URL` and `QDRANT_API_KEY` which you can find on [Qdrant Dashboard](https://cloud.qdrant.io/).
+
+<CodeGroup>
+```python main.py
+from embedchain import App
+
+# load qdrant configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: qdrant
+  config:
+    collection_name: my_qdrant_index
+```
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />
@@ -0,0 +1,24 @@
+---
+title: Weaviate
+---
+
+
+In order to use Weaviate as a vector database, set the environment variables `WEAVIATE_ENDPOINT` and `WEAVIATE_API_KEY` which you can find on [Weaviate dashboard](https://console.weaviate.cloud/dashboard).
+
+<CodeGroup>
+```python main.py
+from embedchain import App
+
+# load weaviate configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: weaviate
+  config:
+    collection_name: my_weaviate_index
+```
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />
@@ -0,0 +1,39 @@
+---
+title: Zilliz
+---
+
+Install related dependencies using the following command:
+
+```bash
+pip install --upgrade 'embedchain[milvus]'
+```
+
+Set the Zilliz environment variables `ZILLIZ_CLOUD_URI` and `ZILLIZ_CLOUD_TOKEN` which you can find it on their [cloud platform](https://cloud.zilliz.com/).
+
+<CodeGroup>
+
+```python main.py
+import os
+from embedchain import App
+
+os.environ['ZILLIZ_CLOUD_URI'] = 'https://xxx.zillizcloud.com'
+os.environ['ZILLIZ_CLOUD_TOKEN'] = 'xxx'
+
+# load zilliz configuration from yaml file
+app = App.from_config(config_path="config.yaml")
+```
+
+```yaml config.yaml
+vectordb:
+  provider: zilliz
+  config:
+    collection_name: 'zilliz_app'
+    uri: https://xxxx.api.gcp-region.zillizcloud.com
+    token: xxx
+    vector_dim: 1536
+    metric_type: L2
+```
+
+</CodeGroup>
+
+<Snippet file="missing-vector-db-tip.mdx" />
@@ -5,7 +5,7 @@ description: 'Deploy your RAG application to embedchain.ai platform'

 ## Deploy on Embedchain Platform

-Embedchain enables developers to deploy their LLM-powered apps in production using the [Embedchain platform](https://app.embedchain.ai). The platform offers free access to context on your data through its REST API. Once the pipeline is deployed, you can update your data sources anytime after deployment.
+Embedchain enables developers to deploy their LLM-powered apps in production using the Embedchain platform. The platform offers free access to context on your data through its REST API. Once the pipeline is deployed, you can update your data sources anytime after deployment.

 Deployment to Embedchain Platform is currently available on an invitation-only basis. To request access, please submit your information via the provided [Google Form](https://forms.gle/vigN11h7b4Ywat668). We will review your request and respond promptly.

@@ -0,0 +1,86 @@
+---
+title: 'Railway.app'
+description: 'Deploy your RAG application to railway.app'
+---
+
+It's easy to host your Embedchain-powered apps and APIs on railway.
+
+Follow the instructions given below to deploy your first application quickly:
+
+## Step-1: Create RAG app
+
+```bash Install embedchain
+pip install embedchain
+```
+
+<Tip>
+**Create a full stack app using Embedchain CLI**
+
+To use your hosted embedchain RAG app, you can easily set up a FastAPI server that can be used anywhere.
+To easily set up a FastAPI server, check out [Get started with Full stack](https://docs.embedchain.ai/get-started/full-stack) page.
+
+Hosting this server on railway is super easy!
+
+</Tip>
+
+## Step-2: Set up your project
+
+### With Docker
+
+You can create a `Dockerfile` in the root of the project, with all the instructions. However, this method is sometimes slower in deployment.
+
+### Without Docker
+
+By default, Railway uses Python 3.7. Embedchain requires the python version to be >3.9 in order to install.
+
+To fix this, create a `.python-version` file in the root directory of your project and specify the correct version
+
+```bash .python-version
+3.10
+```
+
+You also need to create a `requirements.txt` file to specify the requirements.
+
+```bash requirements.txt
+python-dotenv
+embedchain
+fastapi==0.108.0
+uvicorn==0.25.0
+embedchain
+beautifulsoup4
+sentence-transformers
+```
+
+## Step-3: Deploy to Railway 🚀
+
+1. Go to https://railway.app and create an account.
+2. Create a project by clicking on the "Start a new project" button
+
+### With Github
+
+Select `Empty Project` or `Deploy from Github Repo`. 
+
+You should be all set!
+
+### Without Github
+
+You can also use the railway CLI to deploy your apps from the terminal, if you don't want to connect a git repository.
+
+To do this, just run this command in your terminal
+
+```bash Install and set up railway CLI
+npm i -g @railway/cli
+railway login
+railway link [projectID]
+```
+
+Finally, run `railway up` to deploy your app.
+```bash Deploy
+railway up
+```
+
+## Seeking help?
+
+If you run into issues with deployment, please feel free to reach out to us via any of the following methods:
+
+<Snippet file="get-help.mdx" />
@@ -9,6 +9,7 @@ After successfully setting up and testing your RAG app locally, the next step is
  <Card title="Fly.io" href="/deployment/fly_io"></Card>
  <Card title="Modal.com" href="/deployment/modal_com"></Card>
  <Card title="Render.com" href="/deployment/render_com"></Card>
+  <Card title="Railway.app" href="/deployment/railway"></Card>
  <Card title="Streamlit.io" href="/deployment/streamlit_io"></Card>
  <Card title="Gradio.app" href="/deployment/gradio_app"></Card>
  <Card title="Huggingface.co" href="/deployment/huggingface_spaces"></Card>
@@ -4,21 +4,21 @@ title: 📚 Introduction

 ## What is Embedchain?

-Embedchain is an Open Source RAG Framework that makes it easy to create and deploy AI apps. At its core, Embedchain follows the design principle of being *"Conventional but Configurable"* to serve both software engineers and machine learning engineers.
+Embedchain is an Open Source Framework that makes it easy to create and deploy personalized AI apps. At its core, Embedchain follows the design principle of being *"Conventional but Configurable"* to serve both software engineers and machine learning engineers.

-Embedchain streamlines the creation of RAG applications, offering a seamless process for managing various types of unstructured data. It efficiently segments data into manageable chunks, generates relevant embeddings, and stores them in a vector database for optimized retrieval. With a suite of diverse APIs, it enables users to extract contextual information, find precise answers, or engage in interactive chat conversations, all tailored to their own data.
+Embedchain streamlines the creation of personalized LLM applications, offering a seamless process for managing various types of unstructured data. It efficiently segments data into manageable chunks, generates relevant embeddings, and stores them in a vector database for optimized retrieval. With a suite of diverse APIs, it enables users to extract contextual information, find precise answers, or engage in interactive chat conversations, all tailored to their own data.

 ## Who is Embedchain for?

 Embedchain is designed for a diverse range of users, from AI professionals like Data Scientists and Machine Learning Engineers to those just starting their AI journey, including college students, independent developers, and hobbyists. Essentially, it's for anyone with an interest in AI, regardless of their expertise level.

-Our APIs are user-friendly yet adaptable, enabling beginners to effortlessly create LLM-powered applications with as few as 4 lines of code. At the same time, we offer extensive customization options for every aspect of the RAG pipeline. This includes the choice of LLMs, vector databases, loaders and chunkers, retrieval strategies, re-ranking, and more.
+Our APIs are user-friendly yet adaptable, enabling beginners to effortlessly create LLM-powered applications with as few as 4 lines of code. At the same time, we offer extensive customization options for every aspect of building a personalized AI application. This includes the choice of LLMs, vector databases, loaders and chunkers, retrieval strategies, re-ranking, and more.

 Our platform's clear and well-structured abstraction layers ensure that users can tailor the system to meet their specific needs, whether they're crafting a simple project or a complex, nuanced AI application.

 ## Why Use Embedchain?

-Developing a robust and efficient RAG (Retrieval-Augmented Generation) pipeline for production use presents numerous complexities, such as:
+Developing a personalized AI application for production use presents numerous complexities, such as:

 - Integrating and indexing data from diverse sources.
 - Determining optimal data chunking methods for each source.
@@ -48,11 +48,11 @@ When a user asks a question, whether for chatting, searching, or querying, Embed
 2. **Document Retrieval**: These embeddings are then used to find related documents in the database.
 3. **Answer Generation**: The related documents are used by the LLM to craft a precise answer.

-With Embedchain, you don’t have to worry about the complexities of building a RAG pipeline. It offers an easy-to-use interface for developing applications with any kind of data.
+With Embedchain, you don’t have to worry about the complexities of building a personalized AI application. It offers an easy-to-use interface for developing applications with any kind of data.

 ## Getting started

-Checkout our [quickstart guide](/get-started/quickstart) to start your first RAG application.
+Checkout our [quickstart guide](/get-started/quickstart) to start your first AI application.

 ## Support

@@ -1,6 +1,6 @@
 ---
 title: '⚡ Quickstart'
-description: '💡 Create a RAG app on your own data in a minute'
+description: '💡 Create an AI app on your own data in a minute'
 ---

 ## Installation
@@ -31,41 +31,47 @@ This section gives a quickstart example of using Mistral as the Open source LLM
 We are using Mistral hosted at Hugging Face, so will you need a Hugging Face token to run this example. Its *free* and you can create one [here](https://huggingface.co/docs/hub/security-tokens).

 <CodeGroup>
-```python quickstart.py
+```python huggingface_demo.py
 import os
-# replace this with your HF key
+# Replace this with your HF token
 os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "hf_xxxx"

 from embedchain import App
-app = App.from_config("mistral.yaml")
+
+config = {
+  'llm': {
+    'provider': 'huggingface',
+    'config': {
+      'model': 'mistralai/Mistral-7B-Instruct-v0.2',
+      'top_p': 0.5
+    }
+  },
+  'embedder': {
+    'provider': 'huggingface',
+    'config': {
+      'model': 'sentence-transformers/all-mpnet-base-v2'
+    }
+  }
+}
+app = App.from_config(config=config)
 app.add("https://www.forbes.com/profile/elon-musk")
 app.add("https://en.wikipedia.org/wiki/Elon_Musk")
 app.query("What is the net worth of Elon Musk today?")
 # Answer: The net worth of Elon Musk today is $258.7 billion.
 ```
-```yaml mistral.yaml
-llm:
-  provider: huggingface
-  config:
-    model: 'mistralai/Mistral-7B-Instruct-v0.2'
-    top_p: 0.5
-embedder:
-  provider: huggingface
-  config:
-    model: 'sentence-transformers/all-mpnet-base-v2'
-```
 </CodeGroup>

 ## Paid Models

 In this section, we will use both LLM and embedding model from OpenAI.

-```python quickstart.py
+```python openai_demo.py
 import os
-# replace this with your OpenAI key
+from embedchain import App
+
+# Replace this with your OpenAI key
 os.environ["OPENAI_API_KEY"] = "sk-xxxx"

-from embedchain import App
 app = App()
 app.add("https://www.forbes.com/profile/elon-musk")
 app.add("https://en.wikipedia.org/wiki/Elon_Musk")
@@ -5,22 +5,30 @@ description: 'Integrate with Langsmith to debug and monitor your LLM app'

 Embedchain now supports integration with [LangSmith](https://www.langchain.com/langsmith).

-To use langsmith, you need to do the following steps
+To use LangSmith, you need to do the following steps.

-1. Have an account on langsmith and keep the environment variables in handy
-2. Set the environments variables in your app so that embedchain has context about it.
+1. Have an account on LangSmith and keep the environment variables in handy
+2. Set the environment variables in your app so that embedchain has context about it.
 3. Just use embedchain and everything will be logged to LangSmith, so that you can better test and monitor your application.

-Lets cover each step in detail.
+Let's cover each step in detail.

-* First make sure that you a LangSmith account created and have all the necessary variables handy. LangSmith has a [good documentation](https://docs.smith.langchain.com/) on how to get started with their service.

-* Once you have the account setup, we will need the following environment variables
+* First make sure that you have created a LangSmith account and have all the necessary variables handy. LangSmith has a [good documentation](https://docs.smith.langchain.com/) on how to get started with their service.
+
+* Once you have setup the account, we will need the following environment variables

 ```bash
+# Setting environment variable for LangChain Tracing V2 integration.
 export LANGCHAIN_TRACING_V2=true
+
+# Setting the API endpoint for LangChain.
 export LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
+
+# Replace '<your-api-key>' with your LangChain API key.
 export LANGCHAIN_API_KEY=<your-api-key>
+
+# Replace '<your-project>' with your LangChain project name, or it defaults to "default".
 export LANGCHAIN_PROJECT=<your-project>  # if not specified, defaults to "default"
 ```

@@ -29,20 +37,32 @@ If you are using Python, you can use the following code to set environment varia
 ```python
 import os

+# Setting environment variable for LangChain Tracing V2 integration.
 os.environ['LANGCHAIN_TRACING_V2'] = 'true'
+
+# Setting the API endpoint for LangChain.
 os.environ['LANGCHAIN_ENDPOINT'] = 'https://api.smith.langchain.com'
-os.environ['LANGCHAIN_API_KEY'] = <your-api-key>
-os.environ['LANGCHAIN_PROJECT] = <your-project>
+
+# Replace '<your-api-key>' with your LangChain API key.
+os.environ['LANGCHAIN_API_KEY'] = '<your-api-key>'
+
+# Replace '<your-project>' with your LangChain project name.
+os.environ['LANGCHAIN_PROJECT'] = '<your-project>'
 ```

-* Now create an app using embedchain and everything will be automatically visible in the LangSmith
+* Now create an app using Embedchain and everything will be automatically visible in the LangSmith


 ```python
 from embedchain import App

+# Initialize EmbedChain application.
 app = App()
+
+# Add data to your app
 app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+
+# Query your app
 app.query("How many companies did Elon found?")
 ```

@@ -88,9 +88,8 @@
      "pages": [
        "components/introduction",
        {
-          "group": "Data sources",
+          "group": "🗂️ Data sources",
          "pages": [
-
            "components/data-sources/overview",
            {
              "group": "Data types",
@@ -129,8 +128,19 @@
            "components/data-sources/data-type-handling"
          ]
        },
+        {
+          "group": "🗄️ Vector databases",
+          "pages": [
+                "components/vector-databases/chromadb",
+                "components/vector-databases/elasticsearch",
+                "components/vector-databases/pinecone",
+                "components/vector-databases/opensearch",
+                "components/vector-databases/qdrant",
+                "components/vector-databases/weaviate",
+                "components/vector-databases/zilliz"
+          ]
+        },
        "components/llms",
-        "components/vector-databases",
        "components/embedding-models",
        "components/evaluation"
      ]
@@ -142,6 +152,7 @@
        "deployment/fly_io",
        "deployment/modal_com",
        "deployment/render_com",
+        "deployment/railway",
        "deployment/streamlit_io",
        "deployment/gradio_app",
        "deployment/huggingface_spaces",
@@ -7,4 +7,4 @@ from embedchain.client import Client  # noqa: F401
 from embedchain.pipeline import Pipeline  # noqa: F401

 # Setup the user directory if doesn't exist already
-Client.setup_dir()
+Client.setup()
@@ -0,0 +1,116 @@
+# A generic, single database configuration.
+
+[alembic]
+# path to migration scripts
+script_location = embedchain:migrations
+
+# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
+# Uncomment the line below if you want the files to be prepended with date and time
+# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
+# for all available tokens
+# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
+
+# sys.path path, will be prepended to sys.path if present.
+# defaults to the current working directory.
+prepend_sys_path = .
+
+# timezone to use when rendering the date within the migration file
+# as well as the filename.
+# If specified, requires the python>=3.9 or backports.zoneinfo library.
+# Any required deps can installed by adding `alembic[tz]` to the pip requirements
+# string value is passed to ZoneInfo()
+# leave blank for localtime
+# timezone =
+
+# max length of characters to apply to the
+# "slug" field
+# truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+# set to 'true' to allow .pyc and .pyo files without
+# a source .py file to be detected as revisions in the
+# versions/ directory
+# sourceless = false
+
+# version location specification; This defaults
+# to alembic/versions.  When using multiple version
+# directories, initial revisions must be specified with --version-path.
+# The path separator used here should be the separator specified by "version_path_separator" below.
+# version_locations = %(here)s/bar:%(here)s/bat:alembic/versions
+
+# version path separator; As mentioned above, this is the character used to split
+# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
+# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
+# Valid values for version_path_separator are:
+#
+# version_path_separator = :
+# version_path_separator = ;
+# version_path_separator = space
+version_path_separator = os  # Use os.pathsep. Default configuration used for new projects.
+
+# set to 'true' to search source files recursively
+# in each "version_locations" directory
+# new in Alembic version 1.10
+# recursive_version_locations = false
+
+# the output encoding used when revision files
+# are written from script.py.mako
+# output_encoding = utf-8
+
+sqlalchemy.url = driver://user:pass@localhost/dbname
+
+
+[post_write_hooks]
+# post_write_hooks defines scripts or Python functions that are run
+# on newly generated revision scripts.  See the documentation for further
+# detail and examples
+
+# format using "black" - use the console_scripts runner, against the "black" entrypoint
+# hooks = black
+# black.type = console_scripts
+# black.entrypoint = black
+# black.options = -l 79 REVISION_SCRIPT_FILENAME
+
+# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
+# hooks = ruff
+# ruff.type = exec
+# ruff.executable = %(here)s/.venv/bin/ruff
+# ruff.options = --fix REVISION_SCRIPT_FILENAME
+
+# Logging configuration
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARN
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARN
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = WARN
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S
@@ -3,26 +3,29 @@ import concurrent.futures
 import json
 import logging
 import os
-import sqlite3
-import uuid
 from typing import Any, Optional, Union

 import requests
 import yaml
 from tqdm import tqdm

-from embedchain.cache import (Config, ExactMatchEvaluation,
-                              SearchDistanceEvaluation, cache,
-                              gptcache_data_manager, gptcache_pre_function)
+from embedchain.cache import (
+    Config,
+    ExactMatchEvaluation,
+    SearchDistanceEvaluation,
+    cache,
+    gptcache_data_manager,
+    gptcache_pre_function,
+)
 from embedchain.client import Client
 from embedchain.config import AppConfig, CacheConfig, ChunkerConfig
-from embedchain.constants import SQLITE_PATH
+from embedchain.core.db.database import get_session, init_db, setup_engine
+from embedchain.core.db.models import DataSource
 from embedchain.embedchain import EmbedChain
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.embedder.openai import OpenAIEmbedder
 from embedchain.evaluation.base import BaseMetric
-from embedchain.evaluation.metrics import (AnswerRelevance, ContextRelevance,
-                                           Groundedness)
+from embedchain.evaluation.metrics import AnswerRelevance, ContextRelevance, Groundedness
 from embedchain.factory import EmbedderFactory, LlmFactory, VectorDBFactory
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm
@@ -33,8 +36,7 @@ from embedchain.utils.misc import validate_config
 from embedchain.vectordb.base import BaseVectorDB
 from embedchain.vectordb.chroma import ChromaDB

-# Set up the user directory if it doesn't exist already
-Client.setup_dir()
+logger = logging.getLogger(__name__)


@register_deserializable
@@ -54,10 +56,10 @@ class App(EmbedChain):
        embedding_model: BaseEmbedder = None,
        llm: BaseLlm = None,
        config_data: dict = None,
-        log_level=logging.WARN,
        auto_deploy: bool = False,
        chunker: ChunkerConfig = None,
        cache_config: CacheConfig = None,
+        log_level: int = logging.WARN,
    ):
        """
        Initialize a new `App` instance.
@@ -72,8 +74,6 @@ class App(EmbedChain):
        :type llm: BaseLlm, optional
        :param config_data: Config dictionary, defaults to None
        :type config_data: dict, optional
-        :param log_level: Log level to use, defaults to logging.WARN
-        :type log_level: int, optional
        :param auto_deploy: Whether to deploy the pipeline automatically, defaults to False
        :type auto_deploy: bool, optional
        :raises Exception: If an error occurs while creating the pipeline
@@ -87,22 +87,22 @@ class App(EmbedChain):
        if name and config:
            raise Exception("Cannot provide both name and config. Please provide only one of them.")

-        logging.basicConfig(level=log_level, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s")
-        self.logger = logging.getLogger(__name__)
+        # Initialize the metadata db for the app
+        setup_engine(database_uri=os.environ.get("EMBEDCHAIN_DB_URI"))
+        init_db()
+
        self.auto_deploy = auto_deploy
        # Store the dict config as an attribute to be able to send it
        self.config_data = config_data if (config_data and validate_config(config_data)) else None
        self.client = None
        # pipeline_id from the backend
        self.id = None
-        self.chunker = None
-        if chunker:
-            self.chunker = ChunkerConfig(**chunker)
+        self.chunker = ChunkerConfig(**chunker) if chunker else None
        self.cache_config = cache_config

        self.config = config or AppConfig()
        self.name = self.config.name
-        self.config.id = self.local_id = str(uuid.uuid4()) if self.config.id is None else self.config.id
+        self.config.id = self.local_id = "default-app-id" if self.config.id is None else self.config.id

        if id is not None:
            # Init client first since user is trying to fetch the pipeline
@@ -120,6 +120,9 @@ class App(EmbedChain):
        self.llm = llm or OpenAILlm()
        self._init_db()

+        # Session for the metadata db
+        self.db_session = get_session()
+
        # If cache_config is provided, initializing the cache ...
        if self.cache_config is not None:
            self._init_cache()
@@ -127,27 +130,6 @@ class App(EmbedChain):
        # Send anonymous telemetry
        self._telemetry_props = {"class": self.__class__.__name__}
        self.telemetry = AnonymousTelemetry(enabled=self.config.collect_metrics)
-
-        # Establish a connection to the SQLite database
-        self.connection = sqlite3.connect(SQLITE_PATH, check_same_thread=False)
-        self.cursor = self.connection.cursor()
-
-        # Create the 'data_sources' table if it doesn't exist
-        self.cursor.execute(
-            """
-            CREATE TABLE IF NOT EXISTS data_sources (
-                pipeline_id TEXT,
-                hash TEXT,
-                type TEXT,
-                value TEXT,
-                metadata TEXT,
-                is_uploaded INTEGER DEFAULT 0,
-                PRIMARY KEY (pipeline_id, hash)
-            )
-        """
-        )
-        self.connection.commit()
-        # Send anonymous telemetry
        self.telemetry.capture(event_name="init", properties=self._telemetry_props)

        self.user_asks = []
@@ -257,7 +239,7 @@ class App(EmbedChain):
                response.raise_for_status()
                return response.status_code == 200
        except Exception as e:
-            self.logger.exception(f"Error occurred during file upload: {str(e)}")
+            logger.exception(f"Error occurred during file upload: {str(e)}")
            print("❌ Error occurred during file upload!")
            return False

@@ -291,7 +273,7 @@ class App(EmbedChain):
                metadata = {"file_path": data_value, "s3_key": s3_key}
                data_value = presigned_url
            else:
-                self.logger.error(f"File upload failed for hash: {data_hash}")
+                logger.error(f"File upload failed for hash: {data_hash}")
                return False
        else:
            if data_type == "qna_pair":
@@ -307,20 +289,14 @@ class App(EmbedChain):
            return False

    def _mark_data_as_uploaded(self, data_hash):
-        self.cursor.execute(
-            "UPDATE data_sources SET is_uploaded = 1 WHERE hash = ? AND pipeline_id = ?",
-            (data_hash, self.local_id),
-        )
-        self.connection.commit()
+        self.db_session.query(DataSource).filter_by(hash=data_hash, app_id=self.local_id).update({"is_uploaded": 1})

    def get_data_sources(self):
-        db_data = self.cursor.execute("SELECT * FROM data_sources WHERE pipeline_id = ?", (self.local_id,)).fetchall()
-
-        data_sources = []
-        for data in db_data:
-            data_sources.append({"data_type": data[2], "data_value": data[3], "metadata": data[4]})
-
-        return data_sources
+        data_sources = self.db_session.query(DataSource).filter_by(app_id=self.local_id).all()
+        results = []
+        for row in data_sources:
+            results.append({"data_type": row.type, "data_value": row.value, "metadata": row.meta_data})
+        return results

    def deploy(self):
        if self.client is None:
@@ -329,14 +305,11 @@ class App(EmbedChain):
        pipeline_data = self._create_pipeline()
        self.id = pipeline_data["id"]

-        results = self.cursor.execute(
-            "SELECT * FROM data_sources WHERE pipeline_id = ? AND is_uploaded = 0", (self.local_id,)  # noqa:E501
-        ).fetchall()
-
+        results = self.db_session.query(DataSource).filter_by(app_id=self.local_id, is_uploaded=0).all()
        if len(results) > 0:
            print("🛠️ Adding data to your pipeline...")
        for result in results:
-            data_hash, data_type, data_value = result[1], result[2], result[3]
+            data_hash, data_type, data_value = result.hash, result.data_type, result.data_value
            self._process_and_upload_data(data_hash, data_type, data_value)

        # Send anonymous telemetry
@@ -351,18 +324,18 @@ class App(EmbedChain):
        yaml_path: Optional[str] = None,
    ):
        """
-        Instantiate a Pipeline object from a configuration.
+        Instantiate a App object from a configuration.

        :param config_path: Path to the YAML or JSON configuration file.
        :type config_path: Optional[str]
        :param config: A dictionary containing the configuration.
        :type config: Optional[dict[str, Any]]
-        :param auto_deploy: Whether to deploy the pipeline automatically, defaults to False
+        :param auto_deploy: Whether to deploy the app automatically, defaults to False
        :type auto_deploy: bool, optional
        :param yaml_path: (Deprecated) Path to the YAML configuration file. Use config_path instead.
        :type yaml_path: Optional[str]
-        :return: An instance of the Pipeline class.
-        :rtype: Pipeline
+        :return: An instance of the App class.
+        :rtype: App
        """
        # Backward compatibility for yaml_path
        if yaml_path and not config_path:
@@ -385,18 +358,16 @@ class App(EmbedChain):
        elif config and isinstance(config, dict):
            config_data = config
        else:
-            logging.error(
+            logger.error(
                "Please provide either a config file path (YAML or JSON) or a config dictionary. Falling back to defaults because no config is provided.",  # noqa: E501
            )
            config_data = {}

-        try:
-            validate_config(config_data)
-        except Exception as e:
-            raise Exception(f"Error occurred while validating the config. Error: {str(e)}")
+        # Validate the config
+        validate_config(config_data)

        app_config_data = config_data.get("app", {}).get("config", {})
-        db_config_data = config_data.get("vectordb", {})
+        vector_db_config_data = config_data.get("vectordb", {})
        embedding_model_config_data = config_data.get("embedding_model", config_data.get("embedder", {}))
        llm_config_data = config_data.get("llm", {})
        chunker_config_data = config_data.get("chunker", {})
@@ -404,10 +375,14 @@ class App(EmbedChain):

        app_config = AppConfig(**app_config_data)

-        db_provider = db_config_data.get("provider", "chroma")
-        db = VectorDBFactory.create(db_provider, db_config_data.get("config", {}))
+        vector_db_provider = vector_db_config_data.get("provider", "chroma")
+        vector_db = VectorDBFactory.create(vector_db_provider, vector_db_config_data.get("config", {}))

        if llm_config_data:
+            # Initialize the metadata db for the app here since llmfactory needs it for initialization of
+            # the llm memory
+            setup_engine(database_uri=os.environ.get("EMBEDCHAIN_DB_URI"))
+            init_db()
            llm_provider = llm_config_data.get("provider", "openai")
            llm = LlmFactory.create(llm_provider, llm_config_data.get("config", {}))
        else:
@@ -423,14 +398,10 @@ class App(EmbedChain):
        else:
            cache_config = None

-        # Send anonymous telemetry
-        event_properties = {"init_type": "config_data"}
-        AnonymousTelemetry().capture(event_name="init", properties=event_properties)
-
        return cls(
            config=app_config,
            llm=llm,
-            db=db,
+            db=vector_db,
            embedding_model=embedding_model,
            config_data=config_data,
            auto_deploy=auto_deploy,
@@ -505,12 +476,12 @@ class App(EmbedChain):
            EvalMetric.GROUNDEDNESS.value,
        ]

-        logging.info(f"Collecting data from {len(queries)} questions for evaluation...")
+        logger.info(f"Collecting data from {len(queries)} questions for evaluation...")
        dataset = []
        for q, a, c in zip(queries, answers, contexts):
            dataset.append(EvalData(question=q, answer=a, contexts=c))

-        logging.info(f"Evaluating {len(dataset)} data points...")
+        logger.info(f"Evaluating {len(dataset)} data points...")
        result = {}
        with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor:
            future_to_metric = {executor.submit(self._eval, dataset, metric): metric for metric in metrics}
@@ -17,6 +17,8 @@ except ModuleNotFoundError:
    ) from None


+logger = logging.getLogger(__name__)
+
 intents = discord.Intents.default()
 intents.message_content = True
 client = discord.Client(intents=intents)
@@ -37,7 +39,7 @@ class DiscordBot(BaseBot):
            self.add(data)
            response = f"Added data from: {data}"
        except Exception:
-            logging.exception(f"Failed to add data {data}.")
+            logger.exception(f"Failed to add data {data}.")
            response = "Some error occurred while adding data."
        return response

@@ -45,7 +47,7 @@ class DiscordBot(BaseBot):
        try:
            response = self.query(message)
        except Exception:
-            logging.exception(f"Failed to query {message}.")
+            logger.exception(f"Failed to query {message}.")
            response = "An error occurred. Please try again!"
        return response

@@ -60,7 +62,7 @@ class DiscordBot(BaseBot):
 async def query_command(interaction: discord.Interaction, question: str):
    await interaction.response.defer()
    member = client.guilds[0].get_member(client.user.id)
-    logging.info(f"User: {member}, Query: {question}")
+    logger.info(f"User: {member}, Query: {question}")
    try:
        answer = discord_bot.ask_bot(question)
        if args.include_question:
@@ -70,20 +72,20 @@ async def query_command(interaction: discord.Interaction, question: str):
        await interaction.followup.send(response)
    except Exception as e:
        await interaction.followup.send("An error occurred. Please try again!")
-        logging.error("Error occurred during 'query' command:", e)
+        logger.error("Error occurred during 'query' command:", e)


@tree.command(name="add", description="add new content to the embedchain database")
 async def add_command(interaction: discord.Interaction, url_or_text: str):
    await interaction.response.defer()
    member = client.guilds[0].get_member(client.user.id)
-    logging.info(f"User: {member}, Add: {url_or_text}")
+    logger.info(f"User: {member}, Add: {url_or_text}")
    try:
        response = discord_bot.add_data(url_or_text)
        await interaction.followup.send(response)
    except Exception as e:
        await interaction.followup.send("An error occurred. Please try again!")
-        logging.error("Error occurred during 'add' command:", e)
+        logger.error("Error occurred during 'add' command:", e)


@tree.command(name="ping", description="Simple ping pong command")
@@ -96,7 +98,7 @@ async def on_app_command_error(interaction: discord.Interaction, error: discord.
    if isinstance(error, commands.CommandNotFound):
        await interaction.followup.send("Invalid command. Please refer to the documentation for correct syntax.")
    else:
-        logging.error("Error occurred during command execution:", error)
+        logger.error("Error occurred during command execution:", error)


@client.event
@@ -104,8 +106,8 @@ async def on_ready():
    # TODO: Sync in admin command, to not hit rate limits.
    # This might be overkill for most users, and it would require to set a guild or user id, where sync is allowed.
    await tree.sync()
-    logging.debug("Command tree synced")
-    logging.info(f"Logged in as {client.user.name}")
+    logger.debug("Command tree synced")
+    logger.info(f"Logged in as {client.user.name}")


 def start_command():
@@ -19,6 +19,8 @@ except ModuleNotFoundError:
    ) from None


+logger = logging.getLogger(__name__)
+
 SLACK_BOT_TOKEN = os.environ.get("SLACK_BOT_TOKEN")


@@ -42,10 +44,10 @@ class SlackBot(BaseBot):
                    try:
                        response = self.chat_bot.chat(question)
                        self.send_slack_message(message["channel"], response)
-                        logging.info("Query answered successfully!")
+                        logger.info("Query answered successfully!")
                    except Exception as e:
                        self.send_slack_message(message["channel"], "An error occurred. Please try again!")
-                        logging.error("Error occurred during 'query' command:", e)
+                        logger.error("Error occurred during 'query' command:", e)
                elif text.startswith("add"):
                    _, data_type, url_or_text = text.split(" ", 2)
                    if url_or_text.startswith("<") and url_or_text.endswith(">"):
@@ -55,10 +57,10 @@ class SlackBot(BaseBot):
                        self.send_slack_message(message["channel"], f"Added {data_type} : {url_or_text}")
                    except ValueError as e:
                        self.send_slack_message(message["channel"], f"Error: {str(e)}")
-                        logging.error("Error occurred during 'add' command:", e)
+                        logger.error("Error occurred during 'add' command:", e)
                    except Exception as e:
                        self.send_slack_message(message["channel"], f"Failed to add {data_type} : {url_or_text}")
-                        logging.error("Error occurred during 'add' command:", e)
+                        logger.error("Error occurred during 'add' command:", e)

    def send_slack_message(self, channel, message):
        response = self.client.chat_postMessage(channel=channel, text=message)
@@ -68,7 +70,7 @@ class SlackBot(BaseBot):
        app = Flask(__name__)

        def signal_handler(sig, frame):
-            logging.info("\nGracefully shutting down the SlackBot...")
+            logger.info("\nGracefully shutting down the SlackBot...")
            sys.exit(0)

        signal.signal(signal.SIGINT, signal_handler)
@@ -8,6 +8,8 @@ from embedchain.helpers.json_serializable import register_deserializable

 from .base import BaseBot

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class WhatsAppBot(BaseBot):
@@ -35,7 +37,7 @@ class WhatsAppBot(BaseBot):
            self.add(data)
            response = f"Added data from: {data}"
        except Exception:
-            logging.exception(f"Failed to add data {data}.")
+            logger.exception(f"Failed to add data {data}.")
            response = "Some error occurred while adding data."
        return response

@@ -43,7 +45,7 @@ class WhatsAppBot(BaseBot):
        try:
            response = self.query(message)
        except Exception:
-            logging.exception(f"Failed to query {message}.")
+            logger.exception(f"Failed to query {message}.")
            response = "An error occurred. Please try again!"
        return response

@@ -51,7 +53,7 @@ class WhatsAppBot(BaseBot):
        app = self.flask.Flask(__name__)

        def signal_handler(sig, frame):
-            logging.info("\nGracefully shutting down the WhatsAppBot...")
+            logger.info("\nGracefully shutting down the WhatsAppBot...")
            sys.exit(0)

        signal.signal(signal.SIGINT, signal_handler)
@@ -14,6 +14,8 @@ from gptcache.similarity_evaluation.distance import \
 from gptcache.similarity_evaluation.exact_match import \
    ExactMatchEvaluation  # noqa: F401

+logger = logging.getLogger(__name__)
+

 def gptcache_pre_function(data: dict[str, Any], **params: dict[str, Any]):
    return data["input_query"]
@@ -24,12 +26,12 @@ def gptcache_data_manager(vector_dimension):


 def gptcache_data_convert(cache_data):
-    logging.info("[Cache] Cache hit, returning cache data...")
+    logger.info("[Cache] Cache hit, returning cache data...")
    return cache_data


 def gptcache_update_cache_callback(llm_data, update_cache_func, *args, **kwargs):
-    logging.info("[Cache] Cache missed, updating cache...")
+    logger.info("[Cache] Cache missed, updating cache...")
    update_cache_func(Answer(llm_data, CacheDataType.STR))
    return llm_data

@@ -6,6 +6,8 @@ from embedchain.config.add_config import ChunkerConfig
 from embedchain.helpers.json_serializable import JSONSerializable
 from embedchain.models.data_type import DataType

+logger = logging.getLogger(__name__)
+

 class BaseChunker(JSONSerializable):
    def __init__(self, text_splitter):
@@ -27,7 +29,7 @@ class BaseChunker(JSONSerializable):
        chunk_ids = []
        id_map = {}
        min_chunk_size = config.min_chunk_size if config is not None else 1
-        logging.info(f"Skipping chunks smaller than {min_chunk_size} characters")
+        logger.info(f"Skipping chunks smaller than {min_chunk_size} characters")
        data_result = loader.load_data(src)
        data_records = data_result["data"]
        doc_id = data_result["doc_id"]
@@ -7,6 +7,8 @@ import requests

 from embedchain.constants import CONFIG_DIR, CONFIG_FILE

+logger = logging.getLogger(__name__)
+

 class Client:
    def __init__(self, api_key=None, host="https://apiv2.embedchain.ai"):
@@ -24,14 +26,14 @@ class Client:
        else:
            if "api_key" in self.config_data:
                self.api_key = self.config_data["api_key"]
-                logging.info("API key loaded successfully!")
+                logger.info("API key loaded successfully!")
            else:
                raise ValueError(
                    "You are not logged in. Please obtain an API key from https://app.embedchain.ai/settings/keys/"
                )

    @classmethod
-    def setup_dir(cls):
+    def setup(cls):
        """
        Loads the user id from the config file if it exists, otherwise generates a new
        one and saves it to the config file.
@@ -40,6 +42,7 @@ class Client:
        :rtype: str
        """
        os.makedirs(CONFIG_DIR, exist_ok=True)
+
        if os.path.exists(CONFIG_FILE):
            with open(CONFIG_FILE, "r") as f:
                data = json.load(f)
@@ -53,7 +56,7 @@ class Client:
    @classmethod
    def load_config(cls):
        if not os.path.exists(CONFIG_FILE):
-            cls.setup_dir()
+            cls.setup()

        with open(CONFIG_FILE, "r") as config_file:
            return json.load(config_file)
@@ -63,7 +66,7 @@ class Client:
        with open(CONFIG_FILE, "w") as config_file:
            json.dump(self.config_data, config_file, indent=4)

-        logging.info("API key saved successfully!")
+        logger.info("API key saved successfully!")

    def clear(self):
        if "api_key" in self.config_data:
@@ -71,17 +74,17 @@ class Client:
            with open(CONFIG_FILE, "w") as config_file:
                json.dump(self.config_data, config_file, indent=4)
            self.api_key = None
-            logging.info("API key deleted successfully!")
+            logger.info("API key deleted successfully!")
        else:
-            logging.warning("API key not found in the configuration file.")
+            logger.warning("API key not found in the configuration file.")

    def update(self, api_key):
        if self.check(api_key):
            self.api_key = api_key
            self.save()
-            logging.info("API key updated successfully!")
+            logger.info("API key updated successfully!")
        else:
-            logging.warning("Invalid API key provided. API key not updated.")
+            logger.warning("Invalid API key provided. API key not updated.")

    def check(self, api_key):
        validation_url = f"{self.host}/api/v1/accounts/api_keys/validate/"
@@ -89,8 +92,8 @@ class Client:
        if response.status_code == 200:
            return True
        else:
-            logging.warning(f"Response from API: {response.text}")
-            logging.warning("Invalid API key. Unable to validate.")
+            logger.warning(f"Response from API: {response.text}")
+            logger.warning("Invalid API key. Unable to validate.")
            return False

    def get(self):
@@ -5,6 +5,8 @@ from embedchain.config.base_config import BaseConfig
 from embedchain.helpers.json_serializable import JSONSerializable
 from embedchain.vectordb.base import BaseVectorDB

+logger = logging.getLogger(__name__)
+

 class BaseAppConfig(BaseConfig, JSONSerializable):
    """
@@ -36,29 +38,21 @@ class BaseAppConfig(BaseConfig, JSONSerializable):
        defaults to None
        :type collection_name: Optional[str], optional
        """
-        self._setup_logging(log_level)
        self.id = id
        self.collect_metrics = True if (collect_metrics is True or collect_metrics is None) else False
        self.collection_name = collection_name

        if db:
            self._db = db
-            logging.warning(
+            logger.warning(
                "DEPRECATION WARNING: Please supply the database as the second parameter during app init. "
                "Such as `app(config=config, db=db)`."
            )

        if collection_name:
-            logging.warning("DEPRECATION WARNING: Please supply the collection name to the database config.")
+            logger.warning("DEPRECATION WARNING: Please supply the collection name to the database config.")
        return

-    def _setup_logging(self, debug_level):
-        level = logging.WARNING  # Default level
-        if debug_level is not None:
-            level = getattr(logging, debug_level.upper(), None)
-            if not isinstance(level, int):
-                raise ValueError(f"Invalid log level: {debug_level}")
-
-        logging.basicConfig(format="%(asctime)s [%(name)s] [%(levelname)s] %(message)s", level=level)
-        self.logger = logging.getLogger(__name__)
-        return
+    def _setup_logging(self, log_level):
+        logger.basicConfig(format="%(asctime)s [%(name)s] [%(levelname)s] %(message)s", level=log_level)
+        self.logger = logger.getLogger(__name__)
@@ -11,6 +11,7 @@ class BaseEmbedderConfig:
        deployment_name: Optional[str] = None,
        vector_dimension: Optional[int] = None,
        api_key: Optional[str] = None,
+        api_base: Optional[str] = None,
    ):
        """
        Initialize a new instance of an embedder config class.
@@ -24,3 +25,4 @@ class BaseEmbedderConfig:
        self.deployment_name = deployment_name
        self.vector_dimension = vector_dimension
        self.api_key = api_key
+        self.api_base = api_base
@@ -6,40 +6,63 @@ from typing import Any, Optional
 from embedchain.config.base_config import BaseConfig
 from embedchain.helpers.json_serializable import register_deserializable

+logger = logging.getLogger(__name__)
+
 DEFAULT_PROMPT = """
-  Use the following pieces of context to answer the query at the end.
-  If you don't know the answer, just say that you don't know, don't try to make up an answer.
+You are a Q&A expert system. Your responses must always be rooted in the context provided for each query. Here are some guidelines to follow:

-  $context
+1. Refrain from explicitly mentioning the context provided in your response.
+2. The context should silently guide your answers without being directly acknowledged.
+3. Do not use phrases such as 'According to the context provided', 'Based on the context, ...' etc.

-  Query: $query
+Context information:
+----------------------
+$context
+----------------------

-  Helpful Answer:
+Query: $query
+Answer:
 """  # noqa:E501

 DEFAULT_PROMPT_WITH_HISTORY = """
-  Use the following pieces of context to answer the query at the end.
-  If you don't know the answer, just say that you don't know, don't try to make up an answer.
-  I will provide you with our conversation history.
+You are a Q&A expert system. Your responses must always be rooted in the context provided for each query. You are also provided with the conversation history with the user. Make sure to use relevant context from conversation history as needed.

-  $context
+Here are some guidelines to follow:

-  History: $history
+1. Refrain from explicitly mentioning the context provided in your response.
+2. The context should silently guide your answers without being directly acknowledged.
+3. Do not use phrases such as 'According to the context provided', 'Based on the context, ...' etc.

-  Query: $query
+Context information:
+----------------------
+$context
+----------------------

-  Helpful Answer:
+Conversation history:
+----------------------
+$history
+----------------------
+
+Query: $query
+Answer:
 """  # noqa:E501

 DOCS_SITE_DEFAULT_PROMPT = """
-  Use the following pieces of context to answer the query at the end.
-  If you don't know the answer, just say that you don't know, don't try to make up an answer. Wherever possible, give complete code snippet. Dont make up any code snippet on your own.
+You are an expert AI assistant for developer support product. Your responses must always be rooted in the context provided for each query. Wherever possible, give complete code snippet. Dont make up any code snippet on your own.

-  $context
+Here are some guidelines to follow:

-  Query: $query
+1. Refrain from explicitly mentioning the context provided in your response.
+2. The context should silently guide your answers without being directly acknowledged.
+3. Do not use phrases such as 'According to the context provided', 'Based on the context, ...' etc.

-  Helpful Answer:
+Context information:
+----------------------
+$context
+----------------------
+
+Query: $query
+Answer:
 """  # noqa:E501

 DEFAULT_PROMPT_TEMPLATE = Template(DEFAULT_PROMPT)
@@ -72,8 +95,10 @@ class BaseLlmConfig(BaseConfig):
        query_type: Optional[str] = None,
        callbacks: Optional[list] = None,
        api_key: Optional[str] = None,
+        base_url: Optional[str] = None,
        endpoint: Optional[str] = None,
        model_kwargs: Optional[dict[str, Any]] = None,
+        local: Optional[bool] = False,
    ):
        """
        Initializes a configuration class instance for the LLM.
@@ -117,12 +142,14 @@ class BaseLlmConfig(BaseConfig):
        :type callbacks: Optional[list], optional
        :param query_type: The type of query to use, defaults to None
        :type query_type: Optional[str], optional
+        :param local: If True, the model will be run locally, defaults to False (for huggingface provider)
+        :type local: Optional[bool], optional
        :raises ValueError: If the template is not valid as template should
        contain $context and $query (and optionally $history)
        :raises ValueError: Stream is not boolean
        """
        if template is not None:
-            logging.warning(
+            logger.warning(
                "The `template` argument is deprecated and will be removed in a future version. "
                + "Please use `prompt` instead."
            )
@@ -142,8 +169,10 @@ class BaseLlmConfig(BaseConfig):
        self.query_type = query_type
        self.callbacks = callbacks
        self.api_key = api_key
+        self.base_url = base_url
        self.endpoint = endpoint
        self.model_kwargs = model_kwargs
+        self.local = local

        if isinstance(prompt, str):
            prompt = Template(prompt)
@@ -15,6 +15,8 @@ class PineconeDBConfig(BaseVectorDbConfig):
        metric: Optional[str] = "cosine",
        pod_config: Optional[dict[str, any]] = None,
        serverless_config: Optional[dict[str, any]] = None,
+        hybrid_search: bool = False,
+        bm25_encoder: any = None,
        **extra_params: dict[str, any],
    ):
        self.metric = metric
@@ -22,6 +24,8 @@ class PineconeDBConfig(BaseVectorDbConfig):
        self.index_name = index_name
        self.vector_dimension = vector_dimension
        self.extra_params = extra_params
+        self.hybrid_search = hybrid_search
+        self.bm25_encoder = bm25_encoder
        if pod_config is None and serverless_config is None:
            # If no config is provided, use the default pod spec config
            pod_environment = os.environ.get("PINECONE_ENV", "gcp-starter")
@@ -33,4 +37,9 @@ class PineconeDBConfig(BaseVectorDbConfig):
        if self.pod_config and self.serverless_config:
            raise ValueError("Only one of pod_config or serverless_config can be provided.")

+        if self.hybrid_search and self.metric != "dotproduct":
+            raise ValueError(
+                "Hybrid search is only supported with dotproduct metric in Pinecone. See full docs here: https://docs.pinecone.io/docs/hybrid-search#limitations"
+            )  # noqa:E501
+
        super().__init__(collection_name=self.index_name, dir=None)
@@ -6,3 +6,6 @@ HOME_DIR = str(Path.home())
 CONFIG_DIR = os.path.join(HOME_DIR, ".embedchain")
 CONFIG_FILE = os.path.join(CONFIG_DIR, "config.json")
 SQLITE_PATH = os.path.join(CONFIG_DIR, "embedchain.db")
+
+# Set the environment variable for the database URI
+os.environ.setdefault("EMBEDCHAIN_DB_URI", f"sqlite:///{SQLITE_PATH}")
@@ -0,0 +1,88 @@
+import os
+
+from alembic import command
+from alembic.config import Config
+from sqlalchemy import create_engine
+from sqlalchemy.engine.base import Engine
+from sqlalchemy.orm import Session as SQLAlchemySession
+from sqlalchemy.orm import scoped_session, sessionmaker
+
+from .models import Base
+
+
+class DatabaseManager:
+    def __init__(self, echo: bool = False):
+        self.database_uri = os.environ.get("EMBEDCHAIN_DB_URI")
+        self.echo = echo
+        self.engine: Engine = None
+        self._session_factory = None
+
+    def setup_engine(self) -> None:
+        """Initializes the database engine and session factory."""
+        if not self.database_uri:
+            raise RuntimeError("Database URI is not set. Set the EMBEDCHAIN_DB_URI environment variable.")
+        connect_args = {}
+        if self.database_uri.startswith("sqlite"):
+            connect_args["check_same_thread"] = False
+        self.engine = create_engine(self.database_uri, echo=self.echo, connect_args=connect_args)
+        self._session_factory = scoped_session(sessionmaker(bind=self.engine))
+        Base.metadata.bind = self.engine
+
+    def init_db(self) -> None:
+        """Creates all tables defined in the Base metadata."""
+        if not self.engine:
+            raise RuntimeError("Database engine is not initialized. Call setup_engine() first.")
+        Base.metadata.create_all(self.engine)
+
+    def get_session(self) -> SQLAlchemySession:
+        """Provides a session for database operations."""
+        if not self._session_factory:
+            raise RuntimeError("Session factory is not initialized. Call setup_engine() first.")
+        return self._session_factory()
+
+    def close_session(self) -> None:
+        """Closes the current session."""
+        if self._session_factory:
+            self._session_factory.remove()
+
+    def execute_transaction(self, transaction_block):
+        """Executes a block of code within a database transaction."""
+        session = self.get_session()
+        try:
+            transaction_block(session)
+            session.commit()
+        except Exception as e:
+            session.rollback()
+            raise e
+        finally:
+            self.close_session()
+
+
+# Singleton pattern to use throughout the application
+database_manager = DatabaseManager()
+
+
+# Convenience functions for backward compatibility and ease of use
+def setup_engine(database_uri: str, echo: bool = False) -> None:
+    database_manager.database_uri = database_uri
+    database_manager.echo = echo
+    database_manager.setup_engine()
+
+
+def alembic_upgrade() -> None:
+    """Upgrades the database to the latest version."""
+    alembic_config_path = os.path.join(os.path.dirname(__file__), "..", "..", "alembic.ini")
+    alembic_cfg = Config(alembic_config_path)
+    command.upgrade(alembic_cfg, "head")
+
+
+def init_db() -> None:
+    alembic_upgrade()
+
+
+def get_session() -> SQLAlchemySession:
+    return database_manager.get_session()
+
+
+def execute_transaction(transaction_block):
+    database_manager.execute_transaction(transaction_block)
@@ -0,0 +1,31 @@
+import uuid
+
+from sqlalchemy import TIMESTAMP, Column, Integer, String, Text, func
+from sqlalchemy.orm import declarative_base
+
+Base = declarative_base()
+metadata = Base.metadata
+
+
+class DataSource(Base):
+    __tablename__ = "ec_data_sources"
+
+    id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
+    app_id = Column(Text, index=True)
+    hash = Column(Text, index=True)
+    type = Column(Text, index=True)
+    value = Column(Text)
+    meta_data = Column(Text, name="metadata")
+    is_uploaded = Column(Integer, default=0)
+
+
+class ChatHistory(Base):
+    __tablename__ = "ec_chat_history"
+
+    app_id = Column(String, primary_key=True)
+    id = Column(String, primary_key=True)
+    session_id = Column(String, primary_key=True, index=True)
+    question = Column(Text)
+    answer = Column(Text)
+    meta_data = Column(Text, name="metadata")
+    created_at = Column(TIMESTAMP, default=func.current_timestamp(), index=True)
@@ -1,7 +1,6 @@
 import hashlib
 import json
 import logging
-import sqlite3
 from typing import Any, Optional, Union

 from dotenv import load_dotenv
@@ -13,7 +12,7 @@ from embedchain.cache import (adapt, get_gptcache_session,
 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config import AddConfig, BaseLlmConfig, ChunkerConfig
 from embedchain.config.base_app_config import BaseAppConfig
-from embedchain.constants import SQLITE_PATH
+from embedchain.core.db.models import ChatHistory, DataSource
 from embedchain.data_formatter import DataFormatter
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.helpers.json_serializable import JSONSerializable
@@ -21,12 +20,13 @@ from embedchain.llm.base import BaseLlm
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.models.data_type import (DataType, DirectDataType,
                                         IndirectDataType, SpecialDataType)
-from embedchain.telemetry.posthog import AnonymousTelemetry
 from embedchain.utils.misc import detect_datatype, is_valid_json_string
 from embedchain.vectordb.base import BaseVectorDB

 load_dotenv()

+logger = logging.getLogger(__name__)
+

 class EmbedChain(JSONSerializable):
    def __init__(
@@ -53,7 +53,6 @@ class EmbedChain(JSONSerializable):
        :type system_prompt: Optional[str], optional
        :raises ValueError: No database or embedder provided.
        """
-
        self.config = config
        self.cache_config = None
        # Llm
@@ -85,30 +84,6 @@ class EmbedChain(JSONSerializable):
        self.user_asks = []

        self.chunker: Optional[ChunkerConfig] = None
-        # Send anonymous telemetry
-        self._telemetry_props = {"class": self.__class__.__name__}
-        self.telemetry = AnonymousTelemetry(enabled=self.config.collect_metrics)
-        # Establish a connection to the SQLite database
-        self.connection = sqlite3.connect(SQLITE_PATH, check_same_thread=False)
-        self.cursor = self.connection.cursor()
-
-        # Create the 'data_sources' table if it doesn't exist
-        self.cursor.execute(
-            """
-            CREATE TABLE IF NOT EXISTS data_sources (
-                pipeline_id TEXT,
-                hash TEXT,
-                type TEXT,
-                value TEXT,
-                metadata TEXT,
-                is_uploaded INTEGER DEFAULT 0,
-                PRIMARY KEY (pipeline_id, hash)
-            )
-        """
-        )
-        self.connection.commit()
-        # Send anonymous telemetry
-        self.telemetry.capture(event_name="init", properties=self._telemetry_props)

    @property
    def collect_metrics(self):
@@ -170,10 +145,10 @@ class EmbedChain(JSONSerializable):

        try:
            DataType(source)
-            logging.warning(
+            logger.warning(
                f"""Starting from version v0.0.40, Embedchain can automatically detect the data type. So, in the `add` method, the argument order has changed. You no longer need to specify '{source}' for the `source` argument. So the code snippet will be `.add("{data_type}", "{source}")`"""  # noqa #E501
            )
-            logging.warning(
+            logger.warning(
                "Embedchain is swapping the arguments for you. This functionality might be deprecated in the future, so please adjust your code."  # noqa #E501
            )
            source, data_type = data_type, source
@@ -184,7 +159,7 @@ class EmbedChain(JSONSerializable):
            try:
                data_type = DataType(data_type)
            except ValueError:
-                logging.info(
+                logger.info(
                    f"Invalid data_type: '{data_type}', using `custom` instead.\n Check docs to pass the valid data type: `https://docs.embedchain.ai/data-sources/overview`"  # noqa: E501
                )
                data_type = DataType.CUSTOM
@@ -204,21 +179,25 @@ class EmbedChain(JSONSerializable):
        if data_type in {DataType.DOCS_SITE}:
            self.is_docs_site_instance = True

-        # Insert the data into the 'data' table
-        self.cursor.execute(
-            """
-            INSERT OR REPLACE INTO data_sources (hash, pipeline_id, type, value, metadata)
-            VALUES (?, ?, ?, ?, ?)
-        """,
-            (source_hash, self.config.id, data_type.value, str(source), json.dumps(metadata)),
+        # Insert the data into the 'ec_data_sources' table
+        self.db_session.add(
+            DataSource(
+                hash=source_hash,
+                app_id=self.config.id,
+                type=data_type.value,
+                value=source,
+                metadata=json.dumps(metadata),
+            )
        )
-
-        # Commit the transaction
-        self.connection.commit()
+        try:
+            self.db_session.commit()
+        except Exception as e:
+            logger.error(f"Error adding data source: {e}")
+            self.db_session.rollback()

        if dry_run:
            data_chunks_info = {"chunks": documents, "metadata": metadatas, "count": len(documents), "type": data_type}
-            logging.debug(f"Dry run info : {data_chunks_info}")
+            logger.debug(f"Dry run info : {data_chunks_info}")
            return data_chunks_info

        # Send anonymous telemetry
@@ -237,46 +216,6 @@ class EmbedChain(JSONSerializable):

        return source_hash

-    def add_local(
-        self,
-        source: Any,
-        data_type: Optional[DataType] = None,
-        metadata: Optional[dict[str, Any]] = None,
-        config: Optional[AddConfig] = None,
-        **kwargs: Optional[dict[str, Any]],
-    ):
-        """
-        Adds the data from the given URL to the vector db.
-        Loads the data, chunks it, create embedding for each chunk
-        and then stores the embedding to vector database.
-
-        Warning:
-            This method is deprecated and will be removed in future versions. Use `add` instead.
-
-        :param source: The data to embed, can be a URL, local file or raw content, depending on the data type.
-        :type source: Any
-        :param data_type: Automatically detected, but can be forced with this argument. The type of the data to add,
-        defaults to None
-        :type data_type: Optional[DataType], optional
-        :param metadata: Metadata associated with the data source., defaults to None
-        :type metadata: Optional[dict[str, Any]], optional
-        :param config: The `AddConfig` instance to use as configuration options., defaults to None
-        :type config: Optional[AddConfig], optional
-        :raises ValueError: Invalid data type
-        :return: source_hash, a md5-hash of the source, in hexadecimal representation.
-        :rtype: str
-        """
-        logging.warning(
-            "The `add_local` method is deprecated and will be removed in future versions. Please use the `add` method for both local and remote files."  # noqa: E501
-        )
-        return self.add(
-            source=source,
-            data_type=data_type,
-            metadata=metadata,
-            config=config,
-            **kwargs,
-        )
-
    def _get_existing_doc_id(self, chunker: BaseChunker, src: Any):
        """
        Get id of existing document for a given source, based on the data type
@@ -433,10 +372,29 @@ class EmbedChain(JSONSerializable):
        # Count before, to calculate a delta in the end.
        chunks_before_addition = self.db.count()

-        self.db.add(documents=documents, metadatas=metadatas, ids=ids, **kwargs)
-        count_new_chunks = self.db.count() - chunks_before_addition
+        # Filter out empty documents and ensure they meet the API requirements
+        valid_documents = [doc for doc in documents if doc and isinstance(doc, str)]

+        documents = valid_documents
+
+        # Chunk documents into batches of 2048 and handle each batch
+        # helps wigth large loads of embeddings  that hit OpenAI limits
+        document_batches = [documents[i : i + 2048] for i in range(0, len(documents), 2048)]
+        metadata_batches = [metadatas[i : i + 2048] for i in range(0, len(metadatas), 2048)]
+        id_batches = [ids[i : i + 2048] for i in range(0, len(ids), 2048)]
+        for batch_docs, batch_meta, batch_ids in zip(document_batches, metadata_batches, id_batches):
+            try:
+                # Add only valid batches
+                if batch_docs:
+                    self.db.add(documents=batch_docs, metadatas=batch_meta, ids=batch_ids, **kwargs)
+            except Exception as e:
+                print(f"Failed to add batch due to a bad request: {e}")
+                # Handle the error, e.g., by logging, retrying, or skipping
+                pass
+
+        count_new_chunks = self.db.count() - chunks_before_addition
        print(f"Successfully saved {src} ({chunker.data_type}). New chunks count: {count_new_chunks}")
+
        return list(documents), metadatas, ids, count_new_chunks

    @staticmethod
@@ -534,7 +492,7 @@ class EmbedChain(JSONSerializable):
            contexts_data_for_llm_query = contexts

        if self.cache_config is not None:
-            logging.info("Cache enabled. Checking cache...")
+            logger.info("Cache enabled. Checking cache...")
            answer = adapt(
                llm_handler=self.llm.query,
                cache_data_convert=gptcache_data_convert,
@@ -606,7 +564,7 @@ class EmbedChain(JSONSerializable):
        self.llm.update_history(app_id=self.config.id, session_id=session_id)

        if self.cache_config is not None:
-            logging.info("Cache enabled. Checking cache...")
+            logger.debug("Cache enabled. Checking cache...")
            cache_id = f"{session_id}--{self.config.id}"
            answer = adapt(
                llm_handler=self.llm.chat,
@@ -619,6 +577,7 @@ class EmbedChain(JSONSerializable):
                dry_run=dry_run,
            )
        else:
+            logger.debug("Cache disabled. Running chat without cache.")
            answer = self.llm.chat(
                input_query=input_query, contexts=contexts_data_for_llm_query, config=config, dry_run=dry_run
            )
@@ -634,7 +593,7 @@ class EmbedChain(JSONSerializable):
        else:
            return answer

-    def search(self, query, num_documents=3, where=None, raw_filter=None):
+    def search(self, query, num_documents=3, where=None, raw_filter=None, namespace=None):
        """
        Search for similar documents related to the query in the vector database.

@@ -643,6 +602,7 @@ class EmbedChain(JSONSerializable):
            num_documents (int, optional): Number of similar documents to fetch. Defaults to 3.
            where (dict[str, any], optional): Filter criteria for the search.
            raw_filter (dict[str, any], optional): Advanced raw filter criteria for the search.
+            namespace (str, optional): The namespace to search in. Defaults to None.

        Raises:
            ValueError: If both `raw_filter` and `where` are used simultaneously.
@@ -664,6 +624,7 @@ class EmbedChain(JSONSerializable):
            "n_results": num_documents,
            "citations": True,
            "app_id": self.config.id,
+            "namespace": namespace,
            filter_type: filter_criteria,
        }

@@ -689,9 +650,15 @@ class EmbedChain(JSONSerializable):
        Resets the database. Deletes all embeddings irreversibly.
        `App` does not have to be reinitialized after using this method.
        """
+        try:
+            self.db_session.query(DataSource).filter_by(app_id=self.config.id).delete()
+            self.db_session.query(ChatHistory).filter_by(app_id=self.config.id).delete()
+            self.db_session.commit()
+        except Exception as e:
+            logger.error(f"Error deleting data sources: {e}")
+            self.db_session.rollback()
+            return None
        self.db.reset()
-        self.cursor.execute("DELETE FROM data_sources WHERE pipeline_id = ?", (self.config.id,))
-        self.connection.commit()
        self.delete_all_chat_history(app_id=self.config.id)
        # Send anonymous telemetry
        self.telemetry.capture(event_name="reset", properties=self._telemetry_props)
@@ -726,8 +693,15 @@ class EmbedChain(JSONSerializable):
        :param source_hash: The hash of the source.
        :type source_hash: str
        """
+        try:
+            self.db_session.query(DataSource).filter_by(hash=source_id, app_id=self.config.id).delete()
+            self.db_session.commit()
+        except Exception as e:
+            logger.error(f"Error deleting data sources: {e}")
+            self.db_session.rollback()
+            return None
        self.db.delete(where={"hash": source_id})
-        logging.info(f"Successfully deleted {source_id}")
+        logger.info(f"Successfully deleted {source_id}")
        # Send anonymous telemetry
        if self.config.collect_metrics:
            self.telemetry.capture(event_name="delete", properties=self._telemetry_props)
@@ -0,0 +1,19 @@
+from typing import Optional
+
+from langchain_community.embeddings import CohereEmbeddings
+
+from embedchain.config import BaseEmbedderConfig
+from embedchain.embedder.base import BaseEmbedder
+from embedchain.models import VectorDimensions
+
+
+class CohereEmbedder(BaseEmbedder):
+    def __init__(self, config: Optional[BaseEmbedderConfig] = None):
+        super().__init__(config=config)
+
+        embeddings = CohereEmbeddings(model=self.config.model)
+        embedding_fn = BaseEmbedder._langchain_default_concept(embeddings)
+        self.set_embedding_fn(embedding_fn=embedding_fn)
+
+        vector_dimension = self.config.vector_dimension or VectorDimensions.COHERE.value
+        self.set_vector_dimension(vector_dimension=vector_dimension)
@@ -1,6 +1,6 @@
 from typing import Optional

-from langchain.embeddings import HuggingFaceEmbeddings
+from langchain_community.embeddings import HuggingFaceEmbeddings

 from embedchain.config import BaseEmbedderConfig
 from embedchain.embedder.base import BaseEmbedder
@@ -0,0 +1,28 @@
+import logging
+import os
+from typing import Optional
+
+from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings
+
+from embedchain.config import BaseEmbedderConfig
+from embedchain.embedder.base import BaseEmbedder
+from embedchain.models import VectorDimensions
+
+logger = logging.getLogger(__name__)
+
+
+class NvidiaEmbedder(BaseEmbedder):
+    def __init__(self, config: Optional[BaseEmbedderConfig] = None):
+        if "NVIDIA_API_KEY" not in os.environ:
+            raise ValueError("NVIDIA_API_KEY environment variable must be set")
+
+        super().__init__(config=config)
+
+        model = self.config.model or "nvolveqa_40k"
+        logger.info(f"Using NVIDIA embedding model: {model}")
+        embedder = NVIDIAEmbeddings(model=model)
+        embedding_fn = BaseEmbedder._langchain_default_concept(embedder)
+        self.set_embedding_fn(embedding_fn=embedding_fn)
+
+        vector_dimension = self.config.vector_dimension or VectorDimensions.NVIDIA_AI.value
+        self.set_vector_dimension(vector_dimension=vector_dimension)
@@ -2,7 +2,7 @@ import os
 from typing import Optional

 from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
-from langchain.embeddings import AzureOpenAIEmbeddings
+from langchain_community.embeddings import AzureOpenAIEmbeddings

 from embedchain.config import BaseEmbedderConfig
 from embedchain.embedder.base import BaseEmbedder
@@ -17,6 +17,7 @@ class OpenAIEmbedder(BaseEmbedder):
            self.config.model = "text-embedding-ada-002"

        api_key = self.config.api_key or os.environ["OPENAI_API_KEY"]
+        api_base = self.config.api_base or os.environ.get("OPENAI_API_BASE")

        if self.config.deployment_name:
            embeddings = AzureOpenAIEmbeddings(deployment=self.config.deployment_name)
@@ -28,6 +29,7 @@ class OpenAIEmbedder(BaseEmbedder):
                )  # noqa:E501
            embedding_fn = OpenAIEmbeddingFunction(
                api_key=api_key,
+                api_base=api_base,
                organization_id=os.getenv("OPENAI_ORGANIZATION"),
                model_name=self.config.model,
            )
@@ -1,6 +1,6 @@
 from typing import Optional

-from langchain.embeddings import VertexAIEmbeddings
+from langchain_community.embeddings import VertexAIEmbeddings

 from embedchain.config import BaseEmbedderConfig
 from embedchain.embedder.base import BaseEmbedder
@@ -12,6 +12,8 @@ from embedchain.config.evaluation.base import AnswerRelevanceConfig
 from embedchain.evaluation.base import BaseMetric
 from embedchain.utils.evaluation import EvalData, EvalMetric

+logger = logging.getLogger(__name__)
+

 class AnswerRelevance(BaseMetric):
    """
@@ -88,6 +90,6 @@ class AnswerRelevance(BaseMetric):
                try:
                    results.append(future.result())
                except Exception as e:
-                    logging.error(f"Error evaluating answer relevancy for {data}: {e}")
+                    logger.error(f"Error evaluating answer relevancy for {data}: {e}")

        return np.mean(results) if results else 0.0
@@ -12,6 +12,8 @@ from embedchain.config.evaluation.base import GroundednessConfig
 from embedchain.evaluation.base import BaseMetric
 from embedchain.utils.evaluation import EvalData, EvalMetric

+logger = logging.getLogger(__name__)
+

 class Groundedness(BaseMetric):
    """
@@ -97,6 +99,6 @@ class Groundedness(BaseMetric):
                    score = future.result()
                    results.append(score)
                except Exception as e:
-                    logging.error(f"Error while evaluating groundedness for data point {data}: {e}")
+                    logger.error(f"Error while evaluating groundedness for data point {data}: {e}")

        return np.mean(results) if results else 0.0
@@ -23,6 +23,9 @@ class LlmFactory:
        "google": "embedchain.llm.google.GoogleLlm",
        "aws_bedrock": "embedchain.llm.aws_bedrock.AWSBedrockLlm",
        "mistralai": "embedchain.llm.mistralai.MistralAILlm",
+        "groq": "embedchain.llm.groq.GroqLlm",
+        "nvidia": "embedchain.llm.nvidia.NvidiaLlm",
+        "vllm": "embedchain.llm.vllm.VLLM",
    }
    provider_to_config_class = {
        "embedchain": "embedchain.config.llm.base.BaseLlmConfig",
@@ -53,13 +56,15 @@ class EmbedderFactory:
        "vertexai": "embedchain.embedder.vertexai.VertexAIEmbedder",
        "google": "embedchain.embedder.google.GoogleAIEmbedder",
        "mistralai": "embedchain.embedder.mistralai.MistralAIEmbedder",
+        "nvidia": "embedchain.embedder.nvidia.NvidiaEmbedder",
+        "cohere": "embedchain.embedder.cohere.CohereEmbedder",
    }
    provider_to_config_class = {
        "azure_openai": "embedchain.config.embedder.base.BaseEmbedderConfig",
-        "openai": "embedchain.config.embedder.base.BaseEmbedderConfig",
-        "gpt4all": "embedchain.config.embedder.base.BaseEmbedderConfig",
        "google": "embedchain.config.embedder.google.GoogleAIEmbedderConfig",
+        "gpt4all": "embedchain.config.embedder.base.BaseEmbedderConfig",
        "huggingface": "embedchain.config.embedder.base.BaseEmbedderConfig",
+        "openai": "embedchain.config.embedder.base.BaseEmbedderConfig",
    }

    @classmethod
@@ -56,7 +56,7 @@ def generate(rq: queue.Queue):
    ```
    def askQuestion(callback_fn: StreamingStdOutCallbackHandlerYield):
        llm = OpenAI(streaming=True, callbacks=[callback_fn])
-        return llm(prompt="Write a poem about a tree.")
+        return llm.invoke(prompt="Write a poem about a tree.")

    @app.route("/", methods=["GET"])
    def generate_output():
@@ -8,6 +8,8 @@ T = TypeVar("T", bound="JSONSerializable")
 # NOTE: Through inheritance, all of our classes should be children of JSONSerializable. (highest level)
 # NOTE: The @register_deserializable decorator should be added to all user facing child classes. (lowest level)

+logger = logging.getLogger(__name__)
+

 def register_deserializable(cls: Type[T]) -> Type[T]:
    """
@@ -57,7 +59,7 @@ class JSONSerializable:
        try:
            return json.dumps(self, default=self._auto_encoder, ensure_ascii=False)
        except Exception as e:
-            logging.error(f"Serialization error: {e}")
+            logger.error(f"Serialization error: {e}")
            return "{}"

    @classmethod
@@ -79,7 +81,7 @@ class JSONSerializable:
        try:
            return json.loads(json_str, object_hook=cls._auto_decoder)
        except Exception as e:
-            logging.error(f"Deserialization error: {e}")
+            logger.error(f"Deserialization error: {e}")
            # Return a default instance in case of failure
            return cls()

@@ -6,6 +6,8 @@ from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class AnthropicLlm(BaseLlm):
@@ -19,14 +21,14 @@ class AnthropicLlm(BaseLlm):

    @staticmethod
    def _get_answer(prompt: str, config: BaseLlmConfig) -> str:
-        from langchain.chat_models import ChatAnthropic
+        from langchain_community.chat_models import ChatAnthropic

        chat = ChatAnthropic(
            anthropic_api_key=os.environ["ANTHROPIC_API_KEY"], temperature=config.temperature, model=config.model
        )

        if config.max_tokens and config.max_tokens != 1000:
-            logging.warning("Config option `max_tokens` is not supported by this model.")
+            logger.warning("Config option `max_tokens` is not supported by this model.")

        messages = BaseLlm._get_messages(prompt, system_prompt=config.system_prompt)

@@ -1,7 +1,7 @@
 import os
 from typing import Optional

-from langchain.llms import Bedrock
+from langchain_community.llms import Bedrock

 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -46,4 +46,4 @@ class AWSBedrockLlm(BaseLlm):
        else:
            llm = Bedrock(**kwargs)

-        return llm(prompt)
+        return llm.invoke(prompt)
@@ -5,6 +5,8 @@ from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class AzureOpenAILlm(BaseLlm):
@@ -16,7 +18,7 @@ class AzureOpenAILlm(BaseLlm):

    @staticmethod
    def _get_answer(prompt: str, config: BaseLlmConfig) -> str:
-        from langchain.chat_models import AzureChatOpenAI
+        from langchain_community.chat_models import AzureChatOpenAI

        if not config.deployment_name:
            raise ValueError("Deployment name must be provided for Azure OpenAI")
@@ -31,7 +33,7 @@ class AzureOpenAILlm(BaseLlm):
        )

        if config.top_p and config.top_p != 1:
-            logging.warning("Config option `top_p` is not supported by this model.")
+            logger.warning("Config option `top_p` is not supported by this model.")

        messages = BaseLlm._get_messages(prompt, system_prompt=config.system_prompt)

@@ -12,6 +12,8 @@ from embedchain.helpers.json_serializable import JSONSerializable
 from embedchain.memory.base import ChatHistory
 from embedchain.memory.message import ChatMessage

+logger = logging.getLogger(__name__)
+

 class BaseLlm(JSONSerializable):
    def __init__(self, config: Optional[BaseLlmConfig] = None):
@@ -65,6 +67,14 @@ class BaseLlm(JSONSerializable):
        self.memory.add(app_id=app_id, chat_message=chat_message, session_id=session_id)
        self.update_history(app_id=app_id, session_id=session_id)

+    def _format_history(self) -> str:
+        """Format history to be used in prompt
+
+        :return: Formatted history
+        :rtype: str
+        """
+        return "\n".join(self.history)
+
    def generate_prompt(self, input_query: str, contexts: list[str], **kwargs: dict[str, Any]) -> str:
        """
        Generates a prompt based on the given query and context, ready to be
@@ -84,10 +94,8 @@ class BaseLlm(JSONSerializable):

        prompt_contains_history = self.config._validate_prompt_history(self.config.prompt)
        if prompt_contains_history:
-            # Prompt contains history
-            # If there is no history yet, we insert `- no history -`
            prompt = self.config.prompt.substitute(
-                context=context_string, query=input_query, history=self.history or "- no history -"
+                context=context_string, query=input_query, history=self._format_history() or "No history"
            )
        elif self.history and not prompt_contains_history:
            # History is present, but not included in the prompt.
@@ -98,11 +106,11 @@ class BaseLlm(JSONSerializable):
            ):
                # swap in the template with history
                prompt = DEFAULT_PROMPT_WITH_HISTORY_TEMPLATE.substitute(
-                    context=context_string, query=input_query, history=self.history
+                    context=context_string, query=input_query, history=self._format_history()
                )
            else:
                # If we can't swap in the default, we still proceed but tell users that the history is ignored.
-                logging.warning(
+                logger.warning(
                    "Your bot contains a history, but prompt does not include `$history` key. History is ignored."
                )
                prompt = self.config.prompt.substitute(context=context_string, query=input_query)
@@ -153,7 +161,7 @@ class BaseLlm(JSONSerializable):
                'Searching requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'
            ) from None
        search = DuckDuckGoSearchRun()
-        logging.info(f"Access search to get answers for {input_query}")
+        logger.info(f"Access search to get answers for {input_query}")
        return search.run(input_query)

    @staticmethod
@@ -169,7 +177,7 @@ class BaseLlm(JSONSerializable):
        for chunk in answer:
            streamed_answer = streamed_answer + chunk
            yield chunk
-        logging.info(f"Answer: {streamed_answer}")
+        logger.info(f"Answer: {streamed_answer}")

    def query(self, input_query: str, contexts: list[str], config: BaseLlmConfig = None, dry_run=False):
        """
@@ -208,13 +216,13 @@ class BaseLlm(JSONSerializable):
            if self.online:
                k["web_search_result"] = self.access_search_and_get_results(input_query)
            prompt = self.generate_prompt(input_query, contexts, **k)
-            logging.info(f"Prompt: {prompt}")
+            logger.info(f"Prompt: {prompt}")
            if dry_run:
                return prompt

            answer = self.get_answer_from_llm(prompt)
            if isinstance(answer, str):
-                logging.info(f"Answer: {answer}")
+                logger.info(f"Answer: {answer}")
                return answer
            else:
                return self._stream_response(answer)
@@ -264,14 +272,14 @@ class BaseLlm(JSONSerializable):
                k["web_search_result"] = self.access_search_and_get_results(input_query)

            prompt = self.generate_prompt(input_query, contexts, **k)
-            logging.info(f"Prompt: {prompt}")
+            logger.info(f"Prompt: {prompt}")

            if dry_run:
                return prompt

            answer = self.get_answer_from_llm(prompt)
            if isinstance(answer, str):
-                logging.info(f"Answer: {answer}")
+                logger.info(f"Answer: {answer}")
                return answer
            else:
                # this is a streamed response and needs to be handled differently.
@@ -2,7 +2,7 @@ import importlib
 import os
 from typing import Optional

-from langchain.llms.cohere import Cohere
+from langchain_community.llms.cohere import Cohere

 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -40,4 +40,4 @@ class CohereLlm(BaseLlm):
            p=config.top_p,
        )

-        return llm(prompt)
+        return llm.invoke(prompt)
@@ -10,6 +10,8 @@ from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class GoogleLlm(BaseLlm):
@@ -36,7 +38,7 @@ class GoogleLlm(BaseLlm):

    def _get_answer(self, prompt: str) -> Union[str, Generator[Any, Any, None]]:
        model_name = self.config.model or "gemini-pro"
-        logging.info(f"Using Google LLM model: {model_name}")
+        logger.info(f"Using Google LLM model: {model_name}")
        model = genai.GenerativeModel(model_name=model_name)

        generation_config_params = {
@@ -26,7 +26,8 @@ class GPT4ALLLlm(BaseLlm):
    @staticmethod
    def _get_instance(model):
        try:
-            from langchain.llms.gpt4all import GPT4All as LangchainGPT4All
+            from langchain_community.llms.gpt4all import \
+                GPT4All as LangchainGPT4All
        except ModuleNotFoundError:
            raise ModuleNotFoundError(
                "The GPT4All python package is not installed. Please install it with `pip install --upgrade embedchain[opensource]`"  # noqa E501
@@ -0,0 +1,43 @@
+import os
+from typing import Optional
+
+from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
+from langchain.schema import HumanMessage, SystemMessage
+
+try:
+    from langchain_groq import ChatGroq
+except ImportError:
+    raise ImportError("Groq requires extra dependencies. Install with `pip install langchain-groq`") from None
+
+
+from embedchain.config import BaseLlmConfig
+from embedchain.helpers.json_serializable import register_deserializable
+from embedchain.llm.base import BaseLlm
+
+
+@register_deserializable
+class GroqLlm(BaseLlm):
+    def __init__(self, config: Optional[BaseLlmConfig] = None):
+        super().__init__(config=config)
+
+    def get_llm_model_answer(self, prompt) -> str:
+        response = self._get_answer(prompt, self.config)
+        return response
+
+    def _get_answer(self, prompt: str, config: BaseLlmConfig) -> str:
+        messages = []
+        if config.system_prompt:
+            messages.append(SystemMessage(content=config.system_prompt))
+        messages.append(HumanMessage(content=prompt))
+        api_key = config.api_key or os.environ["GROQ_API_KEY"]
+        kwargs = {
+            "model_name": config.model or "mixtral-8x7b-32768",
+            "temperature": config.temperature,
+            "groq_api_key": api_key,
+        }
+        if config.stream:
+            callbacks = config.callbacks if config.callbacks else [StreamingStdOutCallbackHandler()]
+            chat = ChatGroq(**kwargs, streaming=config.stream, callbacks=callbacks, api_key=api_key)
+        else:
+            chat = ChatGroq(**kwargs)
+        return chat.invoke(messages).content
@@ -3,13 +3,16 @@ import logging
 import os
 from typing import Optional

-from langchain.llms.huggingface_endpoint import HuggingFaceEndpoint
-from langchain.llms.huggingface_hub import HuggingFaceHub
+from langchain_community.llms.huggingface_endpoint import HuggingFaceEndpoint
+from langchain_community.llms.huggingface_hub import HuggingFaceHub
+from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline

 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class HuggingFaceLlm(BaseLlm):
@@ -34,12 +37,15 @@ class HuggingFaceLlm(BaseLlm):

    @staticmethod
    def _get_answer(prompt: str, config: BaseLlmConfig) -> str:
-        if config.model:
+        # If the user wants to run the model locally, they can do so by setting the `local` flag to True
+        if config.model and config.local:
+            return HuggingFaceLlm._from_pipeline(prompt=prompt, config=config)
+        elif config.model:
            return HuggingFaceLlm._from_model(prompt=prompt, config=config)
        elif config.endpoint:
            return HuggingFaceLlm._from_endpoint(prompt=prompt, config=config)
        else:
-            raise ValueError("Either `model` or `endpoint` must be set")
+            raise ValueError("Either `model` or `endpoint` must be set in config")

    @staticmethod
    def _from_model(prompt: str, config: BaseLlmConfig) -> str:
@@ -53,15 +59,14 @@ class HuggingFaceLlm(BaseLlm):
        else:
            raise ValueError("`top_p` must be > 0.0 and < 1.0")

-        model = config.model or "google/flan-t5-xxl"
-        logging.info(f"Using HuggingFaceHub with model {model}")
+        model = config.model
+        logger.info(f"Using HuggingFaceHub with model {model}")
        llm = HuggingFaceHub(
            huggingfacehub_api_token=os.environ["HUGGINGFACE_ACCESS_TOKEN"],
            repo_id=model,
            model_kwargs=model_kwargs,
        )
-
-        return llm(prompt)
+        return llm.invoke(prompt)

    @staticmethod
    def _from_endpoint(prompt: str, config: BaseLlmConfig) -> str:
@@ -71,4 +76,23 @@ class HuggingFaceLlm(BaseLlm):
            task="text-generation",
            model_kwargs=config.model_kwargs,
        )
-        return llm(prompt)
+        return llm.invoke(prompt)
+
+    @staticmethod
+    def _from_pipeline(prompt: str, config: BaseLlmConfig) -> str:
+        model_kwargs = {
+            "temperature": config.temperature or 0.1,
+            "max_new_tokens": config.max_tokens,
+        }
+
+        if 0.0 < config.top_p < 1.0:
+            model_kwargs["top_p"] = config.top_p
+        else:
+            raise ValueError("`top_p` must be > 0.0 and < 1.0")
+
+        llm = HuggingFacePipeline.from_model_id(
+            model_id=config.model,
+            task="text-generation",
+            pipeline_kwargs=model_kwargs,
+        )
+        return llm.invoke(prompt)
@@ -1,8 +1,8 @@
 import os
 from typing import Optional

-from langchain.chat_models import JinaChat
 from langchain.schema import HumanMessage, SystemMessage
+from langchain_community.chat_models import JinaChat

 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -2,7 +2,7 @@ import importlib
 import os
 from typing import Optional

-from langchain.llms.replicate import Replicate
+from langchain_community.llms.replicate import Replicate

 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -48,4 +48,4 @@ class Llama2Llm(BaseLlm):
                "top_p": self.config.top_p,
            },
        )
-        return llm(prompt)
+        return llm.invoke(prompt)
@@ -0,0 +1,47 @@
+import os
+from collections.abc import Iterable
+from typing import Optional, Union
+
+from langchain.callbacks.manager import CallbackManager
+from langchain.callbacks.stdout import StdOutCallbackHandler
+from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
+
+try:
+    from langchain_nvidia_ai_endpoints import ChatNVIDIA
+except ImportError:
+    raise ImportError(
+        "NVIDIA AI endpoints requires extra dependencies. Install with `pip install langchain-nvidia-ai-endpoints`"
+    ) from None
+
+from embedchain.config import BaseLlmConfig
+from embedchain.helpers.json_serializable import register_deserializable
+from embedchain.llm.base import BaseLlm
+
+
+@register_deserializable
+class NvidiaLlm(BaseLlm):
+    def __init__(self, config: Optional[BaseLlmConfig] = None):
+        if "NVIDIA_API_KEY" not in os.environ:
+            raise ValueError("NVIDIA_API_KEY environment variable must be set")
+
+        super().__init__(config=config)
+
+    def get_llm_model_answer(self, prompt):
+        return self._get_answer(prompt=prompt, config=self.config)
+
+    @staticmethod
+    def _get_answer(prompt: str, config: BaseLlmConfig) -> Union[str, Iterable]:
+        callback_manager = [StreamingStdOutCallbackHandler()] if config.stream else [StdOutCallbackHandler()]
+        model_kwargs = config.model_kwargs or {}
+        labels = model_kwargs.get("labels", None)
+        params = {"model": config.model}
+        if config.system_prompt:
+            params["system_prompt"] = config.system_prompt
+        if config.temperature:
+            params["temperature"] = config.temperature
+        if config.top_p:
+            params["top_p"] = config.top_p
+        if labels:
+            params["labels"] = labels
+        llm = ChatNVIDIA(**params, callback_manager=CallbackManager(callback_manager))
+        return llm.invoke(prompt).content if labels is None else llm.invoke(prompt, labels=labels).content
@@ -4,7 +4,7 @@ from typing import Optional, Union
 from langchain.callbacks.manager import CallbackManager
 from langchain.callbacks.stdout import StdOutCallbackHandler
 from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
-from langchain.llms.ollama import Ollama
+from langchain_community.llms.ollama import Ollama

 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -31,6 +31,7 @@ class OllamaLlm(BaseLlm):
            temperature=config.temperature,
            top_p=config.top_p,
            callback_manager=CallbackManager(callback_manager),
+            base_url=config.base_url,
        )

-        return llm(prompt)
+        return llm.invoke(prompt)
@@ -1,9 +1,12 @@
 import json
 import os
-from typing import Any, Optional
+from typing import Any, Callable, Dict, Optional, Type, Union

-from langchain.chat_models import ChatOpenAI
-from langchain.schema import AIMessage, HumanMessage, SystemMessage
+from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
+from langchain.schema import BaseMessage, HumanMessage, SystemMessage
+from langchain_core.tools import BaseTool
+from langchain_openai import ChatOpenAI
+from pydantic import BaseModel

 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -12,8 +15,12 @@ from embedchain.llm.base import BaseLlm

@register_deserializable
 class OpenAILlm(BaseLlm):
-    def __init__(self, config: Optional[BaseLlmConfig] = None, functions: Optional[dict[str, Any]] = None):
-        self.functions = functions
+    def __init__(
+        self,
+        config: Optional[BaseLlmConfig] = None,
+        tools: Optional[Union[Dict[str, Any], Type[BaseModel], Callable[..., Any], BaseTool]] = None,
+    ):
+        self.tools = tools
        super().__init__(config=config)

    def get_llm_model_answer(self, prompt) -> str:
@@ -32,29 +39,38 @@ class OpenAILlm(BaseLlm):
            "model_kwargs": {},
        }
        api_key = config.api_key or os.environ["OPENAI_API_KEY"]
+        base_url = config.base_url or os.environ.get("OPENAI_API_BASE", None)
        if config.top_p:
            kwargs["model_kwargs"]["top_p"] = config.top_p
        if config.stream:
-            from langchain.callbacks.streaming_stdout import \
-                StreamingStdOutCallbackHandler
-
            callbacks = config.callbacks if config.callbacks else [StreamingStdOutCallbackHandler()]
-            chat = ChatOpenAI(**kwargs, streaming=config.stream, callbacks=callbacks, api_key=api_key)
-        else:
-            chat = ChatOpenAI(**kwargs, api_key=api_key)
-
-        if self.functions is not None:
-            from langchain.chains.openai_functions import \
-                create_openai_fn_runnable
-            from langchain.prompts import ChatPromptTemplate
-
-            structured_prompt = ChatPromptTemplate.from_messages(messages)
-            runnable = create_openai_fn_runnable(functions=self.functions, prompt=structured_prompt, llm=chat)
-            fn_res = runnable.invoke(
-                {
-                    "input": prompt,
-                }
+            chat = ChatOpenAI(
+                **kwargs,
+                streaming=config.stream,
+                callbacks=callbacks,
+                api_key=api_key,
+                base_url=base_url,
            )
-            messages.append(AIMessage(content=json.dumps(fn_res)))
+        else:
+            chat = ChatOpenAI(**kwargs, api_key=api_key, base_url=base_url)
+        if self.tools:
+            return self._query_function_call(chat, self.tools, messages)

-        return chat(messages).content
+        return chat.invoke(messages).content
+
+    def _query_function_call(
+        self,
+        chat: ChatOpenAI,
+        tools: Optional[Union[Dict[str, Any], Type[BaseModel], Callable[..., Any], BaseTool]],
+        messages: list[BaseMessage],
+    ) -> str:
+        from langchain.output_parsers.openai_tools import JsonOutputToolsParser
+        from langchain_core.utils.function_calling import \
+            convert_to_openai_tool
+
+        openai_tools = [convert_to_openai_tool(tools)]
+        chat = chat.bind(tools=openai_tools).pipe(JsonOutputToolsParser())
+        try:
+            return json.dumps(chat.invoke(messages)[0])
+        except IndexError:
+            return "Input could not be mapped to the function!"
@@ -2,7 +2,7 @@ import importlib
 import os
 from typing import Optional

-from langchain.llms import Together
+from langchain_community.llms import Together

 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
@@ -40,4 +40,4 @@ class TogetherLlm(BaseLlm):
            top_p=config.top_p,
        )

-        return llm(prompt)
+        return llm.invoke(prompt)
@@ -2,10 +2,15 @@ import importlib
 import logging
 from typing import Optional

+from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
+from langchain_google_vertexai import ChatVertexAI
+
 from embedchain.config import BaseLlmConfig
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class VertexAILlm(BaseLlm):
@@ -24,13 +29,17 @@ class VertexAILlm(BaseLlm):

    @staticmethod
    def _get_answer(prompt: str, config: BaseLlmConfig) -> str:
-        from langchain.chat_models import ChatVertexAI
-
-        chat = ChatVertexAI(temperature=config.temperature, model=config.model)
-
        if config.top_p and config.top_p != 1:
-            logging.warning("Config option `top_p` is not supported by this model.")
+            logger.warning("Config option `top_p` is not supported by this model.")

        messages = BaseLlm._get_messages(prompt, system_prompt=config.system_prompt)

-        return chat(messages).content
+        if config.stream:
+            callbacks = config.callbacks if config.callbacks else [StreamingStdOutCallbackHandler()]
+            llm = ChatVertexAI(
+                temperature=config.temperature, model=config.model, callbacks=callbacks, streaming=config.stream
+            )
+        else:
+            llm = ChatVertexAI(temperature=config.temperature, model=config.model)
+
+        return llm.invoke(messages).content
@@ -37,4 +37,4 @@ class VLLM(BaseLlm):
            llm_args.update(config.model_kwargs)

        llm = BaseVLLM(**llm_args)
-        return llm(prompt)
+        return llm.invoke(prompt)
@@ -9,6 +9,8 @@ from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils.misc import is_readable

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class BeehiivLoader(BaseLoader):
@@ -90,9 +92,9 @@ class BeehiivLoader(BaseLoader):
                if is_readable(data):
                    return data
                else:
-                    logging.warning(f"Page is not readable (too many invalid characters): {link}")
+                    logger.warning(f"Page is not readable (too many invalid characters): {link}")
            except ParserRejectedMarkup as e:
-                logging.error(f"Failed to parse {link}: {e}")
+                logger.error(f"Failed to parse {link}: {e}")
            return None

        for link in links:
@@ -10,6 +10,8 @@ from embedchain.loaders.base_loader import BaseLoader
 from embedchain.loaders.text_file import TextFileLoader
 from embedchain.utils.misc import detect_datatype

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class DirectoryLoader(BaseLoader):
@@ -27,12 +29,12 @@ class DirectoryLoader(BaseLoader):
        if not directory_path.is_dir():
            raise ValueError(f"Invalid path: {path}")

-        logging.info(f"Loading data from directory: {path}")
+        logger.info(f"Loading data from directory: {path}")
        data_list = self._process_directory(directory_path)
        doc_id = hashlib.sha256((str(data_list) + str(directory_path)).encode()).hexdigest()

        for error in self.errors:
-            logging.warning(error)
+            logger.warning(error)

        return {"doc_id": doc_id, "data": data_list}

@@ -46,7 +48,7 @@ class DirectoryLoader(BaseLoader):
                loader = self._predict_loader(file_path)
                data_list.extend(loader.load_data(str(file_path))["data"])
            elif file_path.is_dir():
-                logging.info(f"Loading data from directory: {file_path}")
+                logger.info(f"Loading data from directory: {file_path}")
        return data_list

    def _predict_loader(self, file_path: Path) -> BaseLoader:
@@ -5,6 +5,8 @@ import os
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class DiscordLoader(BaseLoader):
@@ -102,7 +104,7 @@ class DiscordLoader(BaseLoader):

        class DiscordClient(discord.Client):
            async def on_ready(self) -> None:
-                logging.info("Logged on as {0}!".format(self.user))
+                logger.info("Logged on as {0}!".format(self.user))
                try:
                    channel = self.get_channel(int(channel_id))
                    if not isinstance(channel, discord.TextChannel):
@@ -121,7 +123,7 @@ class DiscordLoader(BaseLoader):
                                messages.append(DiscordLoader._format_message(thread_message))

                except Exception as e:
-                    logging.error(e)
+                    logger.error(e)
                    await self.close()
                finally:
                    await self.close()
@@ -131,7 +133,7 @@ class DiscordLoader(BaseLoader):
        client = DiscordClient(intents=intents)
        client.run(self.token)

-        meta_data = {
+        metadata = {
            "url": channel_id,
        }

@@ -144,7 +146,7 @@ class DiscordLoader(BaseLoader):
            "data": [
                {
                    "content": messages,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                }
            ],
        }
@@ -8,25 +8,27 @@ import requests
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils.misc import clean_string

+logger = logging.getLogger(__name__)
+

 class DiscourseLoader(BaseLoader):
    def __init__(self, config: Optional[dict[str, Any]] = None):
        super().__init__()
        if not config:
            raise ValueError(
-                "DiscourseLoader requires a config. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+                "DiscourseLoader requires a config. Check the documentation for the correct format - `https://docs.embedchain.ai/components/data-sources/discourse`"  # noqa: E501
            )

        self.domain = config.get("domain")
        if not self.domain:
            raise ValueError(
-                "DiscourseLoader requires a domain. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+                "DiscourseLoader requires a domain. Check the documentation for the correct format - `https://docs.embedchain.ai/components/data-sources/discourse`"  # noqa: E501
            )

    def _check_query(self, query):
        if not query or not isinstance(query, str):
            raise ValueError(
-                "DiscourseLoader requires a query. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+                "DiscourseLoader requires a query. Check the documentation for the correct format - `https://docs.embedchain.ai/components/data-sources/discourse`"  # noqa: E501
            )

    def _load_post(self, post_id):
@@ -35,11 +37,11 @@ class DiscourseLoader(BaseLoader):
        try:
            response.raise_for_status()
        except Exception as e:
-            logging.error(f"Failed to load post {post_id}: {e}")
+            logger.error(f"Failed to load post {post_id}: {e}")
            return
        response_data = response.json()
        post_contents = clean_string(response_data.get("raw"))
-        meta_data = {
+        metadata = {
            "url": post_url,
            "created_at": response_data.get("created_at", ""),
            "username": response_data.get("username", ""),
@@ -48,7 +50,7 @@ class DiscourseLoader(BaseLoader):
        }
        data = {
            "content": post_contents,
-            "meta_data": meta_data,
+            "meta_data": metadata,
        }
        return data

@@ -56,7 +58,7 @@ class DiscourseLoader(BaseLoader):
        self._check_query(query)
        data = []
        data_contents = []
-        logging.info(f"Searching data on discourse url: {self.domain}, for query: {query}")
+        logger.info(f"Searching data on discourse url: {self.domain}, for query: {query}")
        search_url = f"{self.domain}search.json?q={query}"
        response = requests.get(search_url)
        try:
@@ -15,6 +15,8 @@ except ImportError:
 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class DocsSiteLoader(BaseLoader):
@@ -28,7 +30,7 @@ class DocsSiteLoader(BaseLoader):

        response = requests.get(url)
        if response.status_code != 200:
-            logging.info(f"Failed to fetch the website: {response.status_code}")
+            logger.info(f"Failed to fetch the website: {response.status_code}")
            return

        soup = BeautifulSoup(response.text, "html.parser")
@@ -53,7 +55,7 @@ class DocsSiteLoader(BaseLoader):
    def _load_data_from_url(url: str) -> list:
        response = requests.get(url)
        if response.status_code != 200:
-            logging.info(f"Failed to fetch the website: {response.status_code}")
+            logger.info(f"Failed to fetch the website: {response.status_code}")
            return []

        soup = BeautifulSoup(response.content, "html.parser")
@@ -1,7 +1,7 @@
 import hashlib

 try:
-    from langchain.document_loaders import Docx2txtLoader
+    from langchain_community.document_loaders import Docx2txtLoader
 except ImportError:
    raise ImportError(
        'Docx file requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'
@@ -18,9 +18,9 @@ class DocxFileLoader(BaseLoader):
        output = []
        data = loader.load()
        content = data[0].page_content
-        meta_data = data[0].metadata
-        meta_data["url"] = "local"
-        output.append({"content": content, "meta_data": meta_data})
+        metadata = data[0].metadata
+        metadata["url"] = "local"
+        output.append({"content": content, "meta_data": metadata})
        doc_id = hashlib.sha256((content + url).encode()).hexdigest()
        return {
            "doc_id": doc_id,
@@ -1,7 +1,6 @@
 import concurrent.futures
 import hashlib
 import logging
-import os
 import re
 import shlex
 from typing import Any, Optional
@@ -14,7 +13,7 @@ from embedchain.utils.misc import clean_string
 GITHUB_URL = "https://github.com"
 GITHUB_API_URL = "https://api.github.com"

-VALID_SEARCH_TYPES = set(["code", "repo", "pr", "issue", "discussion"])
+VALID_SEARCH_TYPES = set(["code", "repo", "pr", "issue", "discussion", "branch", "file"])


 class GithubLoader(BaseLoader):
@@ -66,85 +65,56 @@ class GithubLoader(BaseLoader):
            )
        return data

-    @staticmethod
-    def _get_github_repo_data(repo_url: str):
-        local_hash = hashlib.sha256(repo_url.encode()).hexdigest()
-        local_path = f"/tmp/{local_hash}"
+    def _get_github_repo_data(self, repo_name: str, branch_name: str = None, file_path: str = None) -> list[dict]:
+        """Get file contents from Repo"""
        data = []

-        def _get_repo_tree(repo_url: str, local_path: str):
-            try:
-                from git import Repo
-            except ImportError as e:
-                raise ValueError(
-                    "GithubLoader requires extra dependencies. Install with `pip install --upgrade 'embedchain[github]'`"  # noqa: E501
-                ) from e
+        repo = self.client.get_repo(repo_name)
+        repo_contents = repo.get_contents("")

-            if os.path.exists(local_path):
-                logging.info("Repository already exists. Fetching updates...")
-                repo = Repo(local_path)
-                logging.info("Fetch completed.")
-            else:
-                logging.info("Cloning repository...")
-                repo = Repo.clone_from(repo_url, local_path)
-                logging.info("Clone completed.")
-            return repo.head.commit.tree
+        if branch_name:
+            repo_contents = repo.get_contents("", ref=branch_name)
+        if file_path:
+            repo_contents = [repo.get_contents(file_path)]

-        def _get_repo_tree_contents(repo_path, tree, progress_bar):
-            for subtree in tree:
-                if subtree.type == "tree":
-                    _get_repo_tree_contents(repo_path, subtree, progress_bar)
-                else:
-                    assert subtree.type == "blob"
+        with tqdm(desc="Loading files:", unit="item") as progress_bar:
+            while repo_contents:
+                file_content = repo_contents.pop(0)
+                if file_content.type == "dir":
                    try:
-                        contents = subtree.data_stream.read().decode("utf-8")
+                        repo_contents.extend(repo.get_contents(file_content.path))
                    except Exception:
-                        logging.warning(f"Failed to read file: {subtree.path}")
-                        progress_bar.update(1) if progress_bar else None
+                        logging.warning(f"Failed to read directory: {file_content.path}")
+                        progress_bar.update(1)
+                        continue
+                else:
+                    try:
+                        file_text = file_content.decoded_content.decode()
+                    except Exception:
+                        logging.warning(f"Failed to read file: {file_content.path}")
+                        progress_bar.update(1)
                        continue

-                    url = f"{repo_url.rstrip('.git')}/blob/main/{subtree.path}"
+                    file_path = file_content.path
                    data.append(
                        {
-                            "content": clean_string(contents),
+                            "content": clean_string(file_text),
                            "meta_data": {
-                                "url": url,
+                                "path": file_path,
                            },
                        }
                    )
-                if progress_bar is not None:
-                    progress_bar.update(1)

-        repo_tree = _get_repo_tree(repo_url, local_path)
-        tree_list = list(repo_tree.traverse())
-        with tqdm(total=len(tree_list), desc="Loading files:", unit="item") as progress_bar:
-            _get_repo_tree_contents(local_path, repo_tree, progress_bar)
+                progress_bar.update(1)

        return data

    def _github_search_repo(self, query: str) -> list[dict]:
        """Search GitHub repo."""
-        data = []
-        logging.info(f"Searching github repos with query: {query}")
-        results = self.client.search_repositories(query)
-        # Add repo urls and descriptions
-        urls = list(map(lambda x: x.html_url, results))
-        descriptions = list(map(lambda x: x.description, results))
-        data.append(
-            {
-                "content": clean_string(desc),
-                "meta_data": {
-                    "url": url,
-                },
-            }
-            for url, desc in zip(urls, descriptions)
-        )

-        # Add repo contents
-        for result in results:
-            clone_url = result.clone_url
-            logging.info(f"Cloning repository: {clone_url}")
-            data = self._get_github_repo_data(clone_url)
+        logging.info(f"Searching github repos with query: {query}")
+        updated_query = query.split(":")[-1]
+        data = self._get_github_repo_data(updated_query)
        return data

    def _github_search_issues_and_pr(self, query: str, type: str) -> list[dict]:
@@ -222,6 +192,43 @@ class GithubLoader(BaseLoader):
                    )
        return data

+    def _get_github_repo_branch(self, query: str, type: str) -> list[dict]:
+        """Get file contents for specific branch"""
+
+        logging.info(f"Searching github repo for query: {query} is:{type}")
+        pattern = r"repo:(\S+) name:(\S+)"
+        match = re.search(pattern, query)
+
+        if match:
+            repo_name = match.group(1)
+            branch_name = match.group(2)
+        else:
+            raise ValueError(
+                f"Repository name and Branch name not found, instead found this \
+                    Repo: {repo_name}, Branch: {branch_name}"
+            )
+
+        data = self._get_github_repo_data(repo_name=repo_name, branch_name=branch_name)
+        return data
+
+    def _get_github_repo_file(self, query: str, type: str) -> list[dict]:
+        """Get specific file content"""
+
+        logging.info(f"Searching github repo for query: {query} is:{type}")
+        pattern = r"repo:(\S+) path:(\S+)"
+        match = re.search(pattern, query)
+
+        if match:
+            repo_name = match.group(1)
+            file_path = match.group(2)
+        else:
+            raise ValueError(
+                f"Repository name and File name not found, instead found this Repo: {repo_name}, File: {file_path}"
+            )
+
+        data = self._get_github_repo_data(repo_name=repo_name, file_path=file_path)
+        return data
+
    def _search_github_data(self, search_type: str, query: str):
        """Search github data."""
        if search_type == "code":
@@ -232,6 +239,10 @@ class GithubLoader(BaseLoader):
            data = self._github_search_issues_and_pr(query, search_type)
        elif search_type == "pr":
            data = self._github_search_issues_and_pr(query, search_type)
+        elif search_type == "branch":
+            data = self._get_github_repo_branch(query, search_type)
+        elif search_type == "file":
+            data = self._get_github_repo_file(query, search_type)
        elif search_type == "discussion":
            raise ValueError("GithubLoader does not support searching discussions yet.")
        else:
@@ -22,6 +22,8 @@ except ImportError:
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils.misc import clean_string

+logger = logging.getLogger(__name__)
+

 class GmailReader:
    SCOPES = ["https://www.googleapis.com/auth/gmail.readonly"]
@@ -114,7 +116,7 @@ class GmailLoader(BaseLoader):
    def load_data(self, query: str):
        reader = GmailReader(query=query)
        emails = reader.load_emails()
-        logging.info(f"Gmail Loader: {len(emails)} emails found for query '{query}'")
+        logger.info(f"Gmail Loader: {len(emails)} emails found for query '{query}'")

        data = []
        for email in emails:
@@ -8,8 +8,8 @@ except ImportError:
        "Google Drive requires extra dependencies. Install with `pip install embedchain[googledrive]`"
    ) from None

-from langchain.document_loaders import GoogleDriveLoader as Loader
-from langchain.document_loaders import UnstructuredFileIOLoader
+from langchain_community.document_loaders import GoogleDriveLoader as Loader
+from langchain_community.document_loaders import UnstructuredFileIOLoader

 from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader
@@ -11,14 +11,14 @@ class LocalQnaPairLoader(BaseLoader):
        question, answer = content
        content = f"Q: {question}\nA: {answer}"
        url = "local"
-        meta_data = {"url": url, "question": question}
+        metadata = {"url": url, "question": question}
        doc_id = hashlib.sha256((content + url).encode()).hexdigest()
        return {
            "doc_id": doc_id,
            "data": [
                {
                    "content": content,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                }
            ],
        }
@@ -9,7 +9,7 @@ class LocalTextLoader(BaseLoader):
    def load_data(self, content):
        """Load data from a local text file."""
        url = "local"
-        meta_data = {
+        metadata = {
            "url": url,
        }
        doc_id = hashlib.sha256((content + url).encode()).hexdigest()
@@ -18,7 +18,7 @@ class LocalTextLoader(BaseLoader):
            "data": [
                {
                    "content": content,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                }
            ],
        }
@@ -10,7 +10,7 @@ class MdxLoader(BaseLoader):
        """Load data from a mdx file."""
        with open(url, "r", encoding="utf-8") as infile:
            content = infile.read()
-        meta_data = {
+        metadata = {
            "url": url,
        }
        doc_id = hashlib.sha256((content + url).encode()).hexdigest()
@@ -19,7 +19,7 @@ class MdxLoader(BaseLoader):
            "data": [
                {
                    "content": content,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                }
            ],
        }
@@ -5,6 +5,8 @@ from typing import Any, Optional
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils.misc import clean_string

+logger = logging.getLogger(__name__)
+

 class MySQLLoader(BaseLoader):
    def __init__(self, config: Optional[dict[str, Any]]):
@@ -32,7 +34,7 @@ class MySQLLoader(BaseLoader):
            self.connection = sqlconnector.connection.MySQLConnection(**config)
            self.cursor = self.connection.cursor()
        except (sqlconnector.Error, IOError) as err:
-            logging.info(f"Connection failed: {err}")
+            logger.info(f"Connection failed: {err}")
            raise ValueError(
                f"Unable to connect with the given config: {config}.",
                "Please provide the correct configuration to load data from you MySQL DB. \
@@ -9,6 +9,8 @@ from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils.misc import clean_string

+logger = logging.getLogger(__name__)
+

 class NotionDocument:
    """
@@ -98,7 +100,7 @@ class NotionLoader(BaseLoader):

        id = source[-32:]
        formatted_id = f"{id[:8]}-{id[8:12]}-{id[12:16]}-{id[16:20]}-{id[20:]}"
-        logging.debug(f"Extracted notion page id as: {formatted_id}")
+        logger.debug(f"Extracted notion page id as: {formatted_id}")

        integration_token = os.getenv("NOTION_INTEGRATION_TOKEN")
        reader = NotionPageLoader(integration_token=integration_token)
@@ -35,8 +35,8 @@ class OpenAPILoader(BaseLoader):
            yaml_data = yaml.load(file, Loader=yaml.SafeLoader)
            for i, (key, value) in enumerate(yaml_data.items()):
                string_data = f"{key}: {value}"
-                meta_data = {"url": file_path, "row": i + 1}
-                data.append({"content": string_data, "meta_data": meta_data})
+                metadata = {"url": file_path, "row": i + 1}
+                data.append({"content": string_data, "meta_data": metadata})
                data_content.append(string_data)
        doc_id = hashlib.sha256((content + ", ".join(data_content)).encode()).hexdigest()
        return {"doc_id": doc_id, "data": data}
@@ -1,7 +1,7 @@
 import hashlib

 try:
-    from langchain.document_loaders import PyPDFLoader
+    from langchain_community.document_loaders import PyPDFLoader
 except ImportError:
    raise ImportError(
        'PDF File requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'
@@ -27,12 +27,12 @@ class PdfFileLoader(BaseLoader):
        for page in pages:
            content = page.page_content
            content = clean_string(content)
-            meta_data = page.metadata
-            meta_data["url"] = url
+            metadata = page.metadata
+            metadata["url"] = url
            data.append(
                {
                    "content": content,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                }
            )
            all_content.append(content)
@@ -4,6 +4,8 @@ from typing import Any, Optional

 from embedchain.loaders.base_loader import BaseLoader

+logger = logging.getLogger(__name__)
+

 class PostgresLoader(BaseLoader):
    def __init__(self, config: Optional[dict[str, Any]] = None):
@@ -32,7 +34,7 @@ class PostgresLoader(BaseLoader):
                conn_params.append(f"{key}={value}")
            config_info = " ".join(conn_params)

-        logging.info(f"Connecting to postrgres sql: {config_info}")
+        logger.info(f"Connecting to postrgres sql: {config_info}")
        self.connection = psycopg.connect(conninfo=config_info)
        self.cursor = self.connection.cursor()

@@ -28,7 +28,7 @@ class RSSFeedLoader(BaseLoader):
    @staticmethod
    def get_rss_content(url: str):
        try:
-            from langchain.document_loaders import \
+            from langchain_community.document_loaders import \
                RSSFeedLoader as LangchainRSSFeedLoader
        except ImportError:
            raise ImportError(
@@ -41,12 +41,12 @@ class RSSFeedLoader(BaseLoader):
        data = loader.load()

        for entry in data:
-            meta_data = RSSFeedLoader.serialize_metadata(entry.metadata)
-            meta_data.update({"url": url})
+            metadata = RSSFeedLoader.serialize_metadata(entry.metadata)
+            metadata.update({"url": url})
            output.append(
                {
                    "content": entry.page_content,
-                    "meta_data": meta_data,
+                    "meta_data": metadata,
                }
            )

@@ -19,6 +19,8 @@ from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.loaders.web_page import WebPageLoader

+logger = logging.getLogger(__name__)
+

@register_deserializable
 class SitemapLoader(BaseLoader):
@@ -41,7 +43,7 @@ class SitemapLoader(BaseLoader):
                response.raise_for_status()
                soup = BeautifulSoup(response.text, "xml")
            except requests.RequestException as e:
-                logging.error(f"Error fetching sitemap from URL: {e}")
+                logger.error(f"Error fetching sitemap from URL: {e}")
                return
        elif os.path.isfile(sitemap_source):
            with open(sitemap_source, "r") as file:
@@ -60,7 +62,7 @@ class SitemapLoader(BaseLoader):
                loader_data = web_page_loader.load_data(link)
                return loader_data.get("data")
            except ParserRejectedMarkup as e:
-                logging.error(f"Failed to parse {link}: {e}")
+                logger.error(f"Failed to parse {link}: {e}")
            return None

        with concurrent.futures.ThreadPoolExecutor() as executor:
@@ -72,6 +74,6 @@ class SitemapLoader(BaseLoader):
                    if data:
                        output.extend(data)
                except Exception as e:
-                    logging.error(f"Error loading page {link}: {e}")
+                    logger.error(f"Error loading page {link}: {e}")

        return {"doc_id": doc_id, "data": output}
@@ -11,6 +11,8 @@ from embedchain.utils.misc import clean_string

 SLACK_API_BASE_URL = "https://www.slack.com/api/"

+logger = logging.getLogger(__name__)
+

 class SlackLoader(BaseLoader):
    def __init__(self, config: Optional[dict[str, Any]] = None):
@@ -38,7 +40,7 @@ class SlackLoader(BaseLoader):
                "SLACK_USER_TOKEN environment variables not provided. Check `https://docs.embedchain.ai/data-sources/slack` to learn more."  # noqa:E501
            )

-        logging.info(f"Creating Slack Loader with config: {config}")
+        logger.info(f"Creating Slack Loader with config: {config}")
        # get slack client config params
        slack_bot_token = os.getenv("SLACK_USER_TOKEN")
        ssl_cert = ssl.create_default_context(cafile=certifi.where())
@@ -54,7 +56,7 @@ class SlackLoader(BaseLoader):
            headers=headers,
            team_id=team_id,
        )
-        logging.info("Slack Loader setup successful!")
+        logger.info("Slack Loader setup successful!")

    @staticmethod
    def _check_query(query):
@@ -69,7 +71,7 @@ class SlackLoader(BaseLoader):
            data = []
            data_content = []

-            logging.info(f"Searching slack conversations for query: {query}")
+            logger.info(f"Searching slack conversations for query: {query}")
            results = self.client.search_messages(
                query=query,
                sort="timestamp",
@@ -79,7 +81,7 @@ class SlackLoader(BaseLoader):

            messages = results.get("messages")
            num_message = len(messages)
-            logging.info(f"Found {num_message} messages for query: {query}")
+            logger.info(f"Found {num_message} messages for query: {query}")

            matches = messages.get("matches", [])
            for message in matches:
@@ -88,16 +90,16 @@ class SlackLoader(BaseLoader):
                content = clean_string(text)

                message_meta_data_keys = ["iid", "team", "ts", "type", "user", "username"]
-                meta_data = {}
+                metadata = {}
                for key in message.keys():
                    if key in message_meta_data_keys:
-                        meta_data[key] = message.get(key)
-                meta_data.update({"url": url})
+                        metadata[key] = message.get(key)
+                metadata.update({"url": url})

                data.append(
                    {
                        "content": content,
-                        "meta_data": meta_data,
+                        "meta_data": metadata,
                    }
                )
                data_content.append(content)
@@ -107,7 +109,7 @@ class SlackLoader(BaseLoader):
                "data": data,
            }
        except Exception as e:
-            logging.warning(f"Error in loading slack data: {e}")
+            logger.warning(f"Error in loading slack data: {e}")
            raise ValueError(
                f"Error in loading slack data: {e}. Check `https://docs.embedchain.ai/data-sources/slack` to learn more."  # noqa:E501
            ) from e
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Deshraj Yadav	8520033d44	[Bug fix] Fix issues related to logging configuration (#1318 )	2024-03-14 00:45:37 -07:00
Deshraj Yadav	ebdce87fde	[Version] Update version to 0.1.96 (#1317 )	2024-03-14 00:02:51 -07:00
Abhishek Sharma	f2122ed696	[Fix] Added missing provider for 'vllm' (#1316 )	2024-03-14 00:01:30 -07:00
Deshraj Yadav	3616eaadb4	[Refactor] Improve logging package wide (#1315 )	2024-03-13 17:13:30 -07:00
berwin joule	ef69c91b60	[Bug fix]: fix Cannot add documents to chromadb with inconsistent sizes. (#1314 )	2024-03-13 11:01:46 -07:00
Dev Khant	117824b32c	Add folder and branch to GitHub (#1308 )	2024-03-12 12:15:37 -07:00
Dev Khant	f77f5b996e	Support for Cohere Embeddings (#1310 )	2024-03-12 12:14:45 -07:00
Deshraj Yadav	a4d32aec24	[Docs] Update docs and readme (#1309 )	2024-03-10 00:23:21 -08:00
Uzair Naeem	9111495fae	Enhance code readability and documentation clarity (#1307 )	2024-03-07 08:12:31 -08:00
Hardik Jindal	ee1e3f0957	[docs] update langsmith.mdx (#1306 )	2024-03-07 08:10:11 -08:00
Deshraj Yadav	4dc5c7348f	[Bug fix]: Fix issue of OPENAI_API_BASE env variable being mandatory (#1305 )	2024-03-05 14:07:44 -08:00
Deshraj Yadav	4428768eaa	[Bug Fix]: Fix test cases and update version to 0.1.93 (#1303 )	2024-03-04 18:35:01 -08:00
Joe	11f4ce8fb6	#1155 : Add support for OpenAI-compatible endpoint in LLM and Embed (#1197 )	2024-03-04 18:17:20 -08:00
Felipe Amaral	6078738d34	Feature: Custom Ollama endpoint base_url (#1301 )	2024-03-04 18:09:45 -08:00
Deshraj Yadav	faacfeb891	[Improvement] Set a default app id if not provided in the app configuration (#1300 )	2024-03-02 15:10:34 -08:00
Deshraj Yadav	8d7e8b6fb9	[Docs] Update pinecone integration docs (#1296 )	2024-03-01 16:49:43 -08:00
Deshraj Yadav	7e1d2ffdd7	[Bug fix] Fix search API for pinecone vector db	2024-03-01 16:44:42 -08:00
Deshraj Yadav	91044ec591	[Bug fix] Fix issue related to get_data_sources() method (#1295 )	2024-03-01 11:54:36 -08:00
Deshraj Yadav	c77a75dfb5	[Feature] Add support for NVIDIA AI LLMs and embedding models (#1293 )	2024-02-29 23:56:25 -08:00
Deshraj Yadav	6518c0c06b	[Docs] Update docs for resetting vector database (#1289 )	2024-02-28 12:19:47 -08:00
Deshraj Yadav	09cdaff9a2	[Improvement] Fix deprecation warnings (#1288 )	2024-02-27 15:10:41 -08:00
Deshraj Yadav	56bf33ab7f	[Feature] Add support for running huggingface models locally (#1287 )	2024-02-27 15:05:17 -08:00
Deshraj Yadav	752f638cfc	[Feature/Improvements] Delete data sources from metadata db when using `app.delete()` (#1286 )	2024-02-26 13:18:42 -08:00
Deshraj Yadav	92dd7edb57	[Feature] Add support for Groq LLMs (#1284 )	2024-02-25 11:58:03 -08:00
Deshraj Yadav	b4bb4cf053	[Bug fix] Fix issue for using any metadata db apart from sqlite (#1282 )	2024-02-22 09:52:40 -08:00
Deshraj Yadav	f0400e928a	[Bug fix] Fix issue related to initalizing the local database engine (#1281 )	2024-02-22 02:40:06 -08:00
Deshraj Yadav	aa5ad625af	Add support for supplying custom db params (#1276 )	2024-02-21 16:15:57 -08:00
Deshraj Yadav	f8f69eab03	[Feature] Add support for python 3.13 and other migration related fixes (#1279 )	2024-02-21 13:04:03 -08:00
João Moura	2b2263acaa	Updating python version to including <=3.13 (#1278 )	2024-02-21 11:48:13 -08:00
Deshraj Yadav	5e2e7fb639	[Feature] Add support to use any sql database as the metadata storage for embedchain apps (#1273 )	2024-02-19 13:04:18 -08:00
Deshraj Yadav	6c12bc9044	[Improvements] Improve the default prompt and data loader util functions (#1272 )	2024-02-18 14:06:32 -08:00
Saurabh Misra	9a11683003	⚡️ Speed up `is_readable by` 101% in `embedchain/utils/misc.py` (#1258 ) Co-authored-by: codeflash-ai[bot] <148906541+codeflash-ai[bot]@users.noreply.github.com>	2024-02-15 23:18:53 -08:00
Deshraj Yadav	38b4e06963	[Feature] Add support for hybrid search for pinecone vector database (#1259 )	2024-02-15 13:20:14 -08:00
Deshraj Yadav	0766a44ccf	[Bug fix] Fix vertex ai integration issue (#1257 )	2024-02-14 11:19:32 -08:00
Deshraj Yadav	036bf3a161	Update version to 0.1.78 (#1256 )	2024-02-12 17:23:14 -08:00
UnMonsieur	41bd258b93	[Feature] OpenAI Function Calling (#1224 )	2024-02-11 17:58:11 -08:00
Deshraj Yadav	38e212c721	[Bug fix] Fix test (#1255 )	2024-02-11 17:57:14 -08:00
Deshraj Yadav	2f285ea00a	[Bug fix] Fix history sequence in prompt (#1254 )	2024-02-11 16:07:36 -08:00
Dhravya Shah	d38120c839	[Docs] Added documentation to deploy to Railway.app (#1250 )	2024-02-11 15:58:42 -08:00
Michael	d94aee812b	[Improvements] Fixes to null data results and OpenAI embedding limits (#1238 )	2024-02-11 15:45:02 -08:00
Rishiraj2594	68d650ec40	[Docs] Typo fixed youtube-video.mdx (#1253 )	2024-02-09 16:34:08 -08:00
Rishiraj2594	769d926f5a	[Docs] Typo fixed in youtube-channel.mdx (#1252 )	2024-02-09 16:33:48 -08:00
Oskar	9478bab04e	Fix links to the Discourse docs in the Discourse Loader (#1251 )	2024-02-09 08:21:52 -08:00