[Feature] Add support for AIAssistant (#938 )

[Bug Fix] fix chromadb where clause for query and delete (#937 )
Co-authored-by: Deven Patel <deven298@yahoo.com>
2023-11-10 16:47:34 -08:00 · 2023-11-10 16:04:25 -08:00 · 2023-11-10 10:15:04 -08:00 · 2023-11-10 10:13:21 -08:00 · 2023-11-09 22:29:41 -08:00 · 2023-11-09 15:17:51 -08:00
46 changed files with 1075 additions and 168 deletions
@@ -1,7 +1,7 @@
 app:
  config:
    id: 'my-app'
-    log_level: 'WARN'
+    log_level: 'WARNING'
    collect_metrics: true
    collection_name: 'my-app'

@@ -30,4 +30,4 @@ embedder:
  provider: openai
  config:
    model: 'text-embedding-ada-002'
-    deployment_name: null
+    deployment_name: 'my-app'
@@ -24,12 +24,11 @@ To use this you need to save `credentials.json` in the directory from where you
 12. Put the `.json` file in your current directory and rename it to `credentials.json`

 ```python
-import os
-from embedchain.apps.app import App
-from embedchain.models.data_type import DataType
+from embedchain import Pipeline as App
+
 app = App()

-query = "to: me label:inbox"
-app.add(query, data_type=DataType.GMAIL)
+gmail_filter = "to: me label:inbox"
+app.add(gmail_filter, data_type="gmail")
 app.query("Summarize my email conversations")
 ```
@@ -2,52 +2,43 @@
 title: '📃 JSON'
 ---

-To add any json file, use the data_type as `json`. Headers are included for each line, so if you have an `age` column, `18` will be added as `age: 18`. Eg:
+To add any json file, use the data_type as `json`. Headers are included for each line, so for example if you have a json like `{"age": 18}`, then it will be added as `age: 18`.

 Here are the supported sources for loading `json`:
+
 ```
 1. URL - valid url to json file that ends with ".json" extension.
 2. Local file - valid url to local json file that ends with ".json" extension.
 3. String - valid json string (e.g. - app.add('{"foo": "bar"}'))
 ```

-If you would like to add other data structures (e.x. list, dict etc.), do:
-```python
-    import json
-    a = {"foo": "bar"}
-    valid_json_string_data = json.dumps(a, indent=0)
+<Tip>
+If you would like to add other data structures (e.g. list, dict etc.), convert it to a valid json first using `json.dumps()` function.
+</Tip>

-    b = [{"foo": "bar"}]
-    valid_json_string_data = json.dumps(b, indent=0)
-```
-Example:
-```python
-import os
+## Example

-from embedchain.apps.app import App
+<CodeGroup>

-os.environ["OPENAI_API_KEY"] = "openai_api_key"
+```python python
+from embedchain import Pipeline as App

 app = App()

-response = app.query("What is the net worth of Elon Musk as of October 2023?")
+# Add json file
+app.add("temp.json")

-print(response)
-"I'm sorry, but I don't have access to real-time information or future predictions. Therefore, I don't know the net worth of Elon Musk as of October 2023."
-
-source_id = app.add("temp.json")
-
-response = app.query("What is the net worth of Elon Musk as of October 2023?")
-
-print(response)
-"As of October 2023, Elon Musk's net worth is $255.2 billion."
+app.query("What is the net worth of Elon Musk as of October 2023?")
+# As of October 2023, Elon Musk's net worth is $255.2 billion.
 ```
-temp.json
-```json
+
+
+```json temp.json
 {
    "question": "What is your net worth, Elon Musk?",
    "answer": "As of October 2023, Elon Musk's net worth is $255.2 billion, making him one of the wealthiest individuals in the world."
 }
 ```
+</CodeGroup>


@@ -2,13 +2,10 @@
 title: 🙌 OpenAPI
 ---

-To add any OpenAPI spec yaml file (currently the json file will be detected as JSON data type), use the data_type as 'openapi'. 'openapi' allows remote urls and conventional file paths. Headers are included for each line, so if you have an `age` column, `18` will be added as `age: 18`. Eg:
+To add any OpenAPI spec yaml file (currently the json file will be detected as JSON data type), use the data_type as 'openapi'. 'openapi' allows remote urls and conventional file paths.

 ```python
-from embedchain.apps.app import App
-import os
-
-os.environ["OPENAI_API_KEY"] = "sk-xxx"
+from embedchain import Pipeline as App

 app = App()

@@ -16,8 +13,10 @@ app.add("https://github.com/openai/openai-openapi/blob/master/openapi.yaml", dat
 # Or add using the local file path
 # app.add("configs/openai_openapi.yaml", data_type="openapi")

-response = app.query("What can OpenAI API endpoint do? Can you list the things it can learn from?")
+app.query("What can OpenAI API endpoint do? Can you list the things it can learn from?")
 # Answer: The OpenAI API endpoint allows users to interact with OpenAI's models and perform various tasks such as generating text, answering questions, summarizing documents, translating languages, and more. The specific capabilities and tasks that the API can learn from may vary depending on the models and features provided by OpenAI. For more detailed information, it is recommended to refer to the OpenAI API documentation at https://platform.openai.com/docs/api-reference.
 ```

-NOTE: The yaml file added to the App must have the required OpenAPI fields otherwise the adding OpenAPI spec will fail. Please refer to [OpenAPI Spec Doc](https://spec.openapis.org/oas/v3.1.0)
+<Note>
+The yaml file added to the App must have the required OpenAPI fields otherwise the adding OpenAPI spec will fail. Please refer to [OpenAPI Spec Doc](https://spec.openapis.org/oas/v3.1.0)
+</Note>
@@ -20,6 +20,7 @@ Embedchain comes with built-in support for various data sources. We handle the c
  <Card title="🙌 OpenAPI" href="/data-sources/openapi"></Card>
  <Card title="🎥📺 youtube video" href="/data-sources/youtube-video"></Card>
  <Card title="📬 Gmail" href="/data-sources/gmail"></Card>
+  <Card title="🐘 Postgres" href="/data-sources/postgres"></Card>
 </CardGroup>

 <br/ >
@@ -0,0 +1,64 @@
+---
+title: '🐘 Postgres'
+---
+
+1. Setup the Postgres loader by configuring the postgres db.
+```Python
+from embedchain.loaders.postgres import PostgresLoader
+
+config = {
+    "host": "host_address",
+    "port": "port_number",
+    "dbname": "database_name",
+    "user": "username",
+    "password": "password",
+}
+
+"""
+config = {
+    "url": "your_postgres_url"
+}
+"""
+
+postgres_loader = PostgresLoader(config=config)
+
+```
+
+You can either setup the loader by passing the postgresql url or by providing the config data.
+For more details on how to setup with valid url and config, check postgres [documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING:~:text=34.1.1.%C2%A0Connection%20Strings-,%23,-Several%20libpq%20functions).
+
+NOTE: if you provide the `url` field in config, all other fields will be ignored.
+
+2. Once you setup the loader, you can create an app and load data using the above postgres loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+
+question = "What is Elon Musk's networth?"
+response = app.query(question)
+# Answer: As of September 2021, Elon Musk's net worth is estimated to be around $250 billion, making him one of the wealthiest individuals in the world. However, please note that net worth can fluctuate over time due to various factors such as stock market changes and business ventures.
+
+app.add("SELECT * FROM table_name;", data_type='postgres', loader=postgres_loader)
+# Adds `(1, 'What is your net worth, Elon Musk?', "As of October 2023, Elon Musk's net worth is $255.2 billion.")`
+
+response = app.query(question)
+# Answer: As of October 2023, Elon Musk's net worth is $255.2 billion.
+```
+
+NOTE: The `add` function of the app will accept any executable query to load data. DO NOT pass the `CREATE`, `INSERT` queries in `add` function as they will result in not adding any data, so it is pointless.
+
+3. We automatically create a chunker to chunk your postgres data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+
+from embedchain.chunkers.postgres import PostgresChunker
+from embedchain.config.add_config import ChunkerConfig
+
+postgres_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+postgres_chunker = PostgresChunker(config=postgres_chunker_config)
+
+app.add("SELECT * FROM table_name;", data_type='postgres', loader=postgres_loader, chunker=postgres_chunker)
+```
@@ -1,21 +1,46 @@
 ---
-title: '📱 Telegram Bot'
+title: "📱 Telegram Bot"
 ---

 ### 🖼️ Template Setup

- Fork [this](https://replit.com/@taranjeetio/EC-Telegram-Bot-Template?v=1#README.md) replit template.
- Set your `OPENAI_API_KEY` in Secrets.
 - Open the Telegram app and search for the `BotFather` user.
 - Start a chat with BotFather and use the `/newbot` command to create a new bot.
 - Follow the instructions to choose a name and username for your bot.
 - Once the bot is created, BotFather will provide you with a unique token for your bot.
- Set this token as `TELEGRAM_BOT_TOKEN` in Secrets.
+
+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name telegram-bot -e OPENAI_API_KEY=sk-xxx -e TELEGRAM_BOT_TOKEN=xxx -p 8000:8000 embedchain/telegram-bot
+        ```
+
+    <Note>
+    If you wish to use **Docker**, you would need to host your bot on a server.
+    You can use [ngrok](https://ngrok.com/) to expose your localhost to the
+    internet and then set the webhook using the ngrok URL.
+    </Note>
+
+    </Tab>
+    <Tab title="replit">
+    <Card>
+        Fork <ins>**[this](https://replit.com/@taranjeetio/EC-Telegram-Bot-Template?v=1#README.md)**</ins> replit template.
+    </Card>
+
+    - Set your `OPENAI_API_KEY` in Secrets.
+    - Set the unique token as `TELEGRAM_BOT_TOKEN` in Secrets.
+
+    </Tab>
+
+</Tabs>
+
 - Click on `Run` in the replit container and a URL will get generated for your bot.
 - Now set your webhook by running the following link in your browser:
+
 ```url
 https://api.telegram.org/bot<Your_Telegram_Bot_Token>/setWebhook?url=<Replit_Generated_URL>
 ```
+
 - When you get a successful response in your browser, your bot is ready to be used.

 ### 🚀 Usage Instructions
@@ -12,10 +12,19 @@ pip install --upgrade embedchain

 2. Launch your WhatsApp bot:

+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name whatsapp-bot -e OPENAI_API_KEY=sk-xxx -p 8000:8000 embedchain/whatsapp-bot
+        ```
+    </Tab>
+    <Tab title="python">
+        ```bash
+        python -m embedchain.bots.whatsapp --port 5000
+        ```
+    </Tab>
+</Tabs>

-```bash
-python -m embedchain.bots.whatsapp --port 5000
-```

 If your bot needs to be accessible online, use your machine's public IP or DNS. Otherwise, employ a proxy server like [ngrok](https://ngrok.com/) to make your local bot accessible.

@@ -8,7 +8,7 @@ Embedchain now supports [OpenAI Assistants API](https://platform.openai.com/docs

 At a high level, an integration of the Assistants API has the following flow:

-1. Create an Assistant in the API by defining it custom instructions and picking a model
+1. Create an Assistant in the API by defining custom instructions and picking a model
 2. Create a Thread when a user starts a conversation
 3. Add Messages to the Thread as the user ask questions
 4. Run the Assistant on the Thread to trigger responses. This automatically calls the relevant tools.
@@ -19,7 +19,7 @@ Creating an OpenAI Assistant using Embedchain is very simple 3 step process.

 Make sure that you have `OPENAI_API_KEY` set in the environment variable.

-```python
+```python Initialize
 from embedchain.store.assistants import OpenAIAssistant

 assistant = OpenAIAssistant(
@@ -28,10 +28,28 @@ assistant = OpenAIAssistant(
 )
 ```

+If you want to use the existing assistant, you can do something like this:
+
+```python Initialize
+# Load an assistant and create a new thread
+assistant = OpenAIAssistant(assistant_id="asst_xxx")
+
+# Load a specific thread for an assistant
+assistant = OpenAIAssistant(assistant_id="asst_xxx", thread_id="thread_xxx")
+```
+
 ### Arguments

-<ResponseField name="assistant_id" type="string" required>
-  Load existing OpenAI Assistant. If you pass this, you don't have to pass other arguments
+<ResponseField name="name" type="string">
+  Name for your AI assistant
+</ResponseField>
+
+<ResponseField name="instructions" type="string">
+  how the Assistant and model should behave or respond
+</ResponseField>
+
+<ResponseField name="assistant_id" type="string">
+  Load existing OpenAI Assistant. If you pass this, you don't have to pass other arguments.
 </ResponseField>

 <ResponseField name="thread_id" type="string">
@@ -50,17 +68,21 @@ assistant = OpenAIAssistant(
  Add data sources to your assistant. You can add in the following format: `[{"source": "https://example.com", "data_type": "web_page"}]`
 </ResponseField>

+<ResponseField name="telemetry" type="boolean" default="True">
+  Anonymous telemetry (doesn't collect any user information or user's files). Used to improve the Embedchain package utilization. Default is `True`.
+</ResponseField>
+
 ## Step-2: Add data to thread

 You can add any custom data source that is supported by Embedchain. Else, you can directly pass the file path on your local system and Embedchain propagates it to OpenAI Assistant.
-```python
+```python Add data
 assistant.add("/path/to/file.pdf")
-assistant.add("https://www.youtube.com/watch?v=U9mJuUkhUzk", data_type="youtube_video")
+assistant.add("https://www.youtube.com/watch?v=U9mJuUkhUzk")
 assistant.add("https://openai.com/blog/new-models-and-developer-products-announced-at-devday")
 ```

 ## Step-3: Chat with your Assistant
-```python
+```python Chat
 assistant.chat("How much OpenAI credits were offered to attendees during OpenAI DevDay?")
 # Response: 'Every attendee of OpenAI DevDay 2023 was offered $500 in OpenAI credits.'
 ```
@@ -2,6 +2,7 @@ from typing import Optional

 import yaml

+from embedchain.client import Client
 from embedchain.config import (AppConfig, BaseEmbedderConfig, BaseLlmConfig,
                               ChunkerConfig)
 from embedchain.config.vectordb.base import BaseVectorDbConfig
@@ -67,6 +68,9 @@ class App(EmbedChain):
        :type system_prompt: Optional[str], optional
        :raises TypeError: LLM, database or embedder or their config is not a valid class instance.
        """
+        # Setup user directory if it doesn't exist already
+        Client.setup_dir()
+
        # Type check configs
        if config and not isinstance(config, AppConfig):
            raise TypeError(
@@ -1,7 +1,7 @@
 from typing import Any

-from embedchain import App
-from embedchain.config import AddConfig, AppConfig, BaseLlmConfig
+from embedchain import Pipeline as App
+from embedchain.config import AddConfig, BaseLlmConfig, PipelineConfig
 from embedchain.embedder.openai import OpenAIEmbedder
 from embedchain.helper.json_serializable import (JSONSerializable,
                                                 register_deserializable)
@@ -12,7 +12,7 @@ from embedchain.vectordb.chroma import ChromaDB
@register_deserializable
 class BaseBot(JSONSerializable):
    def __init__(self):
-        self.app = App(config=AppConfig(), llm=OpenAILlm(), db=ChromaDB(), embedder=OpenAIEmbedder())
+        self.app = App(config=PipelineConfig(), llm=OpenAILlm(), db=ChromaDB(), embedding_model=OpenAIEmbedder())

    def add(self, data: Any, config: AddConfig = None):
        """
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helper.json_serializable import register_deserializable
+
+
+@register_deserializable
+class PostgresChunker(BaseChunker):
+    """Chunker for postgres."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -5,7 +5,7 @@ import uuid

 import requests

-from embedchain.embedchain import CONFIG_DIR, CONFIG_FILE
+from embedchain.constants import CONFIG_DIR, CONFIG_FILE


 class Client:
@@ -0,0 +1,8 @@
+import os
+from pathlib import Path
+
+ABS_PATH = os.getcwd()
+HOME_DIR = str(Path.home())
+CONFIG_DIR = os.path.join(HOME_DIR, ".embedchain")
+CONFIG_FILE = os.path.join(CONFIG_DIR, "config.json")
+SQLITE_PATH = os.path.join(CONFIG_DIR, "embedchain.db")
@@ -1,4 +1,5 @@
 from importlib import import_module
+from typing import Any, Dict

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config import AddConfig
@@ -15,7 +16,7 @@ class DataFormatter(JSONSerializable):
    .add or .add_local method call
    """

-    def __init__(self, data_type: DataType, config: AddConfig):
+    def __init__(self, data_type: DataType, config: AddConfig, kwargs: Dict[str, Any]):
        """
        Initialize a dataformatter, set data type and chunker based on datatype.

@@ -24,15 +25,15 @@ class DataFormatter(JSONSerializable):
        :param config: AddConfig instance with nested loader and chunker config attributes.
        :type config: AddConfig
        """
-        self.loader = self._get_loader(data_type=data_type, config=config.loader)
-        self.chunker = self._get_chunker(data_type=data_type, config=config.chunker)
+        self.loader = self._get_loader(data_type=data_type, config=config.loader, kwargs=kwargs)
+        self.chunker = self._get_chunker(data_type=data_type, config=config.chunker, kwargs=kwargs)

    def _lazy_load(self, module_path: str):
        module_path, class_name = module_path.rsplit(".", 1)
        module = import_module(module_path)
        return getattr(module, class_name)

-    def _get_loader(self, data_type: DataType, config: LoaderConfig) -> BaseLoader:
+    def _get_loader(self, data_type: DataType, config: LoaderConfig, kwargs: Dict[str, Any]) -> BaseLoader:
        """
        Returns the appropriate data loader for the given data type.

@@ -63,13 +64,28 @@ class DataFormatter(JSONSerializable):
            DataType.GMAIL: "embedchain.loaders.gmail.GmailLoader",
            DataType.NOTION: "embedchain.loaders.notion.NotionLoader",
        }
+
+        custom_loaders = set(
+            [
+                DataType.POSTGRES,
+            ]
+        )
+
        if data_type in loaders:
            loader_class: type = self._lazy_load(loaders[data_type])
            return loader_class()
-        else:
-            raise ValueError(f"Unsupported data type: {data_type}")
+        elif data_type in custom_loaders:
+            loader_class: type = kwargs.get("loader", None)
+            if loader_class is not None:
+                return loader_class

-    def _get_chunker(self, data_type: DataType, config: ChunkerConfig) -> BaseChunker:
+        raise ValueError(
+            f"Cant find the loader for {data_type}.\
+                    We recommend to pass the loader to use data_type: {data_type},\
+                        check `https://docs.embedchain.ai/data-sources/overview`."
+        )
+
+    def _get_chunker(self, data_type: DataType, config: ChunkerConfig, kwargs: Dict[str, Any]) -> BaseChunker:
        """Returns the appropriate chunker for the given data type (updated for lazy loading)."""
        chunker_classes = {
            DataType.YOUTUBE_VIDEO: "embedchain.chunkers.youtube_video.YoutubeVideoChunker",
@@ -89,12 +105,21 @@ class DataFormatter(JSONSerializable):
            DataType.OPENAPI: "embedchain.chunkers.openapi.OpenAPIChunker",
            DataType.GMAIL: "embedchain.chunkers.gmail.GmailChunker",
            DataType.NOTION: "embedchain.chunkers.notion.NotionChunker",
+            DataType.POSTGRES: "embedchain.chunkers.postgres.PostgresChunker",
        }

        if data_type in chunker_classes:
-            chunker_class = self._lazy_load(chunker_classes[data_type])
+            if "chunker" in kwargs:
+                chunker_class = kwargs.get("chunker")
+            else:
+                chunker_class = self._lazy_load(chunker_classes[data_type])
+
            chunker = chunker_class(config)
            chunker.set_data_type(data_type)
            return chunker
        else:
-            raise ValueError(f"Unsupported data type: {data_type}")
+            raise ValueError(
+                f"Cant find the chunker for {data_type}.\
+                    We recommend to pass the chunker to use data_type: {data_type},\
+                        check `https://docs.embedchain.ai/data-sources/overview`."
+            )
@@ -1,9 +1,7 @@
 import hashlib
 import json
 import logging
-import os
 import sqlite3
-from pathlib import Path
 from typing import Any, Dict, List, Optional, Tuple, Union

 from dotenv import load_dotenv
@@ -12,6 +10,7 @@ from langchain.docstore.document import Document
 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config import AddConfig, BaseLlmConfig, ChunkerConfig
 from embedchain.config.apps.base_app_config import BaseAppConfig
+from embedchain.constants import SQLITE_PATH
 from embedchain.data_formatter import DataFormatter
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.helper.json_serializable import JSONSerializable
@@ -25,12 +24,6 @@ from embedchain.vectordb.base import BaseVectorDB

 load_dotenv()

-ABS_PATH = os.getcwd()
-HOME_DIR = str(Path.home())
-CONFIG_DIR = os.path.join(HOME_DIR, ".embedchain")
-CONFIG_FILE = os.path.join(CONFIG_DIR, "config.json")
-SQLITE_PATH = os.path.join(CONFIG_DIR, "embedchain.db")
-

 class EmbedChain(JSONSerializable):
    def __init__(
@@ -81,6 +74,9 @@ class EmbedChain(JSONSerializable):
        if system_prompt:
            self.llm.config.system_prompt = system_prompt

+        # Fetch the history from the database if exists
+        self.llm.update_history(app_id=self.config.id)
+
        # Attributes that aren't subclass related.
        self.user_asks = []

@@ -137,6 +133,7 @@ class EmbedChain(JSONSerializable):
        metadata: Optional[Dict[str, Any]] = None,
        config: Optional[AddConfig] = None,
        dry_run=False,
+        **kwargs: Dict[str, Any],
    ):
        """
        Adds the data from the given URL to the vector db.
@@ -180,21 +177,6 @@ class EmbedChain(JSONSerializable):
        if data_type:
            try:
                data_type = DataType(data_type)
-                if data_type == DataType.JSON:
-                    if isinstance(source, str):
-                        if not is_valid_json_string(source):
-                            raise ValueError(
-                                f"Invalid json input: {source}",
-                                "Provide the correct JSON formatted source, \
-                                    refer `https://docs.embedchain.ai/data-sources/json`",
-                            )
-                    elif not isinstance(source, str):
-                        raise ValueError(
-                            "Invaid content input. \
-                            If you want to upload (list, dict, etc.), do \
-                                `json.dump(data, indent=0)` and add the stringified JSON. \
-                                    Check - `https://docs.embedchain.ai/data-sources/json`"
-                        )
            except ValueError:
                raise ValueError(
                    f"Invalid data_type: '{data_type}'.",
@@ -218,8 +200,9 @@ class EmbedChain(JSONSerializable):
            print(f"Data with hash {source_hash} already exists. Skipping addition.")
            return source_hash

-        data_formatter = DataFormatter(data_type, config)
        self.user_asks.append([source, data_type.value, metadata])
+
+        data_formatter = DataFormatter(data_type, config, kwargs)
        documents, metadatas, _ids, new_chunks = self.load_and_embed(
            data_formatter.loader, data_formatter.chunker, source, metadata, source_hash, dry_run
        )
@@ -265,6 +248,7 @@ class EmbedChain(JSONSerializable):
        data_type: Optional[DataType] = None,
        metadata: Optional[Dict[str, Any]] = None,
        config: Optional[AddConfig] = None,
+        **kwargs: Dict[str, Any],
    ):
        """
        Adds the data from the given URL to the vector db.
@@ -290,7 +274,13 @@ class EmbedChain(JSONSerializable):
        logging.warning(
            "The `add_local` method is deprecated and will be removed in future versions. Please use the `add` method for both local and remote files."  # noqa: E501
        )
-        return self.add(source=source, data_type=data_type, metadata=metadata, config=config)
+        return self.add(
+            source=source,
+            data_type=data_type,
+            metadata=metadata,
+            config=config,
+            kwargs=kwargs,
+        )

    def _get_existing_doc_id(self, chunker: BaseChunker, src: Any):
        """
@@ -488,13 +478,13 @@ class EmbedChain(JSONSerializable):
        query_config = config or self.llm.config
        if where is not None:
            where = where
-        elif query_config is not None and query_config.where is not None:
-            where = query_config.where
        else:
            where = {}
+            if query_config is not None and query_config.where is not None:
+                where = query_config.where

-        if self.config.id is not None:
-            where.update({"app_id": self.config.id})
+            if self.config.id is not None:
+                where.update({"app_id": self.config.id})

        # We cannot query the database with the input query in case of an image search. This is because we need
        # to bring down both the image and text to the same dimension to be able to compare them.
@@ -608,6 +598,9 @@ class EmbedChain(JSONSerializable):
            input_query=input_query, contexts=contexts_data_for_llm_query, config=config, dry_run=dry_run
        )

+        # add conversation in memory
+        self.llm.add_history(self.config.id, input_query, answer)
+
        # Send anonymous telemetry
        self.telemetry.capture(event_name="chat", properties=self._telemetry_props)

@@ -651,5 +644,16 @@ class EmbedChain(JSONSerializable):
        self.db.reset()
        self.cursor.execute("DELETE FROM data_sources WHERE pipeline_id = ?", (self.config.id,))
        self.connection.commit()
+        self.delete_history()
        # Send anonymous telemetry
        self.telemetry.capture(event_name="reset", properties=self._telemetry_props)
+
+    def get_history(self, num_rounds: int = 10, display_format: bool = True):
+        return self.llm.memory.get_recent_memories(
+            app_id=self.config.id,
+            num_rounds=num_rounds,
+            display_format=display_format,
+        )
+
+    def delete_history(self):
+        self.llm.memory.delete_chat_history(app_id=self.config.id)
@@ -1,14 +1,15 @@
 import logging
 from typing import Any, Dict, Generator, List, Optional

-from langchain.memory import ConversationBufferMemory
-from langchain.schema import BaseMessage
+from langchain.schema import BaseMessage as LCBaseMessage

 from embedchain.config import BaseLlmConfig
 from embedchain.config.llm.base import (DEFAULT_PROMPT,
                                        DEFAULT_PROMPT_WITH_HISTORY_TEMPLATE,
                                        DOCS_SITE_PROMPT_TEMPLATE)
 from embedchain.helper.json_serializable import JSONSerializable
+from embedchain.memory.base import ECChatMemory
+from embedchain.memory.message import ChatMessage


 class BaseLlm(JSONSerializable):
@@ -23,7 +24,7 @@ class BaseLlm(JSONSerializable):
        else:
            self.config = config

-        self.memory = ConversationBufferMemory()
+        self.memory = ECChatMemory()
        self.is_docs_site_instance = False
        self.online = False
        self.history: Any = None
@@ -44,11 +45,18 @@ class BaseLlm(JSONSerializable):
        """
        self.history = history

-    def update_history(self):
+    def update_history(self, app_id: str):
        """Update class history attribute with history in memory (for chat method)"""
-        chat_history = self.memory.load_memory_variables({})["history"]
+        chat_history = self.memory.get_recent_memories(app_id=app_id, num_rounds=10)
        if chat_history:
-            self.set_history(chat_history)
+            self.set_history([str(history) for history in chat_history])
+
+    def add_history(self, app_id: str, question: str, answer: str, metadata: Optional[Dict[str, Any]] = None):
+        chat_message = ChatMessage()
+        chat_message.add_user_message(question, metadata=metadata)
+        chat_message.add_ai_message(answer, metadata=metadata)
+        self.memory.add(app_id=app_id, chat_message=chat_message)
+        self.update_history(app_id=app_id)

    def generate_prompt(self, input_query: str, contexts: List[str], **kwargs: Dict[str, Any]) -> str:
        """
@@ -165,7 +173,6 @@ class BaseLlm(JSONSerializable):
        for chunk in answer:
            streamed_answer = streamed_answer + chunk
            yield chunk
-        self.memory.chat_memory.add_ai_message(streamed_answer)
        logging.info(f"Answer: {streamed_answer}")

    def query(self, input_query: str, contexts: List[str], config: BaseLlmConfig = None, dry_run=False):
@@ -257,8 +264,6 @@ class BaseLlm(JSONSerializable):
            if self.online:
                k["web_search_result"] = self.access_search_and_get_results(input_query)

-            self.update_history()
-
            prompt = self.generate_prompt(input_query, contexts, **k)
            logging.info(f"Prompt: {prompt}")

@@ -267,16 +272,9 @@ class BaseLlm(JSONSerializable):

            answer = self.get_answer_from_llm(prompt)

-            self.memory.chat_memory.add_user_message(input_query)
-
            if isinstance(answer, str):
-                self.memory.chat_memory.add_ai_message(answer)
                logging.info(f"Answer: {answer}")

-                # NOTE: Adding to history before and after. This could be seen as redundant.
-                # If we change it, we have to change the tests (no big deal).
-                self.update_history()
-
                return answer
            else:
                # this is a streamed response and needs to be handled differently.
@@ -287,7 +285,7 @@ class BaseLlm(JSONSerializable):
                self.config: BaseLlmConfig = BaseLlmConfig.deserialize(prev_config)

    @staticmethod
-    def _get_messages(prompt: str, system_prompt: Optional[str] = None) -> List[BaseMessage]:
+    def _get_messages(prompt: str, system_prompt: Optional[str] = None) -> List[LCBaseMessage]:
        """
        Construct a list of langchain messages

@@ -25,10 +25,21 @@ class JSONLoader(BaseLoader):

        return LLHUBJSONLoader()

+    @staticmethod
+    def _check_content(content):
+        if not isinstance(content, str):
+            raise ValueError(
+                "Invaid content input. \
+                If you want to upload (list, dict, etc.), do \
+                    `json.dump(data, indent=0)` and add the stringified JSON. \
+                        Check - `https://docs.embedchain.ai/data-sources/json`"
+            )
+
    @staticmethod
    def load_data(content):
        """Load a json file. Each data point is a key value pair."""

+        JSONLoader._check_content(content)
        loader = JSONLoader._get_llama_hub_loader()

        data = []
@@ -0,0 +1,73 @@
+import hashlib
+import logging
+from typing import Any, Dict, Optional
+
+from embedchain.loaders.base_loader import BaseLoader
+
+
+class PostgresLoader(BaseLoader):
+    def __init__(self, config: Optional[Dict[str, Any]] = None):
+        super().__init__()
+        if not config:
+            raise ValueError(f"Must provide the valid config. Received: {config}")
+
+        self.connection = None
+        self.cursor = None
+        self._setup_loader(config=config)
+
+    def _setup_loader(self, config: Dict[str, Any]):
+        try:
+            import psycopg
+        except ImportError as e:
+            raise ImportError(
+                "Unable to import required packages. \
+                    Run `pip install --upgrade 'embedchain[postgres]'`"
+            ) from e
+
+        config_info = ""
+        if "url" in config:
+            config_info = config.get("url")
+        else:
+            conn_params = []
+            for key, value in config.items():
+                conn_params.append(f"{key}={value}")
+            config_info = " ".join(conn_params)
+
+        logging.info(f"Connecting to postrgres sql: {config_info}")
+        self.connection = psycopg.connect(conninfo=config_info)
+        self.cursor = self.connection.cursor()
+
+    def _check_query(self, query):
+        if not isinstance(query, str):
+            raise ValueError(
+                f"Invalid postgres query: {query}",
+                "Provide the valid source to add from postgres, \
+                    make sure you are following `https://docs.embedchain.ai/data-sources/postgres`",
+            )
+
+    def load_data(self, query):
+        self._check_query(query)
+        try:
+            data = []
+            data_content = []
+            self.cursor.execute(query)
+            results = self.cursor.fetchall()
+            for result in results:
+                doc_content = str(result)
+                data.append({"content": doc_content, "meta_data": {"url": f"postgres_query-({query})"}})
+                data_content.append(doc_content)
+            doc_id = hashlib.sha256((query + ", ".join(data_content)).encode()).hexdigest()
+            return {
+                "doc_id": doc_id,
+                "data": data,
+            }
+        except Exception as e:
+            raise ValueError(f"Failed to load data using query={query} with: {e}")
+
+    def close_connection(self):
+        if self.cursor:
+            self.cursor.close()
+            self.cursor = None
+        if self.connection:
+            self.connection.close()
+            self.connection = None
@@ -0,0 +1,116 @@
+import json
+import logging
+import sqlite3
+import uuid
+from typing import Any, Dict, List, Optional
+
+from embedchain.constants import SQLITE_PATH
+from embedchain.memory.message import ChatMessage
+from embedchain.memory.utils import merge_metadata_dict
+
+CHAT_MESSAGE_CREATE_TABLE_QUERY = """
+            CREATE TABLE IF NOT EXISTS chat_history (
+                app_id TEXT,
+                id TEXT,
+                question TEXT,
+                answer TEXT,
+                metadata TEXT,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                PRIMARY KEY (id, app_id)
+            )
+            """
+
+
+class ECChatMemory:
+    def __init__(self) -> None:
+        with sqlite3.connect(SQLITE_PATH) as self.connection:
+            self.cursor = self.connection.cursor()
+
+            self.cursor.execute(CHAT_MESSAGE_CREATE_TABLE_QUERY)
+            self.connection.commit()
+
+    def add(self, app_id, chat_message: ChatMessage) -> Optional[str]:
+        memory_id = str(uuid.uuid4())
+        metadata_dict = merge_metadata_dict(chat_message.human_message.metadata, chat_message.ai_message.metadata)
+        if metadata_dict:
+            metadata = self._serialize_json(metadata_dict)
+        ADD_CHAT_MESSAGE_QUERY = """
+            INSERT INTO chat_history (app_id, id, question, answer, metadata)
+            VALUES (?, ?, ?, ?, ?)
+        """
+        self.cursor.execute(
+            ADD_CHAT_MESSAGE_QUERY,
+            (
+                app_id,
+                memory_id,
+                chat_message.human_message.content,
+                chat_message.ai_message.content,
+                metadata if metadata_dict else "{}",
+            ),
+        )
+        self.connection.commit()
+        logging.info(f"Added chat memory to db with id: {memory_id}")
+        return memory_id
+
+    def delete_chat_history(self, app_id: str):
+        DELETE_CHAT_HISTORY_QUERY = """
+            DELETE FROM chat_history WHERE app_id=?
+        """
+        self.cursor.execute(
+            DELETE_CHAT_HISTORY_QUERY,
+            (app_id,),
+        )
+        self.connection.commit()
+
+    def get_recent_memories(self, app_id, num_rounds=10, display_format=False) -> List[ChatMessage]:
+        """
+        Get the most recent num_rounds rounds of conversations
+        between human and AI, for a given app_id.
+        """
+
+        QUERY = """
+            SELECT * FROM chat_history
+            WHERE app_id=?
+            ORDER BY created_at DESC
+            LIMIT ?
+        """
+        self.cursor.execute(
+            QUERY,
+            (app_id, num_rounds),
+        )
+
+        results = self.cursor.fetchall()
+        history = []
+        for result in results:
+            app_id, _, question, answer, metadata, timestamp = result
+            metadata = self._deserialize_json(metadata=metadata)
+            # Return list of dict if display_format is True
+            if display_format:
+                history.append({"human": question, "ai": answer, "metadata": metadata, "timestamp": timestamp})
+            else:
+                memory = ChatMessage()
+                memory.add_user_message(question, metadata=metadata)
+                memory.add_ai_message(answer, metadata=metadata)
+                history.append(memory)
+        return history
+
+    def _serialize_json(self, metadata: Dict[str, Any]):
+        return json.dumps(metadata)
+
+    def _deserialize_json(self, metadata: str):
+        return json.loads(metadata)
+
+    def close_connection(self):
+        self.connection.close()
+
+    def count_history_messages(self, app_id: str):
+        QUERY = """
+        SELECT COUNT(*) FROM chat_history
+        WHERE app_id=?
+        """
+        self.cursor.execute(
+            QUERY,
+            (app_id,),
+        )
+        count = self.cursor.fetchone()[0]
+        return count
@@ -0,0 +1,72 @@
+import logging
+from typing import Any, Dict, Optional
+
+from embedchain.helper.json_serializable import JSONSerializable
+
+
+class BaseMessage(JSONSerializable):
+    """
+    The base abstract message class.
+
+    Messages are the inputs and outputs of Models.
+    """
+
+    # The string content of the message.
+    content: str
+
+    # The creator of the message. AI, Human, Bot etc.
+    by: str
+
+    # Any additional info.
+    metadata: Dict[str, Any]
+
+    def __init__(self, content: str, creator: str, metadata: Optional[Dict[str, Any]] = None) -> None:
+        super().__init__()
+        self.content = content
+        self.creator = creator
+        self.metadata = metadata
+
+    @property
+    def type(self) -> str:
+        """Type of the Message, used for serialization."""
+
+    @classmethod
+    def is_lc_serializable(cls) -> bool:
+        """Return whether this class is serializable."""
+        return True
+
+    def __str__(self) -> str:
+        return f"{self.creator}: {self.content}"
+
+
+class ChatMessage(JSONSerializable):
+    """
+    The base abstract chat message class.
+
+    Chat messages are the pair of (question, answer) conversation
+    between human and model.
+    """
+
+    human_message: Optional[BaseMessage] = None
+    ai_message: Optional[BaseMessage] = None
+
+    def add_user_message(self, message: str, metadata: Optional[dict] = None):
+        if self.human_message:
+            logging.info(
+                "Human message already exists in the chat message,\
+                overwritting it with new message."
+            )
+
+        self.human_message = BaseMessage(content=message, creator="human", metadata=metadata)
+
+    def add_ai_message(self, message: str, metadata: Optional[dict] = None):
+        if self.ai_message:
+            logging.info(
+                "AI message already exists in the chat message,\
+                overwritting it with new message."
+            )
+
+        self.ai_message = BaseMessage(content=message, creator="ai", metadata=metadata)
+
+    def __str__(self) -> str:
+        return f"{self.human_message}\n{self.ai_message}"
@@ -0,0 +1,35 @@
+from typing import Any, Dict, Optional
+
+
+def merge_metadata_dict(left: Optional[Dict[str, Any]], right: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]:
+    """
+    Merge the metadatas of two BaseMessage types.
+
+    Args:
+        left (Dict[str, Any]): metadata of human message
+        right (Dict[str, Any]): metadata of ai message
+
+    Returns:
+        Dict[str, Any]: combined metadata dict with dedup
+        to be saved in db.
+    """
+    if not left and not right:
+        return None
+    elif not left:
+        return right
+    elif not right:
+        return left
+
+    merged = left.copy()
+    for k, v in right.items():
+        if k not in merged:
+            merged[k] = v
+        elif type(merged[k]) != type(v):
+            raise ValueError(f'additional_kwargs["{k}"] already exists in this message,' " but with a different type.")
+        elif isinstance(merged[k], str):
+            merged[k] += v
+        elif isinstance(merged[k], dict):
+            merged[k] = merge_metadata_dict(merged[k], v)
+        else:
+            raise ValueError(f"Additional kwargs key {k} already exists in this message.")
+    return merged
@@ -29,6 +29,7 @@ class IndirectDataType(Enum):
    JSON = "json"
    OPENAPI = "openapi"
    GMAIL = "gmail"
+    POSTGRES = "postgres"


 class SpecialDataType(Enum):
@@ -57,3 +58,4 @@ class DataType(Enum):
    JSON = IndirectDataType.JSON.value
    OPENAPI = IndirectDataType.OPENAPI.value
    GMAIL = IndirectDataType.GMAIL.value
+    POSTGRES = IndirectDataType.POSTGRES.value
@@ -10,7 +10,8 @@ import yaml

 from embedchain import Client
 from embedchain.config import ChunkerConfig, PipelineConfig
-from embedchain.embedchain import CONFIG_DIR, EmbedChain
+from embedchain.constants import SQLITE_PATH
+from embedchain.embedchain import EmbedChain
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.embedder.openai import OpenAIEmbedder
 from embedchain.factory import EmbedderFactory, LlmFactory, VectorDBFactory
@@ -22,8 +23,6 @@ from embedchain.utils import validate_yaml_config
 from embedchain.vectordb.base import BaseVectorDB
 from embedchain.vectordb.chroma import ChromaDB

-SQLITE_PATH = os.path.join(CONFIG_DIR, "embedchain.db")
-

@register_deserializable
 class Pipeline(EmbedChain):
@@ -65,6 +64,9 @@ class Pipeline(EmbedChain):
        :type auto_deploy: bool, optional
        :raises Exception: If an error occurs while creating the pipeline
        """
+        # Setup user directory if it doesn't exist already
+        Client.setup_dir()
+
        if id and yaml_path:
            raise Exception("Cannot provide both id and config. Please provide only one of them.")

@@ -76,22 +78,18 @@ class Pipeline(EmbedChain):

        logging.basicConfig(level=log_level, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s")
        self.logger = logging.getLogger(__name__)
-
        self.auto_deploy = auto_deploy
-
        # Store the yaml config as an attribute to be able to send it
        self.yaml_config = None
        self.client = None
        # pipeline_id from the backend
        self.id = None
-
        self.chunker = None
        if chunker:
            self.chunker = ChunkerConfig(**chunker)

        self.config = config or PipelineConfig()
        self.name = self.config.name
-
        self.config.id = self.local_id = str(uuid.uuid4()) if self.config.id is None else self.config.id

        if yaml_path:
@@ -1,16 +1,20 @@
 import logging
 import os
+import re
 import tempfile
 import time
+import uuid
 from pathlib import Path
 from typing import cast

 from openai import OpenAI
 from openai.types.beta.threads import MessageContentText, ThreadMessage

+from embedchain import Pipeline
 from embedchain.config import AddConfig
 from embedchain.data_formatter import DataFormatter
 from embedchain.models.data_type import DataType
+from embedchain.telemetry.posthog import AnonymousTelemetry
 from embedchain.utils import detect_datatype

 logging.basicConfig(level=logging.WARN)
@@ -27,6 +31,7 @@ class OpenAIAssistant:
        data_sources=None,
        assistant_id=None,
        log_level=logging.WARN,
+        collect_metrics=True,
    ):
        self.name = name or "OpenAI Assistant"
        self.instructions = instructions
@@ -37,14 +42,24 @@ class OpenAIAssistant:
        self._client = OpenAI()
        self._initialize_assistant(assistant_id)
        self.thread_id = thread_id or self._create_thread()
+        self._telemetry_props = {"class": self.__class__.__name__}
+        self.telemetry = AnonymousTelemetry(enabled=collect_metrics)
+        self.telemetry.capture(event_name="init", properties=self._telemetry_props)

    def add(self, source, data_type=None):
        file_path = self._prepare_source_path(source, data_type)
        self._add_file_to_assistant(file_path)
+
+        event_props = {
+            **self._telemetry_props,
+            "data_type": data_type or detect_datatype(source),
+        }
+        self.telemetry.capture(event_name="add", properties=event_props)
        logging.info("Data successfully added to the assistant.")

    def chat(self, message):
        self._send_message(message)
+        self.telemetry.capture(event_name="chat", properties=self._telemetry_props)
        return self._get_latest_response()

    def delete_thread(self):
@@ -70,9 +85,9 @@ class OpenAIAssistant:
        if Path(source).is_file():
            return source
        data_type = data_type or detect_datatype(source)
-        formatter = DataFormatter(data_type=DataType(data_type), config=AddConfig())
+        formatter = DataFormatter(data_type=DataType(data_type), config=AddConfig(), kwargs={})
        data = formatter.loader.load_data(source)["data"]
-        return self._save_temp_data(data[0]["content"].encode())
+        return self._save_temp_data(data=data[0]["content"].encode(), source=source)

    def _add_file_to_assistant(self, file_path):
        file_obj = self._client.files.create(file=open(file_path, "rb"), purpose="assistants")
@@ -117,9 +132,73 @@ class OpenAIAssistant:
        content = [c.text.value for c in thread_message.content if isinstance(c, MessageContentText)]
        return " ".join(content)

-    def _save_temp_data(self, data):
+    def _save_temp_data(self, data, source):
+        special_chars_pattern = r'[\\/:*?"<>|&=% ]+'
+        sanitized_source = re.sub(special_chars_pattern, "_", source)[:256]
        temp_dir = tempfile.mkdtemp()
-        file_path = os.path.join(temp_dir, "temp_data")
+        file_path = os.path.join(temp_dir, sanitized_source)
        with open(file_path, "wb") as file:
            file.write(data)
        return file_path
+
+
+class AIAssistant:
+    def __init__(
+        self,
+        name=None,
+        instructions=None,
+        yaml_path=None,
+        assistant_id=None,
+        thread_id=None,
+        data_sources=None,
+        log_level=logging.WARN,
+        collect_metrics=True,
+    ):
+        logging.basicConfig(level=log_level)
+
+        self.name = name or "AI Assistant"
+        self.data_sources = data_sources or []
+        self.log_level = log_level
+        self.instructions = instructions
+        self.assistant_id = assistant_id or str(uuid.uuid4())
+        self.thread_id = thread_id or str(uuid.uuid4())
+        self.pipeline = Pipeline.from_config(yaml_path=yaml_path) if yaml_path else Pipeline()
+        self.pipeline.local_id = self.pipeline.config.id = self.thread_id
+
+        if self.instructions:
+            self.pipeline.system_prompt = self.instructions
+
+        print(
+            f"🎉 Created AI Assistant with name: {self.name}, assistant_id: {self.assistant_id}, thread_id: {self.thread_id}"  # noqa: E501
+        )
+
+        # telemetry related properties
+        self._telemetry_props = {"class": self.__class__.__name__}
+        self.telemetry = AnonymousTelemetry(enabled=collect_metrics)
+        self.telemetry.capture(event_name="init", properties=self._telemetry_props)
+
+        if self.data_sources:
+            for data_source in self.data_sources:
+                metadata = {"assistant_id": self.assistant_id, "thread_id": "global_knowledge"}
+                self.pipeline.add(data_source["source"], data_source.get("data_type"), metadata=metadata)
+
+    def add(self, source, data_type=None):
+        metadata = {"assistant_id": self.assistant_id, "thread_id": self.thread_id}
+        self.pipeline.add(source, data_type=data_type, metadata=metadata)
+        event_props = {
+            **self._telemetry_props,
+            "data_type": data_type or detect_datatype(source),
+        }
+        self.telemetry.capture(event_name="add", properties=event_props)
+
+    def chat(self, query):
+        where = {
+            "$and": [
+                {"assistant_id": {"$eq": self.assistant_id}},
+                {"thread_id": {"$in": [self.thread_id, "global_knowledge"]}},
+            ]
+        }
+        return self.pipeline.chat(query, where=where)
+
+    def delete(self):
+        self.pipeline.reset()
@@ -309,7 +309,7 @@ def validate_yaml_config(config_data):
                    "gpt4all",
                    "jina",
                    "llama2",
-                    "vertex_ai",
+                    "vertexai",
                ),
                Optional("config"): {
                    Optional("model"): str,
@@ -329,23 +329,17 @@ def validate_yaml_config(config_data):
                Optional("provider"): Or(
                    "chroma", "elasticsearch", "opensearch", "pinecone", "qdrant", "weaviate", "zilliz"
                ),
-                Optional("config"): {
-                    Optional("collection_name"): str,
-                    Optional("dir"): str,
-                    Optional("allow_reset"): bool,
-                    Optional("host"): str,
-                    Optional("port"): str,
-                },
+                Optional("config"): object,  # TODO: add particular config schema for each provider
            },
            Optional("embedder"): {
-                Optional("provider"): Or("openai", "gpt4all", "huggingface", "vertexai"),
+                Optional("provider"): Or("openai", "gpt4all", "huggingface", "vertexai", "azure_openai"),
                Optional("config"): {
                    Optional("model"): Optional(str),
                    Optional("deployment_name"): Optional(str),
                },
            },
            Optional("embedding_model"): {
-                Optional("provider"): Or("openai", "gpt4all", "huggingface", "vertexai"),
+                Optional("provider"): Or("openai", "gpt4all", "huggingface", "vertexai", "azure_openai"),
                Optional("config"): {
                    Optional("model"): str,
                    Optional("deployment_name"): str,
@@ -77,7 +77,7 @@ class ChromaDB(BaseVectorDB):
    def _generate_where_clause(self, where: Dict[str, any]) -> str:
        # If only one filter is supplied, return it as is
        # (no need to wrap in $and based on chroma docs)
-        if len(where.keys()) == 1:
+        if len(where.keys()) <= 1:
            return where
        where_filters = []
        for k, v in where.items():
@@ -224,7 +224,7 @@ class ChromaDB(BaseVectorDB):
                        input_query,
                    ],
                    n_results=n_results,
-                    where=where,
+                    where=self._generate_where_clause(where),
                )
            else:
                result = self.collection.query(
@@ -232,7 +232,7 @@ class ChromaDB(BaseVectorDB):
                        input_query,
                    ],
                    n_results=n_results,
-                    where=where,
+                    where=self._generate_where_clause(where),
                )
        except InvalidDimensionException as e:
            raise InvalidDimensionException(
@@ -275,7 +275,7 @@ class ChromaDB(BaseVectorDB):
        return self.collection.count()

    def delete(self, where):
-        return self.collection.delete(where=where)
+        return self.collection.delete(where=self._generate_where_clause(where))

    def reset(self):
        """
@@ -1,6 +1,6 @@
 fastapi==0.104.0
 uvicorn==0.23.2
-embedchain==0.0.91
-embedchain[streamlit, community, opensource, elasticsearch, opensearch, poe, discord, slack, whatsapp, weaviate, pinecone, qdrant, images, huggingface_hub, cohere, milvus, dataloaders, vertexai, llama2, gmail, json]==0.0.91
+embedchain==0.1.3
+embedchain[streamlit, community, opensource, elasticsearch, opensearch, poe, discord, slack, whatsapp, weaviate, pinecone, qdrant, images, huggingface_hub, cohere, milvus, dataloaders, vertexai, llama2, gmail, json]==0.1.3
 sqlalchemy==2.0.22
 python-multipart==0.0.6
@@ -0,0 +1,2 @@
+TELEGRAM_BOT_TOKEN=
+OPENAI_API_KEY=
@@ -0,0 +1,11 @@
+FROM python:3.11-slim
+
+WORKDIR /usr/src/
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . .
+
+EXPOSE 8000
+
+CMD ["python", "telegram_bot.py"]
@@ -63,4 +63,4 @@ def send_message(chat_id, text):


 if __name__ == "__main__":
-    app.run(host="0.0.0.0", port=5000, debug=False)
+    app.run(host="0.0.0.0", port=8000, debug=False)
@@ -1,2 +0,0 @@
-TELEGRAM_BOT_TOKEN=""
-OPENAI_API_KEY=""
@@ -0,0 +1 @@
+OPENAI_API_KEY=
@@ -0,0 +1,11 @@
+FROM python:3.11-slim
+
+WORKDIR /usr/src/
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . .
+
+EXPOSE 8000
+
+CMD ["python", "whatsapp_bot.py"]
@@ -1 +0,0 @@
-OPENAI_API_KEY=""
@@ -48,4 +48,4 @@ def query(message):


 if __name__ == "__main__":
-    app.run(host="0.0.0.0", port=5000, debug=False)
+    app.run(host="0.0.0.0", port=8000, debug=False)
@@ -4338,6 +4338,117 @@ files = [
 [package.extras]
 test = ["enum34", "ipaddress", "mock", "pywin32", "wmi"]

+[[package]]
+name = "psycopg"
+version = "3.1.12"
+description = "PostgreSQL database adapter for Python"
+optional = true
+python-versions = ">=3.7"
+files = [
+    {file = "psycopg-3.1.12-py3-none-any.whl", hash = "sha256:8ec5230d6a7eb654b4fb3cf2d3eda8871d68f24807b934790504467f1deee9f8"},
+    {file = "psycopg-3.1.12.tar.gz", hash = "sha256:cec7ad2bc6a8510e56c45746c631cf9394148bdc8a9a11fd8cf8554ce129ae78"},
+]
+
+[package.dependencies]
+typing-extensions = ">=4.1"
+tzdata = {version = "*", markers = "sys_platform == \"win32\""}
+
+[package.extras]
+binary = ["psycopg-binary (==3.1.12)"]
+c = ["psycopg-c (==3.1.12)"]
+dev = ["black (>=23.1.0)", "dnspython (>=2.1)", "flake8 (>=4.0)", "mypy (>=1.4.1)", "types-setuptools (>=57.4)", "wheel (>=0.37)"]
+docs = ["Sphinx (>=5.0)", "furo (==2022.6.21)", "sphinx-autobuild (>=2021.3.14)", "sphinx-autodoc-typehints (>=1.12)"]
+pool = ["psycopg-pool"]
+test = ["anyio (>=3.6.2,<4.0)", "mypy (>=1.4.1)", "pproxy (>=2.7)", "pytest (>=6.2.5)", "pytest-cov (>=3.0)", "pytest-randomly (>=3.5)"]
+
+[[package]]
+name = "psycopg-binary"
+version = "3.1.12"
+description = "PostgreSQL database adapter for Python -- C optimisation distribution"
+optional = true
+python-versions = ">=3.7"
+files = [
+    {file = "psycopg_binary-3.1.12-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:29a69f62aae8617361376d9ed1e34966ae9c3a74c4ab3aa430a7ce0c11530862"},
+    {file = "psycopg_binary-3.1.12-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:7308316fdb6796399041b80db0ab9f356504ed26427e46834ade82ba94b067ce"},
+    {file = "psycopg_binary-3.1.12-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:130752b9b2f8d071f179e257b9698cedfe4546be81ad5ecd8ed52cf9d725580d"},
+    {file = "psycopg_binary-3.1.12-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:45bcecc96a6e6fe11e06b75f7ba8005d6f717f16fae7ab1cf5a0aec5191f87c3"},
+    {file = "psycopg_binary-3.1.12-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:bc3f0fcc4fcccffda2450c725bee9fad73bc6c110cfbe3b8a777063845d9c6b9"},
+    {file = "psycopg_binary-3.1.12-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f93749f0fe69cfbfec22af690bb4b241f1a4347c57be26fe2e5b70588f7d602f"},
+    {file = "psycopg_binary-3.1.12-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:36147f708cc6a9d74c2b8d880f8dd3a6d53364b5c487536adaa022d435c90733"},
+    {file = "psycopg_binary-3.1.12-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:2bbcc6fbabc2b92d18d955d9fa104fd9d8bd2dcb97a279c4e788c6b714ffd1af"},
+    {file = "psycopg_binary-3.1.12-cp310-cp310-musllinux_1_1_ppc64le.whl", hash = "sha256:0dee8a1ecc501d9c3db06d08184712459bbb5806a09121c3a25e8cbe91e234d7"},
+    {file = "psycopg_binary-3.1.12-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:49d6acf228edb5bd9000735b89b780b18face776d081b905cf68e149d57dfcc1"},
+    {file = "psycopg_binary-3.1.12-cp310-cp310-win_amd64.whl", hash = "sha256:ee65335781a54f29f4abc28060a6188c41bdd42fdc3cbc1dd84695ed8ef18321"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:d401722aa38bda64d1ba8293f6dad99f6f684711e2c016a93f138f2bbcff2a4b"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:46eac158e8e794d9414a8fe7706beeee9b1ecc4accbea914314825ace8137105"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3f017400679aa38f6cb22b888b8ec198a5b100ec2132e6b3bcfa797b14b5b438"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d176c4614f5208ab9938d5426d61627c8fbc7f8dab53fef42c8bf2ab8605aa51"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c48c4f3fcfd9e75e3fdb18eea320de591e06059a859280ec26ce8d753299353d"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:98fce28d8136bdd883f20d26467bf259b5fb559eb64d8f83695690714cdfdad3"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:e4a0f44bc29fc1b56ee1c865796cbe354078ee1e985f898e4915db185055bf7d"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:6def4f238ca02d6b42336b405d02729c081c978cda9b6ba7549a9c63a91ba823"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-musllinux_1_1_ppc64le.whl", hash = "sha256:000838cb5ab7851116b462e58893a96b0f1e35864135a6283f3242a730ec45d3"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:e7949e1aefe339f04dbecac6aa036c9cd137a58f966c4b96ab933823c340ee12"},
+    {file = "psycopg_binary-3.1.12-cp311-cp311-win_amd64.whl", hash = "sha256:b32922872460575083487de41e17e8cf308c3550da02c704efe42960bc6c19de"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:70054ada2f890d004dc3d5ff908e34aecb085fd599d40db2975c09a39c50dfc3"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7544d6d74f5b5f9daafe8a4ed7d266787d62a2bf16f5120c45d42d1f4a856bc8"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:43197161099cb4e36a9ca44c10657908b619d7263ffcff30932ad4627430dc3c"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:68398cdf3aedd4042b1126b9aba34615f1ab592831483282f19f0159fce5ca75"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:77ae6cda3ffee2425aca9ea7af57296d0c701e2ac5897b48b95dfee050234592"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:278e8888e90fb6ebd7eae8ccb85199eafd712b734e641e0d40f2a903e946102d"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:047c4ba8d3089465b0a69c4c669128df43403867858d78da6b40b33788bfa89f"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:8248b11ac490bb74de80457ab0e9cef31c08164ff7b867031927a17e5c9e19ed"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-musllinux_1_1_ppc64le.whl", hash = "sha256:6979c02acb9783c6134ee516751b8f891a2d4db7f73ebecc9e92750283d6fb99"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:eaf2375b724ad61ee82a5c2a849e57b12b3cb510ec8845084132bbb907cb3335"},
+    {file = "psycopg_binary-3.1.12-cp312-cp312-win_amd64.whl", hash = "sha256:6177cfa6f872a9cc84dbfc7dc163af6ef01639c50acc9a441673f29c2305c37a"},
+    {file = "psycopg_binary-3.1.12-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:b81427fd5a97c9b4ac12f3b8d985870b0c3866b5fc2e72e51cacd3630ffd6466"},
+    {file = "psycopg_binary-3.1.12-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3f17a2c393879aa54f840540009d0e70a30d22ffa0038d81e258ac2c99b15d74"},
+    {file = "psycopg_binary-3.1.12-cp37-cp37m-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6c6a5d125a61101ef5ab7384206e43952fe2a5fca997b96d28a28a752512f900"},
+    {file = "psycopg_binary-3.1.12-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:942a18df448a33d77aa7dff7e93062ace7926608a965db003622cb5f27910ba2"},
+    {file = "psycopg_binary-3.1.12-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3195baff3e3e5d71828400d38af0ffc5a15d7dca2bfaadc9eb615235774b9290"},
+    {file = "psycopg_binary-3.1.12-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:f26bb34e0e9bb83fba00c4835f91f5c5348cdf689df8c8b503571c0d0027c8f5"},
+    {file = "psycopg_binary-3.1.12-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:104bdc85c5c4884b3f900155b635588a28740f561b32a3e27c38bcd249feba41"},
+    {file = "psycopg_binary-3.1.12-cp37-cp37m-musllinux_1_1_ppc64le.whl", hash = "sha256:53464cb71e06faac479f44b8870f115004187e1dfb299b9725d1d7f85d9e5479"},
+    {file = "psycopg_binary-3.1.12-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:052835aac03ee6a9d5b6fe35c468da79084ebe38709e6d3c24ff5b9422fb2947"},
+    {file = "psycopg_binary-3.1.12-cp37-cp37m-win_amd64.whl", hash = "sha256:a21a7fffec1a225b26d72adb960d771fc5a9aba8e1f7dd710abcaa9a980e9740"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:6925a543e88cdfd1a2f679c7a33c08f107de60728a4a3c52f88d4491d40a7f51"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:b04957bd5caff94eac38306357b6d448dd20a6f68fd998e115e3731a55118d83"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f6f55979804853efa5ce84d7ef59ff3772e0823247497f7d4a6870e6527fd791"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7d343e1f564fdc8964e1c08b8a6c1f6ebf4b45ee5631b5241c9cbac793f4500c"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:48c4ba35f717783327931aa9da6e6aab81b6b90f3e6b902b18e269d73e7d0882"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d77c95d6086e0714225764772bf8110bb29dfbc6c32aa56e725a01998ce20e7c"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:6dea80e65c7a97150d555b64744e7279ff4c6b259d27580b756a5b282a7d44e3"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:03a851123d0155e1d6ca5b6cccf624e2fc71c8f7eae76f5100196e0fca047d30"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-musllinux_1_1_ppc64le.whl", hash = "sha256:99ad07b9ef5853713bb63c55e179af52994e96f445c5d66b87d8b986182922ef"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:4441d0f8ecae499a6ac5c79078c9fcd406c0bf70e72cb6cba888aca51aa46943"},
+    {file = "psycopg_binary-3.1.12-cp38-cp38-win_amd64.whl", hash = "sha256:cb45a709b966583773acc3418fffbf6d73b014943b6efceca6a7d3ca960956cf"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:5112245daf98e22046316e72690689a8952a9b078908206a6b16cd28d84cde7c"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:c2eb94bf0bd653c940517cd92dc4f98c85d505f69013b247dda747413bcf0a8b"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d41b03ce52a109858735ac19fe0295e3f77bef0388d6a3e105074ad68f4a9645"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:4fddc3c9beaf745de3da10230f0144a4c667b21c3f7a94a3bb1fb004954c9810"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c5987616698c895ae079fb5e26811b72948cb3b75c2c690446379298e96c1568"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f4ae45d58bd79795a2d23d05be5496b226b09ac2688b9ed9808e13c345e2d542"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:bb98252ac8ba41a121f88979e4232ffc1d6722c953531cbdae2b328322308581"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:ca09e4937c9db24a58951ee9aea7aae7bca11a954b30c59f3b271e9bdebd80d7"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-musllinux_1_1_ppc64le.whl", hash = "sha256:03e321e149d051daa20892ed1bb3beabf0aae98a8c37da30ec80fa12306f9ba9"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:d819cb43cccc10ba501b9d462409fcaaeb19f77b8379b2e7ca0ced4a49446d4a"},
+    {file = "psycopg_binary-3.1.12-cp39-cp39-win_amd64.whl", hash = "sha256:c9eb2ba27760bc1303f0708ba95b9e4f3f3b77a081ef4f7f53375c71da3a1bee"},
+]
+
+[[package]]
+name = "psycopg-pool"
+version = "3.1.8"
+description = "Connection Pool for Psycopg"
+optional = true
+python-versions = ">=3.7"
+files = [
+    {file = "psycopg-pool-3.1.8.tar.gz", hash = "sha256:53d9691503b538d419bf147359028633780294976743b1654dbf5f3a85b675db"},
+    {file = "psycopg_pool-3.1.8-py3-none-any.whl", hash = "sha256:dc9b177e749aae4ad155d22f9d02ccb14fe2bf30792227fb02317361b446ee39"},
+]
+
+[package.dependencies]
+typing-extensions = ">=3.10"
+
 [[package]]
 name = "pulsar-client"
 version = "3.3.0"
@@ -7298,6 +7409,7 @@ opensearch = ["opensearch-py"]
 opensource = ["gpt4all", "sentence-transformers", "torch"]
 pinecone = ["pinecone-client"]
 poe = ["fastapi-poe"]
+postgres = ["psycopg", "psycopg-binary", "psycopg-pool"]
 qdrant = ["qdrant-client"]
 slack = ["flask", "slack-sdk"]
 streamlit = []
@@ -7308,4 +7420,4 @@ whatsapp = ["flask", "twilio"]
 [metadata]
 lock-version = "2.0"
 python-versions = ">=3.9,<3.12"
-content-hash = "9bc0e2292b5ddaea49636fd1183a6b0603b610cbb6d9c708e922be892cbab59c"
+content-hash = "fa041b870ce060414e7c2d2bc21c2c4909aff117b40a161f2eaafafe44136597"
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "embedchain"
-version = "0.1.0"
+version = "0.1.7"
 description = "Data platform for LLMs - Load, index, retrieve and sync any unstructured data"
 authors = [
    "Taranjeet Singh <taranjeet@embedchain.ai>",
@@ -130,6 +130,9 @@ pymilvus = { version = "2.3.1", optional = true }
 google-cloud-aiplatform = { version = "^1.26.1", optional = true }
 replicate = { version = "^0.15.4", optional = true }
 schema = "^0.7.5"
+psycopg = { version = "^3.1.12", optional = true }
+psycopg-binary = { version = "^3.1.12", optional = true }
+psycopg-pool = { version = "^3.1.8", optional = true }

 [tool.poetry.group.dev.dependencies]
 black = "^23.3.0"
@@ -184,9 +187,8 @@ gmail = [
    "google-api-core",
 ]
 json = ["llama-hub"]
+postgres = ["psycopg", "psycopg-binary", "psycopg-pool"]

 [tool.poetry.group.docs.dependencies]

-
-
 [tool.poetry.scripts]
@@ -6,6 +6,7 @@ from embedchain.chunkers.mdx import MdxChunker
 from embedchain.chunkers.notion import NotionChunker
 from embedchain.chunkers.openapi import OpenAPIChunker
 from embedchain.chunkers.pdf_file import PdfFileChunker
+from embedchain.chunkers.postgres import PostgresChunker
 from embedchain.chunkers.qna_pair import QnaPairChunker
 from embedchain.chunkers.sitemap import SitemapChunker
 from embedchain.chunkers.table import TableChunker
@@ -33,6 +34,7 @@ chunker_common_config = {
    JSONChunker: {"chunk_size": 1000, "chunk_overlap": 0, "length_function": len},
    OpenAPIChunker: {"chunk_size": 1000, "chunk_overlap": 0, "length_function": len},
    GmailChunker: {"chunk_size": 1000, "chunk_overlap": 0, "length_function": len},
+    PostgresChunker: {"chunk_size": 1000, "chunk_overlap": 0, "length_function": len},
 }


@@ -7,6 +7,7 @@ from embedchain import App
 from embedchain.config import AppConfig, ChromaDbConfig
 from embedchain.embedchain import EmbedChain
 from embedchain.llm.base import BaseLlm
+from embedchain.memory.base import ECChatMemory

 os.environ["OPENAI_API_KEY"] = "test-api-key"

@@ -25,6 +26,11 @@ def test_whole_app(app_instance, mocker):
    mocker.patch.object(BaseLlm, "get_answer_from_llm", return_value=knowledge)
    mocker.patch.object(BaseLlm, "get_llm_model_answer", return_value=knowledge)
    mocker.patch.object(BaseLlm, "generate_prompt")
+    mocker.patch.object(
+        BaseLlm,
+        "add_history",
+    )
+    mocker.patch.object(ECChatMemory, "delete_chat_history", autospec=True)

    app_instance.add(knowledge, data_type="text")
    app_instance.query("What text did I give you?")
@@ -41,6 +47,10 @@ def test_add_after_reset(app_instance, mocker):
    chroma_config = {"allow_reset": True}

    app_instance = App(config=config, db_config=ChromaDbConfig(**chroma_config))
+
+    # mock delete chat history
+    mocker.patch.object(ECChatMemory, "delete_chat_history", autospec=True)
+
    app_instance.reset()

    app_instance.db.client.heartbeat()
@@ -63,5 +73,5 @@ def test_add_after_reset(app_instance, mocker):
 def test_add_with_incorrect_content(app_instance, mocker):
    content = [{"foo": "bar"}]

-    with pytest.raises(ValueError):
+    with pytest.raises(TypeError):
        app_instance.add(content, data_type="json")
@@ -5,6 +5,8 @@ from unittest.mock import MagicMock, patch
 from embedchain import App
 from embedchain.config import AppConfig, BaseLlmConfig
 from embedchain.llm.base import BaseLlm
+from embedchain.memory.base import ECChatMemory
+from embedchain.memory.message import ChatMessage


 class TestApp(unittest.TestCase):
@@ -31,14 +33,14 @@ class TestApp(unittest.TestCase):
        """
        config = AppConfig(collect_metrics=False)
        app = App(config=config)
-        first_answer = app.chat("Test query 1")
-        self.assertEqual(first_answer, "Test answer")
-        self.assertEqual(len(app.llm.memory.chat_memory.messages), 2)
-        self.assertEqual(len(app.llm.history.splitlines()), 2)
-        second_answer = app.chat("Test query 2")
-        self.assertEqual(second_answer, "Test answer")
-        self.assertEqual(len(app.llm.memory.chat_memory.messages), 4)
-        self.assertEqual(len(app.llm.history.splitlines()), 4)
+        with patch.object(BaseLlm, "add_history") as mock_history:
+            first_answer = app.chat("Test query 1")
+            self.assertEqual(first_answer, "Test answer")
+            mock_history.assert_called_with(app.config.id, "Test query 1", "Test answer")
+
+            second_answer = app.chat("Test query 2")
+            self.assertEqual(second_answer, "Test answer")
+            mock_history.assert_called_with(app.config.id, "Test query 2", "Test answer")

    @patch.object(App, "retrieve_from_database", return_value=["Test context"])
    @patch.object(BaseLlm, "get_answer_from_llm", return_value="Test answer")
@@ -49,16 +51,22 @@ class TestApp(unittest.TestCase):

        Also tests that a dry run does not change the history
        """
-        config = AppConfig(collect_metrics=False)
-        app = App(config=config)
-        first_answer = app.chat("Test query 1")
-        self.assertEqual(first_answer, "Test answer")
-        self.assertEqual(len(app.llm.history.splitlines()), 2)
-        history = app.llm.history
-        dry_run = app.chat("Test query 2", dry_run=True)
-        self.assertIn("History:", dry_run)
-        self.assertEqual(history, app.llm.history)
-        self.assertEqual(len(app.llm.history.splitlines()), 2)
+        with patch.object(ECChatMemory, "get_recent_memories") as mock_memory:
+            mock_message = ChatMessage()
+            mock_message.add_user_message("Test query 1")
+            mock_message.add_ai_message("Test answer")
+            mock_memory.return_value = [mock_message]
+
+            config = AppConfig(collect_metrics=False)
+            app = App(config=config)
+            first_answer = app.chat("Test query 1")
+            self.assertEqual(first_answer, "Test answer")
+            self.assertEqual(len(app.llm.history), 1)
+            history = app.llm.history
+            dry_run = app.chat("Test query 2", dry_run=True)
+            self.assertIn("History:", dry_run)
+            self.assertEqual(history, app.llm.history)
+            self.assertEqual(len(app.llm.history), 1)

    @patch("chromadb.api.models.Collection.Collection.add", MagicMock)
    def test_chat_with_where_in_params(self):
@@ -0,0 +1,60 @@
+from unittest.mock import MagicMock
+
+import psycopg
+import pytest
+
+from embedchain.loaders.postgres import PostgresLoader
+
+
+@pytest.fixture
+def postgres_loader(mocker):
+    with mocker.patch.object(psycopg, "connect"):
+        config = {"url": "postgres://user:password@localhost:5432/database"}
+        loader = PostgresLoader(config=config)
+        yield loader
+
+
+def test_postgres_loader_initialization(postgres_loader):
+    assert postgres_loader.connection is not None
+    assert postgres_loader.cursor is not None
+
+
+def test_postgres_loader_invalid_config():
+    with pytest.raises(ValueError, match="Must provide the valid config. Received: None"):
+        PostgresLoader(config=None)
+
+
+def test_load_data(postgres_loader, monkeypatch):
+    mock_cursor = MagicMock()
+    monkeypatch.setattr(postgres_loader, "cursor", mock_cursor)
+
+    query = "SELECT * FROM table"
+    mock_cursor.fetchall.return_value = [(1, "data1"), (2, "data2")]
+
+    result = postgres_loader.load_data(query)
+
+    assert "doc_id" in result
+    assert "data" in result
+    assert len(result["data"]) == 2
+    assert result["data"][0]["meta_data"]["url"] == f"postgres_query-({query})"
+    assert result["data"][1]["meta_data"]["url"] == f"postgres_query-({query})"
+    assert mock_cursor.execute.called_with(query)
+
+
+def test_load_data_exception(postgres_loader, monkeypatch):
+    mock_cursor = MagicMock()
+    monkeypatch.setattr(postgres_loader, "cursor", mock_cursor)
+
+    _ = "SELECT * FROM table"
+    mock_cursor.execute.side_effect = Exception("Mocked exception")
+
+    with pytest.raises(
+        ValueError, match=r"Failed to load data using query=SELECT \* FROM table with: Mocked exception"
+    ):
+        postgres_loader.load_data("SELECT * FROM table")
+
+
+def test_close_connection(postgres_loader):
+    postgres_loader.close_connection()
+    assert postgres_loader.cursor is None
+    assert postgres_loader.connection is None
@@ -0,0 +1,67 @@
+import pytest
+
+from embedchain.memory.base import ECChatMemory
+from embedchain.memory.message import ChatMessage
+
+
+# Fixture for creating an instance of ECChatMemory
+@pytest.fixture
+def chat_memory_instance():
+    return ECChatMemory()
+
+
+def test_add_chat_memory(chat_memory_instance):
+    app_id = "test_app"
+    human_message = "Hello, how are you?"
+    ai_message = "I'm fine, thank you!"
+
+    chat_message = ChatMessage()
+    chat_message.add_user_message(human_message)
+    chat_message.add_ai_message(ai_message)
+
+    chat_memory_instance.add(app_id, chat_message)
+
+    assert chat_memory_instance.count_history_messages(app_id) == 1
+    chat_memory_instance.delete_chat_history(app_id)
+
+
+def test_get_recent_memories(chat_memory_instance):
+    app_id = "test_app"
+
+    for i in range(1, 7):
+        human_message = f"Question {i}"
+        ai_message = f"Answer {i}"
+
+        chat_message = ChatMessage()
+        chat_message.add_user_message(human_message)
+        chat_message.add_ai_message(ai_message)
+
+        chat_memory_instance.add(app_id, chat_message)
+
+    recent_memories = chat_memory_instance.get_recent_memories(app_id, num_rounds=5)
+
+    assert len(recent_memories) == 5
+
+
+def test_delete_chat_history(chat_memory_instance):
+    app_id = "test_app"
+
+    for i in range(1, 6):
+        human_message = f"Question {i}"
+        ai_message = f"Answer {i}"
+
+        chat_message = ChatMessage()
+        chat_message.add_user_message(human_message)
+        chat_message.add_ai_message(ai_message)
+
+        chat_memory_instance.add(app_id, chat_message)
+
+    chat_memory_instance.delete_chat_history(app_id)
+
+    assert chat_memory_instance.count_history_messages(app_id) == 0
+
+
+@pytest.fixture
+def close_connection(chat_memory_instance):
+    yield
+    chat_memory_instance.close_connection()
@@ -0,0 +1,37 @@
+from embedchain.memory.message import BaseMessage, ChatMessage
+
+
+def test_ec_base_message():
+    content = "Hello, how are you?"
+    creator = "human"
+    metadata = {"key": "value"}
+
+    message = BaseMessage(content=content, creator=creator, metadata=metadata)
+
+    assert message.content == content
+    assert message.creator == creator
+    assert message.metadata == metadata
+    assert message.type is None
+    assert message.is_lc_serializable() is True
+    assert str(message) == f"{creator}: {content}"
+
+
+def test_ec_base_chat_message():
+    human_message_content = "Hello, how are you?"
+    ai_message_content = "I'm fine, thank you!"
+    human_metadata = {"user": "John"}
+    ai_metadata = {"response_time": 0.5}
+
+    chat_message = ChatMessage()
+    chat_message.add_user_message(human_message_content, metadata=human_metadata)
+    chat_message.add_ai_message(ai_message_content, metadata=ai_metadata)
+
+    assert chat_message.human_message.content == human_message_content
+    assert chat_message.human_message.creator == "human"
+    assert chat_message.human_message.metadata == human_metadata
+
+    assert chat_message.ai_message.content == ai_message_content
+    assert chat_message.ai_message.creator == "ai"
+    assert chat_message.ai_message.metadata == ai_metadata
+
+    assert str(chat_message) == f"human: {human_message_content}\nai: {ai_message_content}"
@@ -0,0 +1,36 @@
+import yaml
+
+from embedchain.utils import validate_yaml_config
+
+CONFIG_YAMLS = [
+    "configs/anthropic.yaml",
+    "configs/azure_openai.yaml",
+    "configs/chroma.yaml",
+    "configs/chunker.yaml",
+    "configs/cohere.yaml",
+    "configs/full-stack.yaml",
+    "configs/gpt4all.yaml",
+    "configs/huggingface.yaml",
+    "configs/jina.yaml",
+    "configs/llama2.yaml",
+    "configs/opensearch.yaml",
+    "configs/opensource.yaml",
+    "configs/pinecone.yaml",
+    "configs/vertexai.yaml",
+    "configs/weaviate.yaml",
+]
+
+
+class TestAllConfigYamls:
+    def test_all_config_yamls(self):
+        """Test that all config yamls are valid."""
+        for config_yaml in CONFIG_YAMLS:
+            with open(config_yaml, "r") as f:
+                config = yaml.safe_load(f)
+            assert config is not None
+
+            try:
+                validate_yaml_config(config)
+            except Exception as e:
+                print(f"Error in {config_yaml}: {e}")
+                raise e
Author	SHA1	Message	Date
Deshraj YadavandGitHub	1364975396	[Feature] Add support for AIAssistant (#938 )	2023-11-10 16:47:34 -08:00
Deven Patel GitHub Deven Patel	deaa7f50f8	[Bug Fix] fix chromadb where clause for query and delete (#937 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-10 16:04:25 -08:00
Sidharth MohantyandGitHub	744ab5156f	[Bug fix] missing dir on first init for App (#934 )	2023-11-10 10:15:04 -08:00
Sidharth MohantyandGitHub	c45413969a	[refactor] Use pipeline for bots instead of App (#936 )	2023-11-10 10:13:21 -08:00
Deshraj YadavandGitHub	b314e5e080	[bug] Fix issue of missing user directory on first init (#931 )	2023-11-09 22:29:41 -08:00
Deshraj YadavandGitHub	17129e2eaa	[Improvement] Add support for reloading history for an existing app (#930 )	2023-11-09 15:17:51 -08:00
Deven Patel GitHub Deven Patel	654fd8d74c	[Improvement] Use SQLite for chat memory (#910 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-09 13:56:28 -08:00
Sidharth MohantyandGitHub	9d3568ef75	Update package version to 0.1.3 (#928 )	2023-11-09 11:33:18 -08:00
Sidharth MohantyandGitHub	14712cac88	Deploy remaining bots and fix schema validation (#927 )	2023-11-09 10:44:47 -08:00
Deshraj YadavandGitHub	0d568c758b	[Feat] Add anonymous telemetry to assistant (#924 )	2023-11-09 02:07:47 -08:00
Deshraj YadavandGitHub	7c6b88c7c5	[Docs] Update docs and improve assistant api (#923 )	2023-11-09 01:16:19 -08:00
Deshraj YadavandGitHub	32c93be46e	[chore] update poetry.lock file (#922 )	2023-11-09 00:59:29 -08:00
Deven Patel GitHub Deven Patel	7de8d85199	[Feature] Add Postgres data loader (#918 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-08 23:50:46 -08:00