Bump version to 0.0.38 (#442 )

feat: session id for telemetry (#440 )
fix: notion install error message (#439 )
2023-08-15 03:01:11 +05:30 · 2023-08-15 02:58:09 +05:30 · 2023-08-15 02:57:23 +05:30 · 2023-08-12 06:00:21 +05:30 · 2023-08-12 05:52:27 +05:30 · 2023-08-12 05:32:40 +05:30
126 changed files with 14741 additions and 146 deletions
@@ -1,7 +1,7 @@
 # embedchain

 [![PyPI](https://img.shields.io/pypi/v/embedchain)](https://pypi.org/project/embedchain/)
-[![Discord](https://dcbadge.vercel.app/api/server/nhvCbCtKV?style=flat)](https://discord.gg/6PzXDgEjG5)
+[![Discord](https://dcbadge.vercel.app/api/server/6PzXDgEjG5?style=flat)](https://discord.gg/6PzXDgEjG5)
 [![Twitter](https://img.shields.io/twitter/follow/embedchain)](https://twitter.com/embedchain)
 [![Substack](https://img.shields.io/badge/Substack-%23006f5c.svg?logo=substack)](https://embedchain.substack.com/)
 [![Open in Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/138lMWhENGeEu7Q1-6lNbNTHGLZXBBz_B?usp=sharing)
@@ -85,7 +85,7 @@ If you utilize this repository, please consider citing it with:
 ```
@misc{embedchain,
  author = {Taranjeet Singh},
-  title = {Embechain: Framework to easily create LLM powered bots over any dataset},
+  title = {Embedchain: Framework to easily create LLM powered bots over any dataset},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
@@ -85,11 +85,13 @@ app = CustomApp(config)
    - ANTHPROPIC
    - VERTEX_AI
    - GPT4ALL
+    - AZURE_OPENAI
 - Following embedding functions are available for an embedding function
    - OPENAI
    - HUGGING_FACE
    - VERTEX_AI
    - GPT4ALL
+    - AZURE_OPENAI


 ### PersonApp
@@ -20,6 +20,10 @@ from chromadb.utils import embedding_functions
 config = AppConfig(log_level="DEBUG")
 naval_chat_bot = App(config)

+# Example: specify a custom collection name
+config = AppConfig(collection_name="naval_chat_bot")
+naval_chat_bot = App(config)
+
 # Example: define your own chunker config for `youtube_video`
 chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=100, length_function=len)
 naval_chat_bot.add("youtube_video", "https://www.youtube.com/watch?v=3qHkcs3kG44", AddConfig(chunker=chunker_config))
@@ -64,7 +68,7 @@ einstein_chat_template = Template("""

        Human: $query
        Albert Einstein:""")
-query_config = QueryConfig(einstein_chat_template)
+query_config = QueryConfig(template=einstein_chat_template)
 queries = [
        "Where did you complete your studies?",
        "Why did you win nobel prize?",
@@ -54,6 +54,18 @@ To add any code documentation website as a loader, use the data_type as `docs_si
 app.add("docs_site", "https://docs.embedchain.ai/")
 ```

+### Notion
+To use notion you must install the extra dependencies with `pip install embedchain[notion]`.
+
+To load a notion page, use the data_type as `notion`.
+The next argument must **end** with the `notion page id`. The id is a 32-character string. Eg:
+
+```python
+app.add("notion", "cfbc134ca6464fc980d0391613959196")
+app.add("notion", "my-page-cfbc134ca6464fc980d0391613959196")
+app.add("notion", "https://www.notion.so/my-page-cfbc134ca6464fc980d0391613959196")
+```
+
 ### Text

 To supply your own text, use the data_type as `text` and enter a string. The text is not processed, this can be very versatile. Eg:
@@ -4,11 +4,13 @@ title: '🔍 Query configurations'

 ## AppConfig

-| option      | description           | type                            | default                |
-|-------------|-----------------------|---------------------------------|------------------------|
-| log_level   | log level             | string                          | WARNING                |
+| option    | description           | type                            | default                |
+|-----------|-----------------------|---------------------------------|------------------------|
+| log_level | log level             | string                          | WARNING                |
 | embedding_fn| embedding function    | chromadb.utils.embedding_functions | \{text-embedding-ada-002\} |
-| db          | vector database (experimental) | BaseVectorDB               | ChromaDB               |
+| db        | vector database (experimental) | BaseVectorDB               | ChromaDB               |
+| collection_name | initial collection name for the database | string             | embedchain_store |
+| collect_metrics | collect anonymous telemetry data to improve embedchain | boolean | true |


 ## AddConfig
@@ -45,6 +47,7 @@ Default values of chunker config parameters for different `data_type`:
 |pdf_file|1000|0|len|
 |youtube_video|2000|0|len|
 |docs_site|500|50|len|
+|notion|300|0|len|

 ### LoaderConfig

@@ -54,9 +57,14 @@ _coming soon_

 |option|description|type|default|
 |---|---|---|---|
-|template|custom template for prompt|Template|Template("Use the following pieces of context to answer the query at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. \$context Query: \$query Helpful Answer:")|
-|history|include conversation history from your client or database|any (recommendation: list[str])|None
-|stream|control if response is streamed back to the user|bool|False|
+|number_documents|Absolute number of documents to pull from the database as context.|int|1
+|template|custom template for prompt. If history is used with query, $history has to be included as well.|Template|Template("Use the following pieces of context to answer the query at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. \$context Query: \$query Helpful Answer:")|
+|model|name of the model used.|string|depends on app type|
+|temperature|Controls the randomness of the model's output. Higher values (closer to 1) make output more random, lower values make it more deterministic.|float|0|
+|max_tokens|Controls how many tokens are used. Exact implementation (whether it counts prompt and/or response) depends on the model.|int|1000|
+|top_p|Controls the diversity of words. Higher values (closer to 1) make word selection more diverse, lower values make words less diverse.|float|1|
+|history|include conversation history from your client or database.|any (recommendation: list[str])|None|
+|stream|control if response is streamed back to the user.|bool|False|

 ## ChatConfig

@@ -64,4 +72,4 @@ All options for query and...

 _coming soon_

-History is handled automatically, the config option is not supported.
+`history` is not supported, as that is handled is handled automatically, the config option is not supported.
@@ -10,35 +10,58 @@ Embedchain community has been super active in creating demos on top of Embedchai

 - [Discord Bot for LLM chat](https://github.com/Reidond/discord_bots_playground/tree/c8b0c36541e4b393782ee506804c4b6962426dd6/python/chat-channel-bot) by Reidond
 - [EmbedChain-Streamlit-Docker App](https://github.com/amjadraza/embedchain-streamlit-app) by amjadraza
- [Harry Potter Philosphers Stone Bot](https://github.com/vinayak-kempawad/Harry_Potter_Philosphers_Stone_Bot/) by Vinayak Kempawad, ([linkedin post](https://www.linkedin.com/feed/update/urn:li:activity:7080907532155686912/))
+- [Harry Potter Philosphers Stone Bot](https://github.com/vinayak-kempawad/Harry_Potter_Philosphers_Stone_Bot/) by Vinayak Kempawad, ([LinkedIn post](https://www.linkedin.com/feed/update/urn:li:activity:7080907532155686912/))
 - [LLM bot trained on own messages](https://github.com/Harin329/harinBot) by Hao Wu

 ### Closed Source

 - [Taobot.io](https://taobot.io) - chatbot & knowledgebase hybrid by [cachho](https://github.com/cachho)
+- [Create Instant ChatBot 🤖 using embedchain](https://databutton.com/v/h3e680h9) by Avra, ([Tweet](https://twitter.com/Avra_b/status/1674704745154641920/))
+- [JOBO 🤖 — The AI-driven sidekick to craft your resume](https://try-jobo.com/) by Enrico Willemse, ([LinkedIn Post](https://www.linkedin.com/posts/enrico-willemse_jobai-gptfun-embedchain-activity-7090340080879374336-ueLB/))
+- [Explore Your Knowledge Base: Interactive chats over various forms of documents](https://chatdocs.dkedar.com/) by Kedar Dabhadkar, ([LinkedIn Post](https://www.linkedin.com/posts/dkedar7_machinelearning-llmops-activity-7092524836639424513-2O3L/))
+- [Chatbot trained on 1000+ videos of Ester hicks the co-author behind the famous book Secret](https://ask-abraham.thoughtseed.repl.co) by Mohan Kumar
+

 ## Templates

 ### Replit
 - [Embedchain Chat Bot](https://replit.com/@taranjeet1/Embedchain-Chat-Bot) by taranjeetio
 - [Embedchain Memory Chat Bot Template](https://replit.com/@taranjeetio/Embedchain-Memory-Chat-Bot-Template) by taranjeetio
+- [Chatbot app to demonstrate question-answering using retrieved information](https://replit.com/@AllisonMorrell/EmbedChainlitPublic) by Allison Morrell, ([LinkedIn Post](https://www.linkedin.com/posts/allison-morrell-2889275a_retrievalbot-screenshots-activity-7080339991754649600-wihZ/))

 ## Posts

 ### Blogs

- [Customer Service LINE Bot](https://www.evanlin.com/langchain-embedchain/)
+- [Customer Service LINE Bot](https://www.evanlin.com/langchain-embedchain/) by Evan Lin
+- [Chatbot in Under 5 mins using Embedchain](https://medium.com/@ayush.wattal/chatbot-in-under-5-mins-using-embedchain-a4f161fcf9c5) by Ayush Wattal
+- [Understanding what the LLM framework embedchain does](https://zenn.dev/hijikix/articles/4bc8d60156a436) by Daisuke Hashimoto
+- [In bed with GPT and Node.js](https://dev.to/worldlinetech/in-bed-with-gpt-and-nodejs-4kh2) by Raphaël Semeteys, ([LinkedIn Post](https://www.linkedin.com/posts/raphaelsemeteys_in-bed-with-gpt-and-nodejs-activity-7088113552326029313-nn87/))
+- [Using Embedchain — A powerful LangChain Python wrapper to build Chat Bots even faster!⚡](https://medium.com/@avra42/using-embedchain-a-powerful-langchain-python-wrapper-to-build-chat-bots-even-faster-35c12994a360) by Avra, ([Tweet](https://twitter.com/Avra_b/status/1686767751560310784/))

 ### LinkedIn

 - [What is embedchain](https://www.linkedin.com/posts/activity-7079393104423698432-wRyi/) by Rithesh Sreenivasan
 - [Building a chatbot with EmbedChain](https://www.linkedin.com/posts/activity-7078434598984060928-Zdso/) by Lior Sinclair
 - [Making chatbot without vs with embedchain](https://www.linkedin.com/posts/kalyanksnlp_llms-chatbots-langchain-activity-7077453416221863936-7N1L/) by Kalyan KS
+- [EmbedChain - very intuitive, first you index your data and then query!](https://www.linkedin.com/posts/shubhamsaboo_embedchain-a-framework-to-easily-create-activity-7079535460699557888-ad1X/) by Shubham Saboo
+- [EmbedChain - Harnessing power of LLM](https://www.linkedin.com/posts/uditsaini_chatbotrevolution-llmpoweredbots-embedchainframework-activity-7077520356827181056-FjTK/) by Udit S.
+- [AI assistant for ABBYY Vantage](https://www.linkedin.com/posts/maximevermeir_llm-github-abbyy-activity-7081658972071424000-fXfZ/) by Maxime V.
+- [About embedchain](https://www.linkedin.com/feed/update/urn:li:activity:7080984218914189312/) by Morris Lee
+- [How to use Embedchain](https://www.linkedin.com/posts/nehaabansal_github-embedchainembedchain-framework-activity-7085830340136595456-kbW5/) by Neha Bansal
+- [Youtube/Webpage summary for Energy Study](https://www.linkedin.com/posts/bar%C4%B1%C5%9F-sanl%C4%B1-34b82715_enerji-python-activity-7082735341563977730-Js0U/) by Barış Sanlı, ([Tweet](https://twitter.com/barissanli/status/1676968784979193857/)) 

 ### Twitter

 - [What is embedchain](https://twitter.com/AlphaSignalAI/status/1672668574450847745) by Lior
 - [Building a chatbot with Embedchain](https://twitter.com/Saboo_Shubham_/status/1673537044419686401) by Shubham Saboo
+- [Chatbot docker image behind an API with yaml configs with Embedchain](https://twitter.com/tricalt/status/1678411430192730113/) by Vasilije
+- [Build AI powered PDF chatbot with just five lines of Python code with Embedchain!](https://twitter.com/Saboo_Shubham_/status/1676627104866156544/) by Shubham Saboo
+- [Chatbot against a youtube video using embedchain](https://twitter.com/smaameri/status/1675201443043704834/) by Sami Maameri
+- [Highlights of EmbedChain](https://twitter.com/carl_AIwarts/status/1673542204328120321/) by carl_AIwarts
+- [Build Llama-2 chatbot in less than 5 minutes](https://twitter.com/Saboo_Shubham_/status/1682168956918833152/) by Shubham Saboo
+- [All cool features of embedchain](https://twitter.com/DhravyaShah/status/1683497882438217728/) by Dhravya Shah, ([LinkedIn Post](https://www.linkedin.com/posts/dhravyashah_what-if-i-tell-you-that-you-can-make-an-ai-activity-7089459599287726080-ZIYm/))
+- [Read paid Medium articles for Free using embedchain](https://twitter.com/kumarkaushal_/status/1688952961622585344) by Kaushal Kumar

 ## Videos

@@ -49,6 +72,13 @@ Embedchain community has been super active in creating demos on top of Embedchai
 - [How To Create A Custom Knowledge AI Powered Bot | Install + How To Use](https://www.youtube.com/watch?v=VfCrIiAst-c) by The Ai Solopreneur
 - [Build Custom Chatbot in 6 min with this Framework [Beginner Friendly]](https://www.youtube.com/watch?v=-8HxOpaFySM) by Maya Akim
 - [embedchain-streamlit-app](https://www.youtube.com/watch?v=3-9GVd-3v74) by Amjad Raza
+- [🤖CHAT with ANY ONLINE RESOURCES using EMBEDCHAIN - a LangChain wrapper, in few lines of code !](https://www.youtube.com/watch?v=Mp7zJe4TIdM) by Avra
+- [Building resource-driven LLM-powered bots with Embedchain](https://www.youtube.com/watch?v=IVfcAgxTO4I) by BugBytes
+- [embedchain-streamlit-demo](https://www.youtube.com/watch?v=yJAWB13FhYQ) by Amjad Raza
+- [Embedchain - create your own AI chatbots using open source models](https://www.youtube.com/shorts/O3rJWKwSrWE) by Dhravya Shah
+- [AI ChatBot in 5 lines Python Code](https://www.youtube.com/watch?v=zjWvLJLksv8) by Data Engineering
+- [Interview with Karl Marx](https://www.youtube.com/watch?v=5Y4Tscwj1xk) by Alexander Ray Williams
+- [Vlog where we try to build a bot based on our content on the internet](https://www.youtube.com/watch?v=I2w8CWM3bx4) by DV, ([Tweet](https://twitter.com/dvcoolster/status/1688387017544261632))

 ## Mentions

@@ -0,0 +1,34 @@
+---
+title: '💾 Vector Database'
+---
+
+We support `Chroma` and `Elasticsearch` as two vector database. 
+`Chroma` is used as a default database.
+
+### Elasticsearch
+In order to use `Elasticsearch` as vector database we need to use App type `CustomApp`. 
+```python
+import os
+from embedchain import CustomApp
+from embedchain.config import CustomAppConfig, ElasticsearchDBConfig
+from embedchain.models import Providers, EmbeddingFunctions, VectorDatabases
+
+os.environ["OPENAI_API_KEY"] = 'OPENAI_API_KEY'
+
+es_config = ElasticsearchDBConfig(
+	# elasticsearch url or list of nodes url with different hosts and ports.
+	es_url='http://localhost:9200',
+	# pass named parameters supported by Python Elasticsearch client
+	ca_certs="/path/to/http_ca.crt",
+	basic_auth=("username", "password")
+)
+config = CustomAppConfig(
+	embedding_fn=EmbeddingFunctions.OPENAI, 
+	provider=Providers.OPENAI, 
+	db_type=VectorDatabases.ELASTICSEARCH, 
+	es_config=es_config,
+)
+es_app = CustomApp(config)
+```
+- Set `db_type=VectorDatabases.ELASTICSEARCH` and `es_config=ElasticsearchDBConfig(es_url='')` in `CustomAppConfig`.
+- `ElasticsearchDBConfig` accepts `es_url` as elasticsearch url or as list of nodes url with different hosts and ports. Additionally we can pass named paramaters supported by Python Elasticsearch client.
@@ -0,0 +1,46 @@
+---
+title: '🌍 API Server'
+---
+
+### 🐳 Docker Setup
+
+- Open variables.env, and edit it to add your 🔑 `OPENAI_API_KEY`.
+- To setup your api server using docker, run the following command inside this folder using your terminal.
+
+```bash
+docker-compose up --build
+```
+
+📝 Note: The build command might take a while to install all the packages depending on your system resources.
+
+### 🚀 Usage Instructions
+
+- Your api server is running on [http://localhost:5000/](http://localhost:5000/)
+- To use the api server, make an api call to the endpoints `/add` and `/query` using the json formats discussed below.
+- To add data sources to the bot:
+```json
+// Request
+{
+  "data_type": "your_data_type_here",
+  "url_or_text": "your_url_or_text_here"
+}
+
+// Response
+{
+  "data": "Added data_type: url_or_text"
+}
+```
+- To ask questions from the bot:
+```json
+// Request
+{
+  "question": "your_question_here"
+}
+
+// Response
+{
+  "data": "your_answer_here"
+}
+```
+
+🎉 Happy Chatting! 🎉
@@ -0,0 +1,48 @@
+---
+title: '🤖 Discord Bot'
+---
+
+### 🔑 Keys Setup
+
+- Set your `OPENAI_API_KEY` in your variables.env file.
+- Go to [https://discord.com/developers/applications/](https://discord.com/developers/applications/) and click on `New Application`.
+- Enter the name for your bot, accept the terms and click on `Create`. On the resulting page, enter the details of your bot as you like.
+- On the left sidebar, click on `Bot`. Under the heading `Privileged Gateway Intents`, toggle all 3 options to ON position. Save your changes.
+- Now click on `Reset Token` and copy the token value. Set it as `DISCORD_BOT_TOKEN` in variables.env file.
+- On the left sidebar, click on `OAuth2` and go to `General`.
+- Set `Authorization Method` to `In-app Authorization`. Under `Scopes` select `bot`.
+- Under `Bot Permissions` allow the following and then click on `Save Changes`.
+```text
+Read Messages/View Channel (under General Permissions)
+Send Messages (under Text Permissions)
+Read Message History (under Text Permissions)
+Mention everyone (under Text Permissions)
+```
+- Now under `OAuth2` and go to `URL Generator`. Under `Scopes` select `bot`.
+- Under `Bot Permissions` set the same permissions as above.
+- Now scroll down and copy the `Generated URL`. Paste it in a browser window and select the Server where you want to add the bot.
+- Click on `Continue` and authorize the bot.
+- 🎉 The bot has been successfully added to your server.
+
+### 🐳 Docker Setup
+
+- To setup your discord bot using docker, run the following command inside this folder using your terminal.
+```bash
+docker-compose up --build
+```
+📝 Note: The build command might take a while to install all the packages depending on your system resources.
+
+### 🚀 Usage Instructions
+
+- Go to the server where you have added your bot.
+- You can add data sources to the bot using the command:
+```text
+/ec add <data_type> <url_or_text>
+```
+- You can ask your queries from the bot using the command:
+```text
+/ec query <question>
+```
+📝 Note: To use the bot privately, you can message the bot directly by right clicking the bot and selecting `Message`.
+
+🎉 Happy Chatting! 🎉
@@ -0,0 +1,22 @@
+---
+title: '🌐 Full Stack'
+---
+
+### 🐳 Docker Setup
+
+- To setup full stack app using docker, run the following command inside this folder using your terminal.
+
+```bash
+docker-compose up --build
+```
+
+📝 Note: The build command might take a while to install all the packages depending on your system resources.
+
+### 🚀 Usage Instructions
+
+- Go to [http://localhost:3000/](http://localhost:3000/) in your browser to view the dashboard.
+- Add your `OpenAI API key` 🔑 in the Settings.
+- Create a new bot and you'll be navigated to its page.
+- Here you can add your data sources and then chat with the bot.
+
+🎉 Happy Chatting! 🎉
@@ -32,7 +32,11 @@
    },
    {
      "group": "Advanced",
-      "pages": ["advanced/app_types", "advanced/interface_types", "advanced/adding_data","advanced/data_types", "advanced/query_configuration", "advanced/configuration", "advanced/testing", "advanced/showcase"]
+      "pages": ["advanced/app_types", "advanced/interface_types", "advanced/adding_data","advanced/data_types", "advanced/query_configuration", "advanced/configuration", "advanced/testing", "advanced/vector_database", "advanced/showcase"]
+    },
+    {
+      "group": "Examples",
+      "pages": ["examples/full_stack", "examples/api_server", "examples/discord_bot"]
    },
    {
      "group": "Contribution Guidelines",
@@ -20,7 +20,7 @@ Run your first bot in python using the following code. Make sure to set the `OPE
 ```python
 import os

-from embedchain Import App
+from embedchain import App

 os.environ["OPENAI_API_KEY"] = "xxx"
 elon_musk_bot = App()
@@ -29,7 +29,7 @@ elon_musk_bot = App()
 elon_musk_bot.add("web_page", "https://en.wikipedia.org/wiki/Elon_Musk")
 elon_musk_bot.add("web_page", "https://www.tesla.com/elon-musk")

-response = elon_bot.query("How many companies does Elon Musk run?")
+response = elon_musk_bot.query("How many companies does Elon Musk run?")
 print(response)
 # Answer: 'Elon Musk runs four companies: Tesla, SpaceX, Neuralink, and The Boring Company.'
 ```
@@ -64,6 +64,9 @@ class CustomApp(EmbedChain):
            if self.provider == Providers.GPT4ALL:
                return self.open_source_app._get_gpt4all_answer(prompt, config)

+            if self.provider == Providers.AZURE_OPENAI:
+                return CustomApp._get_azure_openai_answer(prompt, config)
+
        except ImportError as e:
            raise ImportError(e.msg) from None

@@ -71,8 +74,6 @@ class CustomApp(EmbedChain):
    def _get_openai_answer(prompt: str, config: ChatConfig) -> str:
        from langchain.chat_models import ChatOpenAI

-        logging.info(vars(config))
-
        chat = ChatOpenAI(
            temperature=config.temperature,
            model=config.model or "gpt-3.5-turbo",
@@ -113,6 +114,29 @@ class CustomApp(EmbedChain):

        return chat(messages).content

+    @staticmethod
+    def _get_azure_openai_answer(prompt: str, config: ChatConfig) -> str:
+        from langchain.chat_models import AzureChatOpenAI
+
+        if not config.deployment_name:
+            raise ValueError("Deployment name must be provided for Azure OpenAI")
+
+        chat = AzureChatOpenAI(
+            deployment_name=config.deployment_name,
+            openai_api_version="2023-05-15",
+            model_name=config.model or "gpt-3.5-turbo",
+            temperature=config.temperature,
+            max_tokens=config.max_tokens,
+            streaming=config.stream,
+        )
+
+        if config.top_p and config.top_p != 1:
+            logging.warning("Config option `top_p` is not supported by this model.")
+
+        messages = CustomApp._get_messages(prompt)
+
+        return chat(messages).content
+
    @staticmethod
    def _get_messages(prompt: str) -> List[BaseMessage]:
        from langchain.schema import HumanMessage, SystemMessage
@@ -22,6 +22,33 @@ class EmbedChainPersonApp:
        self.person_prompt = f"You are {person}. Whatever you say, you will always say in {person} style."  # noqa:E501
        super().__init__(config)

+    def add_person_template_to_config(self, default_prompt: str, config: ChatConfig = None):
+        """
+        This method checks if the config object contains a prompt template
+        if yes it adds the person prompt to it and return the updated config
+        else it creates a config object with the default prompt added to the person prompt
+
+        :param default_prompt: it is the default prompt for query or chat methods
+        :param config: Optional. The `ChatConfig` instance to use as
+        configuration options.
+        """
+        template = Template(self.person_prompt + " " + default_prompt)
+
+        if config:
+            if config.template:
+                # Add person prompt to custom user template
+                config.template = Template(self.person_prompt + " " + config.template.template)
+            else:
+                # If no user template is present, use person prompt with the default template
+                config.template = template
+        else:
+            # if no config is present at all, initialize the config with person prompt and default template
+            config = QueryConfig(
+                template=template,
+            )
+
+        return config
+

 class PersonApp(EmbedChainPersonApp, App):
    """
@@ -29,19 +56,13 @@ class PersonApp(EmbedChainPersonApp, App):
    Extends functionality from EmbedChainPersonApp and App
    """

-    def query(self, input_query, config: QueryConfig = None):
-        self.template = Template(self.person_prompt + " " + DEFAULT_PROMPT)
-        query_config = QueryConfig(
-            template=self.template,
-        )
-        return super().query(input_query, query_config)
+    def query(self, input_query, config: QueryConfig = None, dry_run=False):
+        config = self.add_person_template_to_config(DEFAULT_PROMPT, config)
+        return super().query(input_query, config, dry_run)

-    def chat(self, input_query, config: ChatConfig = None):
-        self.template = Template(self.person_prompt + " " + DEFAULT_PROMPT_WITH_HISTORY)
-        chat_config = ChatConfig(
-            template=self.template,
-        )
-        return super().chat(input_query, chat_config)
+    def chat(self, input_query, config: ChatConfig = None, dry_run=False):
+        config = self.add_person_template_to_config(DEFAULT_PROMPT_WITH_HISTORY, config)
+        return super().chat(input_query, config, dry_run)


 class PersonOpenSourceApp(EmbedChainPersonApp, OpenSourceApp):
@@ -50,14 +71,10 @@ class PersonOpenSourceApp(EmbedChainPersonApp, OpenSourceApp):
    Extends functionality from EmbedChainPersonApp and OpenSourceApp
    """

-    def query(self, input_query, config: QueryConfig = None):
-        query_config = QueryConfig(
-            template=self.template,
-        )
-        return super().query(input_query, query_config)
+    def query(self, input_query, config: QueryConfig = None, dry_run=False):
+        config = self.add_person_template_to_config(DEFAULT_PROMPT, config)
+        return super().query(input_query, config, dry_run)

-    def chat(self, input_query, config: ChatConfig = None):
-        chat_config = ChatConfig(
-            template=self.template,
-        )
-        return super().chat(input_query, chat_config)
+    def chat(self, input_query, config: ChatConfig = None, dry_run=False):
+        config = self.add_person_template_to_config(DEFAULT_PROMPT_WITH_HISTORY, config)
+        return super().chat(input_query, config, dry_run)
@@ -0,0 +1,20 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.AddConfig import ChunkerConfig
+
+
+class NotionChunker(BaseChunker):
+    """Chunker for notion."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=300, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -33,6 +33,7 @@ class ChatConfig(QueryConfig):
        max_tokens=None,
        top_p=None,
        stream: bool = False,
+        deployment_name=None,
    ):
        """
        Initializes the ChatConfig instance.
@@ -68,6 +69,7 @@ class ChatConfig(QueryConfig):
            top_p=top_p,
            history=[0],
            stream=stream,
+            deployment_name=deployment_name,
        )

    def set_history(self, history):
@@ -62,6 +62,7 @@ class QueryConfig(BaseConfig):
        top_p=None,
        history=None,
        stream: bool = False,
+        deployment_name=None,
    ):
        """
        Initializes the QueryConfig instance.
@@ -106,6 +107,7 @@ class QueryConfig(BaseConfig):
        self.max_tokens = max_tokens if max_tokens else 1000
        self.model = model
        self.top_p = top_p if top_p else 1
+        self.deployment_name = deployment_name

        if self.validate_template(template):
            self.template = template
@@ -5,3 +5,5 @@ from .apps.OpenSourceAppConfig import OpenSourceAppConfig  # noqa: F401
 from .BaseConfig import BaseConfig  # noqa: F401
 from .ChatConfig import ChatConfig  # noqa: F401
 from .QueryConfig import QueryConfig  # noqa: F401
+from .vectordbs.ElasticsearchDBConfig import \
+    ElasticsearchDBConfig  # noqa: F401
@@ -1,6 +1,13 @@
 import os
+from typing import Optional

-from chromadb.utils import embedding_functions
+try:
+    from chromadb.utils import embedding_functions
+except RuntimeError:
+    from embedchain.utils import use_pysqlite3
+
+    use_pysqlite3()
+    from chromadb.utils import embedding_functions

 from .BaseAppConfig import BaseAppConfig

@@ -10,16 +17,32 @@ class AppConfig(BaseAppConfig):
    Config to initialize an embedchain custom `App` instance, with extra config options.
    """

-    def __init__(self, log_level=None, host=None, port=None, id=None):
+    def __init__(
+        self,
+        log_level=None,
+        host=None,
+        port=None,
+        id=None,
+        collection_name=None,
+        collect_metrics: Optional[bool] = None,
+    ):
        """
        :param log_level: Optional. (String) Debug level
        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
-        :param id: Optional. ID of the app. Document metadata will have this id.
        :param host: Optional. Hostname for the database server.
        :param port: Optional. Port for the database server.
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param collection_name: Optional. Collection name for the database.
+        :param collect_metrics: Defaults to True. Send anonymous telemetry to improve embedchain.
        """
        super().__init__(
-            log_level=log_level, embedding_fn=AppConfig.default_embedding_function(), host=host, port=port, id=id
+            log_level=log_level,
+            embedding_fn=AppConfig.default_embedding_function(),
+            host=host,
+            port=port,
+            id=id,
+            collection_name=collection_name,
+            collect_metrics=collect_metrics,
        )

    @staticmethod
@@ -1,6 +1,8 @@
 import logging

 from embedchain.config.BaseConfig import BaseConfig
+from embedchain.config.vectordbs import ElasticsearchDBConfig
+from embedchain.models import VectorDatabases, VectorDimensions


 class BaseAppConfig(BaseConfig):
@@ -8,35 +10,78 @@ class BaseAppConfig(BaseConfig):
    Parent config to initialize an instance of `App`, `OpenSourceApp` or `CustomApp`.
    """

-    def __init__(self, log_level=None, embedding_fn=None, db=None, host=None, port=None, id=None):
+    def __init__(
+        self,
+        log_level=None,
+        embedding_fn=None,
+        db=None,
+        host=None,
+        port=None,
+        id=None,
+        collection_name=None,
+        collect_metrics: bool = True,
+        db_type: VectorDatabases = None,
+        vector_dim: VectorDimensions = None,
+        es_config: ElasticsearchDBConfig = None,
+    ):
        """
        :param log_level: Optional. (String) Debug level
        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
        :param embedding_fn: Embedding function to use.
        :param db: Optional. (Vector) database instance to use for embeddings.
-        :param id: Optional. ID of the app. Document metadata will have this id.
        :param host: Optional. Hostname for the database server.
        :param port: Optional. Port for the database server.
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param collection_name: Optional. Collection name for the database.
+        :param collect_metrics: Defaults to True. Send anonymous telemetry to improve embedchain.
+        :param db_type: Optional. type of Vector database to use
+        :param vector_dim: Vector dimension generated by embedding fn
+        :param es_config: Optional. elasticsearch database config to be used for connection
        """
        self._setup_logging(log_level)
-
-        self.db = db if db else BaseAppConfig.default_db(embedding_fn=embedding_fn, host=host, port=port)
+        self.collection_name = collection_name if collection_name else "embedchain_store"
+        self.db = BaseAppConfig.get_db(
+            db=db,
+            embedding_fn=embedding_fn,
+            host=host,
+            port=port,
+            db_type=db_type,
+            vector_dim=vector_dim,
+            collection_name=self.collection_name,
+            es_config=es_config,
+        )
        self.id = id
+        self.collect_metrics = True if (collect_metrics is True or collect_metrics is None) else False
        return

    @staticmethod
-    def default_db(embedding_fn, host, port):
+    def get_db(db, embedding_fn, host, port, db_type, vector_dim, collection_name, es_config):
        """
-        Sets database to default (`ChromaDb`).
-
+        Get db based on db_type, db with default database (`ChromaDb`)
+        :param Optional. (Vector) database to use for embeddings.
        :param embedding_fn: Embedding function to use in database.
        :param host: Optional. Hostname for the database server.
        :param port: Optional. Port for the database server.
-        :returns: Default database
+        :param db_type: Optional. db type to use. Supported values (`es`, `chroma`)
+        :param vector_dim: Vector dimension generated by embedding fn
+        :param collection_name: Optional. Collection name for the database.
+        :param es_config: Optional. elasticsearch database config to be used for connection
        :raises ValueError: BaseAppConfig knows no default embedding function.
+        :returns: database instance
        """
+        if db:
+            return db
+
        if embedding_fn is None:
            raise ValueError("ChromaDb cannot be instantiated without an embedding function")
+
+        if db_type == VectorDatabases.ELASTICSEARCH:
+            from embedchain.vectordb.elasticsearch_db import ElasticsearchDB
+
+            return ElasticsearchDB(
+                embedding_fn=embedding_fn, vector_dim=vector_dim, collection_name=collection_name, es_config=es_config
+            )
+
        from embedchain.vectordb.chroma_db import ChromaDB

        return ChromaDB(embedding_fn=embedding_fn, host=host, port=port)
@@ -1,9 +1,11 @@
-from typing import Any
+from typing import Any, Optional

 from chromadb.api.types import Documents, Embeddings
 from dotenv import load_dotenv

-from embedchain.models import EmbeddingFunctions, Providers
+from embedchain.config.vectordbs import ElasticsearchDBConfig
+from embedchain.models import (EmbeddingFunctions, Providers, VectorDatabases,
+                               VectorDimensions)

 from .BaseAppConfig import BaseAppConfig

@@ -24,9 +26,13 @@ class CustomAppConfig(BaseAppConfig):
        host=None,
        port=None,
        id=None,
+        collection_name=None,
        provider: Providers = None,
-        model=None,
        open_source_app_config=None,
+        deployment_name=None,
+        collect_metrics: Optional[bool] = None,
+        db_type: VectorDatabases = None,
+        es_config: ElasticsearchDBConfig = None,
    ):
        """
        :param log_level: Optional. (String) Debug level
@@ -34,11 +40,15 @@ class CustomAppConfig(BaseAppConfig):
        :param embedding_fn: Optional. Embedding function to use.
        :param embedding_fn_model: Optional. Model name to use for embedding function.
        :param db: Optional. (Vector) database to use for embeddings.
-        :param id: Optional. ID of the app. Document metadata will have this id.
        :param host: Optional. Hostname for the database server.
        :param port: Optional. Port for the database server.
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param collection_name: Optional. Collection name for the database.
        :param provider: Optional. (Providers): LLM Provider to use.
        :param open_source_app_config: Optional. Config instance needed for open source apps.
+        :param collect_metrics: Defaults to True. Send anonymous telemetry to improve embedchain.
+        :param db_type: Optional. type of Vector database to use.
+        :param es_config: Optional. elasticsearch database config to be used for connection
        """
        if provider:
            self.provider = provider
@@ -49,11 +59,18 @@ class CustomAppConfig(BaseAppConfig):

        super().__init__(
            log_level=log_level,
-            embedding_fn=CustomAppConfig.embedding_function(embedding_function=embedding_fn, model=embedding_fn_model),
+            embedding_fn=CustomAppConfig.embedding_function(
+                embedding_function=embedding_fn, model=embedding_fn_model, deployment_name=deployment_name
+            ),
            db=db,
            host=host,
            port=port,
            id=id,
+            collection_name=collection_name,
+            collect_metrics=collect_metrics,
+            db_type=db_type,
+            vector_dim=CustomAppConfig.get_vector_dimension(embedding_function=embedding_fn),
+            es_config=es_config,
        )

    @staticmethod
@@ -68,7 +85,7 @@ class CustomAppConfig(BaseAppConfig):
        return embed_function

    @staticmethod
-    def embedding_function(embedding_function: EmbeddingFunctions, model: str = None):
+    def embedding_function(embedding_function: EmbeddingFunctions, model: str = None, deployment_name: str = None):
        if not isinstance(embedding_function, EmbeddingFunctions):
            raise ValueError(
                f"Invalid option: '{embedding_function}'. Expecting one of the following options: {list(map(lambda x: x.value, EmbeddingFunctions))}"  # noqa: E501
@@ -80,7 +97,10 @@ class CustomAppConfig(BaseAppConfig):
            if model:
                embeddings = OpenAIEmbeddings(model=model)
            else:
-                embeddings = OpenAIEmbeddings()
+                if deployment_name:
+                    embeddings = OpenAIEmbeddings(deployment=deployment_name)
+                else:
+                    embeddings = OpenAIEmbeddings()
            return CustomAppConfig.langchain_default_concept(embeddings)

        elif embedding_function == EmbeddingFunctions.HUGGING_FACE:
@@ -100,3 +120,20 @@ class CustomAppConfig(BaseAppConfig):
            from chromadb.utils import embedding_functions

            return embedding_functions.SentenceTransformerEmbeddingFunction(model_name=model)
+
+    @staticmethod
+    def get_vector_dimension(embedding_function: EmbeddingFunctions):
+        if not isinstance(embedding_function, EmbeddingFunctions):
+            raise ValueError(f"Invalid option: '{embedding_function}'.")
+
+        if embedding_function == EmbeddingFunctions.OPENAI:
+            return VectorDimensions.OPENAI.value
+
+        elif embedding_function == EmbeddingFunctions.HUGGING_FACE:
+            return VectorDimensions.HUGGING_FACE.value
+
+        elif embedding_function == EmbeddingFunctions.VERTEX_AI:
+            return VectorDimensions.VERTEX_AI.value
+
+        elif embedding_function == EmbeddingFunctions.GPT4ALL:
+            return VectorDimensions.GPT4ALL.value
@@ -1,3 +1,5 @@
+from typing import Optional
+
 from chromadb.utils import embedding_functions

 from .BaseAppConfig import BaseAppConfig
@@ -8,13 +10,24 @@ class OpenSourceAppConfig(BaseAppConfig):
    Config to initialize an embedchain custom `OpenSourceApp` instance, with extra config options.
    """

-    def __init__(self, log_level=None, host=None, port=None, id=None, model=None):
+    def __init__(
+        self,
+        log_level=None,
+        host=None,
+        port=None,
+        id=None,
+        collection_name=None,
+        collect_metrics: Optional[bool] = None,
+        model=None,
+    ):
        """
        :param log_level: Optional. (String) Debug level
        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param collection_name: Optional. Collection name for the database.
        :param host: Optional. Hostname for the database server.
        :param port: Optional. Port for the database server.
+        :param collect_metrics: Defaults to True. Send anonymous telemetry to improve embedchain.
        :param model: Optional. GPT4ALL uses the model to instantiate the class.
        So unlike `App`, it has to be provided before querying.
        """
@@ -26,6 +39,8 @@ class OpenSourceAppConfig(BaseAppConfig):
            host=host,
            port=port,
            id=id,
+            collection_name=collection_name,
+            collect_metrics=collect_metrics,
        )

    @staticmethod
@@ -0,0 +1,15 @@
+from typing import Dict, List, Union
+
+from embedchain.config.BaseConfig import BaseConfig
+
+
+class ElasticsearchDBConfig(BaseConfig):
+    """
+    Config to initialize an elasticsearch client.
+    :param es_url. elasticsearch url or list of nodes url to be used for connection
+    :param ES_EXTRA_PARAMS: extra params dict that can be passed to elasticsearch.
+    """
+
+    def __init__(self, es_url: Union[str, List[str]] = None, **ES_EXTRA_PARAMS: Dict[str, any]):
+        self.ES_URL = es_url
+        self.ES_EXTRA_PARAMS = ES_EXTRA_PARAMS
@@ -1,5 +1,6 @@
 from embedchain.chunkers.docs_site import DocsSiteChunker
 from embedchain.chunkers.docx_file import DocxFileChunker
+from embedchain.chunkers.notion import NotionChunker
 from embedchain.chunkers.pdf_file import PdfFileChunker
 from embedchain.chunkers.qna_pair import QnaPairChunker
 from embedchain.chunkers.text import TextChunker
@@ -36,17 +37,27 @@ class DataFormatter:
        :raises ValueError: If an unsupported data type is provided.
        """
        loaders = {
-            "youtube_video": YoutubeVideoLoader(),
-            "pdf_file": PdfFileLoader(),
-            "web_page": WebPageLoader(),
-            "qna_pair": LocalQnaPairLoader(),
-            "text": LocalTextLoader(),
-            "docx": DocxFileLoader(),
-            "sitemap": SitemapLoader(),
-            "docs_site": DocsSiteLoader(),
+            "youtube_video": YoutubeVideoLoader,
+            "pdf_file": PdfFileLoader,
+            "web_page": WebPageLoader,
+            "qna_pair": LocalQnaPairLoader,
+            "text": LocalTextLoader,
+            "docx": DocxFileLoader,
+            "sitemap": SitemapLoader,
+            "docs_site": DocsSiteLoader,
        }
+        lazy_loaders = ("notion",)
        if data_type in loaders:
-            return loaders[data_type]
+            loader_class = loaders[data_type]
+            loader = loader_class()
+            return loader
+        elif data_type in lazy_loaders:
+            if data_type == "notion":
+                from embedchain.loaders.notion import NotionLoader
+
+                return NotionLoader()
+            else:
+                raise ValueError(f"Unsupported data type: {data_type}")
        else:
            raise ValueError(f"Unsupported data type: {data_type}")

@@ -67,6 +78,7 @@ class DataFormatter:
            "docx": DocxFileChunker,
            "sitemap": WebPageChunker,
            "docs_site": DocsSiteChunker,
+            "notion": NotionChunker,
        }
        if data_type in chunker_classes:
            chunker_class = chunker_classes[data_type]
@@ -1,15 +1,22 @@
+import importlib.metadata
 import logging
 import os
+import threading
+from typing import Optional
+import uuid

-from chromadb.errors import InvalidDimensionException
+import requests
 from dotenv import load_dotenv
 from langchain.docstore.document import Document
 from langchain.memory import ConversationBufferMemory
+from tenacity import retry, stop_after_attempt, wait_fixed

+from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config import AddConfig, ChatConfig, QueryConfig
 from embedchain.config.apps.BaseAppConfig import BaseAppConfig
 from embedchain.config.QueryConfig import DOCS_SITE_PROMPT_TEMPLATE
 from embedchain.data_formatter import DataFormatter
+from embedchain.loaders.base_loader import BaseLoader

 load_dotenv()

@@ -29,12 +36,17 @@ class EmbedChain:
        """

        self.config = config
-        self.db_client = self.config.db.client
-        self.collection = self.config.db.collection
+        self.collection = self.config.db._get_or_create_collection(self.config.collection_name)
+        self.db = self.config.db
        self.user_asks = []
        self.is_docs_site_instance = False
        self.online = False

+        # Send anonymous telemetry
+        self.s_id = self.config.id if self.config.id else str(uuid.uuid4())
+        thread_telemetry = threading.Thread(target=self._send_telemetry_event, args=("init",))
+        thread_telemetry.start()
+
    def add(self, data_type, url, metadata=None, config: AddConfig = None):
        """
        Adds the data from the given URL to the vector db.
@@ -52,10 +64,21 @@ class EmbedChain:

        data_formatter = DataFormatter(data_type, config)
        self.user_asks.append([data_type, url, metadata])
-        self.load_and_embed(data_formatter.loader, data_formatter.chunker, url, metadata)
+        documents, _metadatas, _ids, new_chunks = self.load_and_embed(
+            data_formatter.loader, data_formatter.chunker, url, metadata
+        )
        if data_type in ("docs_site",):
            self.is_docs_site_instance = True

+        # Send anonymous telemetry
+        if self.config.collect_metrics:
+            # it's quicker to check the variable twice than to count words when they won't be submitted.
+            word_count = sum([len(document.split(" ")) for document in documents])
+
+            extra_metadata = {"data_type": data_type, "word_count": word_count, "chunks_count": new_chunks}
+            thread_telemetry = threading.Thread(target=self._send_telemetry_event, args=("add", extra_metadata))
+            thread_telemetry.start()
+
    def add_local(self, data_type, content, metadata=None, config: AddConfig = None):
        """
        Adds the data you supply to the vector db.
@@ -80,7 +103,7 @@ class EmbedChain:
            metadata,
        )

-    def load_and_embed(self, loader, chunker, src, metadata=None):
+    def load_and_embed(self, loader: BaseLoader, chunker: BaseChunker, src, metadata=None):
        """
        Loads the data from the given URL, chunks it, and adds it to database.

@@ -89,6 +112,7 @@ class EmbedChain:
        :param src: The data to be handled by the loader. Can be a URL for
        remote sources or local content for local loaders.
        :param metadata: Optional. Metadata associated with the data source.
+        :return: (List) documents (embedded text), (List) metadata, (list) ids, (int) number of chunks
        """
        embeddings_data = chunker.create_chunks(loader, src)
        documents = embeddings_data["documents"]
@@ -97,11 +121,10 @@ class EmbedChain:
        # get existing ids, and discard doc if any common id exist.
        where = {"app_id": self.config.id} if self.config.id is not None else {}
        # where={"url": src}
-        existing_docs = self.collection.get(
+        existing_ids = self.db.get(
            ids=ids,
            where=where,  # optional filter
        )
-        existing_ids = set(existing_docs["ids"])

        if len(existing_ids):
            data_dict = {id: (doc, meta) for id, doc, meta in zip(ids, documents, metadatas)}
@@ -109,7 +132,8 @@ class EmbedChain:

            if not data_dict:
                print(f"All data from {src} already exists in the database.")
-                return
+                # Make sure to return a matching return type
+                return [], [], [], 0

            ids = list(data_dict.keys())
            documents, metadatas = zip(*data_dict.values())
@@ -118,13 +142,18 @@ class EmbedChain:
        if self.config.id is not None:
            metadatas = [{**m, "app_id": self.config.id} for m in metadatas]

+        # FIXME: Fix the error handling logic when metadatas or metadata is None
+        metadatas = metadatas if metadatas else []
+        metadata = metadata if metadata else {}
        chunks_before_addition = self.count()

        # Add metadata to each document
-        metadatas_with_metadata = [meta or metadata for meta in metadatas]
+        metadatas_with_metadata = [{**meta, **metadata} for meta in metadatas]

-        self.collection.add(documents=documents, metadatas=list(metadatas_with_metadata), ids=ids)
-        print((f"Successfully saved {src}. New chunks count: " f"{self.count() - chunks_before_addition}"))
+        self.db.add(documents=documents, metadatas=metadatas_with_metadata, ids=ids)
+        count_new_chunks = self.count() - chunks_before_addition
+        print((f"Successfully saved {src}. New chunks count: {count_new_chunks}"))
+        return list(documents), metadatas_with_metadata, ids, count_new_chunks

    def _format_result(self, results):
        return [
@@ -151,23 +180,13 @@ class EmbedChain:
        :param config: The query configuration.
        :return: The content of the document that matched your query.
        """
-        try:
-            where = {"app_id": self.config.id} if self.config.id is not None else {}  # optional filter
-            result = self.collection.query(
-                query_texts=[
-                    input_query,
-                ],
-                n_results=config.number_documents,
-                where=where,
-            )
-        except InvalidDimensionException as e:
-            raise InvalidDimensionException(
-                e.message()
-                + ". This is commonly a side-effect when an embedding function, different from the one used to add the embeddings, is used to retrieve an embedding from the database."  # noqa E501
-            ) from None
+        where = {"app_id": self.config.id} if self.config.id is not None else {}  # optional filter
+        contents = self.db.query(
+            input_query=input_query,
+            n_results=config.number_documents,
+            where=where,
+        )

-        results_formatted = self._format_result(result)
-        contents = [result[0].page_content for result in results_formatted]
        return contents

    def _append_search_and_context(self, context, web_search_result):
@@ -247,6 +266,10 @@ class EmbedChain:

        answer = self.get_answer_from_llm(prompt, config)

+        # Send anonymous telemetry
+        thread_telemetry = threading.Thread(target=self._send_telemetry_event, args=("query",))
+        thread_telemetry.start()
+
        if isinstance(answer, str):
            logging.info(f"Answer: {answer}")
            return answer
@@ -286,7 +309,7 @@ class EmbedChain:
        k = {}
        if self.online:
            k["web_search_result"] = self.access_search_and_get_results(input_query)
-        contexts = self.retrieve_from_database(input_query, config, **k)
+        contexts = self.retrieve_from_database(input_query, config)

        global memory
        chat_history = memory.load_memory_variables({})["history"]
@@ -304,6 +327,10 @@ class EmbedChain:

        memory.chat_memory.add_user_message(input_query)

+        # Send anonymous telemetry
+        thread_telemetry = threading.Thread(target=self._send_telemetry_event, args=("chat",))
+        thread_telemetry.start()
+
        if isinstance(answer, str):
            memory.chat_memory.add_ai_message(answer)
            logging.info(f"Answer: {answer}")
@@ -320,17 +347,48 @@ class EmbedChain:
        memory.chat_memory.add_ai_message(streamed_answer)
        logging.info(f"Answer: {streamed_answer}")

-    def count(self):
+    def set_collection(self, collection_name):
+        """
+        Set the collection to use.
+
+        :param collection_name: The name of the collection to use.
+        """
+        self.collection = self.config.db._get_or_create_collection(collection_name)
+
+    def count(self) -> int:
        """
        Count the number of embeddings.

        :return: The number of embeddings.
        """
-        return self.collection.count()
+        return self.db.count()

    def reset(self):
        """
        Resets the database. Deletes all embeddings irreversibly.
        `App` has to be reinitialized after using this method.
        """
-        self.db_client.reset()
+        # Send anonymous telemetry
+        thread_telemetry = threading.Thread(target=self._send_telemetry_event, args=("reset",))
+        thread_telemetry.start()
+
+        self.db.reset()
+
+    @retry(stop=stop_after_attempt(3), wait=wait_fixed(1))
+    def _send_telemetry_event(self, method: str, extra_metadata: Optional[dict] = None):
+        if not self.config.collect_metrics:
+            return
+
+        with threading.Lock():
+            url = "https://api.embedchain.ai/api/v1/telemetry/"
+            metadata = {
+                "s_id": self.s_id,
+                "version": importlib.metadata.version(__package__ or __name__),
+                "method": method,
+                "language": "py",
+            }
+            if extra_metadata:
+                metadata.update(extra_metadata)
+
+            response = requests.post(url, json={"metadata": metadata})
+            response.raise_for_status()
@@ -0,0 +1,9 @@
+class BaseLoader:
+    def __init__(self):
+        pass
+
+    def load_data():
+        """
+        Implemented by child classes
+        """
+        pass
@@ -4,8 +4,10 @@ from urllib.parse import urljoin, urlparse
 import requests
 from bs4 import BeautifulSoup

+from embedchain.loaders.base_loader import BaseLoader

-class DocsSiteLoader:
+
+class DocsSiteLoader(BaseLoader):
    def __init__(self):
        self.visited_links = set()

@@ -1,7 +1,9 @@
 from langchain.document_loaders import Docx2txtLoader

+from embedchain.loaders.base_loader import BaseLoader

-class DocxFileLoader:
+
+class DocxFileLoader(BaseLoader):
    def load_data(self, url):
        """Load data from a .docx file."""
        loader = Docx2txtLoader(url)
@@ -1,4 +1,7 @@
-class LocalQnaPairLoader:
+from embedchain.loaders.base_loader import BaseLoader
+
+
+class LocalQnaPairLoader(BaseLoader):
    def load_data(self, content):
        """Load data from a local QnA pair."""
        question, answer = content
@@ -1,4 +1,7 @@
-class LocalTextLoader:
+from embedchain.loaders.base_loader import BaseLoader
+
+
+class LocalTextLoader(BaseLoader):
    def load_data(self, content):
        """Load data from a local text file."""
        meta_data = {
@@ -0,0 +1,41 @@
+import logging
+import os
+
+try:
+    from llama_index import download_loader
+except ImportError:
+    raise ImportError("Notion requires extra dependencies. Install with `pip install embedchain[community]`") from None
+
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+
+class NotionLoader(BaseLoader):
+    def load_data(self, source):
+        """Load data from a PDF file."""
+
+        NotionPageReader = download_loader("NotionPageReader")
+
+        # Reformat Id to match notion expectation
+        id = source[-32:]
+        formatted_id = f"{id[:8]}-{id[8:12]}-{id[12:16]}-{id[16:20]}-{id[20:]}"
+        logging.debug(f"Extracted notion page id as: {formatted_id}")
+
+        # Get page through the notion api
+        integration_token = os.getenv("NOTION_INTEGRATION_TOKEN")
+        reader = NotionPageReader(integration_token=integration_token)
+        documents = reader.load_data(page_ids=[formatted_id])
+
+        # Extract text
+        raw_text = documents[0].text
+
+        # Clean text
+        text = clean_string(raw_text)
+
+        return [
+            {
+                "content": text,
+                "meta_data": {"url": f"notion-{formatted_id}"},
+            }
+        ]
@@ -1,9 +1,10 @@
 from langchain.document_loaders import PyPDFLoader

+from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils import clean_string


-class PdfFileLoader:
+class PdfFileLoader(BaseLoader):
    def load_data(self, url):
        """Load data from a PDF file."""
        loader = PyPDFLoader(url)
@@ -4,11 +4,12 @@ import requests
 from bs4 import BeautifulSoup
 from bs4.builder import ParserRejectedMarkup

+from embedchain.loaders.base_loader import BaseLoader
 from embedchain.loaders.web_page import WebPageLoader
 from embedchain.utils import is_readable


-class SitemapLoader:
+class SitemapLoader(BaseLoader):
    def load_data(self, sitemap_url):
        """
        This method takes a sitemap URL as input and retrieves
@@ -3,10 +3,11 @@ import logging
 import requests
 from bs4 import BeautifulSoup

+from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils import clean_string


-class WebPageLoader:
+class WebPageLoader(BaseLoader):
    def load_data(self, url):
        """Load data from a web page."""
        response = requests.get(url)
@@ -1,9 +1,10 @@
 from langchain.document_loaders import YoutubeLoader

+from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils import clean_string


-class YoutubeVideoLoader:
+class YoutubeVideoLoader(BaseLoader):
    def load_data(self, url):
        """Load data from a Youtube video."""
        loader = YoutubeLoader.from_youtube_url(url, add_video_info=True)
@@ -6,3 +6,4 @@ class Providers(Enum):
    ANTHROPHIC = "ANTHPROPIC"
    VERTEX_AI = "VERTEX_AI"
    GPT4ALL = "GPT4ALL"
+    AZURE_OPENAI = "AZURE_OPENAI"
@@ -0,0 +1,6 @@
+from enum import Enum
+
+
+class VectorDatabases(Enum):
+    CHROMADB = "CHROMADB"
+    ELASTICSEARCH = "ELASTICSEARCH"
@@ -0,0 +1,9 @@
+from enum import Enum
+
+
+# vector length created by embedding fn
+class VectorDimensions(Enum):
+    GPT4ALL = 384
+    OPENAI = 1536
+    VERTEX_AI = 768
+    HUGGING_FACE = 384
@@ -1,2 +1,4 @@
 from .EmbeddingFunctions import EmbeddingFunctions  # noqa: F401
 from .Providers import Providers  # noqa: F401
+from .VectorDatabases import VectorDatabases  # noqa: F401
+from .VectorDimensions import VectorDimensions  # noqa: F401
@@ -1,3 +1,4 @@
+import logging
 import re
 import string

@@ -43,5 +44,48 @@ def is_readable(s):
    :param s: string
    :return: True if the string is more than 95% printable.
    """
-    printable_ratio = sum(c in string.printable for c in s) / len(s)
+    try:
+        printable_ratio = sum(c in string.printable for c in s) / len(s)
+    except ZeroDivisionError:
+        logging.warning("Empty string processed as unreadable")
+        printable_ratio = 0
    return printable_ratio > 0.95  # 95% of characters are printable
+
+
+def use_pysqlite3():
+    """
+    Swap std-lib sqlite3 with pysqlite3.
+    """
+    import platform
+    import sqlite3
+
+    if platform.system() == "Linux" and sqlite3.sqlite_version_info < (3, 35, 0):
+        try:
+            # According to the Chroma team, this patch only works on Linux
+            import datetime
+            import subprocess
+            import sys
+
+            subprocess.check_call(
+                [sys.executable, "-m", "pip", "install", "pysqlite3-binary", "--quiet", "--disable-pip-version-check"]
+            )
+
+            __import__("pysqlite3")
+            sys.modules["sqlite3"] = sys.modules.pop("pysqlite3")
+
+            # Let the user know what happened.
+            current_time = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S,%f")[:-3]
+            print(
+                f"{current_time} [embedchain] [INFO]",
+                "Swapped std-lib sqlite3 with pysqlite3 for ChromaDb compatibility.",
+                f"Your original version was {sqlite3.sqlite_version}.",
+            )
+        except Exception as e:
+            # Escape all exceptions
+            current_time = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S,%f")[:-3]
+            print(
+                f"{current_time} [embedchain] [ERROR]",
+                "Failed to swap std-lib sqlite3 with pysqlite3 for ChromaDb compatibility.",
+                "Error:",
+                e,
+            )
@@ -3,7 +3,6 @@ class BaseVectorDB:

    def __init__(self):
        self.client = self._get_or_create_db()
-        self.collection = self._get_or_create_collection()

    def _get_or_create_db(self):
        """Get or create the database."""
@@ -11,3 +10,18 @@ class BaseVectorDB:

    def _get_or_create_collection(self):
        raise NotImplementedError
+
+    def get(self):
+        raise NotImplementedError
+
+    def add(self):
+        raise NotImplementedError
+
+    def query(self):
+        raise NotImplementedError
+
+    def count(self):
+        raise NotImplementedError
+
+    def reset(self):
+        raise NotImplementedError
@@ -1,6 +1,18 @@
 import logging
+from typing import Any, Dict, List

-import chromadb
+from chromadb.errors import InvalidDimensionException
+from langchain.docstore.document import Document
+
+try:
+    import chromadb
+except RuntimeError:
+    from embedchain.utils import use_pysqlite3
+
+    use_pysqlite3()
+    import chromadb
+
+from chromadb.config import Settings

 from embedchain.vectordb.base_vector_db import BaseVectorDB

@@ -16,28 +28,91 @@ class ChromaDB(BaseVectorDB):

        if host and port:
            logging.info(f"Connecting to ChromaDB server: {host}:{port}")
-            self.client_settings = chromadb.config.Settings(
-                chroma_api_impl="rest",
-                chroma_server_host=host,
-                chroma_server_http_port=port,
-            )
+            self.settings = Settings(chroma_server_host=host, chroma_server_http_port=port)
+            self.client = chromadb.HttpClient(self.settings)
        else:
            if db_dir is None:
                db_dir = "db"
-            self.client_settings = chromadb.config.Settings(
-                chroma_db_impl="duckdb+parquet",
-                persist_directory=db_dir,
-                anonymized_telemetry=False,
+            self.settings = Settings(anonymized_telemetry=False, allow_reset=True)
+            self.client = chromadb.PersistentClient(
+                path=db_dir,
+                settings=self.settings,
            )
        super().__init__()

    def _get_or_create_db(self):
        """Get or create the database."""
-        return chromadb.Client(self.client_settings)
+        return self.client

-    def _get_or_create_collection(self):
+    def _get_or_create_collection(self, name):
        """Get or create the collection."""
-        return self.client.get_or_create_collection(
-            "embedchain_store",
+        self.collection = self.client.get_or_create_collection(
+            name=name,
            embedding_function=self.embedding_fn,
        )
+        return self.collection
+
+    def get(self, ids: List[str], where: Dict[str, any]) -> List[str]:
+        """
+        Get existing doc ids present in vector database
+        :param ids: list of doc ids to check for existance
+        :param where: Optional. to filter data
+        """
+        existing_docs = self.collection.get(
+            ids=ids,
+            where=where,  # optional filter
+        )
+
+        return set(existing_docs["ids"])
+
+    def add(self, documents: List[str], metadatas: List[object], ids: List[str]) -> Any:
+        """
+        add data in vector database
+        :param documents: list of texts to add
+        :param metadatas: list of metadata associated with docs
+        :param ids: ids of docs
+        """
+        self.collection.add(documents=documents, metadatas=metadatas, ids=ids)
+
+    def _format_result(self, results):
+        return [
+            (Document(page_content=result[0], metadata=result[1] or {}), result[2])
+            for result in zip(
+                results["documents"][0],
+                results["metadatas"][0],
+                results["distances"][0],
+            )
+        ]
+
+    def query(self, input_query: List[str], n_results: int, where: Dict[str, any]) -> List[str]:
+        """
+        query contents from vector data base based on vector similarity
+        :param input_query: list of query string
+        :param n_results: no of similar documents to fetch from database
+        :param where: Optional. to filter data
+        :return: The content of the document that matched your query.
+        """
+        try:
+            result = self.collection.query(
+                query_texts=[
+                    input_query,
+                ],
+                n_results=n_results,
+                where=where,
+            )
+        except InvalidDimensionException as e:
+            raise InvalidDimensionException(
+                e.message()
+                + ". This is commonly a side-effect when an embedding function, different from the one used to add the embeddings, is used to retrieve an embedding from the database."  # noqa E501
+            ) from None
+
+        results_formatted = self._format_result(result)
+        contents = [result[0].page_content for result in results_formatted]
+        return contents
+
+    def count(self) -> int:
+        return self.collection.count()
+
+    def reset(self):
+        # Delete all data from the database
+        self.client.reset()
@@ -0,0 +1,136 @@
+from typing import Any, Callable, Dict, List
+
+try:
+    from elasticsearch import Elasticsearch
+    from elasticsearch.helpers import bulk
+except ImportError:
+    raise ImportError(
+        "Elasticsearch requires extra dependencies. Install with `pip install embedchain[elasticsearch]`"
+    ) from None
+
+from embedchain.config import ElasticsearchDBConfig
+from embedchain.models.VectorDimensions import VectorDimensions
+from embedchain.vectordb.base_vector_db import BaseVectorDB
+
+
+class ElasticsearchDB(BaseVectorDB):
+    def __init__(
+        self,
+        es_config: ElasticsearchDBConfig = None,
+        embedding_fn: Callable[[list[str]], list[str]] = None,
+        vector_dim: VectorDimensions = None,
+        collection_name: str = None,
+    ):
+        """
+        Elasticsearch as vector database
+        :param es_config. elasticsearch database config to be used for connection
+        :param embedding_fn: Function to generate embedding vectors.
+        :param vector_dim: Vector dimension generated by embedding fn
+        :param collection_name: Optional. Collection name for the database.
+        """
+        if not hasattr(embedding_fn, "__call__"):
+            raise ValueError("Embedding function is not a function")
+        if es_config is None:
+            raise ValueError("ElasticsearchDBConfig is required")
+        if vector_dim is None:
+            raise ValueError("Vector Dimension is required to refer correct index and mapping")
+        if collection_name is None:
+            raise ValueError("collection name is required. It cannot be empty")
+        self.embedding_fn = embedding_fn
+        self.client = Elasticsearch(es_config.ES_URL, **es_config.ES_EXTRA_PARAMS)
+        self.vector_dim = vector_dim
+        self.es_index = f"{collection_name}_{self.vector_dim}"
+        index_settings = {
+            "mappings": {
+                "properties": {
+                    "text": {"type": "text"},
+                    "embeddings": {"type": "dense_vector", "index": False, "dims": self.vector_dim},
+                }
+            }
+        }
+        if not self.client.indices.exists(index=self.es_index):
+            # create index if not exist
+            print("Creating index", self.es_index, index_settings)
+            self.client.indices.create(index=self.es_index, body=index_settings)
+        super().__init__()
+
+    def _get_or_create_db(self):
+        return self.client
+
+    def _get_or_create_collection(self, name):
+        """Note: nothing to return here. Discuss later"""
+
+    def get(self, ids: List[str], where: Dict[str, any]) -> List[str]:
+        """
+        Get existing doc ids present in vector database
+        :param ids: list of doc ids to check for existance
+        :param where: Optional. to filter data
+        """
+        query = {"bool": {"must": [{"ids": {"values": ids}}]}}
+        if "app_id" in where:
+            app_id = where["app_id"]
+            query["bool"]["must"].append({"term": {"metadata.app_id": app_id}})
+        response = self.client.search(index=self.es_index, query=query, _source=False)
+        docs = response["hits"]["hits"]
+        ids = [doc["_id"] for doc in docs]
+        return set(ids)
+
+    def add(self, documents: List[str], metadatas: List[object], ids: List[str]) -> Any:
+        """
+        add data in vector database
+        :param documents: list of texts to add
+        :param metadatas: list of metadata associated with docs
+        :param ids: ids of docs
+        """
+        docs = []
+        embeddings = self.embedding_fn(documents)
+        for id, text, metadata, embeddings in zip(ids, documents, metadatas, embeddings):
+            docs.append(
+                {
+                    "_index": self.es_index,
+                    "_id": id,
+                    "_source": {"text": text, "metadata": metadata, "embeddings": embeddings},
+                }
+            )
+        bulk(self.client, docs)
+        self.client.indices.refresh(index=self.es_index)
+        return
+
+    def query(self, input_query: List[str], n_results: int, where: Dict[str, any]) -> List[str]:
+        """
+        query contents from vector data base based on vector similarity
+        :param input_query: list of query string
+        :param n_results: no of similar documents to fetch from database
+        :param where: Optional. to filter data
+        """
+        input_query_vector = self.embedding_fn(input_query)
+        query_vector = input_query_vector[0]
+        query = {
+            "script_score": {
+                "query": {"bool": {"must": [{"exists": {"field": "text"}}]}},
+                "script": {
+                    "source": "cosineSimilarity(params.input_query_vector, 'embeddings') + 1.0",
+                    "params": {"input_query_vector": query_vector},
+                },
+            }
+        }
+        if "app_id" in where:
+            app_id = where["app_id"]
+            query["script_score"]["query"]["bool"]["must"] = [{"term": {"metadata.app_id": app_id}}]
+        _source = ["text"]
+        response = self.client.search(index=self.es_index, query=query, _source=_source, size=n_results)
+        docs = response["hits"]["hits"]
+        contents = [doc["_source"]["text"] for doc in docs]
+        return contents
+
+    def count(self) -> int:
+        query = {"match_all": {}}
+        response = self.client.count(index=self.es_index, query=query)
+        doc_count = response["count"]
+        return doc_count
+
+    def reset(self):
+        # Delete all data from the database
+        if self.client.indices.exists(index=self.es_index):
+            # delete index in Es
+            self.client.indices.delete(index=self.es_index)
@@ -0,0 +1,8 @@
+__pycache__/
+database
+db
+pyenv
+venv
+.env
+.git
+trash_files/
@@ -0,0 +1,8 @@
+__pycache__
+db
+database
+pyenv
+venv
+.env
+trash_files/
+.ideas.md
@@ -0,0 +1,11 @@
+FROM python:3.11 AS backend
+
+WORKDIR /usr/src/api
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . .
+
+EXPOSE 5000
+
+CMD ["python", "api_server.py"]
@@ -0,0 +1,42 @@
+from flask import Flask, jsonify, request
+
+from embedchain import App
+
+app = Flask(__name__)
+
+
+def initialize_chat_bot():
+    global chat_bot
+    chat_bot = App()
+
+
+@app.route("/add", methods=["POST"])
+def add():
+    data = request.get_json()
+    data_type = data.get("data_type")
+    url_or_text = data.get("url_or_text")
+    if data_type and url_or_text:
+        try:
+            chat_bot.add(data_type, url_or_text)
+            return jsonify({"data": f"Added {data_type}: {url_or_text}"}), 200
+        except Exception:
+            return jsonify({"error": f"Failed to add {data_type}: {url_or_text}"}), 500
+    return jsonify({"error": "Invalid request. Please provide 'data_type' and 'url_or_text' in JSON format."}), 400
+
+
+@app.route("/query", methods=["POST"])
+def query():
+    data = request.get_json()
+    question = data.get("question")
+    if question:
+        try:
+            response = chat_bot.chat(question)
+            return jsonify({"data": response}), 200
+        except Exception:
+            return jsonify({"error": "An error occurred. Please try again!"}), 500
+    return jsonify({"error": "Invalid request. Please provide 'question' in JSON format."}), 400
+
+
+if __name__ == "__main__":
+    initialize_chat_bot()
+    app.run(host="0.0.0.0", port=5000, debug=False)
@@ -0,0 +1,13 @@
+version: "3.9"
+
+services:
+  backend:
+    container_name: embedchain_api
+    restart: unless-stopped
+    build:
+      context: .
+      dockerfile: Dockerfile
+    env_file:
+      - variables.env
+    ports:
+      - "5000:5000"
@@ -0,0 +1,2 @@
+flask==2.3.2
+embedchain==0.0.30
@@ -0,0 +1 @@
+OPENAI_API_KEY=""
@@ -0,0 +1,8 @@
+__pycache__/
+database
+db
+pyenv
+venv
+.env
+.git
+trash_files/
@@ -0,0 +1,7 @@
+__pycache__
+db
+database
+pyenv
+venv
+.env
+trash_files/
@@ -0,0 +1,9 @@
+FROM python:3.11 AS backend
+
+WORKDIR /usr/src/discord_bot
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . .
+
+CMD ["python", "discord_bot.py"]
@@ -0,0 +1,65 @@
+import os
+
+import discord
+from discord.ext import commands
+from dotenv import load_dotenv
+
+from embedchain import App
+
+load_dotenv()
+intents = discord.Intents.default()
+intents.message_content = True
+
+bot = commands.Bot(command_prefix="/ec ", intents=intents)
+root_folder = os.getcwd()
+
+
+def initialize_chat_bot():
+    global chat_bot
+    chat_bot = App()
+
+
+@bot.event
+async def on_ready():
+    print(f"Logged in as {bot.user.name}")
+    initialize_chat_bot()
+
+
+@bot.event
+async def on_command_error(ctx, error):
+    if isinstance(error, commands.CommandNotFound):
+        await send_response(ctx, "Invalid command. Please refer to the documentation for correct syntax.")
+    else:
+        print("Error occurred during command execution:", error)
+
+
+@bot.command()
+async def add(ctx, data_type: str, *, url_or_text: str):
+    print(f"User: {ctx.author.name}, Data Type: {data_type}, URL/Text: {url_or_text}")
+    try:
+        chat_bot.add(data_type, url_or_text)
+        await send_response(ctx, f"Added {data_type} : {url_or_text}")
+    except Exception as e:
+        await send_response(ctx, f"Failed to add {data_type} : {url_or_text}")
+        print("Error occurred during 'add' command:", e)
+
+
+@bot.command()
+async def query(ctx, *, question: str):
+    print(f"User: {ctx.author.name}, Query: {question}")
+    try:
+        response = chat_bot.chat(question)
+        await send_response(ctx, response)
+    except Exception as e:
+        await send_response(ctx, "An error occurred. Please try again!")
+        print("Error occurred during 'query' command:", e)
+
+
+async def send_response(ctx, message):
+    if ctx.guild is None:
+        await ctx.send(message)
+    else:
+        await ctx.reply(message)
+
+
+bot.run(os.environ["DISCORD_BOT_TOKEN"])
@@ -0,0 +1,11 @@
+version: "3.9"
+
+services:
+  backend:
+    container_name: embedchain_discord_bot
+    restart: unless-stopped
+    build:
+      context: .
+      dockerfile: Dockerfile
+    env_file:
+      - variables.env
@@ -0,0 +1,3 @@
+discord==2.3.1
+embedchain==0.0.30
+python-dotenv==1.0.0
@@ -0,0 +1,2 @@
+OPENAI_API_KEY=""
+DISCORD_BOT_TOKEN=""
@@ -0,0 +1 @@
+.git
@@ -0,0 +1,18 @@
+## 🐳 Docker Setup
+
+- To setup full stack app using docker, run the following command inside this folder using your terminal.
+
+```bash
+docker-compose up --build
+```
+
+📝 Note: The build command might take a while to install all the packages depending on your system resources.
+
+## 🚀 Usage Instructions
+
+- Go to [http://localhost:3000/](http://localhost:3000/) in your browser to view the dashboard.
+- Add your `OpenAI API key` 🔑 in the Settings.
+- Create a new bot and you'll be navigated to its page.
+- Here you can add your data sources and then chat with the bot.
+
+🎉 Happy Chatting! 🎉
@@ -0,0 +1,7 @@
+__pycache__/
+database
+pyenv
+venv
+.env
+.git
+trash_files/
@@ -0,0 +1,6 @@
+__pycache__
+database
+pyenv
+venv
+.env
+trash_files/
@@ -0,0 +1,11 @@
+FROM python:3.11 AS backend
+
+WORKDIR /usr/src/app/backend
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . .
+
+EXPOSE 8000
+
+CMD ["python", "server.py"]
@@ -0,0 +1,14 @@
+from flask_sqlalchemy import SQLAlchemy
+
+db = SQLAlchemy()
+
+
+class APIKey(db.Model):
+    id = db.Column(db.Integer, primary_key=True)
+    key = db.Column(db.String(255), nullable=False)
+
+
+class BotList(db.Model):
+    id = db.Column(db.Integer, primary_key=True)
+    name = db.Column(db.String(255), nullable=False)
+    slug = db.Column(db.String(255), nullable=False, unique=True)
@@ -0,0 +1,5 @@
+import os
+
+ROOT_DIRECTORY = os.getcwd()
+DB_DIRECTORY_OPEN_AI = os.path.join(os.getcwd(), "database", "open_ai")
+DB_DIRECTORY_OPEN_SOURCE = os.path.join(os.getcwd(), "database", "open_source")
@@ -0,0 +1,32 @@
+import os
+
+from flask import Blueprint, jsonify, make_response, request
+from models import APIKey
+from paths import DB_DIRECTORY_OPEN_AI
+
+from embedchain import App
+
+chat_response_bp = Blueprint("chat_response", __name__)
+
+
+# Chat Response for user query
+@chat_response_bp.route("/api/get_answer", methods=["POST"])
+def get_answer():
+    try:
+        data = request.get_json()
+        query = data.get("query")
+        embedding_model = data.get("embedding_model")
+        app_type = data.get("app_type")
+
+        if embedding_model == "open_ai":
+            os.chdir(DB_DIRECTORY_OPEN_AI)
+            api_key = APIKey.query.first().key
+            os.environ["OPENAI_API_KEY"] = api_key
+            if app_type == "app":
+                chat_bot = App()
+
+        response = chat_bot.chat(query)
+        return make_response(jsonify({"response": response}), 200)
+
+    except Exception as e:
+        return make_response(jsonify({"error": str(e)}), 400)
@@ -0,0 +1,72 @@
+from flask import Blueprint, jsonify, make_response, request
+from models import APIKey, BotList, db
+
+dashboard_bp = Blueprint("dashboard", __name__)
+
+
+# Set Open AI Key
+@dashboard_bp.route("/api/set_key", methods=["POST"])
+def set_key():
+    data = request.get_json()
+    api_key = data["openAIKey"]
+    existing_key = APIKey.query.first()
+    if existing_key:
+        existing_key.key = api_key
+    else:
+        new_key = APIKey(key=api_key)
+        db.session.add(new_key)
+    db.session.commit()
+    return make_response(jsonify(message="API key saved successfully"), 200)
+
+
+# Check OpenAI Key
+@dashboard_bp.route("/api/check_key", methods=["GET"])
+def check_key():
+    existing_key = APIKey.query.first()
+    if existing_key:
+        return make_response(jsonify(status="ok", message="OpenAI Key exists"), 200)
+    else:
+        return make_response(jsonify(status="fail", message="No OpenAI Key present"), 200)
+
+
+# Create a bot
+@dashboard_bp.route("/api/create_bot", methods=["POST"])
+def create_bot():
+    data = request.get_json()
+    name = data["name"]
+    slug = name.lower().replace(" ", "_")
+    existing_bot = BotList.query.filter_by(slug=slug).first()
+    if existing_bot:
+        return (make_response(jsonify(message="Bot already exists"), 400),)
+    new_bot = BotList(name=name, slug=slug)
+    db.session.add(new_bot)
+    db.session.commit()
+    return make_response(jsonify(message="Bot created successfully"), 200)
+
+
+# Delete a bot
+@dashboard_bp.route("/api/delete_bot", methods=["POST"])
+def delete_bot():
+    data = request.get_json()
+    slug = data.get("slug")
+    bot = BotList.query.filter_by(slug=slug).first()
+    if bot:
+        db.session.delete(bot)
+        db.session.commit()
+        return make_response(jsonify(message="Bot deleted successfully"), 200)
+    return make_response(jsonify(message="Bot not found"), 400)
+
+
+# Get the list of bots
+@dashboard_bp.route("/api/get_bots", methods=["GET"])
+def get_bots():
+    bots = BotList.query.all()
+    bot_list = []
+    for bot in bots:
+        bot_list.append(
+            {
+                "name": bot.name,
+                "slug": bot.slug,
+            }
+        )
+    return jsonify(bot_list)
@@ -0,0 +1,27 @@
+import os
+
+from flask import Blueprint, jsonify, make_response, request
+from models import APIKey
+from paths import DB_DIRECTORY_OPEN_AI
+
+from embedchain import App
+
+sources_bp = Blueprint("sources", __name__)
+
+
+# API route to add data sources
+@sources_bp.route("/api/add_sources", methods=["POST"])
+def add_sources():
+    try:
+        embedding_model = request.json.get("embedding_model")
+        name = request.json.get("name")
+        value = request.json.get("value")
+        if embedding_model == "open_ai":
+            os.chdir(DB_DIRECTORY_OPEN_AI)
+            api_key = APIKey.query.first().key
+            os.environ["OPENAI_API_KEY"] = api_key
+            chat_bot = App()
+        chat_bot.add(name, value)
+        return make_response(jsonify(message="Sources added successfully"), 200)
+    except Exception as e:
+        return make_response(jsonify(message=f"Error adding sources: {str(e)}"), 400)
@@ -0,0 +1,27 @@
+import os
+
+from flask import Flask
+from models import db
+from paths import DB_DIRECTORY_OPEN_AI, ROOT_DIRECTORY
+from routes.chat_response import chat_response_bp
+from routes.dashboard import dashboard_bp
+from routes.sources import sources_bp
+
+app = Flask(__name__)
+app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///" + os.path.join(ROOT_DIRECTORY, "database", "user_data.db")
+app.register_blueprint(dashboard_bp)
+app.register_blueprint(sources_bp)
+app.register_blueprint(chat_response_bp)
+
+
+# Initialize the app on startup
+def load_app():
+    os.makedirs(DB_DIRECTORY_OPEN_AI, exist_ok=True)
+    db.init_app(app)
+    with app.app_context():
+        db.create_all()
+
+
+if __name__ == "__main__":
+    load_app()
+    app.run(host="0.0.0.0", debug=True, port=8000)
@@ -0,0 +1,22 @@
+version: "3.9"
+
+services:
+  backend:
+    container_name: embedchain_backend
+    restart: unless-stopped
+    build:
+      context: backend
+      dockerfile: Dockerfile
+    ports:
+      - "8000:8000"
+
+  frontend:
+    container_name: embedchain_frontend
+    restart: unless-stopped
+    build:
+      context: frontend
+      dockerfile: Dockerfile
+    ports:
+      - "3000:3000"
+    depends_on:
+      - "backend"
@@ -0,0 +1,7 @@
+node_modules/
+build
+dist
+.env
+.git
+.next/
+trash_files/
@@ -0,0 +1,3 @@
+{
+  "extends": ["next/babel", "next/core-web-vitals"]
+}
@@ -0,0 +1,38 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# local env files
+.env*.local
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts
+
+vscode/
+trash_files/
@@ -0,0 +1,14 @@
+FROM node:18 AS frontend
+
+WORKDIR /usr/src/app/frontend
+COPY package.json .
+COPY package-lock.json .
+RUN npm install
+
+COPY . .
+
+RUN npm run build
+
+EXPOSE 3000
+
+CMD ["npm", "start"]
@@ -0,0 +1,7 @@
+{
+  "compilerOptions": {
+    "paths": {
+      "@/*": ["./src/*"]
+    }
+  }
+}
@@ -0,0 +1,26 @@
+/** @type {import('next').NextConfig} */
+const nextConfig = {
+  async rewrites() {
+    return [
+      {
+        source: "/api/:path*",
+        destination: "http://backend:8000/api/:path*",
+      },
+    ];
+  },
+  reactStrictMode: true,
+  experimental: {
+    proxyTimeout: 6000000,
+  },
+  webpack(config) {
+    config.module.rules.push({
+      test: /\.svg$/i,
+      issuer: /\.[jt]sx?$/,
+      use: ["@svgr/webpack"],
+    });
+
+    return config;
+  },
+};
+
+module.exports = nextConfig;
@@ -0,0 +1,25 @@
+{
+  "name": "frontend",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint"
+  },
+  "dependencies": {
+    "autoprefixer": "^10.4.14",
+    "eslint": "8.44.0",
+    "eslint-config-next": "13.4.9",
+    "flowbite": "^1.7.0",
+    "next": "13.4.9",
+    "postcss": "8.4.25",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "tailwindcss": "3.3.2"
+  },
+  "devDependencies": {
+    "@svgr/webpack": "^8.0.1"
+  }
+}
@@ -0,0 +1,6 @@
+module.exports = {
+  plugins: {
+    tailwindcss: {},
+    autoprefixer: {},
+  },
+}
@@ -0,0 +1,20 @@
+<svg
+  viewBox="0 0 24 24"
+  fill="currentColor"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path
+      fill-rule="evenodd"
+      clip-rule="evenodd"
+      d="M14 2C14 2.74028 13.5978 3.38663 13 3.73244V4H20C21.6569 4 23 5.34315 23 7V19C23 20.6569 21.6569 22 20 22H4C2.34315 22 1 20.6569 1 19V7C1 5.34315 2.34315 4 4 4H11V3.73244C10.4022 3.38663 10 2.74028 10 2C10 0.895431 10.8954 0 12 0C13.1046 0 14 0.895431 14 2ZM4 6H11H13H20C20.5523 6 21 6.44772 21 7V19C21 19.5523 20.5523 20 20 20H4C3.44772 20 3 19.5523 3 19V7C3 6.44772 3.44772 6 4 6ZM15 11.5C15 10.6716 15.6716 10 16.5 10C17.3284 10 18 10.6716 18 11.5C18 12.3284 17.3284 13 16.5 13C15.6716 13 15 12.3284 15 11.5ZM16.5 8C14.567 8 13 9.567 13 11.5C13 13.433 14.567 15 16.5 15C18.433 15 20 13.433 20 11.5C20 9.567 18.433 8 16.5 8ZM7.5 10C6.67157 10 6 10.6716 6 11.5C6 12.3284 6.67157 13 7.5 13C8.32843 13 9 12.3284 9 11.5C9 10.6716 8.32843 10 7.5 10ZM4 11.5C4 9.567 5.567 8 7.5 8C9.433 8 11 9.567 11 11.5C11 13.433 9.433 15 7.5 15C5.567 15 4 13.433 4 11.5ZM10.8944 16.5528C10.6474 16.0588 10.0468 15.8586 9.55279 16.1056C9.05881 16.3526 8.85858 16.9532 9.10557 17.4472C9.68052 18.5971 10.9822 19 12 19C13.0178 19 14.3195 18.5971 14.8944 17.4472C15.1414 16.9532 14.9412 16.3526 14.4472 16.1056C13.9532 15.8586 13.3526 16.0588 13.1056 16.5528C13.0139 16.7362 12.6488 17 12 17C11.3512 17 10.9861 16.7362 10.8944 16.5528Z"
+      fill="currentColor"
+    ></path>
+  </g>
+</svg>
@@ -0,0 +1,14 @@
+<svg
+aria-hidden="true"
+xmlns="http://www.w3.org/2000/svg"
+fill="none"
+viewBox="0 0 14 14"
+>
+<path
+    stroke="currentColor"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+    stroke-width="2"
+    d="m1 1 6 6m0 0 6 6M7 7l6-6M7 7l-6 6"
+/>
+</svg>
@@ -0,0 +1,15 @@
+<svg
+  fill="currentColor"
+  viewBox="0 0 32 32"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path d="M18.8,16l5.5-5.5c0.8-0.8,0.8-2,0-2.8l0,0C24,7.3,23.5,7,23,7c-0.5,0-1,0.2-1.4,0.6L16,13.2l-5.5-5.5 c-0.8-0.8-2.1-0.8-2.8,0C7.3,8,7,8.5,7,9.1s0.2,1,0.6,1.4l5.5,5.5l-5.5,5.5C7.3,21.9,7,22.4,7,23c0,0.5,0.2,1,0.6,1.4 C8,24.8,8.5,25,9,25c0.5,0,1-0.2,1.4-0.6l5.5-5.5l5.5,5.5c0.8,0.8,2.1,0.8,2.8,0c0.8-0.8,0.8-2.1,0-2.8L18.8,16z"></path>
+  </g>
+</svg>
@@ -0,0 +1,9 @@
+<svg
+  aria-hidden="true"
+  xmlns="http://www.w3.org/2000/svg"
+  fill="currentColor"
+  viewBox="0 0 22 21"
+>
+  <path d="M16.975 11H10V4.025a1 1 0 0 0-1.066-.998 8.5 8.5 0 1 0 9.039 9.039.999.999 0 0 0-1-1.066h.002Z" />
+  <path d="M12.5 0c-.157 0-.311.01-.565.027A1 1 0 0 0 11 1.02V10h8.975a1 1 0 0 0 1-.935c.013-.188.028-.374.028-.565A8.51 8.51 0 0 0 12.5 0Z" />
+</svg>
@@ -0,0 +1,15 @@
+<svg
+  fill="currentColor"
+  viewBox="0 0 56 56"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path d="M 15.5547 53.125 L 40.4453 53.125 C 45.2969 53.125 47.7109 50.6640 47.7109 45.7890 L 47.7109 24.5078 L 30.7422 24.5078 C 27.7422 24.5078 26.3359 23.0781 26.3359 20.0781 L 26.3359 2.8750 L 15.5547 2.8750 C 10.7266 2.8750 8.2891 5.3594 8.2891 10.2344 L 8.2891 45.7890 C 8.2891 50.6875 10.7266 53.125 15.5547 53.125 Z M 30.8125 21.2969 L 47.4531 21.2969 C 47.2891 20.3359 46.6094 19.3984 45.5078 18.2500 L 32.5703 5.1015 C 31.4922 3.9766 30.5078 3.2969 29.5234 3.1328 L 29.5234 20.0313 C 29.5234 20.875 29.9687 21.2969 30.8125 21.2969 Z M 18.9766 34.6562 C 18.0156 34.6562 17.3359 33.9766 17.3359 33.0625 C 17.3359 32.1484 18.0156 31.4687 18.9766 31.4687 L 37.0469 31.4687 C 37.9844 31.4687 38.7109 32.1484 38.7109 33.0625 C 38.7109 33.9766 37.9844 34.6562 37.0469 34.6562 Z M 18.9766 43.5859 C 18.0156 43.5859 17.3359 42.9062 17.3359 41.9922 C 17.3359 41.0781 18.0156 40.3984 18.9766 40.3984 L 37.0469 40.3984 C 37.9844 40.3984 38.7109 41.0781 38.7109 41.9922 C 38.7109 42.9062 37.9844 43.5859 37.0469 43.5859 Z"></path>
+  </g>
+</svg>
@@ -0,0 +1,12 @@
+<svg
+    aria-hidden="true"
+    fill="currentColor"
+    viewBox="0 0 20 20"
+    xmlns="http://www.w3.org/2000/svg"
+>
+    <path
+    clip-rule="evenodd"
+    fill-rule="evenodd"
+    d="M2 4.75A.75.75 0 012.75 4h14.5a.75.75 0 010 1.5H2.75A.75.75 0 012 4.75zm0 10.5a.75.75 0 01.75-.75h7.5a.75.75 0 010 1.5h-7.5a.75.75 0 01-.75-.75zM2 10a.75.75 0 01.75-.75h14.5a.75.75 0 010 1.5H2.75A.75.75 0 012 10z"
+    ></path>
+</svg>
@@ -0,0 +1,14 @@
+<svg
+aria-hidden="true"
+xmlns="http://www.w3.org/2000/svg"
+fill="none"
+viewBox="0 0 10 6"
+>
+<path
+    stroke="currentColor"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+    stroke-width="2"
+    d="m1 1 4 4 4-4"
+/>
+</svg>
@@ -0,0 +1,14 @@
+<svg
+    aria-hidden="true"
+    xmlns="http://www.w3.org/2000/svg"
+    fill="none"
+    viewBox="0 0 10 6"
+>
+    <path
+    stroke="currentColor"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+    stroke-width="2"
+    d="M9 5 5 1 1 5"
+    />
+</svg>
@@ -0,0 +1,39 @@
+<svg
+viewBox="0 0 20 20"
+version="1.1"
+xmlns="http://www.w3.org/2000/svg"
+xmlns:xlink="http://www.w3.org/1999/xlink"
+fill="currentColor"
+>
+<g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+<g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+></g>
+<g id="SVGRepo_iconCarrier">
+    <title>github [#142]</title> <desc>Created with Sketch.</desc>
+    <defs> </defs>
+    <g
+    id="Page-1"
+    stroke="none"
+    stroke-width="1"
+    fill="none"
+    fill-rule="evenodd"
+    >
+    <g
+        id="Dribbble-Light-Preview"
+        transform="translate(-140.000000, -7559.000000)"
+        fill="currentColor"
+    >
+        <g id="icons" transform="translate(56.000000, 160.000000)">   
+        <path
+            d="M94,7399 C99.523,7399 104,7403.59 104,7409.253 C104,7413.782 101.138,7417.624 97.167,7418.981 C96.66,7419.082 96.48,7418.762 96.48,7418.489 C96.48,7418.151 96.492,7417.047 96.492,7415.675 C96.492,7414.719 96.172,7414.095 95.813,7413.777 C98.04,7413.523 100.38,7412.656 100.38,7408.718 C100.38,7407.598 99.992,7406.684 99.35,7405.966 C99.454,7405.707 99.797,7404.664 99.252,7403.252 C99.252,7403.252 98.414,7402.977 96.505,7404.303 C95.706,7404.076 94.85,7403.962 94,7403.958 C93.15,7403.962 92.295,7404.076 91.497,7404.303 C89.586,7402.977 88.746,7403.252 88.746,7403.252 C88.203,7404.664 88.546,7405.707 88.649,7405.966 C88.01,7406.684 87.619,7407.598 87.619,7408.718 C87.619,7412.646 89.954,7413.526 92.175,7413.785 C91.889,7414.041 91.63,7414.493 91.54,7415.156 C90.97,7415.418 89.522,7415.871 88.63,7414.304 C88.63,7414.304 88.101,7413.319 87.097,7413.247 C87.097,7413.247 86.122,7413.234 87.029,7413.87 C87.029,7413.87 87.684,7414.185 88.139,7415.37 C88.139,7415.37 88.726,7417.2 91.508,7416.58 C91.513,7417.437 91.522,7418.245 91.522,7418.489 C91.522,7418.76 91.338,7419.077 90.839,7418.982 C86.865,7417.627 84,7413.783 84,7409.253 C84,7403.59 88.478,7399 94,7399"
+            id="github-[#142]"
+        >       
+        </path>
+        </g>
+    </g>
+    </g>
+</g>
+</svg>
@@ -0,0 +1,17 @@
+<svg
+  fill="currentColor"
+  viewBox="0 0 32 32"
+  version="1.1"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <title>linkedin</title>
+    <path d="M28.778 1.004h-25.56c-0.008-0-0.017-0-0.027-0-1.199 0-2.172 0.964-2.186 2.159v25.672c0.014 1.196 0.987 2.161 2.186 2.161 0.010 0 0.019-0 0.029-0h25.555c0.008 0 0.018 0 0.028 0 1.2 0 2.175-0.963 2.194-2.159l0-0.002v-25.67c-0.019-1.197-0.994-2.161-2.195-2.161-0.010 0-0.019 0-0.029 0h0.001zM9.9 26.562h-4.454v-14.311h4.454zM7.674 10.293c-1.425 0-2.579-1.155-2.579-2.579s1.155-2.579 2.579-2.579c1.424 0 2.579 1.154 2.579 2.578v0c0 0.001 0 0.002 0 0.004 0 1.423-1.154 2.577-2.577 2.577-0.001 0-0.002 0-0.003 0h0zM26.556 26.562h-4.441v-6.959c0-1.66-0.034-3.795-2.314-3.795-2.316 0-2.669 1.806-2.669 3.673v7.082h-4.441v-14.311h4.266v1.951h0.058c0.828-1.395 2.326-2.315 4.039-2.315 0.061 0 0.121 0.001 0.181 0.003l-0.009-0c4.5 0 5.332 2.962 5.332 6.817v7.855z"></path>
+  </g>
+</svg>
@@ -0,0 +1,28 @@
+<svg
+  viewBox="0 0 15 15"
+  fill="none"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path
+      d="M3.5 8H3V7H3.5C3.77614 7 4 7.22386 4 7.5C4 7.77614 3.77614 8 3.5 8Z"
+      fill="currentColor"
+    ></path>
+    <path
+      d="M7 10V7H7.5C7.77614 7 8 7.22386 8 7.5V9.5C8 9.77614 7.77614 10 7.5 10H7Z"
+      fill="currentColor"
+    ></path>
+    <path
+      fill-rule="evenodd"
+      clip-rule="evenodd"
+      d="M1 1.5C1 0.671573 1.67157 0 2.5 0H10.7071L14 3.29289V13.5C14 14.3284 13.3284 15 12.5 15H2.5C1.67157 15 1 14.3284 1 13.5V1.5ZM3.5 6H2V11H3V9H3.5C4.32843 9 5 8.32843 5 7.5C5 6.67157 4.32843 6 3.5 6ZM7.5 6H6V11H7.5C8.32843 11 9 10.3284 9 9.5V7.5C9 6.67157 8.32843 6 7.5 6ZM10 11V6H13V7H11V8H12V9H11V11H10Z"
+      fill="currentColor"
+    ></path>
+  </g>
+</svg>
@@ -0,0 +1,13 @@
+<svg
+xmlns="http://www.w3.org/2000/svg"
+fill="currentColor"
+viewBox="0 0 24 24"
+stroke="currentColor"
+>
+<path
+    strokeLinecap="round"
+    strokeLinejoin="round"
+    strokeWidth="2"
+    d="M12 6v6m0 0v6m0-6h6m-6 0H6"
+/>
+</svg>
@@ -0,0 +1,57 @@
+<svg
+    viewBox="-0.5 0 25 25"
+    fill="none"
+    xmlns="http://www.w3.org/2000/svg"
+>
+    <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+    <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+    ></g>
+    <g id="SVGRepo_iconCarrier">
+    {" "}
+    <path
+        d="M12 7.82001H22"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    <path
+        d="M2 7.82001H4"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    <path
+        d="M20 16.82H22"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    <path
+        d="M2 16.82H12"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    <path
+        d="M8 11.82C10.2091 11.82 12 10.0291 12 7.82001C12 5.61087 10.2091 3.82001 8 3.82001C5.79086 3.82001 4 5.61087 4 7.82001C4 10.0291 5.79086 11.82 8 11.82Z"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    <path
+        d="M16 20.82C18.2091 20.82 20 19.0291 20 16.82C20 14.6109 18.2091 12.82 16 12.82C13.7909 12.82 12 14.6109 12 16.82C12 19.0291 13.7909 20.82 16 20.82Z"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    </g>
+</svg>
@@ -0,0 +1,61 @@
+<svg
+  viewBox="0 0 24 24"
+  fill="none"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <rect
+      height="4"
+      stroke="currentColor"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      stroke-width="2"
+      width="4"
+      x="10"
+      y="4"
+    ></rect>
+    <rect
+      height="4"
+      stroke="currentColor"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      stroke-width="2"
+      width="4"
+      x="10"
+      y="16"
+    ></rect>
+    <rect
+      height="4"
+      stroke="currentColor"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      stroke-width="2"
+      width="4"
+      x="3"
+      y="16"
+    ></rect>
+    <rect
+      height="4"
+      stroke="currentColor"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      stroke-width="2"
+      width="4"
+      x="17"
+      y="16"
+    ></rect>
+    <path
+      d="M12 8V12M12 16V12M12 12H17C18.1046 12 19 12.8954 19 14V16M12 12H7C5.89543 12 5 12.8954 5 14V16"
+      stroke="currentColor"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      stroke-width="2"
+    ></path>
+  </g>
+</svg>
@@ -0,0 +1,21 @@
+<svg
+  viewBox="0 0 24 24"
+  fill="none"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path
+      d="M12 3V21M9 21H15M19 6V3H5V6"
+      stroke="currentColor"
+      stroke-width="2"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+    ></path>
+  </g>
+</svg>
@@ -0,0 +1,24 @@
+<svg
+  fill="currentColor"
+  version="1.1"
+  xmlns="http://www.w3.org/2000/svg"
+  xmlns:xlink="http://www.w3.org/1999/xlink"
+  viewBox="0 0 512 512"
+  xml:space="preserve"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <g id="7935ec95c421cee6d86eb22ecd12f847">
+      <path
+        style="display: inline;"
+        d="M459.186,151.787c0.203,4.501,0.305,9.023,0.305,13.565 c0,138.542-105.461,298.285-298.274,298.285c-59.209,0-114.322-17.357-160.716-47.104c8.212,0.973,16.546,1.47,25.012,1.47 c49.121,0,94.318-16.759,130.209-44.884c-45.887-0.841-84.596-31.154-97.938-72.804c6.408,1.227,12.968,1.886,19.73,1.886 c9.55,0,18.816-1.287,27.617-3.68c-47.955-9.633-84.1-52.001-84.1-102.795c0-0.446,0-0.882,0.011-1.318 c14.133,7.847,30.294,12.562,47.488,13.109c-28.134-18.796-46.637-50.885-46.637-87.262c0-19.212,5.16-37.218,14.193-52.7 c51.707,63.426,128.941,105.156,216.072,109.536c-1.784-7.675-2.718-15.674-2.718-23.896c0-57.891,46.941-104.832,104.832-104.832 c30.173,0,57.404,12.734,76.525,33.102c23.887-4.694,46.313-13.423,66.569-25.438c-7.827,24.485-24.434,45.025-46.089,58.002 c21.209-2.535,41.426-8.171,60.222-16.505C497.448,118.542,479.666,137.004,459.186,151.787z"
+      > 
+      </path>
+    </g>
+  </g>
+</svg>
@@ -0,0 +1,20 @@
+<svg
+  fill="currentColor"
+  version="1.1"
+  id="Layer_1"
+  xmlns="http://www.w3.org/2000/svg"
+  xmlns:xlink="http://www.w3.org/1999/xlink"
+  viewBox="0 0 24 24"
+  enable-background="new 0 0 24 24"
+  xml:space="preserve"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path d="M18.41,0.01H4V9h2V2.01h10v6h6v14H4v2h20V5.6L18.41,0.01z M18,2.43l3.59,3.59H18V2.43z M21,18v2h-1h-1h-1v-8h2v6H21z M7,12 H6v2h1v6h2v-6h1v-2H9H7z M15,12l-1,1.61L13,12v0h-2v8h2v-4.21l1,1.61l1-1.61V20h2v-8L15,12L15,12z M3,15H2v-3H0v8h2v-3h1v3h2v-8H3 V15z"></path>
+  </g>
+</svg>
@@ -0,0 +1,39 @@
+<svg
+  viewBox="0 -3 20 20"
+  version="1.1"
+  xmlns="http://www.w3.org/2000/svg"
+  xmlns:xlink="http://www.w3.org/1999/xlink"
+  fill="currentColor"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">    
+    <title>youtube [#168]</title> <desc>Created with Sketch.</desc>
+    <defs> </defs>
+    <g
+      id="Page-1"
+      stroke="none"
+      stroke-width="1"
+      fill="none"
+      fill-rule="evenodd"
+    >      
+      <g
+        id="Dribbble-Light-Preview"
+        transform="translate(-300.000000, -7442.000000)"
+        fill="currentColor"
+      >
+        <g id="icons" transform="translate(56.000000, 160.000000)">   
+          <path
+            d="M251.988432,7291.58588 L251.988432,7285.97425 C253.980638,7286.91168 255.523602,7287.8172 257.348463,7288.79353 C255.843351,7289.62824 253.980638,7290.56468 251.988432,7291.58588 M263.090998,7283.18289 C262.747343,7282.73013 262.161634,7282.37809 261.538073,7282.26141 C259.705243,7281.91336 248.270974,7281.91237 246.439141,7282.26141 C245.939097,7282.35515 245.493839,7282.58153 245.111335,7282.93357 C243.49964,7284.42947 244.004664,7292.45151 244.393145,7293.75096 C244.556505,7294.31342 244.767679,7294.71931 245.033639,7294.98558 C245.376298,7295.33761 245.845463,7295.57995 246.384355,7295.68865 C247.893451,7296.0008 255.668037,7296.17532 261.506198,7295.73552 C262.044094,7295.64178 262.520231,7295.39147 262.895762,7295.02447 C264.385932,7293.53455 264.28433,7285.06174 263.090998,7283.18289"
+            id="youtube-[#168]"
+          >
+          </path>
+        </g>
+      </g>
+    </g>
+  </g>
+</svg>
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Taranjeet Singh	09c02954ba	Bump version to 0.0.38 (#442 )	2023-08-15 03:01:11 +05:30
cachho	66b661660b	feat: session id for telemetry (#440 )	2023-08-15 02:58:09 +05:30
cachho	c26559a2d3	fix: notion install error message (#439 )	2023-08-15 02:57:23 +05:30
Taranjeet Singh	c5da46f8b0	update: discord bot docs (#435 )	2023-08-12 06:00:21 +05:30
Sahil Kumar Yadav	0b72269e18	add: support for discord bot (#412 )	2023-08-12 05:52:27 +05:30
cachho	a232d1b779	refactor: do not instantiate all loaders (#418 )	2023-08-12 05:32:40 +05:30
Taranjeet Singh	3cab4415b7	Bump version to 0.0.37 (#431 )	2023-08-12 05:07:51 +05:30
Taranjeet Singh	d494d99c06	docs: update variable name (#430 )	2023-08-12 04:58:37 +05:30
cachho	163f437582	feat: anonymous telemetry (#423 )	2023-08-12 04:57:11 +05:30
cachho	1e0d967bb5	chore: linting (#428 )	2023-08-10 23:42:38 -07:00
Sahil Kumar Yadav	a86deb2675	add: API server(#422 ) Add an example of api server so that devs can quickly get up a bot running along with its api	2023-08-11 11:30:51 +05:30
Taranjeet Singh	d51c508b40	Bump version to 0.0.36 (#426 )	2023-08-11 09:50:39 +05:30
aryankhanna475	af8b3081fa	feat: Update showcase section in the docs (#421 )	2023-08-11 09:44:45 +05:30
Taranjeet Singh	1dbe7daac1	fix: Update embedding field name for Elastiscearch mapping (#425 )	2023-08-11 09:43:52 +05:30
Taranjeet Singh	e56f91a239	Bump version to 0.0.35 (#424 )	2023-08-11 09:29:31 +05:30
Prashant Chaudhary	0179141b2e	feat: add support for Elastcisearch as vector data source (#402 )	2023-08-11 09:23:56 +05:30
cachho	f0abfea55d	chore: linting (#414 )	2023-08-11 01:53:42 +05:30
Taranjeet Singh	77e223be52	Bump version to 0.0.34 (#420 )	2023-08-10 04:48:34 +05:30
Taranjeet Singh	c96df72cd0	Fix: lazy load Notion loader (#419 )	2023-08-10 04:44:02 +05:30
cachho	ce6eb39009	feat: notion loader (#405 )	2023-08-09 13:15:22 +05:30
Jonas	eeac84e2d9	feat: collection name everywhere (#310 ) Co-authored-by: cachho <admin@ch-webdev.com>	2023-08-09 13:08:35 +05:30
Taranjeet Singh	1ee1e671d1	Bump version to 0.0.33 (#411 )	2023-08-09 12:59:45 +05:30
cachho	2ef7c0b736	fix: escape pysqlite swapping (#410 )	2023-08-09 12:54:41 +05:30
aryankhanna475	f2b563e42a	Additions to the community showcase (#401 ) Co-authored-by: Sahil Kumar Yadav <sahilyadav902@gmail.com>	2023-08-09 12:39:36 +05:30
Taranjeet Singh	7a718643a3	bump version to 0.0.32 (#409 )	2023-08-09 12:28:35 +05:30
Taranjeet Singh	1f0f0c93b7	fix: Pass deployment name as param for azure api (#406 )	2023-08-09 12:25:26 +05:30
Taranjeet Singh	030e3521a9	Bump version to 0.0.31 (#408 )	2023-08-09 12:17:57 +05:30
cachho	fdf5d1928d	test: added chunker unit tests (#325 )	2023-08-09 09:12:30 +05:30
cachho	65011a67d4	fix: is readable - zero division error (#383 )	2023-08-09 09:06:26 +05:30
Sahil Kumar Yadav	ec09a8a6fc	example: embedchain playground (#384 )	2023-08-08 08:43:05 -07:00
cachho	5e94980aaa	fix: no logging in pysqlite replacement (#378 )	2023-07-27 07:00:01 -07:00
aryankhanna475	8b619756b6	update: docs showcase (#377 )	2023-07-27 06:58:22 -07:00
cachho	35b43edb20	fix: typo in readme example (#373 )	2023-07-27 06:56:48 -07:00
cachho	02cbde2fc1	docs: fix argument out of order (#374 )	2023-07-27 06:56:18 -07:00
cachho	a868fce036	fix: remove debug logging (#379 )	2023-07-27 06:55:00 -07:00
cachho	079e35b205	fix: chroma pysqlite version (#350 )	2023-07-27 13:12:27 +05:30
Deshraj Yadav	8c91b75b98	[Feature]: Add support for azure openai model (#372 )	2023-07-27 13:03:32 +05:30
cachho	55bfd7cafe	refactor: loader chunker typing (#324 )	2023-07-26 23:14:57 +05:30
cachho	a8552686b4	docs: add back query config (#365 )	2023-07-26 23:13:56 +05:30
Alessandro Panzieri	12a2f78dcb	add "d" on "Embedchain" in citation section title (#369 )	2023-07-26 22:57:01 +05:30
aaishikdutta	cce6d5ddab	fix: Personapp not working with config (#368 )	2023-07-26 22:04:11 +05:30
cachho	088346c4fc	docs: fix template (#364 )	2023-07-25 00:46:50 -07:00
aaishikdutta	7fa7b9e199	Update discord badge in readme.md (#361 )	2023-07-24 09:19:54 -07:00
aaishikdutta	cac15c147f	added fix for documentation (#362 )	2023-07-24 01:18:37 -07:00
aaishikdutta	c54dd1e7bb	Fixed test case for chroma db (#358 )	2023-07-22 16:17:01 -07:00
aaishikdutta	c9c56a4b26	fixed dry_run not working in PersonApp (#357 )	2023-07-21 22:59:20 -07:00
Taranjeet Singh	acbdb800d3	Bump version to 0.0.30 (#355 )	2023-07-22 07:44:38 +05:30
Taranjeet Singh	49711c92b7	bug: fix online bug in chat endpoint (#354 )	2023-07-22 07:42:17 +05:30
aaishikdutta	c4797eb121	fix: fix PersonOpenSourceApp query error (#353 )	2023-07-22 00:14:07 +05:30
Taranjeet Singh	bea4e9e6d1	Bump version to 0.0.29 (#349 )	2023-07-21 13:10:21 +05:30
Taranjeet Singh	9687ec2c1a	fix: init metadata as empty (#348 )	2023-07-21 13:08:46 +05:30
Taranjeet Singh	09da4a3002	Fix package version in setup.py (#345 )	2023-07-21 08:21:30 +05:30
Taranjeet Singh	93004306f2	Bump version to 0.0.28 (#344 )	2023-07-21 08:16:40 +05:30
juaneloDev	aed894246f	fix: add metadata in add and add_local (#343 )	2023-07-21 08:10:01 +05:30
Taranjeet Singh	fa63a16591	bug: Chroma needs diff client as per settings (#342 )	2023-07-21 06:47:01 +05:30
Taranjeet Singh	40643663cb	Bump chroma version to 0.4 in setup.py (#341 )	2023-07-21 03:28:24 +05:30
Candido Sales Gomes	d590e4423b	update: chroma v0.4.0 (#330 )	2023-07-21 02:47:53 +05:30
Taranjeet Singh	cdbf75e0ec	Bump version to 0.0.27 (#337 )	2023-07-21 00:06:54 +05:30