Bump version to 0.0.36 (#426)

Co-authored-by: cachho <admin@ch-webdev.com>

.github/PULL_REQUEST_TEMPLATE.md

@@ -11,14 +11,17 @@ Please delete options that are not relevant.
 - [ ] Bug fix (non-breaking change which fixes an issue)
 - [ ] New feature (non-breaking change which adds functionality)
 - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
+- [ ] Refactor (does not change functionality, e.g. code style improvements, linting)
 - [ ] Documentation update
 
 ## How Has This Been Tested?
 
 Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
 
-- [ ] Test A
-- [ ] Test B
+Please delete options that are not relevant.
+
+- [ ] Unit Test
+- [ ] Test Script (please provide)
 
 ## Checklist:
 

README.md

@@ -1,7 +1,7 @@
 # embedchain
 
 [![PyPI](https://img.shields.io/pypi/v/embedchain)](https://pypi.org/project/embedchain/)
-[![Discord](https://dcbadge.vercel.app/api/server/nhvCbCtKV?style=flat)](https://discord.gg/6PzXDgEjG5)
+[![Discord](https://dcbadge.vercel.app/api/server/6PzXDgEjG5?style=flat)](https://discord.gg/6PzXDgEjG5)
 [![Twitter](https://img.shields.io/twitter/follow/embedchain)](https://twitter.com/embedchain)
 [![Substack](https://img.shields.io/badge/Substack-%23006f5c.svg?logo=substack)](https://embedchain.substack.com/)
 [![Open in Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/138lMWhENGeEu7Q1-6lNbNTHGLZXBBz_B?usp=sharing)
@@ -14,6 +14,29 @@ Embedchain is a framework to easily create LLM powered bots over any dataset. If
 pip install embedchain
 ```
 
+## 🔥 Latest
+
+- **[2023/07/19]** Released support for 🦙 `llama2` model. Start creating your `llama2` based bots like this:
+
+  ```python
+  import os
+
+  from embedchain import Llama2App
+
+  os.environ['REPLICATE_API_TOKEN'] = "REPLICATE API TOKEN"
+
+  zuck_bot = Llama2App()
+
+  # Embed your data
+  zuck_bot.add("youtube_video", "https://www.youtube.com/watch?v=Ff4fRgnuFgQ")
+  zuck_bot.add("web_page", "https://en.wikipedia.org/wiki/Mark_Zuckerberg")
+
+  # Nice, your bot is ready now. Start asking questions to your bot.
+  zuck_bot.query("Who is Mark Zuckerberg?")
+  # Answer: Mark Zuckerberg is an American internet entrepreneur and business magnate. He is the co-founder and CEO of Facebook.
+  ```
+
+
 ## 🔍 Demo
 
 Try out embedchain in your browser:
@@ -62,7 +85,7 @@ If you utilize this repository, please consider citing it with:
 ```
 @misc{embedchain,
   author = {Taranjeet Singh},
-  title = {Embechain: Framework to easily create LLM powered bots over any dataset},
+  title = {Embedchain: Framework to easily create LLM powered bots over any dataset},
   year = {2023},
   publisher = {GitHub},
   journal = {GitHub repository},

docs/advanced/adding_data.mdx

@@ -0,0 +1,25 @@
+---
+title: '➕ Adding Data'
+---
+
+## Add Dataset
+
+- This step assumes that you have already created an `app` instance by either using `App`, `OpenSourceApp` or `CustomApp`. We are calling our app instance as `naval_chat_bot` 🤖
+
+- Now use `.add()` function to add any dataset.
+
+```python
+# naval_chat_bot = App() or
+# naval_chat_bot = OpenSourceApp()
+
+# Embed Online Resources
+naval_chat_bot.add("youtube_video", "https://www.youtube.com/watch?v=3qHkcs3kG44")
+naval_chat_bot.add("pdf_file", "https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
+naval_chat_bot.add("web_page", "https://nav.al/feedback")
+naval_chat_bot.add("web_page", "https://nav.al/agi")
+
+# Embed Local Resources
+naval_chat_bot.add_local("qna_pair", ("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."))
+```
+
+The possible formats to add data can be found on the [Supported Data Formats](/advanced/data_types) page.

docs/advanced/app_types.mdx

@@ -2,12 +2,6 @@
 title: '📱 App types'
 ---
 
-Creating a chatbot involves 3 steps:
-
-- ⚙️ Import the App instance
-- 🗃️ Add Dataset
-- 💬 Query or Chat on the dataset and get answers (Interface Types)
-
 ## App Types
 
 We have three types of App.
@@ -16,13 +10,12 @@ We have three types of App.
 
 ```python
 from embedchain import App
-naval_chat_bot = App()
+app = App()
 ```
 
 - `App` uses OpenAI's model, so these are paid models. 💸 You will be charged for embedding model usage and LLM usage.
-
 - `App` uses OpenAI's embedding model to create embeddings for chunks and ChatGPT API as LLM to get answer given the relevant docs. Make sure that you have an OpenAI account and an API key. If you have don't have an API key, you can create one by visiting [this link](https://platform.openai.com/account/api-keys).
-
+- `App` is opinionated. It uses the best embedding model and LLM on the market.
 - Once you have the API key, set it in an environment variable called `OPENAI_API_KEY`
 
 ```python
@@ -30,16 +23,76 @@ import os
 os.environ["OPENAI_API_KEY"] = "sk-xxxx"
 ```
 
+### Llama2App
+
+```python
+import os
+
+from embedchain import Llama2App
+
+os.environ['REPLICATE_API_TOKEN'] = "REPLICATE API TOKEN"
+
+zuck_bot = Llama2App()
+
+# Embed your data
+zuck_bot.add("youtube_video", "https://www.youtube.com/watch?v=Ff4fRgnuFgQ")
+zuck_bot.add("web_page", "https://en.wikipedia.org/wiki/Mark_Zuckerberg")
+
+# Nice, your bot is ready now. Start asking questions to your bot.
+zuck_bot.query("Who is Mark Zuckerberg?")
+# Answer: Mark Zuckerberg is an American internet entrepreneur and business magnate. He is the co-founder and CEO of Facebook. Born in 1984, he dropped out of Harvard University to focus on his social media platform, which has since grown to become one of the largest and most influential technology companies in the world.
+
+# Enable web search for your bot
+zuck_bot.online = True # enable internet access for the bot
+zuck_bot.query("Who owns the new threads app and when it was founded?")
+# Answer: Based on the context provided, the new Threads app is owned by Meta, the parent company of Facebook, Instagram, and WhatsApp.
+```
+
+- `Llama2App` uses Replicate's LLM model, so these are paid models. You can get the `REPLICATE_API_TOKEN` by registering on [their website](https://replicate.com/account).
+- `Llama2App` uses OpenAI's embedding model to create embeddings for chunks. Make sure that you have an OpenAI account and an API key. If you have don't have an API key, you can create one by visiting [this link](https://platform.openai.com/account/api-keys).
+
+
 ### OpenSourceApp
 
 ```python
 from embedchain import OpenSourceApp
-naval_chat_bot = OpenSourceApp()
+app = OpenSourceApp()
 ```
 
 - `OpenSourceApp` uses open source embedding and LLM model. It uses `all-MiniLM-L6-v2` from Sentence Transformers library as the embedding model and `gpt4all` as the LLM.
 - Here there is no need to setup any api keys. You just need to install embedchain package and these will get automatically installed. 📦
 - Once you have imported and instantiated the app, every functionality from here onwards is the same for either type of app. 📚
+- `OpenSourceApp` is opinionated. It uses the best open source embedding model and LLM on the market.
+
+### CustomApp
+
+```python
+from embedchain import CustomApp
+from embedchain.config import CustomAppConfig
+from embedchain.models import Providers, EmbeddingFunctions
+
+config = CustomAppConfig(embedding_fn=EmbeddingFunctions.OPENAI, provider=Providers.OPENAI)
+app = CustomApp(config)
+```
+
+- `CustomApp` is not opinionated.
+- Configuration required. It's for advanced users who want to mix and match different embedding models and LLMs. Configuration required.
+- while it's doing that, it's still providing abstractions through `Providers`.
+- paid and free/open source providers included.
+- Once you have imported and instantiated the app, every functionality from here onwards is the same for either type of app. 📚
+- Following providers are available for an LLM
+    - OPENAI
+    - ANTHPROPIC
+    - VERTEX_AI
+    - GPT4ALL
+    - AZURE_OPENAI
+- Following embedding functions are available for an embedding function
+    - OPENAI
+    - HUGGING_FACE
+    - VERTEX_AI
+    - GPT4ALL
+    - AZURE_OPENAI
+
 
 ### PersonApp
 
@@ -57,25 +110,7 @@ import os
 os.environ["OPENAI_API_KEY"] = "sk-xxxx"
 ```
 
-## Add Dataset
-
-- This step assumes that you have already created an `app` instance by either using `App` or `OpenSourceApp`. We are calling our app instance as `naval_chat_bot` 🤖
-
-- Now use `.add()` function to add any dataset.
-
-```python
-# naval_chat_bot = App() or
-# naval_chat_bot = OpenSourceApp()
-
-# Embed Online Resources
-naval_chat_bot.add("youtube_video", "https://www.youtube.com/watch?v=3qHkcs3kG44")
-naval_chat_bot.add("pdf_file", "https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
-naval_chat_bot.add("web_page", "https://nav.al/feedback")
-naval_chat_bot.add("web_page", "https://nav.al/agi")
-
-# Embed Local Resources
-naval_chat_bot.add_local("qna_pair", ("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."))
-```
+#### Compatibility with other apps
 
 - If there is any other app instance in your script or app, you can change the import as
 
@@ -90,49 +125,3 @@ from embedchain import App as ECApp
 from embedchain import OpenSourceApp as ECOSApp
 from embedchain import PersonApp as ECPApp
 ```
-
-## Interface Types
-
-### Query Interface
-
-- This interface is like a question answering bot. It takes a question and gets the answer. It does not maintain context about the previous chats.❓
-
-- To use this, call `.query()` function to get the answer for any query.
-
-```python
-print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
-# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
-```
-
-### Chat Interface
-
-- This interface is chat interface where it remembers previous conversation. Right now it remembers 5 conversation by default. 💬
-
-- To use this, call `.chat` function to get the answer for any query.
-
-```python
-print(naval_chat_bot.chat("How to be happy in life?"))
-# answer: The most important trick to being happy is to realize happiness is a skill you develop and a choice you make. You choose to be happy, and then you work at it. It's just like building muscles or succeeding at your job. It's about recognizing the abundance and gifts around you at all times.
-
-print(naval_chat_bot.chat("who is naval ravikant?"))
-# answer: Naval Ravikant is an Indian-American entrepreneur and investor.
-
-print(naval_chat_bot.chat("what did the author say about happiness?"))
-# answer: The author, Naval Ravikant, believes that happiness is a choice you make and a skill you develop. He compares the mind to the body, stating that just as the body can be molded and changed, so can the mind. He emphasizes the importance of being present in the moment and not getting caught up in regrets of the past or worries about the future. By being present and grateful for where you are, you can experience true happiness.
-```
-
-### Stream Response
-
-- You can add config to your query method to stream responses like ChatGPT does. You would require a downstream handler to render the chunk in your desirable format. Supports both OpenAI model and OpenSourceApp. 📊
-
-- To use this, instantiate a `QueryConfig` or `ChatConfig` object with `stream=True`. Then pass it to the `.chat()` or `.query()` method. The following example iterates through the chunks and prints them as they appear.
-
-```python
-app = App()
-query_config = QueryConfig(stream = True)
-resp = app.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", query_config)
-
-for chunk in resp:
-    print(chunk, end="", flush=True)
-# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
-```

docs/advanced/configuration.mdx

@@ -6,22 +6,22 @@ Embedchain is made to work out of the box. However, for advanced users we're als
 
 ## Examples
 
-### Custom embedding function
+### General
 
 Here's the readme example with configuration options.
 
 ```python
 import os
 from embedchain import App
-from embedchain.config import InitConfig, AddConfig, QueryConfig, ChunkerConfig
+from embedchain.config import AppConfig, AddConfig, QueryConfig, ChunkerConfig
 from chromadb.utils import embedding_functions
 
-# Example: use your own embedding function
-config = InitConfig(ef=embedding_functions.OpenAIEmbeddingFunction(
-                api_key=os.getenv("OPENAI_API_KEY"),
-                organization_id=os.getenv("OPENAI_ORGANIZATION"),
-                model_name="text-embedding-ada-002"
-            ))
+# Example: set the log level for debugging
+config = AppConfig(log_level="DEBUG")
+naval_chat_bot = App(config)
+
+# Example: specify a custom collection name
+config = AppConfig(collection_name="naval_chat_bot")
 naval_chat_bot = App(config)
 
 # Example: define your own chunker config for `youtube_video`
@@ -35,7 +35,7 @@ naval_chat_bot.add("web_page", "https://nav.al/agi", add_config)
 
 naval_chat_bot.add_local("qna_pair", ("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."), add_config)
 
-query_config = QueryConfig() # Currently no options
+query_config = QueryConfig()
 print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", query_config))
 ```
 
@@ -68,7 +68,7 @@ einstein_chat_template = Template("""
 
         Human: $query
         Albert Einstein:""")
-query_config = QueryConfig(einstein_chat_template)
+query_config = QueryConfig(template=einstein_chat_template)
 queries = [
         "Where did you complete your studies?",
         "Why did you win nobel prize?",
@@ -87,22 +87,3 @@ for query in queries:
 # Query:  Why did you divorce your first wife?
 # Response:  We divorced due to living apart for five years.
 ```
-
-## Other methods
-
-### Reset
-
-Resets the database and deletes all embeddings. Irreversible. Requires reinitialization afterwards.
-
-```python
-app.reset()
-```
-
-### Count
-
-Counts the number of embeddings (chunks) in the database.
-
-```python
-print(app.count())
-# returns: 481
-```

docs/advanced/data_types.mdx

@@ -30,6 +30,14 @@ To add any web page, use the data_type as `web_page`. Eg:
 app.add('web_page', 'a_valid_web_page_url')
 ```
 
+### Sitemap
+
+Add all web pages from an xml-sitemap. Filters non-text files. Use the data_type as `sitemap`. Eg:
+
+```python
+app.add('sitemap', 'https://example.com/sitemap.xml')
+```
+
 ### Doc file
 
 To add any doc/docx file, use the data_type as `docx`. Eg:
@@ -46,6 +54,18 @@ To add any code documentation website as a loader, use the data_type as `docs_si
 app.add("docs_site", "https://docs.embedchain.ai/")
 ```
 
+### Notion
+To use notion you must install the extra dependencies with `pip install embedchain[notion]`.
+
+To load a notion page, use the data_type as `notion`.
+The next argument must **end** with the `notion page id`. The id is a 32-character string. Eg:
+
+```python
+app.add("notion", "cfbc134ca6464fc980d0391613959196")
+app.add("notion", "my-page-cfbc134ca6464fc980d0391613959196")
+app.add("notion", "https://www.notion.so/my-page-cfbc134ca6464fc980d0391613959196")
+```
+
 ### Text
 
 To supply your own text, use the data_type as `text` and enter a string. The text is not processed, this can be very versatile. Eg:

docs/advanced/interface_types.mdx

@@ -0,0 +1,75 @@
+---
+title: '🤝 Interface types'
+---
+
+## Interface Types
+
+The embedchain app exposes the following methods.
+
+### Query Interface
+
+- This interface is like a question answering bot. It takes a question and gets the answer. It does not maintain context about the previous chats.❓
+
+- To use this, call `.query()` function to get the answer for any query.
+
+```python
+print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
+# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
+```
+
+### Chat Interface
+
+- This interface is chat interface where it remembers previous conversation. Right now it remembers 5 conversation by default. 💬
+
+- To use this, call `.chat` function to get the answer for any query.
+
+```python
+print(naval_chat_bot.chat("How to be happy in life?"))
+# answer: The most important trick to being happy is to realize happiness is a skill you develop and a choice you make. You choose to be happy, and then you work at it. It's just like building muscles or succeeding at your job. It's about recognizing the abundance and gifts around you at all times.
+
+print(naval_chat_bot.chat("who is naval ravikant?"))
+# answer: Naval Ravikant is an Indian-American entrepreneur and investor.
+
+print(naval_chat_bot.chat("what did the author say about happiness?"))
+# answer: The author, Naval Ravikant, believes that happiness is a choice you make and a skill you develop. He compares the mind to the body, stating that just as the body can be molded and changed, so can the mind. He emphasizes the importance of being present in the moment and not getting caught up in regrets of the past or worries about the future. By being present and grateful for where you are, you can experience true happiness.
+```
+
+#### Dry Run
+
+Dry Run is an option in the `query` and `chat` methods that allows the user to not send their constructed prompt to the LLM, to save money. It's used for [testing](/advanced/testing#dry-run).
+
+
+### Stream Response
+
+- You can add config to your query method to stream responses like ChatGPT does. You would require a downstream handler to render the chunk in your desirable format. Supports both OpenAI model and OpenSourceApp. 📊
+
+- To use this, instantiate a `QueryConfig` or `ChatConfig` object with `stream=True`. Then pass it to the `.chat()` or `.query()` method. The following example iterates through the chunks and prints them as they appear.
+
+```python
+app = App()
+query_config = QueryConfig(stream = True)
+resp = app.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", query_config)
+
+for chunk in resp:
+    print(chunk, end="", flush=True)
+# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
+```
+
+### Other Methods
+
+#### Reset
+
+Resets the database and deletes all embeddings. Irreversible. Requires reinitialization afterwards.
+
+```python
+app.reset()
+```
+
+#### Count
+
+Counts the number of embeddings (chunks) in the database.
+
+```python
+print(app.count())
+# returns: 481
+```

docs/advanced/query_configuration.mdx

@@ -2,13 +2,14 @@
 title: '🔍 Query configurations'
 ---
 
-## InitConfig
+## AppConfig
 
 | option    | description           | type                            | default                |
 |-----------|-----------------------|---------------------------------|------------------------|
 | log_level | log level             | string                          | WARNING                |
-| ef        | embedding function    | chromadb.utils.embedding_functions | \{text-embedding-ada-002\} |
+| embedding_fn| embedding function    | chromadb.utils.embedding_functions | \{text-embedding-ada-002\} |
 | db        | vector database (experimental) | BaseVectorDB               | ChromaDB               |
+| collection_name | initial collection name for the database | string             | embedchain_store |
 
 
 ## AddConfig
@@ -18,6 +19,14 @@ title: '🔍 Query configurations'
 |chunker|chunker config|ChunkerConfig|Default values for chunker depends on the `data_type`. Please refer [ChunkerConfig](#chunker-config)|
 |loader|loader config|LoaderConfig|None|
 
+Yes, you are passing `ChunkerConfig` to `AddConfig`, like so:
+
+```python
+chunker_config = ChunkerConfig(chunk_size=100)
+add_config = AddConfig(chunker=chunker_config)
+app.add_local("text", "lorem ipsum", config=add_config)
+```
+
 ### ChunkerConfig
 
 |option|description|type|default|
@@ -36,6 +45,8 @@ Default values of chunker config parameters for different `data_type`:
 |web_page|500|0|len|
 |pdf_file|1000|0|len|
 |youtube_video|2000|0|len|
+|docs_site|500|50|len|
+|notion|300|0|len|
 
 ### LoaderConfig
 
@@ -45,9 +56,14 @@ _coming soon_
 
 |option|description|type|default|
 |---|---|---|---|
-|template|custom template for prompt|Template|Template("Use the following pieces of context to answer the query at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. \$context Query: \$query Helpful Answer:")|
-|history|include conversation history from your client or database|any (recommendation: list[str])|None
-|stream|control if response is streamed back to the user|bool|False|
+|number_documents|Absolute number of documents to pull from the database as context.|int|1
+|template|custom template for prompt. If history is used with query, $history has to be included as well.|Template|Template("Use the following pieces of context to answer the query at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. \$context Query: \$query Helpful Answer:")|
+|model|name of the model used.|string|depends on app type|
+|temperature|Controls the randomness of the model's output. Higher values (closer to 1) make output more random, lower values make it more deterministic.|float|0|
+|max_tokens|Controls how many tokens are used. Exact implementation (whether it counts prompt and/or response) depends on the model.|int|1000|
+|top_p|Controls the diversity of words. Higher values (closer to 1) make word selection more diverse, lower values make words less diverse.|float|1|
+|history|include conversation history from your client or database.|any (recommendation: list[str])|None|
+|stream|control if response is streamed back to the user.|bool|False|
 
 ## ChatConfig
 
@@ -55,4 +71,4 @@ All options for query and...
 
 _coming soon_
 
-History is handled automatically, the config option is not supported.
+`history` is not supported, as that is handled is handled automatically, the config option is not supported.

docs/advanced/showcase.mdx

@@ -10,35 +10,58 @@ Embedchain community has been super active in creating demos on top of Embedchai
 
 - [Discord Bot for LLM chat](https://github.com/Reidond/discord_bots_playground/tree/c8b0c36541e4b393782ee506804c4b6962426dd6/python/chat-channel-bot) by Reidond
 - [EmbedChain-Streamlit-Docker App](https://github.com/amjadraza/embedchain-streamlit-app) by amjadraza
-- [Harry Potter Philosphers Stone Bot](https://github.com/vinayak-kempawad/Harry_Potter_Philosphers_Stone_Bot/) by Vinayak Kempawad, ([linkedin post](https://www.linkedin.com/feed/update/urn:li:activity:7080907532155686912/))
+- [Harry Potter Philosphers Stone Bot](https://github.com/vinayak-kempawad/Harry_Potter_Philosphers_Stone_Bot/) by Vinayak Kempawad, ([LinkedIn post](https://www.linkedin.com/feed/update/urn:li:activity:7080907532155686912/))
 - [LLM bot trained on own messages](https://github.com/Harin329/harinBot) by Hao Wu
 
 ### Closed Source
 
 - [Taobot.io](https://taobot.io) - chatbot & knowledgebase hybrid by [cachho](https://github.com/cachho)
+- [Create Instant ChatBot 🤖 using embedchain](https://databutton.com/v/h3e680h9) by Avra, ([Tweet](https://twitter.com/Avra_b/status/1674704745154641920/))
+- [JOBO 🤖 — The AI-driven sidekick to craft your resume](https://try-jobo.com/) by Enrico Willemse, ([LinkedIn Post](https://www.linkedin.com/posts/enrico-willemse_jobai-gptfun-embedchain-activity-7090340080879374336-ueLB/))
+- [Explore Your Knowledge Base: Interactive chats over various forms of documents](https://chatdocs.dkedar.com/) by Kedar Dabhadkar, ([LinkedIn Post](https://www.linkedin.com/posts/dkedar7_machinelearning-llmops-activity-7092524836639424513-2O3L/))
+- [Chatbot trained on 1000+ videos of Ester hicks the co-author behind the famous book Secret](https://ask-abraham.thoughtseed.repl.co) by Mohan Kumar
+
 
 ## Templates
 
 ### Replit
 - [Embedchain Chat Bot](https://replit.com/@taranjeet1/Embedchain-Chat-Bot) by taranjeetio
 - [Embedchain Memory Chat Bot Template](https://replit.com/@taranjeetio/Embedchain-Memory-Chat-Bot-Template) by taranjeetio
+- [Chatbot app to demonstrate question-answering using retrieved information](https://replit.com/@AllisonMorrell/EmbedChainlitPublic) by Allison Morrell, ([LinkedIn Post](https://www.linkedin.com/posts/allison-morrell-2889275a_retrievalbot-screenshots-activity-7080339991754649600-wihZ/))
 
 ## Posts
 
 ### Blogs
 
-- [Customer Service LINE Bot](https://www.evanlin.com/langchain-embedchain/)
+- [Customer Service LINE Bot](https://www.evanlin.com/langchain-embedchain/) by Evan Lin
+- [Chatbot in Under 5 mins using Embedchain](https://medium.com/@ayush.wattal/chatbot-in-under-5-mins-using-embedchain-a4f161fcf9c5) by Ayush Wattal
+- [Understanding what the LLM framework embedchain does](https://zenn.dev/hijikix/articles/4bc8d60156a436) by Daisuke Hashimoto
+- [In bed with GPT and Node.js](https://dev.to/worldlinetech/in-bed-with-gpt-and-nodejs-4kh2) by Raphaël Semeteys, ([LinkedIn Post](https://www.linkedin.com/posts/raphaelsemeteys_in-bed-with-gpt-and-nodejs-activity-7088113552326029313-nn87/))
+- [Using Embedchain — A powerful LangChain Python wrapper to build Chat Bots even faster!⚡](https://medium.com/@avra42/using-embedchain-a-powerful-langchain-python-wrapper-to-build-chat-bots-even-faster-35c12994a360) by Avra, ([Tweet](https://twitter.com/Avra_b/status/1686767751560310784/))
 
 ### LinkedIn
 
 - [What is embedchain](https://www.linkedin.com/posts/activity-7079393104423698432-wRyi/) by Rithesh Sreenivasan
 - [Building a chatbot with EmbedChain](https://www.linkedin.com/posts/activity-7078434598984060928-Zdso/) by Lior Sinclair
 - [Making chatbot without vs with embedchain](https://www.linkedin.com/posts/kalyanksnlp_llms-chatbots-langchain-activity-7077453416221863936-7N1L/) by Kalyan KS
+- [EmbedChain - very intuitive, first you index your data and then query!](https://www.linkedin.com/posts/shubhamsaboo_embedchain-a-framework-to-easily-create-activity-7079535460699557888-ad1X/) by Shubham Saboo
+- [EmbedChain - Harnessing power of LLM](https://www.linkedin.com/posts/uditsaini_chatbotrevolution-llmpoweredbots-embedchainframework-activity-7077520356827181056-FjTK/) by Udit S.
+- [AI assistant for ABBYY Vantage](https://www.linkedin.com/posts/maximevermeir_llm-github-abbyy-activity-7081658972071424000-fXfZ/) by Maxime V.
+- [About embedchain](https://www.linkedin.com/feed/update/urn:li:activity:7080984218914189312/) by Morris Lee
+- [How to use Embedchain](https://www.linkedin.com/posts/nehaabansal_github-embedchainembedchain-framework-activity-7085830340136595456-kbW5/) by Neha Bansal
+- [Youtube/Webpage summary for Energy Study](https://www.linkedin.com/posts/bar%C4%B1%C5%9F-sanl%C4%B1-34b82715_enerji-python-activity-7082735341563977730-Js0U/) by Barış Sanlı, ([Tweet](https://twitter.com/barissanli/status/1676968784979193857/)) 
 
 ### Twitter
 
 - [What is embedchain](https://twitter.com/AlphaSignalAI/status/1672668574450847745) by Lior
 - [Building a chatbot with Embedchain](https://twitter.com/Saboo_Shubham_/status/1673537044419686401) by Shubham Saboo
+- [Chatbot docker image behind an API with yaml configs with Embedchain](https://twitter.com/tricalt/status/1678411430192730113/) by Vasilije
+- [Build AI powered PDF chatbot with just five lines of Python code with Embedchain!](https://twitter.com/Saboo_Shubham_/status/1676627104866156544/) by Shubham Saboo
+- [Chatbot against a youtube video using embedchain](https://twitter.com/smaameri/status/1675201443043704834/) by Sami Maameri
+- [Highlights of EmbedChain](https://twitter.com/carl_AIwarts/status/1673542204328120321/) by carl_AIwarts
+- [Build Llama-2 chatbot in less than 5 minutes](https://twitter.com/Saboo_Shubham_/status/1682168956918833152/) by Shubham Saboo
+- [All cool features of embedchain](https://twitter.com/DhravyaShah/status/1683497882438217728/) by Dhravya Shah, ([LinkedIn Post](https://www.linkedin.com/posts/dhravyashah_what-if-i-tell-you-that-you-can-make-an-ai-activity-7089459599287726080-ZIYm/))
+- [Read paid Medium articles for Free using embedchain](https://twitter.com/kumarkaushal_/status/1688952961622585344) by Kaushal Kumar
 
 ## Videos
 
@@ -49,6 +72,13 @@ Embedchain community has been super active in creating demos on top of Embedchai
 - [How To Create A Custom Knowledge AI Powered Bot | Install + How To Use](https://www.youtube.com/watch?v=VfCrIiAst-c) by The Ai Solopreneur
 - [Build Custom Chatbot in 6 min with this Framework [Beginner Friendly]](https://www.youtube.com/watch?v=-8HxOpaFySM) by Maya Akim
 - [embedchain-streamlit-app](https://www.youtube.com/watch?v=3-9GVd-3v74) by Amjad Raza
+- [🤖CHAT with ANY ONLINE RESOURCES using EMBEDCHAIN - a LangChain wrapper, in few lines of code !](https://www.youtube.com/watch?v=Mp7zJe4TIdM) by Avra
+- [Building resource-driven LLM-powered bots with Embedchain](https://www.youtube.com/watch?v=IVfcAgxTO4I) by BugBytes
+- [embedchain-streamlit-demo](https://www.youtube.com/watch?v=yJAWB13FhYQ) by Amjad Raza
+- [Embedchain - create your own AI chatbots using open source models](https://www.youtube.com/shorts/O3rJWKwSrWE) by Dhravya Shah
+- [AI ChatBot in 5 lines Python Code](https://www.youtube.com/watch?v=zjWvLJLksv8) by Data Engineering
+- [Interview with Karl Marx](https://www.youtube.com/watch?v=5Y4Tscwj1xk) by Alexander Ray Williams
+- [Vlog where we try to build a bot based on our content on the internet](https://www.youtube.com/watch?v=I2w8CWM3bx4) by DV, ([Tweet](https://twitter.com/dvcoolster/status/1688387017544261632))
 
 ## Mentions
 

docs/advanced/testing.mdx

@@ -2,14 +2,18 @@
 title: '🧪 Testing'
 ---
 
+## Methods for testing
+
+### Dry Run
+
 Before you consume valueable tokens, you should make sure that the embedding you have done works and that it's receiving the correct document from the database.
 
-For this you can use the `dry_run` method.
+For this you can use the `dry_run` option in your `query` or `chat` method.
 
 Following the example above, add this to your script:
 
 ```python
-print(naval_chat_bot.dry_run('Can you tell me who Naval Ravikant is?'))
+print(naval_chat_bot.query('Can you tell me who Naval Ravikant is?', dry_run=True))
 
 '''
 Use the following pieces of context to answer the query at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

docs/advanced/vector_database.mdx

@@ -0,0 +1,34 @@
+---
+title: '💾 Vector Database'
+---
+
+We support `Chroma` and `Elasticsearch` as two vector database. 
+`Chroma` is used as a default database.
+
+### Elasticsearch
+In order to use `Elasticsearch` as vector database we need to use App type `CustomApp`. 
+```python
+import os
+from embedchain import CustomApp
+from embedchain.config import CustomAppConfig, ElasticsearchDBConfig
+from embedchain.models import Providers, EmbeddingFunctions, VectorDatabases
+
+os.environ["OPENAI_API_KEY"] = 'OPENAI_API_KEY'
+
+es_config = ElasticsearchDBConfig(
+	# elasticsearch url or list of nodes url with different hosts and ports.
+	es_url='http://localhost:9200',
+	# pass named parameters supported by Python Elasticsearch client
+	ca_certs="/path/to/http_ca.crt",
+	basic_auth=("username", "password")
+)
+config = CustomAppConfig(
+	embedding_fn=EmbeddingFunctions.OPENAI, 
+	provider=Providers.OPENAI, 
+	db_type=VectorDatabases.ELASTICSEARCH, 
+	es_config=es_config,
+)
+es_app = CustomApp(config)
+```
+- Set `db_type=VectorDatabases.ELASTICSEARCH` and `es_config=ElasticsearchDBConfig(es_url='')` in `CustomAppConfig`.
+- `ElasticsearchDBConfig` accepts `es_url` as elasticsearch url or as list of nodes url with different hosts and ports. Additionally we can pass named paramaters supported by Python Elasticsearch client.

docs/examples/full_stack.mdx

@@ -0,0 +1,22 @@
+---
+title: '🌐 Full Stack'
+---
+
+### 🐳 Docker Setup
+
+- To setup full stack app using docker, run the following command inside this folder using your terminal.
+
+```bash
+docker-compose up --build
+```
+
+📝 Note: The build command might take a while to install all the packages depending on your system resources.
+
+### 🚀 Usage Instructions
+
+- Go to [http://localhost:3000/](http://localhost:3000/) in your browser to view the dashboard.
+- Add your `OpenAI API key` 🔑 in the Settings.
+- Create a new bot and you'll be navigated to its page.
+- Here you can add your data sources and then chat with the bot.
+
+🎉 Happy Chatting! 🎉

docs/mint.json

@@ -32,7 +32,11 @@
     },
     {
       "group": "Advanced",
-      "pages": ["advanced/app_types", "advanced/data_types", "advanced/query_configuration", "advanced/configuration", "advanced/testing", "advanced/showcase"]
+      "pages": ["advanced/app_types", "advanced/interface_types", "advanced/adding_data","advanced/data_types", "advanced/query_configuration", "advanced/configuration", "advanced/testing", "advanced/vector_database", "advanced/showcase"]
+    },
+    {
+      "group": "Examples",
+      "pages": ["examples/full_stack"]
     },
     {
       "group": "Contribution Guidelines",

docs/quickstart.mdx

@@ -9,12 +9,18 @@ Install embedchain python package:
 pip install embedchain
 ```
 
+Creating a chatbot involves 3 steps:
+
+- ⚙️ Import the App instance
+- 🗃️ Add Dataset
+- 💬 Query or Chat on the dataset and get answers (Interface Types)
+
 Run your first bot in python using the following code. Make sure to set the `OPENAI_API_KEY` 🔑 environment variable in the code.
 
 ```python
 import os
 
-from embedchain Import App
+from embedchain import App
 
 os.environ["OPENAI_API_KEY"] = "xxx"
 elon_musk_bot = App()
@@ -23,7 +29,7 @@ elon_musk_bot = App()
 elon_musk_bot.add("web_page", "https://en.wikipedia.org/wiki/Elon_Musk")
 elon_musk_bot.add("web_page", "https://www.tesla.com/elon-musk")
 
-response = elon_bot.query("How many companies does Elon Musk run?")
+response = elon_musk_bot.query("How many companies does Elon Musk run?")
 print(response)
 # Answer: 'Elon Musk runs four companies: Tesla, SpaceX, Neuralink, and The Boring Company.'
 ```

embedchain/__init__.py

@@ -2,7 +2,9 @@ import importlib.metadata
 
 __version__ = importlib.metadata.version(__package__ or __name__)
 
-from .embedchain import App  # noqa: F401
-from .embedchain import OpenSourceApp  # noqa: F401
-from .embedchain import PersonApp  # noqa: F401
-from .embedchain import PersonOpenSourceApp  # noqa: F401
+from embedchain.apps.App import App  # noqa: F401
+from embedchain.apps.CustomApp import CustomApp  # noqa: F401
+from embedchain.apps.Llama2App import Llama2App  # noqa: F401
+from embedchain.apps.OpenSourceApp import OpenSourceApp  # noqa: F401
+from embedchain.apps.PersonApp import (PersonApp,  # noqa: F401
+                                       PersonOpenSourceApp)

embedchain/apps/App.py

@@ -0,0 +1,49 @@
+import openai
+
+from embedchain.config import AppConfig, ChatConfig
+from embedchain.embedchain import EmbedChain
+
+
+class App(EmbedChain):
+    """
+    The EmbedChain app.
+    Has two functions: add and query.
+
+    adds(data_type, url): adds the data from the given URL to the vector db.
+    query(query): finds answer to the given query using vector database and LLM.
+    dry_run(query): test your prompt without consuming tokens.
+    """
+
+    def __init__(self, config: AppConfig = None):
+        """
+        :param config: AppConfig instance to load as configuration. Optional.
+        """
+        if config is None:
+            config = AppConfig()
+
+        super().__init__(config)
+
+    def get_llm_model_answer(self, prompt, config: ChatConfig):
+        messages = []
+        messages.append({"role": "user", "content": prompt})
+        response = openai.ChatCompletion.create(
+            model=config.model or "gpt-3.5-turbo-0613",
+            messages=messages,
+            temperature=config.temperature,
+            max_tokens=config.max_tokens,
+            top_p=config.top_p,
+            stream=config.stream,
+        )
+
+        if config.stream:
+            return self._stream_llm_model_response(response)
+        else:
+            return response["choices"][0]["message"]["content"]
+
+    def _stream_llm_model_response(self, response):
+        """
+        This is a generator for streaming response from the OpenAI completions API
+        """
+        for line in response:
+            chunk = line["choices"][0].get("delta", {}).get("content", "")
+            yield chunk

embedchain/apps/CustomApp.py

@@ -0,0 +1,152 @@
+import logging
+from typing import List
+
+from langchain.schema import BaseMessage
+
+from embedchain.config import ChatConfig, CustomAppConfig
+from embedchain.embedchain import EmbedChain
+from embedchain.models import Providers
+
+
+class CustomApp(EmbedChain):
+    """
+    The custom EmbedChain app.
+    Has two functions: add and query.
+
+    adds(data_type, url): adds the data from the given URL to the vector db.
+    query(query): finds answer to the given query using vector database and LLM.
+    dry_run(query): test your prompt without consuming tokens.
+    """
+
+    def __init__(self, config: CustomAppConfig = None):
+        """
+        :param config: Optional. `CustomAppConfig` instance to load as configuration.
+        :raises ValueError: Config must be provided for custom app
+        """
+        if config is None:
+            raise ValueError("Config must be provided for custom app")
+
+        self.provider = config.provider
+
+        if config.provider == Providers.GPT4ALL:
+            from embedchain import OpenSourceApp
+
+            # Because these models run locally, they should have an instance running when the custom app is created
+            self.open_source_app = OpenSourceApp(config=config.open_source_app_config)
+
+        super().__init__(config)
+
+    def set_llm_model(self, provider: Providers):
+        self.provider = provider
+        if provider == Providers.GPT4ALL:
+            raise ValueError(
+                "GPT4ALL needs to be instantiated with the model known, please create a new app instance instead"
+            )
+
+    def get_llm_model_answer(self, prompt, config: ChatConfig):
+        # TODO: Quitting the streaming response here for now.
+        # Idea: https://gist.github.com/jvelezmagic/03ddf4c452d011aae36b2a0f73d72f68
+        if config.stream:
+            raise NotImplementedError(
+                "Streaming responses have not been implemented for this model yet. Please disable."
+            )
+
+        try:
+            if self.provider == Providers.OPENAI:
+                return CustomApp._get_openai_answer(prompt, config)
+
+            if self.provider == Providers.ANTHROPHIC:
+                return CustomApp._get_athrophic_answer(prompt, config)
+
+            if self.provider == Providers.VERTEX_AI:
+                return CustomApp._get_vertex_answer(prompt, config)
+
+            if self.provider == Providers.GPT4ALL:
+                return self.open_source_app._get_gpt4all_answer(prompt, config)
+
+            if self.provider == Providers.AZURE_OPENAI:
+                return CustomApp._get_azure_openai_answer(prompt, config)
+
+        except ImportError as e:
+            raise ImportError(e.msg) from None
+
+    @staticmethod
+    def _get_openai_answer(prompt: str, config: ChatConfig) -> str:
+        from langchain.chat_models import ChatOpenAI
+
+        chat = ChatOpenAI(
+            temperature=config.temperature,
+            model=config.model or "gpt-3.5-turbo",
+            max_tokens=config.max_tokens,
+            streaming=config.stream,
+        )
+
+        if config.top_p and config.top_p != 1:
+            logging.warning("Config option `top_p` is not supported by this model.")
+
+        messages = CustomApp._get_messages(prompt)
+
+        return chat(messages).content
+
+    @staticmethod
+    def _get_athrophic_answer(prompt: str, config: ChatConfig) -> str:
+        from langchain.chat_models import ChatAnthropic
+
+        chat = ChatAnthropic(temperature=config.temperature, model=config.model)
+
+        if config.max_tokens and config.max_tokens != 1000:
+            logging.warning("Config option `max_tokens` is not supported by this model.")
+
+        messages = CustomApp._get_messages(prompt)
+
+        return chat(messages).content
+
+    @staticmethod
+    def _get_vertex_answer(prompt: str, config: ChatConfig) -> str:
+        from langchain.chat_models import ChatVertexAI
+
+        chat = ChatVertexAI(temperature=config.temperature, model=config.model, max_output_tokens=config.max_tokens)
+
+        if config.top_p and config.top_p != 1:
+            logging.warning("Config option `top_p` is not supported by this model.")
+
+        messages = CustomApp._get_messages(prompt)
+
+        return chat(messages).content
+
+    @staticmethod
+    def _get_azure_openai_answer(prompt: str, config: ChatConfig) -> str:
+        from langchain.chat_models import AzureChatOpenAI
+
+        if not config.deployment_name:
+            raise ValueError("Deployment name must be provided for Azure OpenAI")
+
+        chat = AzureChatOpenAI(
+            deployment_name=config.deployment_name,
+            openai_api_version="2023-05-15",
+            model_name=config.model or "gpt-3.5-turbo",
+            temperature=config.temperature,
+            max_tokens=config.max_tokens,
+            streaming=config.stream,
+        )
+
+        if config.top_p and config.top_p != 1:
+            logging.warning("Config option `top_p` is not supported by this model.")
+
+        messages = CustomApp._get_messages(prompt)
+
+        return chat(messages).content
+
+    @staticmethod
+    def _get_messages(prompt: str) -> List[BaseMessage]:
+        from langchain.schema import HumanMessage, SystemMessage
+
+        return [SystemMessage(content="You are a helpful assistant."), HumanMessage(content=prompt)]
+
+    def _stream_llm_model_response(self, response):
+        """
+        This is a generator for streaming response from the OpenAI completions API
+        """
+        for line in response:
+            chunk = line["choices"][0].get("delta", {}).get("content", "")
+            yield chunk

embedchain/apps/Llama2App.py

@@ -0,0 +1,36 @@
+import os
+
+from langchain.llms import Replicate
+
+from embedchain.config import AppConfig
+from embedchain.embedchain import EmbedChain
+
+
+class Llama2App(EmbedChain):
+    """
+    The EmbedChain Llama2App class.
+    Has two functions: add and query.
+
+    adds(data_type, url): adds the data from the given URL to the vector db.
+    query(query): finds answer to the given query using vector database and LLM.
+    """
+
+    def __init__(self, config: AppConfig = None):
+        """
+        :param config: AppConfig instance to load as configuration. Optional.
+        """
+        if "REPLICATE_API_TOKEN" not in os.environ:
+            raise ValueError("Please set the REPLICATE_API_TOKEN environment variable.")
+
+        if config is None:
+            config = AppConfig()
+
+        super().__init__(config)
+
+    def get_llm_model_answer(self, prompt, config: AppConfig = None):
+        # TODO: Move the model and other inputs into config
+        llm = Replicate(
+            model="a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5",
+            input={"temperature": 0.75, "max_length": 500, "top_p": 1},
+        )
+        return llm(prompt)

embedchain/apps/OpenSourceApp.py

@@ -0,0 +1,65 @@
+import logging
+from typing import Iterable, Union
+
+from embedchain.config import ChatConfig, OpenSourceAppConfig
+from embedchain.embedchain import EmbedChain
+
+gpt4all_model = None
+
+
+class OpenSourceApp(EmbedChain):
+    """
+    The OpenSource app.
+    Same as App, but uses an open source embedding model and LLM.
+
+    Has two function: add and query.
+
+    adds(data_type, url): adds the data from the given URL to the vector db.
+    query(query): finds answer to the given query using vector database and LLM.
+    """
+
+    def __init__(self, config: OpenSourceAppConfig = None):
+        """
+        :param config: OpenSourceAppConfig instance to load as configuration. Optional.
+        `ef` defaults to open source.
+        """
+        logging.info("Loading open source embedding model. This may take some time...")  # noqa:E501
+        if not config:
+            config = OpenSourceAppConfig()
+
+        if not config.model:
+            raise ValueError("OpenSourceApp needs a model to be instantiated. Maybe you passed the wrong config type?")
+
+        self.instance = OpenSourceApp._get_instance(config.model)
+
+        logging.info("Successfully loaded open source embedding model.")
+        super().__init__(config)
+
+    def get_llm_model_answer(self, prompt, config: ChatConfig):
+        return self._get_gpt4all_answer(prompt=prompt, config=config)
+
+    @staticmethod
+    def _get_instance(model):
+        try:
+            from gpt4all import GPT4All
+        except ModuleNotFoundError:
+            raise ValueError(
+                "The GPT4All python package is not installed. Please install it with `pip install GPT4All`"
+            ) from None
+
+        return GPT4All(model)
+
+    def _get_gpt4all_answer(self, prompt: str, config: ChatConfig) -> Union[str, Iterable]:
+        if config.model and config.model != self.config.model:
+            raise RuntimeError(
+                "OpenSourceApp does not support switching models at runtime. Please create a new app instance."
+            )
+
+        response = self.instance.generate(
+            prompt=prompt,
+            streaming=config.stream,
+            top_p=config.top_p,
+            max_tokens=config.max_tokens,
+            temp=config.temperature,
+        )
+        return response

embedchain/apps/PersonApp.py

@@ -0,0 +1,80 @@
+from string import Template
+
+from embedchain.apps.App import App
+from embedchain.apps.OpenSourceApp import OpenSourceApp
+from embedchain.config import ChatConfig, QueryConfig
+from embedchain.config.apps.BaseAppConfig import BaseAppConfig
+from embedchain.config.QueryConfig import (DEFAULT_PROMPT,
+                                           DEFAULT_PROMPT_WITH_HISTORY)
+
+
+class EmbedChainPersonApp:
+    """
+    Base class to create a person bot.
+    This bot behaves and speaks like a person.
+
+    :param person: name of the person, better if its a well known person.
+    :param config: BaseAppConfig instance to load as configuration.
+    """
+
+    def __init__(self, person, config: BaseAppConfig = None):
+        self.person = person
+        self.person_prompt = f"You are {person}. Whatever you say, you will always say in {person} style."  # noqa:E501
+        super().__init__(config)
+
+    def add_person_template_to_config(self, default_prompt: str, config: ChatConfig = None):
+        """
+        This method checks if the config object contains a prompt template
+        if yes it adds the person prompt to it and return the updated config
+        else it creates a config object with the default prompt added to the person prompt
+
+        :param default_prompt: it is the default prompt for query or chat methods
+        :param config: Optional. The `ChatConfig` instance to use as
+        configuration options.
+        """
+        template = Template(self.person_prompt + " " + default_prompt)
+
+        if config:
+            if config.template:
+                # Add person prompt to custom user template
+                config.template = Template(self.person_prompt + " " + config.template.template)
+            else:
+                # If no user template is present, use person prompt with the default template
+                config.template = template
+        else:
+            # if no config is present at all, initialize the config with person prompt and default template
+            config = QueryConfig(
+                template=template,
+            )
+
+        return config
+
+
+class PersonApp(EmbedChainPersonApp, App):
+    """
+    The Person app.
+    Extends functionality from EmbedChainPersonApp and App
+    """
+
+    def query(self, input_query, config: QueryConfig = None, dry_run=False):
+        config = self.add_person_template_to_config(DEFAULT_PROMPT, config)
+        return super().query(input_query, config, dry_run)
+
+    def chat(self, input_query, config: ChatConfig = None, dry_run=False):
+        config = self.add_person_template_to_config(DEFAULT_PROMPT_WITH_HISTORY, config)
+        return super().chat(input_query, config, dry_run)
+
+
+class PersonOpenSourceApp(EmbedChainPersonApp, OpenSourceApp):
+    """
+    The Person app.
+    Extends functionality from EmbedChainPersonApp and OpenSourceApp
+    """
+
+    def query(self, input_query, config: QueryConfig = None, dry_run=False):
+        config = self.add_person_template_to_config(DEFAULT_PROMPT, config)
+        return super().query(input_query, config, dry_run)
+
+    def chat(self, input_query, config: ChatConfig = None, dry_run=False):
+        config = self.add_person_template_to_config(DEFAULT_PROMPT_WITH_HISTORY, config)
+        return super().chat(input_query, config, dry_run)

embedchain/chunkers/base_chunker.py

@@ -5,6 +5,7 @@ class BaseChunker:
     def __init__(self, text_splitter):
         """Initialize the chunker."""
         self.text_splitter = text_splitter
+        self.data_type = None
 
     def create_chunks(self, loader, src):
         """
@@ -22,7 +23,10 @@ class BaseChunker:
         metadatas = []
         for data in datas:
             content = data["content"]
+
             meta_data = data["meta_data"]
+            # add data type to meta data to allow query using data type
+            meta_data["data_type"] = self.data_type
             url = meta_data["url"]
 
             chunks = self.get_chunks(content)
@@ -47,3 +51,9 @@ class BaseChunker:
         Override in child class if custom logic.
         """
         return self.text_splitter.split_text(content)
+
+    def set_data_type(self, data_type):
+        """
+        set the data type of chunker
+        """
+        self.data_type = data_type

embedchain/chunkers/docs_site.py

@@ -5,18 +5,16 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter
 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.AddConfig import ChunkerConfig
 
-TEXT_SPLITTER_CHUNK_PARAMS = {
-    "chunk_size": 500,
-    "chunk_overlap": 50,
-    "length_function": len,
-}
-
 
 class DocsSiteChunker(BaseChunker):
     """Chunker for code docs site."""
 
     def __init__(self, config: Optional[ChunkerConfig] = None):
         if config is None:
-            config = TEXT_SPLITTER_CHUNK_PARAMS
-        text_splitter = RecursiveCharacterTextSplitter(**config)
+            config = ChunkerConfig(chunk_size=500, chunk_overlap=50, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
         super().__init__(text_splitter)

embedchain/chunkers/notion.py

@@ -0,0 +1,20 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.AddConfig import ChunkerConfig
+
+
+class NotionChunker(BaseChunker):
+    """Chunker for notion."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=300, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)

embedchain/config/ChatConfig.py

@@ -33,6 +33,7 @@ class ChatConfig(QueryConfig):
         max_tokens=None,
         top_p=None,
         stream: bool = False,
+        deployment_name=None,
     ):
         """
         Initializes the ChatConfig instance.
@@ -68,6 +69,7 @@ class ChatConfig(QueryConfig):
             top_p=top_p,
             history=[0],
             stream=stream,
+            deployment_name=deployment_name,
         )
 
     def set_history(self, history):

embedchain/config/InitConfig.py

@@ -1,74 +0,0 @@
-import logging
-import os
-
-from chromadb.utils import embedding_functions
-
-from embedchain.config.BaseConfig import BaseConfig
-
-
-class InitConfig(BaseConfig):
-    """
-    Config to initialize an embedchain `App` instance.
-    """
-
-    def __init__(self, log_level=None, ef=None, db=None, host=None, port=None, id=None):
-        """
-        :param log_level: Optional. (String) Debug level
-        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
-        :param ef: Optional. Embedding function to use.
-        :param db: Optional. (Vector) database to use for embeddings.
-        :param id: Optional. ID of the app. Document metadata will have this id.
-        :param host: Optional. Hostname for the database server.
-        :param port: Optional. Port for the database server.
-        """
-        self._setup_logging(log_level)
-        self.ef = ef
-        self.db = db
-        self.host = host
-        self.port = port
-        self.id = id
-        return
-
-    def _set_embedding_function(self, ef):
-        self.ef = ef
-        return
-
-    def _set_embedding_function_to_default(self):
-        """
-        Sets embedding function to default (`text-embedding-ada-002`).
-
-        :raises ValueError: If the template is not valid as template should contain
-        $context and $query
-        """
-        if os.getenv("OPENAI_API_KEY") is None and os.getenv("OPENAI_ORGANIZATION") is None:
-            raise ValueError("OPENAI_API_KEY or OPENAI_ORGANIZATION environment variables not provided")  # noqa:E501
-        self.ef = embedding_functions.OpenAIEmbeddingFunction(
-            api_key=os.getenv("OPENAI_API_KEY"),
-            organization_id=os.getenv("OPENAI_ORGANIZATION"),
-            model_name="text-embedding-ada-002",
-        )
-        return
-
-    def _set_db(self, db):
-        if db:
-            self.db = db
-        return
-
-    def _set_db_to_default(self):
-        """
-        Sets database to default (`ChromaDb`).
-        """
-        from embedchain.vectordb.chroma_db import ChromaDB
-
-        self.db = ChromaDB(ef=self.ef, host=self.host, port=self.port)
-
-    def _setup_logging(self, debug_level):
-        level = logging.WARNING  # Default level
-        if debug_level is not None:
-            level = getattr(logging, debug_level.upper(), None)
-            if not isinstance(level, int):
-                raise ValueError(f"Invalid log level: {debug_level}")
-
-        logging.basicConfig(format="%(asctime)s [%(name)s] [%(levelname)s] %(message)s", level=level)
-        self.logger = logging.getLogger(__name__)
-        return

embedchain/config/QueryConfig.py

@@ -62,6 +62,7 @@ class QueryConfig(BaseConfig):
         top_p=None,
         history=None,
         stream: bool = False,
+        deployment_name=None,
     ):
         """
         Initializes the QueryConfig instance.
@@ -104,8 +105,9 @@ class QueryConfig(BaseConfig):
 
         self.temperature = temperature if temperature else 0
         self.max_tokens = max_tokens if max_tokens else 1000
-        self.model = model if model else "gpt-3.5-turbo-0613"
+        self.model = model
         self.top_p = top_p if top_p else 1
+        self.deployment_name = deployment_name
 
         if self.validate_template(template):
             self.template = template

embedchain/config/__init__.py

@@ -1,5 +1,8 @@
 from .AddConfig import AddConfig, ChunkerConfig  # noqa: F401
+from .apps.AppConfig import AppConfig  # noqa: F401
+from .apps.CustomAppConfig import CustomAppConfig  # noqa: F401
+from .apps.OpenSourceAppConfig import OpenSourceAppConfig  # noqa: F401
 from .BaseConfig import BaseConfig  # noqa: F401
 from .ChatConfig import ChatConfig  # noqa: F401
-from .InitConfig import InitConfig  # noqa: F401
 from .QueryConfig import QueryConfig  # noqa: F401
+from .vectordbs.ElasticsearchDBConfig import ElasticsearchDBConfig  # noqa: F401

embedchain/config/apps/AppConfig.py

@@ -0,0 +1,52 @@
+import os
+
+try:
+    from chromadb.utils import embedding_functions
+except RuntimeError:
+    from embedchain.utils import use_pysqlite3
+
+    use_pysqlite3()
+    from chromadb.utils import embedding_functions
+
+from .BaseAppConfig import BaseAppConfig
+
+
+class AppConfig(BaseAppConfig):
+    """
+    Config to initialize an embedchain custom `App` instance, with extra config options.
+    """
+
+    def __init__(self, log_level=None, host=None, port=None, id=None, collection_name=None):
+        """
+        :param log_level: Optional. (String) Debug level
+        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
+        :param host: Optional. Hostname for the database server.
+        :param port: Optional. Port for the database server.
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param collection_name: Optional. Collection name for the database.
+        """
+        super().__init__(
+            log_level=log_level,
+            embedding_fn=AppConfig.default_embedding_function(),
+            host=host,
+            port=port,
+            id=id,
+            collection_name=collection_name,
+        )
+
+    @staticmethod
+    def default_embedding_function():
+        """
+        Sets embedding function to default (`text-embedding-ada-002`).
+
+        :raises ValueError: If the template is not valid as template should contain
+        $context and $query
+        :returns: The default embedding function for the app class.
+        """
+        if os.getenv("OPENAI_API_KEY") is None and os.getenv("OPENAI_ORGANIZATION") is None:
+            raise ValueError("OPENAI_API_KEY or OPENAI_ORGANIZATION environment variables not provided")  # noqa:E501
+        return embedding_functions.OpenAIEmbeddingFunction(
+            api_key=os.getenv("OPENAI_API_KEY"),
+            organization_id=os.getenv("OPENAI_ORGANIZATION"),
+            model_name="text-embedding-ada-002",
+        )

embedchain/config/apps/BaseAppConfig.py

@@ -0,0 +1,95 @@
+import logging
+
+from embedchain.config.BaseConfig import BaseConfig
+from embedchain.config.vectordbs import ElasticsearchDBConfig
+from embedchain.models import VectorDatabases, VectorDimensions
+
+
+class BaseAppConfig(BaseConfig):
+    """
+    Parent config to initialize an instance of `App`, `OpenSourceApp` or `CustomApp`.
+    """
+
+    def __init__(
+        self,
+        log_level=None,
+        embedding_fn=None,
+        db=None,
+        host=None,
+        port=None,
+        id=None,
+        collection_name=None,
+        db_type: VectorDatabases = None,
+        vector_dim: VectorDimensions = None,
+        es_config: ElasticsearchDBConfig = None,
+    ):
+        """
+        :param log_level: Optional. (String) Debug level
+        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
+        :param embedding_fn: Embedding function to use.
+        :param db: Optional. (Vector) database instance to use for embeddings.
+        :param host: Optional. Hostname for the database server.
+        :param port: Optional. Port for the database server.
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param collection_name: Optional. Collection name for the database.
+        :param db_type: Optional. type of Vector database to use
+        :param vector_dim: Vector dimension generated by embedding fn
+        :param es_config: Optional. elasticsearch database config to be used for connection
+        """
+        self._setup_logging(log_level)
+        self.collection_name = collection_name if collection_name else "embedchain_store"
+        self.db = BaseAppConfig.get_db(
+            db=db,
+            embedding_fn=embedding_fn,
+            host=host,
+            port=port,
+            db_type=db_type,
+            vector_dim=vector_dim,
+            collection_name=self.collection_name,
+            es_config=es_config,
+        )
+        self.id = id
+        return
+
+    @staticmethod
+    def get_db(db, embedding_fn, host, port, db_type, vector_dim, collection_name, es_config):
+        """
+        Get db based on db_type, db with default database (`ChromaDb`)
+        :param Optional. (Vector) database to use for embeddings.
+        :param embedding_fn: Embedding function to use in database.
+        :param host: Optional. Hostname for the database server.
+        :param port: Optional. Port for the database server.
+        :param db_type: Optional. db type to use. Supported values (`es`, `chroma`)
+        :param vector_dim: Vector dimension generated by embedding fn
+        :param collection_name: Optional. Collection name for the database.
+        :param es_config: Optional. elasticsearch database config to be used for connection
+        :raises ValueError: BaseAppConfig knows no default embedding function.
+        :returns: database instance
+        """
+        if db:
+            return db
+
+        if embedding_fn is None:
+            raise ValueError("ChromaDb cannot be instantiated without an embedding function")
+
+        if db_type == VectorDatabases.ELASTICSEARCH:
+            from embedchain.vectordb.elasticsearch_db import ElasticsearchDB
+
+            return ElasticsearchDB(
+                embedding_fn=embedding_fn, vector_dim=vector_dim, collection_name=collection_name, es_config=es_config
+            )
+
+        from embedchain.vectordb.chroma_db import ChromaDB
+
+        return ChromaDB(embedding_fn=embedding_fn, host=host, port=port)
+
+    def _setup_logging(self, debug_level):
+        level = logging.WARNING  # Default level
+        if debug_level is not None:
+            level = getattr(logging, debug_level.upper(), None)
+            if not isinstance(level, int):
+                raise ValueError(f"Invalid log level: {debug_level}")
+
+        logging.basicConfig(format="%(asctime)s [%(name)s] [%(levelname)s] %(message)s", level=level)
+        self.logger = logging.getLogger(__name__)
+        return

embedchain/config/apps/CustomAppConfig.py

@@ -0,0 +1,135 @@
+from typing import Any
+
+from chromadb.api.types import Documents, Embeddings
+from dotenv import load_dotenv
+
+from embedchain.config.vectordbs import ElasticsearchDBConfig
+from embedchain.models import EmbeddingFunctions, Providers, VectorDatabases, VectorDimensions
+
+from .BaseAppConfig import BaseAppConfig
+
+load_dotenv()
+
+
+class CustomAppConfig(BaseAppConfig):
+    """
+    Config to initialize an embedchain custom `App` instance, with extra config options.
+    """
+
+    def __init__(
+        self,
+        log_level=None,
+        embedding_fn: EmbeddingFunctions = None,
+        embedding_fn_model=None,
+        db=None,
+        host=None,
+        port=None,
+        id=None,
+        collection_name=None,
+        provider: Providers = None,
+        open_source_app_config=None,
+        deployment_name=None,
+        db_type: VectorDatabases = None,
+        es_config: ElasticsearchDBConfig = None,
+    ):
+        """
+        :param log_level: Optional. (String) Debug level
+        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
+        :param embedding_fn: Optional. Embedding function to use.
+        :param embedding_fn_model: Optional. Model name to use for embedding function.
+        :param db: Optional. (Vector) database to use for embeddings.
+        :param host: Optional. Hostname for the database server.
+        :param port: Optional. Port for the database server.
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param collection_name: Optional. Collection name for the database.
+        :param provider: Optional. (Providers): LLM Provider to use.
+        :param open_source_app_config: Optional. Config instance needed for open source apps.
+        :param db_type: Optional. type of Vector database to use.
+        :param es_config: Optional. elasticsearch database config to be used for connection
+        """
+        if provider:
+            self.provider = provider
+        else:
+            raise ValueError("CustomApp must have a provider assigned.")
+
+        self.open_source_app_config = open_source_app_config
+
+        super().__init__(
+            log_level=log_level,
+            embedding_fn=CustomAppConfig.embedding_function(
+                embedding_function=embedding_fn, model=embedding_fn_model, deployment_name=deployment_name
+            ),
+            db=db,
+            host=host,
+            port=port,
+            id=id,
+            collection_name=collection_name,
+            db_type=db_type,
+            vector_dim=CustomAppConfig.get_vector_dimension(embedding_function=embedding_fn),
+            es_config=es_config,
+        )
+
+    @staticmethod
+    def langchain_default_concept(embeddings: Any):
+        """
+        Langchains default function layout for embeddings.
+        """
+
+        def embed_function(texts: Documents) -> Embeddings:
+            return embeddings.embed_documents(texts)
+
+        return embed_function
+
+    @staticmethod
+    def embedding_function(embedding_function: EmbeddingFunctions, model: str = None, deployment_name: str = None):
+        if not isinstance(embedding_function, EmbeddingFunctions):
+            raise ValueError(
+                f"Invalid option: '{embedding_function}'. Expecting one of the following options: {list(map(lambda x: x.value, EmbeddingFunctions))}"  # noqa: E501
+            )
+
+        if embedding_function == EmbeddingFunctions.OPENAI:
+            from langchain.embeddings import OpenAIEmbeddings
+
+            if model:
+                embeddings = OpenAIEmbeddings(model=model)
+            else:
+                if deployment_name:
+                    embeddings = OpenAIEmbeddings(deployment=deployment_name)
+                else:
+                    embeddings = OpenAIEmbeddings()
+            return CustomAppConfig.langchain_default_concept(embeddings)
+
+        elif embedding_function == EmbeddingFunctions.HUGGING_FACE:
+            from langchain.embeddings import HuggingFaceEmbeddings
+
+            embeddings = HuggingFaceEmbeddings(model_name=model)
+            return CustomAppConfig.langchain_default_concept(embeddings)
+
+        elif embedding_function == EmbeddingFunctions.VERTEX_AI:
+            from langchain.embeddings import VertexAIEmbeddings
+
+            embeddings = VertexAIEmbeddings(model_name=model)
+            return CustomAppConfig.langchain_default_concept(embeddings)
+
+        elif embedding_function == EmbeddingFunctions.GPT4ALL:
+            # Note: We could use langchains GPT4ALL embedding, but it's not available in all versions.
+            from chromadb.utils import embedding_functions
+
+            return embedding_functions.SentenceTransformerEmbeddingFunction(model_name=model)
+
+    @staticmethod
+    def get_vector_dimension(embedding_function: EmbeddingFunctions):
+        if not isinstance(embedding_function, EmbeddingFunctions):
+            raise ValueError(f"Invalid option: '{embedding_function}'.")
+
+        if embedding_function == EmbeddingFunctions.OPENAI:
+            return VectorDimensions.OPENAI.value
+
+        elif embedding_function == EmbeddingFunctions.HUGGING_FACE:
+            return VectorDimensions.HUGGING_FACE.value
+
+        elif embedding_function == EmbeddingFunctions.VERTEX_AI:
+            return VectorDimensions.VERTEX_AI.value
+
+        elif embedding_function == EmbeddingFunctions.GPT4ALL:
+            return VectorDimensions.GPT4ALL.value

embedchain/config/apps/OpenSourceAppConfig.py

@@ -0,0 +1,40 @@
+from chromadb.utils import embedding_functions
+
+from .BaseAppConfig import BaseAppConfig
+
+
+class OpenSourceAppConfig(BaseAppConfig):
+    """
+    Config to initialize an embedchain custom `OpenSourceApp` instance, with extra config options.
+    """
+
+    def __init__(self, log_level=None, host=None, port=None, id=None, collection_name=None, model=None):
+        """
+        :param log_level: Optional. (String) Debug level
+        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param collection_name: Optional. Collection name for the database.
+        :param host: Optional. Hostname for the database server.
+        :param port: Optional. Port for the database server.
+        :param model: Optional. GPT4ALL uses the model to instantiate the class.
+        So unlike `App`, it has to be provided before querying.
+        """
+        self.model = model or "orca-mini-3b.ggmlv3.q4_0.bin"
+
+        super().__init__(
+            log_level=log_level,
+            embedding_fn=OpenSourceAppConfig.default_embedding_function(),
+            host=host,
+            port=port,
+            id=id,
+            collection_name=collection_name,
+        )
+
+    @staticmethod
+    def default_embedding_function():
+        """
+        Sets embedding function to default (`all-MiniLM-L6-v2`).
+
+        :returns: The default embedding function
+        """
+        return embedding_functions.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2")

embedchain/config/vectordbs/ElasticsearchDBConfig.py

@@ -0,0 +1,15 @@
+from typing import Dict, List, Union
+
+from embedchain.config.BaseConfig import BaseConfig
+
+
+class ElasticsearchDBConfig(BaseConfig):
+    """
+    Config to initialize an elasticsearch client.
+    :param es_url. elasticsearch url or list of nodes url to be used for connection
+    :param ES_EXTRA_PARAMS: extra params dict that can be passed to elasticsearch.
+    """
+
+    def __init__(self, es_url: Union[str, List[str]] = None, **ES_EXTRA_PARAMS: Dict[str, any]):
+        self.ES_URL = es_url
+        self.ES_EXTRA_PARAMS = ES_EXTRA_PARAMS

embedchain/data_formatter/data_formatter.py

@@ -1,5 +1,6 @@
 from embedchain.chunkers.docs_site import DocsSiteChunker
 from embedchain.chunkers.docx_file import DocxFileChunker
+from embedchain.chunkers.notion import NotionChunker
 from embedchain.chunkers.pdf_file import PdfFileChunker
 from embedchain.chunkers.qna_pair import QnaPairChunker
 from embedchain.chunkers.text import TextChunker
@@ -45,8 +46,16 @@ class DataFormatter:
             "sitemap": SitemapLoader(),
             "docs_site": DocsSiteLoader(),
         }
+        lazy_loaders = ("notion",)
         if data_type in loaders:
             return loaders[data_type]
+        elif data_type in lazy_loaders:
+            if data_type == "notion":
+                from embedchain.loaders.notion import NotionLoader
+
+                return NotionLoader()
+            else:
+                raise ValueError(f"Unsupported data type: {data_type}")
         else:
             raise ValueError(f"Unsupported data type: {data_type}")
 
@@ -58,17 +67,21 @@ class DataFormatter:
         :return: The chunker for the given data type.
         :raises ValueError: If an unsupported data type is provided.
         """
-        chunkers = {
-            "youtube_video": YoutubeVideoChunker(config),
-            "pdf_file": PdfFileChunker(config),
-            "web_page": WebPageChunker(config),
-            "qna_pair": QnaPairChunker(config),
-            "text": TextChunker(config),
-            "docx": DocxFileChunker(config),
-            "sitemap": WebPageChunker(config),
-            "docs_site": DocsSiteChunker(config),
+        chunker_classes = {
+            "youtube_video": YoutubeVideoChunker,
+            "pdf_file": PdfFileChunker,
+            "web_page": WebPageChunker,
+            "qna_pair": QnaPairChunker,
+            "text": TextChunker,
+            "docx": DocxFileChunker,
+            "sitemap": WebPageChunker,
+            "docs_site": DocsSiteChunker,
+            "notion": NotionChunker,
         }
-        if data_type in chunkers:
-            return chunkers[data_type]
+        if data_type in chunker_classes:
+            chunker_class = chunker_classes[data_type]
+            chunker = chunker_class(config)
+            chunker.set_data_type(data_type)
+            return chunker
         else:
             raise ValueError(f"Unsupported data type: {data_type}")

embedchain/embedchain.py

@@ -1,18 +1,16 @@
 import logging
 import os
-from string import Template
 
-import openai
-from chromadb.utils import embedding_functions
 from dotenv import load_dotenv
 from langchain.docstore.document import Document
 from langchain.memory import ConversationBufferMemory
 
-from embedchain.config import AddConfig, ChatConfig, InitConfig, QueryConfig
-from embedchain.config.QueryConfig import DOCS_SITE_PROMPT_TEMPLATE, DEFAULT_PROMPT, DEFAULT_PROMPT_WITH_HISTORY
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config import AddConfig, ChatConfig, QueryConfig
+from embedchain.config.apps.BaseAppConfig import BaseAppConfig
+from embedchain.config.QueryConfig import DOCS_SITE_PROMPT_TEMPLATE
 from embedchain.data_formatter import DataFormatter
-
-gpt4all_model = None
+from embedchain.loaders.base_loader import BaseLoader
 
 load_dotenv()
 
@@ -23,17 +21,17 @@ memory = ConversationBufferMemory()
 
 
 class EmbedChain:
-    def __init__(self, config: InitConfig):
+    def __init__(self, config: BaseAppConfig):
         """
         Initializes the EmbedChain instance, sets up a vector DB client and
         creates a collection.
 
-        :param config: InitConfig instance to load as configuration.
+        :param config: BaseAppConfig instance to load as configuration.
         """
 
         self.config = config
-        self.db_client = self.config.db.client
-        self.collection = self.config.db.collection
+        self.collection = self.config.db._get_or_create_collection(self.config.collection_name)
+        self.db = self.config.db
         self.user_asks = []
         self.is_docs_site_instance = False
         self.online = False
@@ -83,7 +81,7 @@ class EmbedChain:
             metadata,
         )
 
-    def load_and_embed(self, loader, chunker, src, metadata=None):
+    def load_and_embed(self, loader: BaseLoader, chunker: BaseChunker, src, metadata=None):
         """
         Loads the data from the given URL, chunks it, and adds it to database.
 
@@ -100,11 +98,10 @@ class EmbedChain:
         # get existing ids, and discard doc if any common id exist.
         where = {"app_id": self.config.id} if self.config.id is not None else {}
         # where={"url": src}
-        existing_docs = self.collection.get(
+        existing_ids = self.db.get(
             ids=ids,
             where=where,  # optional filter
         )
-        existing_ids = set(existing_docs["ids"])
 
         if len(existing_ids):
             data_dict = {id: (doc, meta) for id, doc, meta in zip(ids, documents, metadatas)}
@@ -121,12 +118,15 @@ class EmbedChain:
         if self.config.id is not None:
             metadatas = [{**m, "app_id": self.config.id} for m in metadatas]
 
+        # FIXME: Fix the error handling logic when metadatas or metadata is None
+        metadatas = metadatas if metadatas else []
+        metadata = metadata if metadata else {}
         chunks_before_addition = self.count()
 
         # Add metadata to each document
-        metadatas_with_metadata = [meta or metadata for meta in metadatas]
+        metadatas_with_metadata = [{**meta, **metadata} for meta in metadatas]
 
-        self.collection.add(documents=documents, metadatas=list(metadatas_with_metadata), ids=ids)
+        self.db.add(documents=documents, metadatas=list(metadatas_with_metadata), ids=ids)
         print((f"Successfully saved {src}. New chunks count: " f"{self.count() - chunks_before_addition}"))
 
     def _format_result(self, results):
@@ -139,7 +139,10 @@ class EmbedChain:
             )
         ]
 
-    def get_llm_model_answer(self, prompt):
+    def get_llm_model_answer(self):
+        """
+        Usually implemented by child class
+        """
         raise NotImplementedError
 
     def retrieve_from_database(self, input_query, config: QueryConfig):
@@ -152,15 +155,12 @@ class EmbedChain:
         :return: The content of the document that matched your query.
         """
         where = {"app_id": self.config.id} if self.config.id is not None else {}  # optional filter
-        result = self.collection.query(
-            query_texts=[
-                input_query,
-            ],
+        contents = self.db.query(
+            input_query=input_query,
             n_results=config.number_documents,
             where=where,
         )
-        results_formatted = self._format_result(result)
-        contents = [result[0].page_content for result in results_formatted]
+
         return contents
 
     def _append_search_and_context(self, context, web_search_result):
@@ -206,7 +206,7 @@ class EmbedChain:
         logging.info(f"Access search to get answers for {input_query}")
         return search.run(input_query)
 
-    def query(self, input_query, config: QueryConfig = None):
+    def query(self, input_query, config: QueryConfig = None, dry_run=False):
         """
         Queries the vector database based on the given input query.
         Gets relevant doc based on the query and then passes it to an
@@ -215,6 +215,12 @@ class EmbedChain:
         :param input_query: The query to use.
         :param config: Optional. The `QueryConfig` instance to use as
         configuration options.
+        :param dry_run: Optional. A dry run does everything except send the resulting prompt to
+        the LLM. The purpose is to test the prompt, not the response.
+        You can use it to test your prompt, including the context provided
+        by the vector database's doc retrieval.
+        The only thing the dry run does not consider is the cut-off due to
+        the `max_tokens` parameter.
         :return: The answer to the query.
         """
         if config is None:
@@ -229,6 +235,9 @@ class EmbedChain:
         prompt = self.generate_prompt(input_query, contexts, config, **k)
         logging.info(f"Prompt: {prompt}")
 
+        if dry_run:
+            return prompt
+
         answer = self.get_answer_from_llm(prompt, config)
 
         if isinstance(answer, str):
@@ -244,7 +253,7 @@ class EmbedChain:
             yield chunk
         logging.info(f"Answer: {streamed_answer}")
 
-    def chat(self, input_query, config: ChatConfig = None):
+    def chat(self, input_query, config: ChatConfig = None, dry_run=False):
         """
         Queries the vector database on the given input query.
         Gets relevant doc based on the query and then passes it to an
@@ -254,6 +263,12 @@ class EmbedChain:
         :param input_query: The query to use.
         :param config: Optional. The `ChatConfig` instance to use as
         configuration options.
+        :param dry_run: Optional. A dry run does everything except send the resulting prompt to
+        the LLM. The purpose is to test the prompt, not the response.
+        You can use it to test your prompt, including the context provided
+        by the vector database's doc retrieval.
+        The only thing the dry run does not consider is the cut-off due to
+        the `max_tokens` parameter.
         :return: The answer to the query.
         """
         if config is None:
@@ -264,7 +279,7 @@ class EmbedChain:
         k = {}
         if self.online:
             k["web_search_result"] = self.access_search_and_get_results(input_query)
-        contexts = self.retrieve_from_database(input_query, config, **k)
+        contexts = self.retrieve_from_database(input_query, config)
 
         global memory
         chat_history = memory.load_memory_variables({})["history"]
@@ -274,6 +289,10 @@ class EmbedChain:
 
         prompt = self.generate_prompt(input_query, contexts, config, **k)
         logging.info(f"Prompt: {prompt}")
+
+        if dry_run:
+            return prompt
+
         answer = self.get_answer_from_llm(prompt, config)
 
         memory.chat_memory.add_user_message(input_query)
@@ -294,26 +313,13 @@ class EmbedChain:
         memory.chat_memory.add_ai_message(streamed_answer)
         logging.info(f"Answer: {streamed_answer}")
 
-    def dry_run(self, input_query, config: QueryConfig = None):
+    def set_collection(self, collection_name):
         """
-        A dry run does everything except send the resulting prompt to
-        the LLM. The purpose is to test the prompt, not the response.
-        You can use it to test your prompt, including the context provided
-        by the vector database's doc retrieval.
-        The only thing the dry run does not consider is the cut-off due to
-        the `max_tokens` parameter.
+        Set the collection to use.
 
-        :param input_query: The query to use.
-        :param config: Optional. The `QueryConfig` instance to use as
-        configuration options.
-        :return: The prompt that would be sent to the LLM
+        :param collection_name: The name of the collection to use.
         """
-        if config is None:
-            config = QueryConfig()
-        contexts = self.retrieve_from_database(input_query, config)
-        prompt = self.generate_prompt(input_query, contexts, config)
-        logging.info(f"Prompt: {prompt}")
-        return prompt
+        self.collection = self.config.db._get_or_create_collection(collection_name)
 
     def count(self):
         """
@@ -321,160 +327,11 @@ class EmbedChain:
 
         :return: The number of embeddings.
         """
-        return self.collection.count()
+        return self.db.count()
 
     def reset(self):
         """
         Resets the database. Deletes all embeddings irreversibly.
         `App` has to be reinitialized after using this method.
         """
-        self.db_client.reset()
-
-
-class App(EmbedChain):
-    """
-    The EmbedChain app.
-    Has two functions: add and query.
-
-    adds(data_type, url): adds the data from the given URL to the vector db.
-    query(query): finds answer to the given query using vector database and LLM.
-    dry_run(query): test your prompt without consuming tokens.
-    """
-
-    def __init__(self, config: InitConfig = None):
-        """
-        :param config: InitConfig instance to load as configuration. Optional.
-        """
-        if config is None:
-            config = InitConfig()
-
-        if not config.ef:
-            config._set_embedding_function_to_default()
-
-        if not config.db:
-            config._set_db_to_default()
-
-        super().__init__(config)
-
-    def get_llm_model_answer(self, prompt, config: ChatConfig):
-        messages = []
-        messages.append({"role": "user", "content": prompt})
-        response = openai.ChatCompletion.create(
-            model=config.model,
-            messages=messages,
-            temperature=config.temperature,
-            max_tokens=config.max_tokens,
-            top_p=config.top_p,
-            stream=config.stream,
-        )
-
-        if config.stream:
-            return self._stream_llm_model_response(response)
-        else:
-            return response["choices"][0]["message"]["content"]
-
-    def _stream_llm_model_response(self, response):
-        """
-        This is a generator for streaming response from the OpenAI completions API
-        """
-        for line in response:
-            chunk = line["choices"][0].get("delta", {}).get("content", "")
-            yield chunk
-
-
-class OpenSourceApp(EmbedChain):
-    """
-    The OpenSource app.
-    Same as App, but uses an open source embedding model and LLM.
-
-    Has two function: add and query.
-
-    adds(data_type, url): adds the data from the given URL to the vector db.
-    query(query): finds answer to the given query using vector database and LLM.
-    """
-
-    def __init__(self, config: InitConfig = None):
-        """
-        :param config: InitConfig instance to load as configuration. Optional.
-        `ef` defaults to open source.
-        """
-        print("Loading open source embedding model. This may take some time...")  # noqa:E501
-        if not config:
-            config = InitConfig()
-
-        if not config.ef:
-            config._set_embedding_function(
-                embedding_functions.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2")
-            )
-
-        if not config.db:
-            config._set_db_to_default()
-
-        print("Successfully loaded open source embedding model.")
-        super().__init__(config)
-
-    def get_llm_model_answer(self, prompt, config: ChatConfig):
-        from gpt4all import GPT4All
-
-        global gpt4all_model
-        if gpt4all_model is None:
-            gpt4all_model = GPT4All("orca-mini-3b.ggmlv3.q4_0.bin")
-        response = gpt4all_model.generate(prompt=prompt, streaming=config.stream)
-        return response
-
-
-class EmbedChainPersonApp:
-    """
-    Base class to create a person bot.
-    This bot behaves and speaks like a person.
-
-    :param person: name of the person, better if its a well known person.
-    :param config: InitConfig instance to load as configuration.
-    """
-
-    def __init__(self, person, config: InitConfig = None):
-        self.person = person
-        self.person_prompt = f"You are {person}. Whatever you say, you will always say in {person} style."  # noqa:E501
-        if config is None:
-            config = InitConfig()
-        super().__init__(config)
-
-
-class PersonApp(EmbedChainPersonApp, App):
-    """
-    The Person app.
-    Extends functionality from EmbedChainPersonApp and App
-    """
-
-    def query(self, input_query, config: QueryConfig = None):
-        self.template = Template(self.person_prompt + " " + DEFAULT_PROMPT)
-        query_config = QueryConfig(
-            template=self.template,
-        )
-        return super().query(input_query, query_config)
-
-    def chat(self, input_query, config: ChatConfig = None):
-        self.template = Template(self.person_prompt + " " + DEFAULT_PROMPT_WITH_HISTORY)
-        chat_config = ChatConfig(
-            template=self.template,
-        )
-        return super().chat(input_query, chat_config)
-
-
-class PersonOpenSourceApp(EmbedChainPersonApp, OpenSourceApp):
-    """
-    The Person app.
-    Extends functionality from EmbedChainPersonApp and OpenSourceApp
-    """
-
-    def query(self, input_query, config: QueryConfig = None):
-        query_config = QueryConfig(
-            template=self.template,
-        )
-        return super().query(input_query, query_config)
-
-    def chat(self, input_query, config: ChatConfig = None):
-        chat_config = ChatConfig(
-            template=self.template,
-        )
-        return super().chat(input_query, chat_config)
+        self.db.reset()

embedchain/loaders/base_loader.py

@@ -0,0 +1,9 @@
+class BaseLoader:
+    def __init__(self):
+        pass
+
+    def load_data():
+        """
+        Implemented by child classes
+        """
+        pass

embedchain/loaders/docs_site_loader.py

@@ -4,8 +4,10 @@ from urllib.parse import urljoin, urlparse
 import requests
 from bs4 import BeautifulSoup
 
+from embedchain.loaders.base_loader import BaseLoader
 
-class DocsSiteLoader:
+
+class DocsSiteLoader(BaseLoader):
     def __init__(self):
         self.visited_links = set()
 

embedchain/loaders/docx_file.py

@@ -1,7 +1,9 @@
 from langchain.document_loaders import Docx2txtLoader
 
+from embedchain.loaders.base_loader import BaseLoader
 
-class DocxFileLoader:
+
+class DocxFileLoader(BaseLoader):
     def load_data(self, url):
         """Load data from a .docx file."""
         loader = Docx2txtLoader(url)

embedchain/loaders/local_qna_pair.py

@@ -1,4 +1,7 @@
-class LocalQnaPairLoader:
+from embedchain.loaders.base_loader import BaseLoader
+
+
+class LocalQnaPairLoader(BaseLoader):
     def load_data(self, content):
         """Load data from a local QnA pair."""
         question, answer = content

embedchain/loaders/local_text.py

@@ -1,4 +1,7 @@
-class LocalTextLoader:
+from embedchain.loaders.base_loader import BaseLoader
+
+
+class LocalTextLoader(BaseLoader):
     def load_data(self, content):
         """Load data from a local text file."""
         meta_data = {

embedchain/loaders/notion.py

@@ -0,0 +1,41 @@
+import logging
+import os
+
+try:
+    from llama_index import download_loader
+except ImportError:
+    raise ImportError("Notion requires extra dependencies. Install with `pip install embedchain[notion]`") from None
+
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+
+class NotionLoader(BaseLoader):
+    def load_data(self, source):
+        """Load data from a PDF file."""
+
+        NotionPageReader = download_loader("NotionPageReader")
+
+        # Reformat Id to match notion expectation
+        id = source[-32:]
+        formatted_id = f"{id[:8]}-{id[8:12]}-{id[12:16]}-{id[16:20]}-{id[20:]}"
+        logging.debug(f"Extracted notion page id as: {formatted_id}")
+
+        # Get page through the notion api
+        integration_token = os.getenv("NOTION_INTEGRATION_TOKEN")
+        reader = NotionPageReader(integration_token=integration_token)
+        documents = reader.load_data(page_ids=[formatted_id])
+
+        # Extract text
+        raw_text = documents[0].text
+
+        # Clean text
+        text = clean_string(raw_text)
+
+        return [
+            {
+                "content": text,
+                "meta_data": {"url": f"notion-{formatted_id}"},
+            }
+        ]

embedchain/loaders/pdf_file.py

@@ -1,9 +1,10 @@
 from langchain.document_loaders import PyPDFLoader
 
+from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils import clean_string
 
 
-class PdfFileLoader:
+class PdfFileLoader(BaseLoader):
     def load_data(self, url):
         """Load data from a PDF file."""
         loader = PyPDFLoader(url)

embedchain/loaders/sitemap.py

@@ -4,10 +4,12 @@ import requests
 from bs4 import BeautifulSoup
 from bs4.builder import ParserRejectedMarkup
 
+from embedchain.loaders.base_loader import BaseLoader
 from embedchain.loaders.web_page import WebPageLoader
+from embedchain.utils import is_readable
 
 
-class SitemapLoader:
+class SitemapLoader(BaseLoader):
     def load_data(self, sitemap_url):
         """
         This method takes a sitemap URL as input and retrieves
@@ -20,11 +22,20 @@ class SitemapLoader:
         response.raise_for_status()
 
         soup = BeautifulSoup(response.text, "xml")
-        links = [link.text for link in soup.find_all("loc")]
+
+        links = [link.text for link in soup.find_all("loc") if link.parent.name == "url"]
+        if len(links) == 0:
+            # Get all <loc> tags as a fallback. This might include images.
+            links = [link.text for link in soup.find_all("loc")]
+
         for link in links:
             try:
                 each_load_data = web_page_loader.load_data(link)
-                output.append(each_load_data)
+
+                if is_readable(each_load_data[0].get("content")):
+                    output.append(each_load_data)
+                else:
+                    logging.warning(f"Page is not readable (too many invalid characters): {link}")
             except ParserRejectedMarkup as e:
                 logging.error(f"Failed to parse {link}: {e}")
         return [data[0] for data in output]

embedchain/loaders/web_page.py

@@ -3,10 +3,11 @@ import logging
 import requests
 from bs4 import BeautifulSoup
 
+from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils import clean_string
 
 
-class WebPageLoader:
+class WebPageLoader(BaseLoader):
     def load_data(self, url):
         """Load data from a web page."""
         response = requests.get(url)

embedchain/loaders/youtube_video.py

@@ -1,9 +1,10 @@
 from langchain.document_loaders import YoutubeLoader
 
+from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils import clean_string
 
 
-class YoutubeVideoLoader:
+class YoutubeVideoLoader(BaseLoader):
     def load_data(self, url):
         """Load data from a Youtube video."""
         loader = YoutubeLoader.from_youtube_url(url, add_video_info=True)

embedchain/models/EmbeddingFunctions.py

@@ -0,0 +1,8 @@
+from enum import Enum
+
+
+class EmbeddingFunctions(Enum):
+    OPENAI = "OPENAI"
+    HUGGING_FACE = "HUGGING_FACE"
+    VERTEX_AI = "VERTEX_AI"
+    GPT4ALL = "GPT4ALL"

embedchain/models/Providers.py

@@ -0,0 +1,9 @@
+from enum import Enum
+
+
+class Providers(Enum):
+    OPENAI = "OPENAI"
+    ANTHROPHIC = "ANTHPROPIC"
+    VERTEX_AI = "VERTEX_AI"
+    GPT4ALL = "GPT4ALL"
+    AZURE_OPENAI = "AZURE_OPENAI"

embedchain/models/VectorDatabases.py

@@ -0,0 +1,6 @@
+from enum import Enum
+
+
+class VectorDatabases(Enum):
+    CHROMADB = "CHROMADB"
+    ELASTICSEARCH = "ELASTICSEARCH"

embedchain/models/VectorDimensions.py

@@ -0,0 +1,9 @@
+from enum import Enum
+
+
+# vector length created by embedding fn
+class VectorDimensions(Enum):
+    GPT4ALL = 384
+    OPENAI = 1536
+    VERTEX_AI = 768
+    HUGGING_FACE = 384

embedchain/models/__init__.py

@@ -0,0 +1,4 @@
+from .EmbeddingFunctions import EmbeddingFunctions  # noqa: F401
+from .Providers import Providers  # noqa: F401
+from .VectorDatabases import VectorDatabases  # noqa: F401
+from .VectorDimensions import VectorDimensions  # noqa: F401

embedchain/utils.py

@@ -1,4 +1,6 @@
+import logging
 import re
+import string
 
 
 def clean_string(text):
@@ -33,3 +35,57 @@ def clean_string(text):
     cleaned_text = re.sub(r"([^\w\s])\1*", r"\1", cleaned_text)
 
     return cleaned_text
+
+
+def is_readable(s):
+    """
+    Heuristic to determine if a string is "readable" (mostly contains printable characters and forms meaningful words)
+
+    :param s: string
+    :return: True if the string is more than 95% printable.
+    """
+    try:
+        printable_ratio = sum(c in string.printable for c in s) / len(s)
+    except ZeroDivisionError:
+        logging.warning("Empty string processed as unreadable")
+        printable_ratio = 0
+    return printable_ratio > 0.95  # 95% of characters are printable
+
+
+def use_pysqlite3():
+    """
+    Swap std-lib sqlite3 with pysqlite3.
+    """
+    import platform
+    import sqlite3
+
+    if platform.system() == "Linux" and sqlite3.sqlite_version_info < (3, 35, 0):
+        try:
+            # According to the Chroma team, this patch only works on Linux
+            import datetime
+            import subprocess
+            import sys
+
+            subprocess.check_call(
+                [sys.executable, "-m", "pip", "install", "pysqlite3-binary", "--quiet", "--disable-pip-version-check"]
+            )
+
+            __import__("pysqlite3")
+            sys.modules["sqlite3"] = sys.modules.pop("pysqlite3")
+
+            # Let the user know what happened.
+            current_time = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S,%f")[:-3]
+            print(
+                f"{current_time} [embedchain] [INFO]",
+                "Swapped std-lib sqlite3 with pysqlite3 for ChromaDb compatibility.",
+                f"Your original version was {sqlite3.sqlite_version}.",
+            )
+        except Exception as e:
+            # Escape all exceptions
+            current_time = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S,%f")[:-3]
+            print(
+                f"{current_time} [embedchain] [ERROR]",
+                "Failed to swap std-lib sqlite3 with pysqlite3 for ChromaDb compatibility.",
+                "Error:",
+                e,
+            )

embedchain/vectordb/base_vector_db.py

@@ -3,7 +3,6 @@ class BaseVectorDB:
 
     def __init__(self):
         self.client = self._get_or_create_db()
-        self.collection = self._get_or_create_collection()
 
     def _get_or_create_db(self):
         """Get or create the database."""
@@ -11,3 +10,18 @@ class BaseVectorDB:
 
     def _get_or_create_collection(self):
         raise NotImplementedError
+
+    def get(self):
+        raise NotImplementedError
+
+    def add(self):
+        raise NotImplementedError
+
+    def query(self):
+        raise NotImplementedError
+
+    def count(self):
+        raise NotImplementedError
+
+    def reset(self):
+        raise NotImplementedError

embedchain/vectordb/chroma_db.py

@@ -1,6 +1,18 @@
 import logging
+from typing import Any, Dict, List
 
-import chromadb
+from chromadb.errors import InvalidDimensionException
+from langchain.docstore.document import Document
+
+try:
+    import chromadb
+except RuntimeError:
+    from embedchain.utils import use_pysqlite3
+
+    use_pysqlite3()
+    import chromadb
+
+from chromadb.config import Settings
 
 from embedchain.vectordb.base_vector_db import BaseVectorDB
 
@@ -8,33 +20,99 @@ from embedchain.vectordb.base_vector_db import BaseVectorDB
 class ChromaDB(BaseVectorDB):
     """Vector database using ChromaDB."""
 
-    def __init__(self, db_dir=None, ef=None, host=None, port=None):
-        self.ef = ef
+    def __init__(self, db_dir=None, embedding_fn=None, host=None, port=None):
+        self.embedding_fn = embedding_fn
+
+        if not hasattr(embedding_fn, "__call__"):
+            raise ValueError("Embedding function is not a function")
 
         if host and port:
             logging.info(f"Connecting to ChromaDB server: {host}:{port}")
-            self.client_settings = chromadb.config.Settings(
-                chroma_api_impl="rest",
-                chroma_server_host=host,
-                chroma_server_http_port=port,
-            )
+            self.settings = Settings(chroma_server_host=host, chroma_server_http_port=port)
+            self.client = chromadb.HttpClient(self.settings)
         else:
             if db_dir is None:
                 db_dir = "db"
-            self.client_settings = chromadb.config.Settings(
-                chroma_db_impl="duckdb+parquet",
-                persist_directory=db_dir,
-                anonymized_telemetry=False,
+            self.settings = Settings(anonymized_telemetry=False, allow_reset=True)
+            self.client = chromadb.PersistentClient(
+                path=db_dir,
+                settings=self.settings,
             )
         super().__init__()
 
     def _get_or_create_db(self):
         """Get or create the database."""
-        return chromadb.Client(self.client_settings)
+        return self.client
 
-    def _get_or_create_collection(self):
+    def _get_or_create_collection(self, name):
         """Get or create the collection."""
-        return self.client.get_or_create_collection(
-            "embedchain_store",
-            embedding_function=self.ef,
+        self.collection = self.client.get_or_create_collection(
+            name=name,
+            embedding_function=self.embedding_fn,
         )
+        return self.collection
+
+    def get(self, ids: List[str], where: Dict[str, any]) -> List[str]:
+        """
+        Get existing doc ids present in vector database
+        :param ids: list of doc ids to check for existance
+        :param where: Optional. to filter data
+        """
+        existing_docs = self.collection.get(
+            ids=ids,
+            where=where,  # optional filter
+        )
+
+        return set(existing_docs["ids"])
+
+    def add(self, documents: List[str], metadatas: List[object], ids: List[str]) -> Any:
+        """
+        add data in vector database
+        :param documents: list of texts to add
+        :param metadatas: list of metadata associated with docs
+        :param ids: ids of docs
+        """
+        self.collection.add(documents=documents, metadatas=metadatas, ids=ids)
+
+    def _format_result(self, results):
+        return [
+            (Document(page_content=result[0], metadata=result[1] or {}), result[2])
+            for result in zip(
+                results["documents"][0],
+                results["metadatas"][0],
+                results["distances"][0],
+            )
+        ]
+
+    def query(self, input_query: List[str], n_results: int, where: Dict[str, any]) -> List[str]:
+        """
+        query contents from vector data base based on vector similarity
+        :param input_query: list of query string
+        :param n_results: no of similar documents to fetch from database
+        :param where: Optional. to filter data
+        :return: The content of the document that matched your query.
+        """
+        try:
+            result = self.collection.query(
+                query_texts=[
+                    input_query,
+                ],
+                n_results=n_results,
+                where=where,
+            )
+        except InvalidDimensionException as e:
+            raise InvalidDimensionException(
+                e.message()
+                + ". This is commonly a side-effect when an embedding function, different from the one used to add the embeddings, is used to retrieve an embedding from the database."  # noqa E501
+            ) from None
+
+        results_formatted = self._format_result(result)
+        contents = [result[0].page_content for result in results_formatted]
+        return contents
+
+    def count(self) -> int:
+        return self.collection.count()
+
+    def reset(self):
+        # Delete all data from the database
+        self.client.reset()

embedchain/vectordb/elasticsearch_db.py

@@ -0,0 +1,136 @@
+from typing import Any, Callable, Dict, List
+
+try:
+    from elasticsearch import Elasticsearch
+    from elasticsearch.helpers import bulk
+except ImportError:
+    raise ImportError(
+        "Elasticsearch requires extra dependencies. Install with `pip install embedchain[elasticsearch]`"
+    ) from None
+
+from embedchain.config import ElasticsearchDBConfig
+from embedchain.models.VectorDimensions import VectorDimensions
+from embedchain.vectordb.base_vector_db import BaseVectorDB
+
+
+class ElasticsearchDB(BaseVectorDB):
+    def __init__(
+        self,
+        es_config: ElasticsearchDBConfig = None,
+        embedding_fn: Callable[[list[str]], list[str]] = None,
+        vector_dim: VectorDimensions = None,
+        collection_name: str = None,
+    ):
+        """
+        Elasticsearch as vector database
+        :param es_config. elasticsearch database config to be used for connection
+        :param embedding_fn: Function to generate embedding vectors.
+        :param vector_dim: Vector dimension generated by embedding fn
+        :param collection_name: Optional. Collection name for the database.
+        """
+        if not hasattr(embedding_fn, "__call__"):
+            raise ValueError("Embedding function is not a function")
+        if es_config is None:
+            raise ValueError("ElasticsearchDBConfig is required")
+        if vector_dim is None:
+            raise ValueError("Vector Dimension is required to refer correct index and mapping")
+        if collection_name is None:
+            raise ValueError("collection name is required. It cannot be empty")
+        self.embedding_fn = embedding_fn
+        self.client = Elasticsearch(es_config.ES_URL, **es_config.ES_EXTRA_PARAMS)
+        self.vector_dim = vector_dim
+        self.es_index = f"{collection_name}_{self.vector_dim}"
+        index_settings = {
+            "mappings": {
+                "properties": {
+                    "text": {"type": "text"},
+                    "embeddings": {"type": "dense_vector", "index": False, "dims": self.vector_dim},
+                }
+            }
+        }
+        if not self.client.indices.exists(index=self.es_index):
+            # create index if not exist
+            print("Creating index", self.es_index, index_settings)
+            self.client.indices.create(index=self.es_index, body=index_settings)
+        super().__init__()
+
+    def _get_or_create_db(self):
+        return self.client
+
+    def _get_or_create_collection(self, name):
+        """Note: nothing to return here. Discuss later"""
+
+    def get(self, ids: List[str], where: Dict[str, any]) -> List[str]:
+        """
+        Get existing doc ids present in vector database
+        :param ids: list of doc ids to check for existance
+        :param where: Optional. to filter data
+        """
+        query = {"bool": {"must": [{"ids": {"values": ids}}]}}
+        if "app_id" in where:
+            app_id = where["app_id"]
+            query["bool"]["must"].append({"term": {"metadata.app_id": app_id}})
+        response = self.client.search(index=self.es_index, query=query, _source=False)
+        docs = response["hits"]["hits"]
+        ids = [doc["_id"] for doc in docs]
+        return set(ids)
+
+    def add(self, documents: List[str], metadatas: List[object], ids: List[str]) -> Any:
+        """
+        add data in vector database
+        :param documents: list of texts to add
+        :param metadatas: list of metadata associated with docs
+        :param ids: ids of docs
+        """
+        docs = []
+        embeddings = self.embedding_fn(documents)
+        for id, text, metadata, embeddings in zip(ids, documents, metadatas, embeddings):
+            docs.append(
+                {
+                    "_index": self.es_index,
+                    "_id": id,
+                    "_source": {"text": text, "metadata": metadata, "embeddings": embeddings},
+                }
+            )
+        bulk(self.client, docs)
+        self.client.indices.refresh(index=self.es_index)
+        return
+
+    def query(self, input_query: List[str], n_results: int, where: Dict[str, any]) -> List[str]:
+        """
+        query contents from vector data base based on vector similarity
+        :param input_query: list of query string
+        :param n_results: no of similar documents to fetch from database
+        :param where: Optional. to filter data
+        """
+        input_query_vector = self.embedding_fn(input_query)
+        query_vector = input_query_vector[0]
+        query = {
+            "script_score": {
+                "query": {"bool": {"must": [{"exists": {"field": "text"}}]}},
+                "script": {
+                    "source": "cosineSimilarity(params.input_query_vector, 'embeddings') + 1.0",
+                    "params": {"input_query_vector": query_vector},
+                },
+            }
+        }
+        if "app_id" in where:
+            app_id = where["app_id"]
+            query["script_score"]["query"]["bool"]["must"] = [{"term": {"metadata.app_id": app_id}}]
+        _source = ["text"]
+        response = self.client.search(index=self.es_index, query=query, _source=_source, size=n_results)
+        docs = response["hits"]["hits"]
+        contents = [doc["_source"]["text"] for doc in docs]
+        return contents
+
+    def count(self) -> int:
+        query = {"match_all": {}}
+        response = self.client.count(index=self.es_index, query=query)
+        doc_count = response["count"]
+        return doc_count
+
+    def reset(self):
+        # Delete all data from the database
+        if self.client.indices.exists(index=self.es_index):
+            # delete index in Es
+            self.client.indices.delete(index=self.es_index)

examples/full_stack/.dockerignore

@@ -0,0 +1 @@
+.git

examples/full_stack/README.md

@@ -0,0 +1,18 @@
+## 🐳 Docker Setup
+
+- To setup full stack app using docker, run the following command inside this folder using your terminal.
+
+```bash
+docker-compose up --build
+```
+
+📝 Note: The build command might take a while to install all the packages depending on your system resources.
+
+## 🚀 Usage Instructions
+
+- Go to [http://localhost:3000/](http://localhost:3000/) in your browser to view the dashboard.
+- Add your `OpenAI API key` 🔑 in the Settings.
+- Create a new bot and you'll be navigated to its page.
+- Here you can add your data sources and then chat with the bot.
+
+🎉 Happy Chatting! 🎉

examples/full_stack/backend/.dockerignore

@@ -0,0 +1,7 @@
+__pycache__/
+database
+pyenv
+venv
+.env
+.git
+trash_files/

examples/full_stack/backend/.gitignore

@@ -0,0 +1,6 @@
+__pycache__
+database
+pyenv
+venv
+.env
+trash_files/

examples/full_stack/backend/Dockerfile

@@ -0,0 +1,11 @@
+FROM python:3.11 AS backend
+
+WORKDIR /usr/src/app/backend
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . .
+
+EXPOSE 8000
+
+CMD ["python", "server.py"]

examples/full_stack/backend/models.py

@@ -0,0 +1,14 @@
+from flask_sqlalchemy import SQLAlchemy
+
+db = SQLAlchemy()
+
+
+class APIKey(db.Model):
+    id = db.Column(db.Integer, primary_key=True)
+    key = db.Column(db.String(255), nullable=False)
+
+
+class BotList(db.Model):
+    id = db.Column(db.Integer, primary_key=True)
+    name = db.Column(db.String(255), nullable=False)
+    slug = db.Column(db.String(255), nullable=False, unique=True)

examples/full_stack/backend/paths.py

@@ -0,0 +1,5 @@
+import os
+
+ROOT_DIRECTORY = os.getcwd()
+DB_DIRECTORY_OPEN_AI = os.path.join(os.getcwd(), "database", "open_ai")
+DB_DIRECTORY_OPEN_SOURCE = os.path.join(os.getcwd(), "database", "open_source")

examples/full_stack/backend/routes/chat_response.py

@@ -0,0 +1,32 @@
+import os
+
+from flask import Blueprint, jsonify, make_response, request
+from models import APIKey
+from paths import DB_DIRECTORY_OPEN_AI
+
+from embedchain import App
+
+chat_response_bp = Blueprint("chat_response", __name__)
+
+
+# Chat Response for user query
+@chat_response_bp.route("/api/get_answer", methods=["POST"])
+def get_answer():
+    try:
+        data = request.get_json()
+        query = data.get("query")
+        embedding_model = data.get("embedding_model")
+        app_type = data.get("app_type")
+
+        if embedding_model == "open_ai":
+            os.chdir(DB_DIRECTORY_OPEN_AI)
+            api_key = APIKey.query.first().key
+            os.environ["OPENAI_API_KEY"] = api_key
+            if app_type == "app":
+                chat_bot = App()
+
+        response = chat_bot.chat(query)
+        return make_response(jsonify({"response": response}), 200)
+
+    except Exception as e:
+        return make_response(jsonify({"error": str(e)}), 400)

examples/full_stack/backend/routes/dashboard.py

@@ -0,0 +1,72 @@
+from flask import Blueprint, jsonify, make_response, request
+from models import APIKey, BotList, db
+
+dashboard_bp = Blueprint("dashboard", __name__)
+
+
+# Set Open AI Key
+@dashboard_bp.route("/api/set_key", methods=["POST"])
+def set_key():
+    data = request.get_json()
+    api_key = data["openAIKey"]
+    existing_key = APIKey.query.first()
+    if existing_key:
+        existing_key.key = api_key
+    else:
+        new_key = APIKey(key=api_key)
+        db.session.add(new_key)
+    db.session.commit()
+    return make_response(jsonify(message="API key saved successfully"), 200)
+
+
+# Check OpenAI Key
+@dashboard_bp.route("/api/check_key", methods=["GET"])
+def check_key():
+    existing_key = APIKey.query.first()
+    if existing_key:
+        return make_response(jsonify(status="ok", message="OpenAI Key exists"), 200)
+    else:
+        return make_response(jsonify(status="fail", message="No OpenAI Key present"), 200)
+
+
+# Create a bot
+@dashboard_bp.route("/api/create_bot", methods=["POST"])
+def create_bot():
+    data = request.get_json()
+    name = data["name"]
+    slug = name.lower().replace(" ", "_")
+    existing_bot = BotList.query.filter_by(slug=slug).first()
+    if existing_bot:
+        return (make_response(jsonify(message="Bot already exists"), 400),)
+    new_bot = BotList(name=name, slug=slug)
+    db.session.add(new_bot)
+    db.session.commit()
+    return make_response(jsonify(message="Bot created successfully"), 200)
+
+
+# Delete a bot
+@dashboard_bp.route("/api/delete_bot", methods=["POST"])
+def delete_bot():
+    data = request.get_json()
+    slug = data.get("slug")
+    bot = BotList.query.filter_by(slug=slug).first()
+    if bot:
+        db.session.delete(bot)
+        db.session.commit()
+        return make_response(jsonify(message="Bot deleted successfully"), 200)
+    return make_response(jsonify(message="Bot not found"), 400)
+
+
+# Get the list of bots
+@dashboard_bp.route("/api/get_bots", methods=["GET"])
+def get_bots():
+    bots = BotList.query.all()
+    bot_list = []
+    for bot in bots:
+        bot_list.append(
+            {
+                "name": bot.name,
+                "slug": bot.slug,
+            }
+        )
+    return jsonify(bot_list)

examples/full_stack/backend/routes/sources.py

@@ -0,0 +1,27 @@
+import os
+
+from flask import Blueprint, jsonify, make_response, request
+from models import APIKey
+from paths import DB_DIRECTORY_OPEN_AI
+
+from embedchain import App
+
+sources_bp = Blueprint("sources", __name__)
+
+
+# API route to add data sources
+@sources_bp.route("/api/add_sources", methods=["POST"])
+def add_sources():
+    try:
+        embedding_model = request.json.get("embedding_model")
+        name = request.json.get("name")
+        value = request.json.get("value")
+        if embedding_model == "open_ai":
+            os.chdir(DB_DIRECTORY_OPEN_AI)
+            api_key = APIKey.query.first().key
+            os.environ["OPENAI_API_KEY"] = api_key
+            chat_bot = App()
+        chat_bot.add(name, value)
+        return make_response(jsonify(message="Sources added successfully"), 200)
+    except Exception as e:
+        return make_response(jsonify(message=f"Error adding sources: {str(e)}"), 400)

examples/full_stack/backend/server.py

@@ -0,0 +1,27 @@
+import os
+
+from flask import Flask
+from models import db
+from paths import DB_DIRECTORY_OPEN_AI, ROOT_DIRECTORY
+from routes.chat_response import chat_response_bp
+from routes.dashboard import dashboard_bp
+from routes.sources import sources_bp
+
+app = Flask(__name__)
+app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///" + os.path.join(ROOT_DIRECTORY, "database", "user_data.db")
+app.register_blueprint(dashboard_bp)
+app.register_blueprint(sources_bp)
+app.register_blueprint(chat_response_bp)
+
+
+# Initialize the app on startup
+def load_app():
+    os.makedirs(DB_DIRECTORY_OPEN_AI, exist_ok=True)
+    db.init_app(app)
+    with app.app_context():
+        db.create_all()
+
+
+if __name__ == "__main__":
+    load_app()
+    app.run(host="0.0.0.0", debug=True, port=8000)

examples/full_stack/docker-compose.yml

@@ -0,0 +1,22 @@
+version: "3.9"
+
+services:
+  backend:
+    container_name: embedchain_backend
+    restart: unless-stopped
+    build:
+      context: backend
+      dockerfile: Dockerfile
+    ports:
+      - "8000:8000"
+
+  frontend:
+    container_name: embedchain_frontend
+    restart: unless-stopped
+    build:
+      context: frontend
+      dockerfile: Dockerfile
+    ports:
+      - "3000:3000"
+    depends_on:
+      - "backend"

examples/full_stack/frontend/.dockerignore

@@ -0,0 +1,7 @@
+node_modules/
+build
+dist
+.env
+.git
+.next/
+trash_files/

examples/full_stack/frontend/.eslintrc.json

@@ -0,0 +1,3 @@
+{
+  "extends": ["next/babel", "next/core-web-vitals"]
+}

examples/full_stack/frontend/.gitignore

@@ -0,0 +1,38 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# local env files
+.env*.local
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts
+
+vscode/
+trash_files/

examples/full_stack/frontend/Dockerfile

@@ -0,0 +1,14 @@
+FROM node:18 AS frontend
+
+WORKDIR /usr/src/app/frontend
+COPY package.json .
+COPY package-lock.json .
+RUN npm install
+
+COPY . .
+
+RUN npm run build
+
+EXPOSE 3000
+
+CMD ["npm", "start"]

examples/full_stack/frontend/jsconfig.json

@@ -0,0 +1,7 @@
+{
+  "compilerOptions": {
+    "paths": {
+      "@/*": ["./src/*"]
+    }
+  }
+}

examples/full_stack/frontend/next.config.js

@@ -0,0 +1,26 @@
+/** @type {import('next').NextConfig} */
+const nextConfig = {
+  async rewrites() {
+    return [
+      {
+        source: "/api/:path*",
+        destination: "http://backend:8000/api/:path*",
+      },
+    ];
+  },
+  reactStrictMode: true,
+  experimental: {
+    proxyTimeout: 6000000,
+  },
+  webpack(config) {
+    config.module.rules.push({
+      test: /\.svg$/i,
+      issuer: /\.[jt]sx?$/,
+      use: ["@svgr/webpack"],
+    });
+
+    return config;
+  },
+};
+
+module.exports = nextConfig;

examples/full_stack/frontend/package.json

@@ -0,0 +1,25 @@
+{
+  "name": "frontend",
+  "version": "0.1.0",
+  "private": true,
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "start": "next start",
+    "lint": "next lint"
+  },
+  "dependencies": {
+    "autoprefixer": "^10.4.14",
+    "eslint": "8.44.0",
+    "eslint-config-next": "13.4.9",
+    "flowbite": "^1.7.0",
+    "next": "13.4.9",
+    "postcss": "8.4.25",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "tailwindcss": "3.3.2"
+  },
+  "devDependencies": {
+    "@svgr/webpack": "^8.0.1"
+  }
+}

examples/full_stack/frontend/postcss.config.js

@@ -0,0 +1,6 @@
+module.exports = {
+  plugins: {
+    tailwindcss: {},
+    autoprefixer: {},
+  },
+}

examples/full_stack/frontend/public/icons/bot.svg

@@ -0,0 +1,20 @@
+<svg
+  viewBox="0 0 24 24"
+  fill="currentColor"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path
+      fill-rule="evenodd"
+      clip-rule="evenodd"
+      d="M14 2C14 2.74028 13.5978 3.38663 13 3.73244V4H20C21.6569 4 23 5.34315 23 7V19C23 20.6569 21.6569 22 20 22H4C2.34315 22 1 20.6569 1 19V7C1 5.34315 2.34315 4 4 4H11V3.73244C10.4022 3.38663 10 2.74028 10 2C10 0.895431 10.8954 0 12 0C13.1046 0 14 0.895431 14 2ZM4 6H11H13H20C20.5523 6 21 6.44772 21 7V19C21 19.5523 20.5523 20 20 20H4C3.44772 20 3 19.5523 3 19V7C3 6.44772 3.44772 6 4 6ZM15 11.5C15 10.6716 15.6716 10 16.5 10C17.3284 10 18 10.6716 18 11.5C18 12.3284 17.3284 13 16.5 13C15.6716 13 15 12.3284 15 11.5ZM16.5 8C14.567 8 13 9.567 13 11.5C13 13.433 14.567 15 16.5 15C18.433 15 20 13.433 20 11.5C20 9.567 18.433 8 16.5 8ZM7.5 10C6.67157 10 6 10.6716 6 11.5C6 12.3284 6.67157 13 7.5 13C8.32843 13 9 12.3284 9 11.5C9 10.6716 8.32843 10 7.5 10ZM4 11.5C4 9.567 5.567 8 7.5 8C9.433 8 11 9.567 11 11.5C11 13.433 9.433 15 7.5 15C5.567 15 4 13.433 4 11.5ZM10.8944 16.5528C10.6474 16.0588 10.0468 15.8586 9.55279 16.1056C9.05881 16.3526 8.85858 16.9532 9.10557 17.4472C9.68052 18.5971 10.9822 19 12 19C13.0178 19 14.3195 18.5971 14.8944 17.4472C15.1414 16.9532 14.9412 16.3526 14.4472 16.1056C13.9532 15.8586 13.3526 16.0588 13.1056 16.5528C13.0139 16.7362 12.6488 17 12 17C11.3512 17 10.9861 16.7362 10.8944 16.5528Z"
+      fill="currentColor"
+    ></path>
+  </g>
+</svg>

examples/full_stack/frontend/public/icons/close.svg

@@ -0,0 +1,14 @@
+<svg
+aria-hidden="true"
+xmlns="http://www.w3.org/2000/svg"
+fill="none"
+viewBox="0 0 14 14"
+>
+<path
+    stroke="currentColor"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+    stroke-width="2"
+    d="m1 1 6 6m0 0 6 6M7 7l6-6M7 7l-6 6"
+/>
+</svg>

examples/full_stack/frontend/public/icons/cross.svg

@@ -0,0 +1,15 @@
+<svg
+  fill="currentColor"
+  viewBox="0 0 32 32"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path d="M18.8,16l5.5-5.5c0.8-0.8,0.8-2,0-2.8l0,0C24,7.3,23.5,7,23,7c-0.5,0-1,0.2-1.4,0.6L16,13.2l-5.5-5.5 c-0.8-0.8-2.1-0.8-2.8,0C7.3,8,7,8.5,7,9.1s0.2,1,0.6,1.4l5.5,5.5l-5.5,5.5C7.3,21.9,7,22.4,7,23c0,0.5,0.2,1,0.6,1.4 C8,24.8,8.5,25,9,25c0.5,0,1-0.2,1.4-0.6l5.5-5.5l5.5,5.5c0.8,0.8,2.1,0.8,2.8,0c0.8-0.8,0.8-2.1,0-2.8L18.8,16z"></path>
+  </g>
+</svg>

examples/full_stack/frontend/public/icons/dashboard.svg

@@ -0,0 +1,9 @@
+<svg
+  aria-hidden="true"
+  xmlns="http://www.w3.org/2000/svg"
+  fill="currentColor"
+  viewBox="0 0 22 21"
+>
+  <path d="M16.975 11H10V4.025a1 1 0 0 0-1.066-.998 8.5 8.5 0 1 0 9.039 9.039.999.999 0 0 0-1-1.066h.002Z" />
+  <path d="M12.5 0c-.157 0-.311.01-.565.027A1 1 0 0 0 11 1.02V10h8.975a1 1 0 0 0 1-.935c.013-.188.028-.374.028-.565A8.51 8.51 0 0 0 12.5 0Z" />
+</svg>

examples/full_stack/frontend/public/icons/doc.svg

@@ -0,0 +1,15 @@
+<svg
+  fill="currentColor"
+  viewBox="0 0 56 56"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path d="M 15.5547 53.125 L 40.4453 53.125 C 45.2969 53.125 47.7109 50.6640 47.7109 45.7890 L 47.7109 24.5078 L 30.7422 24.5078 C 27.7422 24.5078 26.3359 23.0781 26.3359 20.0781 L 26.3359 2.8750 L 15.5547 2.8750 C 10.7266 2.8750 8.2891 5.3594 8.2891 10.2344 L 8.2891 45.7890 C 8.2891 50.6875 10.7266 53.125 15.5547 53.125 Z M 30.8125 21.2969 L 47.4531 21.2969 C 47.2891 20.3359 46.6094 19.3984 45.5078 18.2500 L 32.5703 5.1015 C 31.4922 3.9766 30.5078 3.2969 29.5234 3.1328 L 29.5234 20.0313 C 29.5234 20.875 29.9687 21.2969 30.8125 21.2969 Z M 18.9766 34.6562 C 18.0156 34.6562 17.3359 33.9766 17.3359 33.0625 C 17.3359 32.1484 18.0156 31.4687 18.9766 31.4687 L 37.0469 31.4687 C 37.9844 31.4687 38.7109 32.1484 38.7109 33.0625 C 38.7109 33.9766 37.9844 34.6562 37.0469 34.6562 Z M 18.9766 43.5859 C 18.0156 43.5859 17.3359 42.9062 17.3359 41.9922 C 17.3359 41.0781 18.0156 40.3984 18.9766 40.3984 L 37.0469 40.3984 C 37.9844 40.3984 38.7109 41.0781 38.7109 41.9922 C 38.7109 42.9062 37.9844 43.5859 37.0469 43.5859 Z"></path>
+  </g>
+</svg>

examples/full_stack/frontend/public/icons/drawer.svg

@@ -0,0 +1,12 @@
+<svg
+    aria-hidden="true"
+    fill="currentColor"
+    viewBox="0 0 20 20"
+    xmlns="http://www.w3.org/2000/svg"
+>
+    <path
+    clip-rule="evenodd"
+    fill-rule="evenodd"
+    d="M2 4.75A.75.75 0 012.75 4h14.5a.75.75 0 010 1.5H2.75A.75.75 0 012 4.75zm0 10.5a.75.75 0 01.75-.75h7.5a.75.75 0 010 1.5h-7.5a.75.75 0 01-.75-.75zM2 10a.75.75 0 01.75-.75h14.5a.75.75 0 010 1.5H2.75A.75.75 0 012 10z"
+    ></path>
+</svg>

examples/full_stack/frontend/public/icons/dropdown.svg

@@ -0,0 +1,14 @@
+<svg
+aria-hidden="true"
+xmlns="http://www.w3.org/2000/svg"
+fill="none"
+viewBox="0 0 10 6"
+>
+<path
+    stroke="currentColor"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+    stroke-width="2"
+    d="m1 1 4 4 4-4"
+/>
+</svg>

examples/full_stack/frontend/public/icons/dropup.svg

@@ -0,0 +1,14 @@
+<svg
+    aria-hidden="true"
+    xmlns="http://www.w3.org/2000/svg"
+    fill="none"
+    viewBox="0 0 10 6"
+>
+    <path
+    stroke="currentColor"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+    stroke-width="2"
+    d="M9 5 5 1 1 5"
+    />
+</svg>

examples/full_stack/frontend/public/icons/github.svg

@@ -0,0 +1,39 @@
+<svg
+viewBox="0 0 20 20"
+version="1.1"
+xmlns="http://www.w3.org/2000/svg"
+xmlns:xlink="http://www.w3.org/1999/xlink"
+fill="currentColor"
+>
+<g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+<g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+></g>
+<g id="SVGRepo_iconCarrier">
+    <title>github [#142]</title> <desc>Created with Sketch.</desc>
+    <defs> </defs>
+    <g
+    id="Page-1"
+    stroke="none"
+    stroke-width="1"
+    fill="none"
+    fill-rule="evenodd"
+    >
+    <g
+        id="Dribbble-Light-Preview"
+        transform="translate(-140.000000, -7559.000000)"
+        fill="currentColor"
+    >
+        <g id="icons" transform="translate(56.000000, 160.000000)">   
+        <path
+            d="M94,7399 C99.523,7399 104,7403.59 104,7409.253 C104,7413.782 101.138,7417.624 97.167,7418.981 C96.66,7419.082 96.48,7418.762 96.48,7418.489 C96.48,7418.151 96.492,7417.047 96.492,7415.675 C96.492,7414.719 96.172,7414.095 95.813,7413.777 C98.04,7413.523 100.38,7412.656 100.38,7408.718 C100.38,7407.598 99.992,7406.684 99.35,7405.966 C99.454,7405.707 99.797,7404.664 99.252,7403.252 C99.252,7403.252 98.414,7402.977 96.505,7404.303 C95.706,7404.076 94.85,7403.962 94,7403.958 C93.15,7403.962 92.295,7404.076 91.497,7404.303 C89.586,7402.977 88.746,7403.252 88.746,7403.252 C88.203,7404.664 88.546,7405.707 88.649,7405.966 C88.01,7406.684 87.619,7407.598 87.619,7408.718 C87.619,7412.646 89.954,7413.526 92.175,7413.785 C91.889,7414.041 91.63,7414.493 91.54,7415.156 C90.97,7415.418 89.522,7415.871 88.63,7414.304 C88.63,7414.304 88.101,7413.319 87.097,7413.247 C87.097,7413.247 86.122,7413.234 87.029,7413.87 C87.029,7413.87 87.684,7414.185 88.139,7415.37 C88.139,7415.37 88.726,7417.2 91.508,7416.58 C91.513,7417.437 91.522,7418.245 91.522,7418.489 C91.522,7418.76 91.338,7419.077 90.839,7418.982 C86.865,7417.627 84,7413.783 84,7409.253 C84,7403.59 88.478,7399 94,7399"
+            id="github-[#142]"
+        >       
+        </path>
+        </g>
+    </g>
+    </g>
+</g>
+</svg>

examples/full_stack/frontend/public/icons/linkedin.svg

@@ -0,0 +1,17 @@
+<svg
+  fill="currentColor"
+  viewBox="0 0 32 32"
+  version="1.1"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <title>linkedin</title>
+    <path d="M28.778 1.004h-25.56c-0.008-0-0.017-0-0.027-0-1.199 0-2.172 0.964-2.186 2.159v25.672c0.014 1.196 0.987 2.161 2.186 2.161 0.010 0 0.019-0 0.029-0h25.555c0.008 0 0.018 0 0.028 0 1.2 0 2.175-0.963 2.194-2.159l0-0.002v-25.67c-0.019-1.197-0.994-2.161-2.195-2.161-0.010 0-0.019 0-0.029 0h0.001zM9.9 26.562h-4.454v-14.311h4.454zM7.674 10.293c-1.425 0-2.579-1.155-2.579-2.579s1.155-2.579 2.579-2.579c1.424 0 2.579 1.154 2.579 2.578v0c0 0.001 0 0.002 0 0.004 0 1.423-1.154 2.577-2.577 2.577-0.001 0-0.002 0-0.003 0h0zM26.556 26.562h-4.441v-6.959c0-1.66-0.034-3.795-2.314-3.795-2.316 0-2.669 1.806-2.669 3.673v7.082h-4.441v-14.311h4.266v1.951h0.058c0.828-1.395 2.326-2.315 4.039-2.315 0.061 0 0.121 0.001 0.181 0.003l-0.009-0c4.5 0 5.332 2.962 5.332 6.817v7.855z"></path>
+  </g>
+</svg>

examples/full_stack/frontend/public/icons/pdf.svg

@@ -0,0 +1,28 @@
+<svg
+  viewBox="0 0 15 15"
+  fill="none"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path
+      d="M3.5 8H3V7H3.5C3.77614 7 4 7.22386 4 7.5C4 7.77614 3.77614 8 3.5 8Z"
+      fill="currentColor"
+    ></path>
+    <path
+      d="M7 10V7H7.5C7.77614 7 8 7.22386 8 7.5V9.5C8 9.77614 7.77614 10 7.5 10H7Z"
+      fill="currentColor"
+    ></path>
+    <path
+      fill-rule="evenodd"
+      clip-rule="evenodd"
+      d="M1 1.5C1 0.671573 1.67157 0 2.5 0H10.7071L14 3.29289V13.5C14 14.3284 13.3284 15 12.5 15H2.5C1.67157 15 1 14.3284 1 13.5V1.5ZM3.5 6H2V11H3V9H3.5C4.32843 9 5 8.32843 5 7.5C5 6.67157 4.32843 6 3.5 6ZM7.5 6H6V11H7.5C8.32843 11 9 10.3284 9 9.5V7.5C9 6.67157 8.32843 6 7.5 6ZM10 11V6H13V7H11V8H12V9H11V11H10Z"
+      fill="currentColor"
+    ></path>
+  </g>
+</svg>

examples/full_stack/frontend/public/icons/plus.svg

@@ -0,0 +1,13 @@
+<svg
+xmlns="http://www.w3.org/2000/svg"
+fill="currentColor"
+viewBox="0 0 24 24"
+stroke="currentColor"
+>
+<path
+    strokeLinecap="round"
+    strokeLinejoin="round"
+    strokeWidth="2"
+    d="M12 6v6m0 0v6m0-6h6m-6 0H6"
+/>
+</svg>

examples/full_stack/frontend/public/icons/settings.svg

@@ -0,0 +1,57 @@
+<svg
+    viewBox="-0.5 0 25 25"
+    fill="none"
+    xmlns="http://www.w3.org/2000/svg"
+>
+    <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+    <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+    ></g>
+    <g id="SVGRepo_iconCarrier">
+    {" "}
+    <path
+        d="M12 7.82001H22"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    <path
+        d="M2 7.82001H4"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    <path
+        d="M20 16.82H22"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    <path
+        d="M2 16.82H12"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    <path
+        d="M8 11.82C10.2091 11.82 12 10.0291 12 7.82001C12 5.61087 10.2091 3.82001 8 3.82001C5.79086 3.82001 4 5.61087 4 7.82001C4 10.0291 5.79086 11.82 8 11.82Z"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    <path
+        d="M16 20.82C18.2091 20.82 20 19.0291 20 16.82C20 14.6109 18.2091 12.82 16 12.82C13.7909 12.82 12 14.6109 12 16.82C12 19.0291 13.7909 20.82 16 20.82Z"
+        stroke="currentColor"
+        stroke-width="1.5"
+        stroke-linecap="round"
+        stroke-linejoin="round"
+    ></path>{" "}
+    </g>
+</svg>

examples/full_stack/frontend/public/icons/sitemap.svg

@@ -0,0 +1,61 @@
+<svg
+  viewBox="0 0 24 24"
+  fill="none"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <rect
+      height="4"
+      stroke="currentColor"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      stroke-width="2"
+      width="4"
+      x="10"
+      y="4"
+    ></rect>
+    <rect
+      height="4"
+      stroke="currentColor"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      stroke-width="2"
+      width="4"
+      x="10"
+      y="16"
+    ></rect>
+    <rect
+      height="4"
+      stroke="currentColor"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      stroke-width="2"
+      width="4"
+      x="3"
+      y="16"
+    ></rect>
+    <rect
+      height="4"
+      stroke="currentColor"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      stroke-width="2"
+      width="4"
+      x="17"
+      y="16"
+    ></rect>
+    <path
+      d="M12 8V12M12 16V12M12 12H17C18.1046 12 19 12.8954 19 14V16M12 12H7C5.89543 12 5 12.8954 5 14V16"
+      stroke="currentColor"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+      stroke-width="2"
+    ></path>
+  </g>
+</svg>

examples/full_stack/frontend/public/icons/text.svg

@@ -0,0 +1,21 @@
+<svg
+  viewBox="0 0 24 24"
+  fill="none"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path
+      d="M12 3V21M9 21H15M19 6V3H5V6"
+      stroke="currentColor"
+      stroke-width="2"
+      stroke-linecap="round"
+      stroke-linejoin="round"
+    ></path>
+  </g>
+</svg>

examples/full_stack/frontend/public/icons/twitter.svg

@@ -0,0 +1,24 @@
+<svg
+  fill="currentColor"
+  version="1.1"
+  xmlns="http://www.w3.org/2000/svg"
+  xmlns:xlink="http://www.w3.org/1999/xlink"
+  viewBox="0 0 512 512"
+  xml:space="preserve"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <g id="7935ec95c421cee6d86eb22ecd12f847">
+      <path
+        style="display: inline;"
+        d="M459.186,151.787c0.203,4.501,0.305,9.023,0.305,13.565 c0,138.542-105.461,298.285-298.274,298.285c-59.209,0-114.322-17.357-160.716-47.104c8.212,0.973,16.546,1.47,25.012,1.47 c49.121,0,94.318-16.759,130.209-44.884c-45.887-0.841-84.596-31.154-97.938-72.804c6.408,1.227,12.968,1.886,19.73,1.886 c9.55,0,18.816-1.287,27.617-3.68c-47.955-9.633-84.1-52.001-84.1-102.795c0-0.446,0-0.882,0.011-1.318 c14.133,7.847,30.294,12.562,47.488,13.109c-28.134-18.796-46.637-50.885-46.637-87.262c0-19.212,5.16-37.218,14.193-52.7 c51.707,63.426,128.941,105.156,216.072,109.536c-1.784-7.675-2.718-15.674-2.718-23.896c0-57.891,46.941-104.832,104.832-104.832 c30.173,0,57.404,12.734,76.525,33.102c23.887-4.694,46.313-13.423,66.569-25.438c-7.827,24.485-24.434,45.025-46.089,58.002 c21.209-2.535,41.426-8.171,60.222-16.505C497.448,118.542,479.666,137.004,459.186,151.787z"
+      > 
+      </path>
+    </g>
+  </g>
+</svg>

examples/full_stack/frontend/public/icons/web.svg

@@ -0,0 +1,20 @@
+<svg
+  fill="currentColor"
+  version="1.1"
+  id="Layer_1"
+  xmlns="http://www.w3.org/2000/svg"
+  xmlns:xlink="http://www.w3.org/1999/xlink"
+  viewBox="0 0 24 24"
+  enable-background="new 0 0 24 24"
+  xml:space="preserve"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">
+    <path d="M18.41,0.01H4V9h2V2.01h10v6h6v14H4v2h20V5.6L18.41,0.01z M18,2.43l3.59,3.59H18V2.43z M21,18v2h-1h-1h-1v-8h2v6H21z M7,12 H6v2h1v6h2v-6h1v-2H9H7z M15,12l-1,1.61L13,12v0h-2v8h2v-4.21l1,1.61l1-1.61V20h2v-8L15,12L15,12z M3,15H2v-3H0v8h2v-3h1v3h2v-8H3 V15z"></path>
+  </g>
+</svg>

examples/full_stack/frontend/public/icons/youtube.svg

@@ -0,0 +1,39 @@
+<svg
+  viewBox="0 -3 20 20"
+  version="1.1"
+  xmlns="http://www.w3.org/2000/svg"
+  xmlns:xlink="http://www.w3.org/1999/xlink"
+  fill="currentColor"
+>
+  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+  <g
+    id="SVGRepo_tracerCarrier"
+    stroke-linecap="round"
+    stroke-linejoin="round"
+  ></g>
+  <g id="SVGRepo_iconCarrier">    
+    <title>youtube [#168]</title> <desc>Created with Sketch.</desc>
+    <defs> </defs>
+    <g
+      id="Page-1"
+      stroke="none"
+      stroke-width="1"
+      fill="none"
+      fill-rule="evenodd"
+    >      
+      <g
+        id="Dribbble-Light-Preview"
+        transform="translate(-300.000000, -7442.000000)"
+        fill="currentColor"
+      >
+        <g id="icons" transform="translate(56.000000, 160.000000)">   
+          <path
+            d="M251.988432,7291.58588 L251.988432,7285.97425 C253.980638,7286.91168 255.523602,7287.8172 257.348463,7288.79353 C255.843351,7289.62824 253.980638,7290.56468 251.988432,7291.58588 M263.090998,7283.18289 C262.747343,7282.73013 262.161634,7282.37809 261.538073,7282.26141 C259.705243,7281.91336 248.270974,7281.91237 246.439141,7282.26141 C245.939097,7282.35515 245.493839,7282.58153 245.111335,7282.93357 C243.49964,7284.42947 244.004664,7292.45151 244.393145,7293.75096 C244.556505,7294.31342 244.767679,7294.71931 245.033639,7294.98558 C245.376298,7295.33761 245.845463,7295.57995 246.384355,7295.68865 C247.893451,7296.0008 255.668037,7296.17532 261.506198,7295.73552 C262.044094,7295.64178 262.520231,7295.39147 262.895762,7295.02447 C264.385932,7293.53455 264.28433,7285.06174 263.090998,7283.18289"
+            id="youtube-[#168]"
+          >
+          </path>
+        </g>
+      </g>
+    </g>
+  </g>
+</svg>

examples/full_stack/frontend/src/components/PageWrapper.js

@@ -0,0 +1,9 @@
+export default function PageWrapper({ children }) {
+  return (
+    <>
+      <div className="flex pt-4 px-4 sm:ml-64 min-h-screen">
+        <div className="flex-grow pt-4 px-4 rounded-lg">{children}</div>
+      </div>
+    </>
+  );
+}

examples/full_stack/frontend/src/components/chat/BotWrapper.js

@@ -0,0 +1,16 @@
+export default function BotWrapper({ children }) {
+  return (
+    <>
+      <div className="rounded-lg">
+        <div className="flex flex-row items-center">
+          <div className="flex items-center justify-center h-10 w-10 rounded-full bg-black text-white flex-shrink-0">
+            B
+          </div>
+          <div className="ml-3 text-sm bg-white py-2 px-4 shadow-lg rounded-xl">
+            <div>{children}</div>
+          </div>
+        </div>
+      </div>
+    </>
+  );
+}