Bump version to 0.0.26 (#333)

Co-authored-by: Aaishik Dutta <aaishikdutta@Aaishiks-MacBook-Pro.local>

.github/PULL_REQUEST_TEMPLATE.md

@@ -11,14 +11,17 @@ Please delete options that are not relevant.
 - [ ] Bug fix (non-breaking change which fixes an issue)
 - [ ] New feature (non-breaking change which adds functionality)
 - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
+- [ ] Refactor (does not change functionality, e.g. code style improvements, linting)
 - [ ] Documentation update
 
 ## How Has This Been Tested?
 
 Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
 
-- [ ] Test A
-- [ ] Test B
+Please delete options that are not relevant.
+
+- [ ] Unit Test
+- [ ] Test Script (please provide)
 
 ## Checklist:
 

README.md

@@ -14,6 +14,29 @@ Embedchain is a framework to easily create LLM powered bots over any dataset. If
 pip install embedchain
 ```
 
+## 🔥 Latest
+
+- **[2023/07/19]** Released support for 🦙 `llama2` model. Start creating your `llama2` based bots like this:
+
+  ```python
+  import os
+
+  from embedchain import Llama2App
+
+  os.environ['REPLICATE_API_TOKEN'] = "REPLICATE API TOKEN"
+
+  zuck_bot = Llama2App()
+
+  # Embed your data
+  zuck_bot.add("youtube_video", "https://www.youtube.com/watch?v=Ff4fRgnuFgQ")
+  zuck_bot.add("web_page", "https://en.wikipedia.org/wiki/Mark_Zuckerberg")
+
+  # Nice, your bot is ready now. Start asking questions to your bot.
+  zuck_bot.query("Who is Mark Zuckerberg?")
+  # Answer: Mark Zuckerberg is an American internet entrepreneur and business magnate. He is the co-founder and CEO of Facebook.
+  ```
+
+
 ## 🔍 Demo
 
 Try out embedchain in your browser:

docs/advanced/adding_data.mdx

@@ -0,0 +1,25 @@
+---
+title: '➕ Adding Data'
+---
+
+## Add Dataset
+
+- This step assumes that you have already created an `app` instance by either using `App`, `OpenSourceApp` or `CustomApp`. We are calling our app instance as `naval_chat_bot` 🤖
+
+- Now use `.add()` function to add any dataset.
+
+```python
+# naval_chat_bot = App() or
+# naval_chat_bot = OpenSourceApp()
+
+# Embed Online Resources
+naval_chat_bot.add("youtube_video", "https://www.youtube.com/watch?v=3qHkcs3kG44")
+naval_chat_bot.add("pdf_file", "https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
+naval_chat_bot.add("web_page", "https://nav.al/feedback")
+naval_chat_bot.add("web_page", "https://nav.al/agi")
+
+# Embed Local Resources
+naval_chat_bot.add_local("qna_pair", ("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."))
+```
+
+The possible formats to add data can be found on the [Supported Data Formats](/advanced/data_types) page.

docs/advanced/app_types.mdx

@@ -2,12 +2,6 @@
 title: '📱 App types'
 ---
 
-Creating a chatbot involves 3 steps:
-
-- ⚙️ Import the App instance
-- 🗃️ Add Dataset
-- 💬 Query or Chat on the dataset and get answers (Interface Types)
-
 ## App Types
 
 We have three types of App.
@@ -16,13 +10,12 @@ We have three types of App.
 
 ```python
 from embedchain import App
-naval_chat_bot = App()
+app = App()
 ```
 
 - `App` uses OpenAI's model, so these are paid models. 💸 You will be charged for embedding model usage and LLM usage.
-
 - `App` uses OpenAI's embedding model to create embeddings for chunks and ChatGPT API as LLM to get answer given the relevant docs. Make sure that you have an OpenAI account and an API key. If you have don't have an API key, you can create one by visiting [this link](https://platform.openai.com/account/api-keys).
-
+- `App` is opinionated. It uses the best embedding model and LLM on the market.
 - Once you have the API key, set it in an environment variable called `OPENAI_API_KEY`
 
 ```python
@@ -30,16 +23,74 @@ import os
 os.environ["OPENAI_API_KEY"] = "sk-xxxx"
 ```
 
+### Llama2App
+
+```python
+import os
+
+from embedchain import Llama2App
+
+os.environ['REPLICATE_API_TOKEN'] = "REPLICATE API TOKEN"
+
+zuck_bot = Llama2App()
+
+# Embed your data
+zuck_bot.add("youtube_video", "https://www.youtube.com/watch?v=Ff4fRgnuFgQ")
+zuck_bot.add("web_page", "https://en.wikipedia.org/wiki/Mark_Zuckerberg")
+
+# Nice, your bot is ready now. Start asking questions to your bot.
+zuck_bot.query("Who is Mark Zuckerberg?")
+# Answer: Mark Zuckerberg is an American internet entrepreneur and business magnate. He is the co-founder and CEO of Facebook. Born in 1984, he dropped out of Harvard University to focus on his social media platform, which has since grown to become one of the largest and most influential technology companies in the world.
+
+# Enable web search for your bot
+zuck_bot.online = True # enable internet access for the bot
+zuck_bot.query("Who owns the new threads app and when it was founded?")
+# Answer: Based on the context provided, the new Threads app is owned by Meta, the parent company of Facebook, Instagram, and WhatsApp.
+```
+
+- `Llama2App` uses Replicate's LLM model, so these are paid models. You can get the `REPLICATE_API_TOKEN` by registering on [their website](https://replicate.com/account).
+- `Llama2App` uses OpenAI's embedding model to create embeddings for chunks. Make sure that you have an OpenAI account and an API key. If you have don't have an API key, you can create one by visiting [this link](https://platform.openai.com/account/api-keys).
+
+
 ### OpenSourceApp
 
 ```python
 from embedchain import OpenSourceApp
-naval_chat_bot = OpenSourceApp()
+app = OpenSourceApp()
 ```
 
 - `OpenSourceApp` uses open source embedding and LLM model. It uses `all-MiniLM-L6-v2` from Sentence Transformers library as the embedding model and `gpt4all` as the LLM.
 - Here there is no need to setup any api keys. You just need to install embedchain package and these will get automatically installed. 📦
 - Once you have imported and instantiated the app, every functionality from here onwards is the same for either type of app. 📚
+- `OpenSourceApp` is opinionated. It uses the best open source embedding model and LLM on the market.
+
+### CustomApp
+
+```python
+from embedchain import CustomApp
+from embedchain.config import CustomAppConfig
+from embedchain.models import Providers, EmbeddingFunctions
+
+config = CustomAppConfig(embedding_fn=EmbeddingFunctions.OPENAI, provider=Providers.OPENAI)
+app = CustomApp(config)
+```
+
+- `CustomApp` is not opinionated.
+- Configuration required. It's for advanced users who want to mix and match different embedding models and LLMs. Configuration required.
+- while it's doing that, it's still providing abstractions through `Providers`.
+- paid and free/open source providers included.
+- Once you have imported and instantiated the app, every functionality from here onwards is the same for either type of app. 📚
+- Following providers are available for an LLM
+    - OPENAI
+    - ANTHPROPIC
+    - VERTEX_AI
+    - GPT4ALL
+- Following embedding functions are available for an embedding function
+    - OPENAI
+    - HUGGING_FACE
+    - VERTEX_AI
+    - GPT4ALL
+
 
 ### PersonApp
 
@@ -57,25 +108,7 @@ import os
 os.environ["OPENAI_API_KEY"] = "sk-xxxx"
 ```
 
-## Add Dataset
-
-- This step assumes that you have already created an `app` instance by either using `App` or `OpenSourceApp`. We are calling our app instance as `naval_chat_bot` 🤖
-
-- Now use `.add()` function to add any dataset.
-
-```python
-# naval_chat_bot = App() or
-# naval_chat_bot = OpenSourceApp()
-
-# Embed Online Resources
-naval_chat_bot.add("youtube_video", "https://www.youtube.com/watch?v=3qHkcs3kG44")
-naval_chat_bot.add("pdf_file", "https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
-naval_chat_bot.add("web_page", "https://nav.al/feedback")
-naval_chat_bot.add("web_page", "https://nav.al/agi")
-
-# Embed Local Resources
-naval_chat_bot.add_local("qna_pair", ("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."))
-```
+#### Compatibility with other apps
 
 - If there is any other app instance in your script or app, you can change the import as
 
@@ -90,49 +123,3 @@ from embedchain import App as ECApp
 from embedchain import OpenSourceApp as ECOSApp
 from embedchain import PersonApp as ECPApp
 ```
-
-## Interface Types
-
-### Query Interface
-
-- This interface is like a question answering bot. It takes a question and gets the answer. It does not maintain context about the previous chats.❓
-
-- To use this, call `.query()` function to get the answer for any query.
-
-```python
-print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
-# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
-```
-
-### Chat Interface
-
-- This interface is chat interface where it remembers previous conversation. Right now it remembers 5 conversation by default. 💬
-
-- To use this, call `.chat` function to get the answer for any query.
-
-```python
-print(naval_chat_bot.chat("How to be happy in life?"))
-# answer: The most important trick to being happy is to realize happiness is a skill you develop and a choice you make. You choose to be happy, and then you work at it. It's just like building muscles or succeeding at your job. It's about recognizing the abundance and gifts around you at all times.
-
-print(naval_chat_bot.chat("who is naval ravikant?"))
-# answer: Naval Ravikant is an Indian-American entrepreneur and investor.
-
-print(naval_chat_bot.chat("what did the author say about happiness?"))
-# answer: The author, Naval Ravikant, believes that happiness is a choice you make and a skill you develop. He compares the mind to the body, stating that just as the body can be molded and changed, so can the mind. He emphasizes the importance of being present in the moment and not getting caught up in regrets of the past or worries about the future. By being present and grateful for where you are, you can experience true happiness.
-```
-
-### Stream Response
-
-- You can add config to your query method to stream responses like ChatGPT does. You would require a downstream handler to render the chunk in your desirable format. Supports both OpenAI model and OpenSourceApp. 📊
-
-- To use this, instantiate a `QueryConfig` or `ChatConfig` object with `stream=True`. Then pass it to the `.chat()` or `.query()` method. The following example iterates through the chunks and prints them as they appear.
-
-```python
-app = App()
-query_config = QueryConfig(stream = True)
-resp = app.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", query_config)
-
-for chunk in resp:
-    print(chunk, end="", flush=True)
-# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
-```

docs/advanced/configuration.mdx

@@ -6,22 +6,18 @@ Embedchain is made to work out of the box. However, for advanced users we're als
 
 ## Examples
 
-### Custom embedding function
+### General
 
 Here's the readme example with configuration options.
 
 ```python
 import os
 from embedchain import App
-from embedchain.config import InitConfig, AddConfig, QueryConfig, ChunkerConfig
+from embedchain.config import AppConfig, AddConfig, QueryConfig, ChunkerConfig
 from chromadb.utils import embedding_functions
 
-# Example: use your own embedding function
-config = InitConfig(ef=embedding_functions.OpenAIEmbeddingFunction(
-                api_key=os.getenv("OPENAI_API_KEY"),
-                organization_id=os.getenv("OPENAI_ORGANIZATION"),
-                model_name="text-embedding-ada-002"
-            ))
+# Example: set the log level for debugging
+config = AppConfig(log_level="DEBUG")
 naval_chat_bot = App(config)
 
 # Example: define your own chunker config for `youtube_video`
@@ -35,7 +31,7 @@ naval_chat_bot.add("web_page", "https://nav.al/agi", add_config)
 
 naval_chat_bot.add_local("qna_pair", ("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."), add_config)
 
-query_config = QueryConfig() # Currently no options
+query_config = QueryConfig()
 print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", query_config))
 ```
 
@@ -87,22 +83,3 @@ for query in queries:
 # Query:  Why did you divorce your first wife?
 # Response:  We divorced due to living apart for five years.
 ```
-
-## Other methods
-
-### Reset
-
-Resets the database and deletes all embeddings. Irreversible. Requires reinitialization afterwards.
-
-```python
-app.reset()
-```
-
-### Count
-
-Counts the number of embeddings (chunks) in the database.
-
-```python
-print(app.count())
-# returns: 481
-```

docs/advanced/data_types.mdx

@@ -30,6 +30,14 @@ To add any web page, use the data_type as `web_page`. Eg:
 app.add('web_page', 'a_valid_web_page_url')
 ```
 
+### Sitemap
+
+Add all web pages from an xml-sitemap. Filters non-text files. Use the data_type as `sitemap`. Eg:
+
+```python
+app.add('sitemap', 'https://example.com/sitemap.xml')
+```
+
 ### Doc file
 
 To add any doc/docx file, use the data_type as `docx`. Eg:

docs/advanced/interface_types.mdx

@@ -0,0 +1,75 @@
+---
+title: '🤝 Interface types'
+---
+
+## Interface Types
+
+The embedchain app exposes the following methods.
+
+### Query Interface
+
+- This interface is like a question answering bot. It takes a question and gets the answer. It does not maintain context about the previous chats.❓
+
+- To use this, call `.query()` function to get the answer for any query.
+
+```python
+print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
+# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
+```
+
+### Chat Interface
+
+- This interface is chat interface where it remembers previous conversation. Right now it remembers 5 conversation by default. 💬
+
+- To use this, call `.chat` function to get the answer for any query.
+
+```python
+print(naval_chat_bot.chat("How to be happy in life?"))
+# answer: The most important trick to being happy is to realize happiness is a skill you develop and a choice you make. You choose to be happy, and then you work at it. It's just like building muscles or succeeding at your job. It's about recognizing the abundance and gifts around you at all times.
+
+print(naval_chat_bot.chat("who is naval ravikant?"))
+# answer: Naval Ravikant is an Indian-American entrepreneur and investor.
+
+print(naval_chat_bot.chat("what did the author say about happiness?"))
+# answer: The author, Naval Ravikant, believes that happiness is a choice you make and a skill you develop. He compares the mind to the body, stating that just as the body can be molded and changed, so can the mind. He emphasizes the importance of being present in the moment and not getting caught up in regrets of the past or worries about the future. By being present and grateful for where you are, you can experience true happiness.
+```
+
+#### Dry Run
+
+Dry Run is an option in the `query` and `chat` methods that allows the user to not send their constructed prompt to the LLM, to save money. It's used for [testing](/advanced/testing#dry-run).
+
+
+### Stream Response
+
+- You can add config to your query method to stream responses like ChatGPT does. You would require a downstream handler to render the chunk in your desirable format. Supports both OpenAI model and OpenSourceApp. 📊
+
+- To use this, instantiate a `QueryConfig` or `ChatConfig` object with `stream=True`. Then pass it to the `.chat()` or `.query()` method. The following example iterates through the chunks and prints them as they appear.
+
+```python
+app = App()
+query_config = QueryConfig(stream = True)
+resp = app.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", query_config)
+
+for chunk in resp:
+    print(chunk, end="", flush=True)
+# answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
+```
+
+### Other Methods
+
+#### Reset
+
+Resets the database and deletes all embeddings. Irreversible. Requires reinitialization afterwards.
+
+```python
+app.reset()
+```
+
+#### Count
+
+Counts the number of embeddings (chunks) in the database.
+
+```python
+print(app.count())
+# returns: 481
+```

docs/advanced/query_configuration.mdx

@@ -2,13 +2,13 @@
 title: '🔍 Query configurations'
 ---
 
-## InitConfig
+## AppConfig
 
-| option    | description           | type                            | default                |
-|-----------|-----------------------|---------------------------------|------------------------|
-| log_level | log level             | string                          | WARNING                |
-| ef        | embedding function    | chromadb.utils.embedding_functions | \{text-embedding-ada-002\} |
-| db        | vector database (experimental) | BaseVectorDB               | ChromaDB               |
+| option      | description           | type                            | default                |
+|-------------|-----------------------|---------------------------------|------------------------|
+| log_level   | log level             | string                          | WARNING                |
+| embedding_fn| embedding function    | chromadb.utils.embedding_functions | \{text-embedding-ada-002\} |
+| db          | vector database (experimental) | BaseVectorDB               | ChromaDB               |
 
 
 ## AddConfig
@@ -18,6 +18,14 @@ title: '🔍 Query configurations'
 |chunker|chunker config|ChunkerConfig|Default values for chunker depends on the `data_type`. Please refer [ChunkerConfig](#chunker-config)|
 |loader|loader config|LoaderConfig|None|
 
+Yes, you are passing `ChunkerConfig` to `AddConfig`, like so:
+
+```python
+chunker_config = ChunkerConfig(chunk_size=100)
+add_config = AddConfig(chunker=chunker_config)
+app.add_local("text", "lorem ipsum", config=add_config)
+```
+
 ### ChunkerConfig
 
 |option|description|type|default|
@@ -36,6 +44,7 @@ Default values of chunker config parameters for different `data_type`:
 |web_page|500|0|len|
 |pdf_file|1000|0|len|
 |youtube_video|2000|0|len|
+|docs_site|500|50|len|
 
 ### LoaderConfig
 

docs/advanced/testing.mdx

@@ -2,14 +2,18 @@
 title: '🧪 Testing'
 ---
 
+## Methods for testing
+
+### Dry Run
+
 Before you consume valueable tokens, you should make sure that the embedding you have done works and that it's receiving the correct document from the database.
 
-For this you can use the `dry_run` method.
+For this you can use the `dry_run` option in your `query` or `chat` method.
 
 Following the example above, add this to your script:
 
 ```python
-print(naval_chat_bot.dry_run('Can you tell me who Naval Ravikant is?'))
+print(naval_chat_bot.query('Can you tell me who Naval Ravikant is?', dry_run=True))
 
 '''
 Use the following pieces of context to answer the query at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

docs/mint.json

@@ -32,7 +32,7 @@
     },
     {
       "group": "Advanced",
-      "pages": ["advanced/app_types", "advanced/data_types", "advanced/query_configuration", "advanced/configuration", "advanced/testing", "advanced/showcase"]
+      "pages": ["advanced/app_types", "advanced/interface_types", "advanced/adding_data","advanced/data_types", "advanced/query_configuration", "advanced/configuration", "advanced/testing", "advanced/showcase"]
     },
     {
       "group": "Contribution Guidelines",

docs/quickstart.mdx

@@ -9,6 +9,12 @@ Install embedchain python package:
 pip install embedchain
 ```
 
+Creating a chatbot involves 3 steps:
+
+- ⚙️ Import the App instance
+- 🗃️ Add Dataset
+- 💬 Query or Chat on the dataset and get answers (Interface Types)
+
 Run your first bot in python using the following code. Make sure to set the `OPENAI_API_KEY` 🔑 environment variable in the code.
 
 ```python

embedchain/__init__.py

@@ -2,7 +2,9 @@ import importlib.metadata
 
 __version__ = importlib.metadata.version(__package__ or __name__)
 
-from .embedchain import App  # noqa: F401
-from .embedchain import OpenSourceApp  # noqa: F401
-from .embedchain import PersonApp  # noqa: F401
-from .embedchain import PersonOpenSourceApp  # noqa: F401
+from embedchain.apps.App import App  # noqa: F401
+from embedchain.apps.CustomApp import CustomApp  # noqa: F401
+from embedchain.apps.Llama2App import Llama2App  # noqa: F401
+from embedchain.apps.OpenSourceApp import OpenSourceApp  # noqa: F401
+from embedchain.apps.PersonApp import (PersonApp,  # noqa: F401
+                                       PersonOpenSourceApp)

embedchain/apps/App.py

@@ -0,0 +1,49 @@
+import openai
+
+from embedchain.config import AppConfig, ChatConfig
+from embedchain.embedchain import EmbedChain
+
+
+class App(EmbedChain):
+    """
+    The EmbedChain app.
+    Has two functions: add and query.
+
+    adds(data_type, url): adds the data from the given URL to the vector db.
+    query(query): finds answer to the given query using vector database and LLM.
+    dry_run(query): test your prompt without consuming tokens.
+    """
+
+    def __init__(self, config: AppConfig = None):
+        """
+        :param config: AppConfig instance to load as configuration. Optional.
+        """
+        if config is None:
+            config = AppConfig()
+
+        super().__init__(config)
+
+    def get_llm_model_answer(self, prompt, config: ChatConfig):
+        messages = []
+        messages.append({"role": "user", "content": prompt})
+        response = openai.ChatCompletion.create(
+            model=config.model or "gpt-3.5-turbo-0613",
+            messages=messages,
+            temperature=config.temperature,
+            max_tokens=config.max_tokens,
+            top_p=config.top_p,
+            stream=config.stream,
+        )
+
+        if config.stream:
+            return self._stream_llm_model_response(response)
+        else:
+            return response["choices"][0]["message"]["content"]
+
+    def _stream_llm_model_response(self, response):
+        """
+        This is a generator for streaming response from the OpenAI completions API
+        """
+        for line in response:
+            chunk = line["choices"][0].get("delta", {}).get("content", "")
+            yield chunk

embedchain/apps/CustomApp.py

@@ -0,0 +1,128 @@
+import logging
+from typing import List
+
+from langchain.schema import BaseMessage
+
+from embedchain.config import ChatConfig, CustomAppConfig
+from embedchain.embedchain import EmbedChain
+from embedchain.models import Providers
+
+
+class CustomApp(EmbedChain):
+    """
+    The custom EmbedChain app.
+    Has two functions: add and query.
+
+    adds(data_type, url): adds the data from the given URL to the vector db.
+    query(query): finds answer to the given query using vector database and LLM.
+    dry_run(query): test your prompt without consuming tokens.
+    """
+
+    def __init__(self, config: CustomAppConfig = None):
+        """
+        :param config: Optional. `CustomAppConfig` instance to load as configuration.
+        :raises ValueError: Config must be provided for custom app
+        """
+        if config is None:
+            raise ValueError("Config must be provided for custom app")
+
+        self.provider = config.provider
+
+        if config.provider == Providers.GPT4ALL:
+            from embedchain import OpenSourceApp
+
+            # Because these models run locally, they should have an instance running when the custom app is created
+            self.open_source_app = OpenSourceApp(config=config.open_source_app_config)
+
+        super().__init__(config)
+
+    def set_llm_model(self, provider: Providers):
+        self.provider = provider
+        if provider == Providers.GPT4ALL:
+            raise ValueError(
+                "GPT4ALL needs to be instantiated with the model known, please create a new app instance instead"
+            )
+
+    def get_llm_model_answer(self, prompt, config: ChatConfig):
+        # TODO: Quitting the streaming response here for now.
+        # Idea: https://gist.github.com/jvelezmagic/03ddf4c452d011aae36b2a0f73d72f68
+        if config.stream:
+            raise NotImplementedError(
+                "Streaming responses have not been implemented for this model yet. Please disable."
+            )
+
+        try:
+            if self.provider == Providers.OPENAI:
+                return CustomApp._get_openai_answer(prompt, config)
+
+            if self.provider == Providers.ANTHROPHIC:
+                return CustomApp._get_athrophic_answer(prompt, config)
+
+            if self.provider == Providers.VERTEX_AI:
+                return CustomApp._get_vertex_answer(prompt, config)
+
+            if self.provider == Providers.GPT4ALL:
+                return self.open_source_app._get_gpt4all_answer(prompt, config)
+
+        except ImportError as e:
+            raise ImportError(e.msg) from None
+
+    @staticmethod
+    def _get_openai_answer(prompt: str, config: ChatConfig) -> str:
+        from langchain.chat_models import ChatOpenAI
+
+        logging.info(vars(config))
+
+        chat = ChatOpenAI(
+            temperature=config.temperature,
+            model=config.model or "gpt-3.5-turbo",
+            max_tokens=config.max_tokens,
+            streaming=config.stream,
+        )
+
+        if config.top_p and config.top_p != 1:
+            logging.warning("Config option `top_p` is not supported by this model.")
+
+        messages = CustomApp._get_messages(prompt)
+
+        return chat(messages).content
+
+    @staticmethod
+    def _get_athrophic_answer(prompt: str, config: ChatConfig) -> str:
+        from langchain.chat_models import ChatAnthropic
+
+        chat = ChatAnthropic(temperature=config.temperature, model=config.model)
+
+        if config.max_tokens and config.max_tokens != 1000:
+            logging.warning("Config option `max_tokens` is not supported by this model.")
+
+        messages = CustomApp._get_messages(prompt)
+
+        return chat(messages).content
+
+    @staticmethod
+    def _get_vertex_answer(prompt: str, config: ChatConfig) -> str:
+        from langchain.chat_models import ChatVertexAI
+
+        chat = ChatVertexAI(temperature=config.temperature, model=config.model, max_output_tokens=config.max_tokens)
+
+        if config.top_p and config.top_p != 1:
+            logging.warning("Config option `top_p` is not supported by this model.")
+
+        messages = CustomApp._get_messages(prompt)
+
+        return chat(messages).content
+
+    @staticmethod
+    def _get_messages(prompt: str) -> List[BaseMessage]:
+        from langchain.schema import HumanMessage, SystemMessage
+
+        return [SystemMessage(content="You are a helpful assistant."), HumanMessage(content=prompt)]
+
+    def _stream_llm_model_response(self, response):
+        """
+        This is a generator for streaming response from the OpenAI completions API
+        """
+        for line in response:
+            chunk = line["choices"][0].get("delta", {}).get("content", "")
+            yield chunk

embedchain/apps/Llama2App.py

@@ -0,0 +1,36 @@
+import os
+
+from langchain.llms import Replicate
+
+from embedchain.config import AppConfig
+from embedchain.embedchain import EmbedChain
+
+
+class Llama2App(EmbedChain):
+    """
+    The EmbedChain Llama2App class.
+    Has two functions: add and query.
+
+    adds(data_type, url): adds the data from the given URL to the vector db.
+    query(query): finds answer to the given query using vector database and LLM.
+    """
+
+    def __init__(self, config: AppConfig = None):
+        """
+        :param config: AppConfig instance to load as configuration. Optional.
+        """
+        if "REPLICATE_API_TOKEN" not in os.environ:
+            raise ValueError("Please set the REPLICATE_API_TOKEN environment variable.")
+
+        if config is None:
+            config = AppConfig()
+
+        super().__init__(config)
+
+    def get_llm_model_answer(self, prompt, config: AppConfig = None):
+        # TODO: Move the model and other inputs into config
+        llm = Replicate(
+            model="a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5",
+            input={"temperature": 0.75, "max_length": 500, "top_p": 1},
+        )
+        return llm(prompt)

embedchain/apps/OpenSourceApp.py

@@ -0,0 +1,65 @@
+import logging
+from typing import Iterable, Union
+
+from embedchain.config import ChatConfig, OpenSourceAppConfig
+from embedchain.embedchain import EmbedChain
+
+gpt4all_model = None
+
+
+class OpenSourceApp(EmbedChain):
+    """
+    The OpenSource app.
+    Same as App, but uses an open source embedding model and LLM.
+
+    Has two function: add and query.
+
+    adds(data_type, url): adds the data from the given URL to the vector db.
+    query(query): finds answer to the given query using vector database and LLM.
+    """
+
+    def __init__(self, config: OpenSourceAppConfig = None):
+        """
+        :param config: OpenSourceAppConfig instance to load as configuration. Optional.
+        `ef` defaults to open source.
+        """
+        logging.info("Loading open source embedding model. This may take some time...")  # noqa:E501
+        if not config:
+            config = OpenSourceAppConfig()
+
+        if not config.model:
+            raise ValueError("OpenSourceApp needs a model to be instantiated. Maybe you passed the wrong config type?")
+
+        self.instance = OpenSourceApp._get_instance(config.model)
+
+        logging.info("Successfully loaded open source embedding model.")
+        super().__init__(config)
+
+    def get_llm_model_answer(self, prompt, config: ChatConfig):
+        return self._get_gpt4all_answer(prompt=prompt, config=config)
+
+    @staticmethod
+    def _get_instance(model):
+        try:
+            from gpt4all import GPT4All
+        except ModuleNotFoundError:
+            raise ValueError(
+                "The GPT4All python package is not installed. Please install it with `pip install GPT4All`"
+            ) from None
+
+        return GPT4All(model)
+
+    def _get_gpt4all_answer(self, prompt: str, config: ChatConfig) -> Union[str, Iterable]:
+        if config.model and config.model != self.config.model:
+            raise RuntimeError(
+                "OpenSourceApp does not support switching models at runtime. Please create a new app instance."
+            )
+
+        response = self.instance.generate(
+            prompt=prompt,
+            streaming=config.stream,
+            top_p=config.top_p,
+            max_tokens=config.max_tokens,
+            temp=config.temperature,
+        )
+        return response

embedchain/apps/PersonApp.py

@@ -0,0 +1,63 @@
+from string import Template
+
+from embedchain.apps.App import App
+from embedchain.apps.OpenSourceApp import OpenSourceApp
+from embedchain.config import ChatConfig, QueryConfig
+from embedchain.config.apps.BaseAppConfig import BaseAppConfig
+from embedchain.config.QueryConfig import (DEFAULT_PROMPT,
+                                           DEFAULT_PROMPT_WITH_HISTORY)
+
+
+class EmbedChainPersonApp:
+    """
+    Base class to create a person bot.
+    This bot behaves and speaks like a person.
+
+    :param person: name of the person, better if its a well known person.
+    :param config: BaseAppConfig instance to load as configuration.
+    """
+
+    def __init__(self, person, config: BaseAppConfig = None):
+        self.person = person
+        self.person_prompt = f"You are {person}. Whatever you say, you will always say in {person} style."  # noqa:E501
+        super().__init__(config)
+
+
+class PersonApp(EmbedChainPersonApp, App):
+    """
+    The Person app.
+    Extends functionality from EmbedChainPersonApp and App
+    """
+
+    def query(self, input_query, config: QueryConfig = None):
+        self.template = Template(self.person_prompt + " " + DEFAULT_PROMPT)
+        query_config = QueryConfig(
+            template=self.template,
+        )
+        return super().query(input_query, query_config)
+
+    def chat(self, input_query, config: ChatConfig = None):
+        self.template = Template(self.person_prompt + " " + DEFAULT_PROMPT_WITH_HISTORY)
+        chat_config = ChatConfig(
+            template=self.template,
+        )
+        return super().chat(input_query, chat_config)
+
+
+class PersonOpenSourceApp(EmbedChainPersonApp, OpenSourceApp):
+    """
+    The Person app.
+    Extends functionality from EmbedChainPersonApp and OpenSourceApp
+    """
+
+    def query(self, input_query, config: QueryConfig = None):
+        query_config = QueryConfig(
+            template=self.template,
+        )
+        return super().query(input_query, query_config)
+
+    def chat(self, input_query, config: ChatConfig = None):
+        chat_config = ChatConfig(
+            template=self.template,
+        )
+        return super().chat(input_query, chat_config)

embedchain/chunkers/base_chunker.py

@@ -5,6 +5,7 @@ class BaseChunker:
     def __init__(self, text_splitter):
         """Initialize the chunker."""
         self.text_splitter = text_splitter
+        self.data_type = None
 
     def create_chunks(self, loader, src):
         """
@@ -22,7 +23,10 @@ class BaseChunker:
         metadatas = []
         for data in datas:
             content = data["content"]
+
             meta_data = data["meta_data"]
+            # add data type to meta data to allow query using data type
+            meta_data["data_type"] = self.data_type
             url = meta_data["url"]
 
             chunks = self.get_chunks(content)
@@ -47,3 +51,9 @@ class BaseChunker:
         Override in child class if custom logic.
         """
         return self.text_splitter.split_text(content)
+
+    def set_data_type(self, data_type):
+        """
+        set the data type of chunker
+        """
+        self.data_type = data_type

embedchain/chunkers/docs_site.py

@@ -5,18 +5,16 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter
 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.AddConfig import ChunkerConfig
 
-TEXT_SPLITTER_CHUNK_PARAMS = {
-    "chunk_size": 500,
-    "chunk_overlap": 50,
-    "length_function": len,
-}
-
 
 class DocsSiteChunker(BaseChunker):
     """Chunker for code docs site."""
 
     def __init__(self, config: Optional[ChunkerConfig] = None):
         if config is None:
-            config = TEXT_SPLITTER_CHUNK_PARAMS
-        text_splitter = RecursiveCharacterTextSplitter(**config)
+            config = ChunkerConfig(chunk_size=500, chunk_overlap=50, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
         super().__init__(text_splitter)

embedchain/config/InitConfig.py

@@ -1,74 +0,0 @@
-import logging
-import os
-
-from chromadb.utils import embedding_functions
-
-from embedchain.config.BaseConfig import BaseConfig
-
-
-class InitConfig(BaseConfig):
-    """
-    Config to initialize an embedchain `App` instance.
-    """
-
-    def __init__(self, log_level=None, ef=None, db=None, host=None, port=None, id=None):
-        """
-        :param log_level: Optional. (String) Debug level
-        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
-        :param ef: Optional. Embedding function to use.
-        :param db: Optional. (Vector) database to use for embeddings.
-        :param id: Optional. ID of the app. Document metadata will have this id.
-        :param host: Optional. Hostname for the database server.
-        :param port: Optional. Port for the database server.
-        """
-        self._setup_logging(log_level)
-        self.ef = ef
-        self.db = db
-        self.host = host
-        self.port = port
-        self.id = id
-        return
-
-    def _set_embedding_function(self, ef):
-        self.ef = ef
-        return
-
-    def _set_embedding_function_to_default(self):
-        """
-        Sets embedding function to default (`text-embedding-ada-002`).
-
-        :raises ValueError: If the template is not valid as template should contain
-        $context and $query
-        """
-        if os.getenv("OPENAI_API_KEY") is None and os.getenv("OPENAI_ORGANIZATION") is None:
-            raise ValueError("OPENAI_API_KEY or OPENAI_ORGANIZATION environment variables not provided")  # noqa:E501
-        self.ef = embedding_functions.OpenAIEmbeddingFunction(
-            api_key=os.getenv("OPENAI_API_KEY"),
-            organization_id=os.getenv("OPENAI_ORGANIZATION"),
-            model_name="text-embedding-ada-002",
-        )
-        return
-
-    def _set_db(self, db):
-        if db:
-            self.db = db
-        return
-
-    def _set_db_to_default(self):
-        """
-        Sets database to default (`ChromaDb`).
-        """
-        from embedchain.vectordb.chroma_db import ChromaDB
-
-        self.db = ChromaDB(ef=self.ef, host=self.host, port=self.port)
-
-    def _setup_logging(self, debug_level):
-        level = logging.WARNING  # Default level
-        if debug_level is not None:
-            level = getattr(logging, debug_level.upper(), None)
-            if not isinstance(level, int):
-                raise ValueError(f"Invalid log level: {debug_level}")
-
-        logging.basicConfig(format="%(asctime)s [%(name)s] [%(levelname)s] %(message)s", level=level)
-        self.logger = logging.getLogger(__name__)
-        return

embedchain/config/QueryConfig.py

@@ -104,7 +104,7 @@ class QueryConfig(BaseConfig):
 
         self.temperature = temperature if temperature else 0
         self.max_tokens = max_tokens if max_tokens else 1000
-        self.model = model if model else "gpt-3.5-turbo-0613"
+        self.model = model
         self.top_p = top_p if top_p else 1
 
         if self.validate_template(template):

embedchain/config/__init__.py

@@ -1,5 +1,7 @@
 from .AddConfig import AddConfig, ChunkerConfig  # noqa: F401
+from .apps.AppConfig import AppConfig  # noqa: F401
+from .apps.CustomAppConfig import CustomAppConfig  # noqa: F401
+from .apps.OpenSourceAppConfig import OpenSourceAppConfig  # noqa: F401
 from .BaseConfig import BaseConfig  # noqa: F401
 from .ChatConfig import ChatConfig  # noqa: F401
-from .InitConfig import InitConfig  # noqa: F401
 from .QueryConfig import QueryConfig  # noqa: F401

embedchain/config/apps/AppConfig.py

@@ -0,0 +1,40 @@
+import os
+
+from chromadb.utils import embedding_functions
+
+from .BaseAppConfig import BaseAppConfig
+
+
+class AppConfig(BaseAppConfig):
+    """
+    Config to initialize an embedchain custom `App` instance, with extra config options.
+    """
+
+    def __init__(self, log_level=None, host=None, port=None, id=None):
+        """
+        :param log_level: Optional. (String) Debug level
+        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param host: Optional. Hostname for the database server.
+        :param port: Optional. Port for the database server.
+        """
+        super().__init__(
+            log_level=log_level, embedding_fn=AppConfig.default_embedding_function(), host=host, port=port, id=id
+        )
+
+    @staticmethod
+    def default_embedding_function():
+        """
+        Sets embedding function to default (`text-embedding-ada-002`).
+
+        :raises ValueError: If the template is not valid as template should contain
+        $context and $query
+        :returns: The default embedding function for the app class.
+        """
+        if os.getenv("OPENAI_API_KEY") is None and os.getenv("OPENAI_ORGANIZATION") is None:
+            raise ValueError("OPENAI_API_KEY or OPENAI_ORGANIZATION environment variables not provided")  # noqa:E501
+        return embedding_functions.OpenAIEmbeddingFunction(
+            api_key=os.getenv("OPENAI_API_KEY"),
+            organization_id=os.getenv("OPENAI_ORGANIZATION"),
+            model_name="text-embedding-ada-002",
+        )

embedchain/config/apps/BaseAppConfig.py

@@ -0,0 +1,53 @@
+import logging
+
+from embedchain.config.BaseConfig import BaseConfig
+
+
+class BaseAppConfig(BaseConfig):
+    """
+    Parent config to initialize an instance of `App`, `OpenSourceApp` or `CustomApp`.
+    """
+
+    def __init__(self, log_level=None, embedding_fn=None, db=None, host=None, port=None, id=None):
+        """
+        :param log_level: Optional. (String) Debug level
+        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
+        :param embedding_fn: Embedding function to use.
+        :param db: Optional. (Vector) database instance to use for embeddings.
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param host: Optional. Hostname for the database server.
+        :param port: Optional. Port for the database server.
+        """
+        self._setup_logging(log_level)
+
+        self.db = db if db else BaseAppConfig.default_db(embedding_fn=embedding_fn, host=host, port=port)
+        self.id = id
+        return
+
+    @staticmethod
+    def default_db(embedding_fn, host, port):
+        """
+        Sets database to default (`ChromaDb`).
+
+        :param embedding_fn: Embedding function to use in database.
+        :param host: Optional. Hostname for the database server.
+        :param port: Optional. Port for the database server.
+        :returns: Default database
+        :raises ValueError: BaseAppConfig knows no default embedding function.
+        """
+        if embedding_fn is None:
+            raise ValueError("ChromaDb cannot be instantiated without an embedding function")
+        from embedchain.vectordb.chroma_db import ChromaDB
+
+        return ChromaDB(embedding_fn=embedding_fn, host=host, port=port)
+
+    def _setup_logging(self, debug_level):
+        level = logging.WARNING  # Default level
+        if debug_level is not None:
+            level = getattr(logging, debug_level.upper(), None)
+            if not isinstance(level, int):
+                raise ValueError(f"Invalid log level: {debug_level}")
+
+        logging.basicConfig(format="%(asctime)s [%(name)s] [%(levelname)s] %(message)s", level=level)
+        self.logger = logging.getLogger(__name__)
+        return

embedchain/config/apps/CustomAppConfig.py

@@ -0,0 +1,102 @@
+from typing import Any
+
+from chromadb.api.types import Documents, Embeddings
+from dotenv import load_dotenv
+
+from embedchain.models import EmbeddingFunctions, Providers
+
+from .BaseAppConfig import BaseAppConfig
+
+load_dotenv()
+
+
+class CustomAppConfig(BaseAppConfig):
+    """
+    Config to initialize an embedchain custom `App` instance, with extra config options.
+    """
+
+    def __init__(
+        self,
+        log_level=None,
+        embedding_fn: EmbeddingFunctions = None,
+        embedding_fn_model=None,
+        db=None,
+        host=None,
+        port=None,
+        id=None,
+        provider: Providers = None,
+        model=None,
+        open_source_app_config=None,
+    ):
+        """
+        :param log_level: Optional. (String) Debug level
+        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
+        :param embedding_fn: Optional. Embedding function to use.
+        :param embedding_fn_model: Optional. Model name to use for embedding function.
+        :param db: Optional. (Vector) database to use for embeddings.
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param host: Optional. Hostname for the database server.
+        :param port: Optional. Port for the database server.
+        :param provider: Optional. (Providers): LLM Provider to use.
+        :param open_source_app_config: Optional. Config instance needed for open source apps.
+        """
+        if provider:
+            self.provider = provider
+        else:
+            raise ValueError("CustomApp must have a provider assigned.")
+
+        self.open_source_app_config = open_source_app_config
+
+        super().__init__(
+            log_level=log_level,
+            embedding_fn=CustomAppConfig.embedding_function(embedding_function=embedding_fn, model=embedding_fn_model),
+            db=db,
+            host=host,
+            port=port,
+            id=id,
+        )
+
+    @staticmethod
+    def langchain_default_concept(embeddings: Any):
+        """
+        Langchains default function layout for embeddings.
+        """
+
+        def embed_function(texts: Documents) -> Embeddings:
+            return embeddings.embed_documents(texts)
+
+        return embed_function
+
+    @staticmethod
+    def embedding_function(embedding_function: EmbeddingFunctions, model: str = None):
+        if not isinstance(embedding_function, EmbeddingFunctions):
+            raise ValueError(
+                f"Invalid option: '{embedding_function}'. Expecting one of the following options: {list(map(lambda x: x.value, EmbeddingFunctions))}"  # noqa: E501
+            )
+
+        if embedding_function == EmbeddingFunctions.OPENAI:
+            from langchain.embeddings import OpenAIEmbeddings
+
+            if model:
+                embeddings = OpenAIEmbeddings(model=model)
+            else:
+                embeddings = OpenAIEmbeddings()
+            return CustomAppConfig.langchain_default_concept(embeddings)
+
+        elif embedding_function == EmbeddingFunctions.HUGGING_FACE:
+            from langchain.embeddings import HuggingFaceEmbeddings
+
+            embeddings = HuggingFaceEmbeddings(model_name=model)
+            return CustomAppConfig.langchain_default_concept(embeddings)
+
+        elif embedding_function == EmbeddingFunctions.VERTEX_AI:
+            from langchain.embeddings import VertexAIEmbeddings
+
+            embeddings = VertexAIEmbeddings(model_name=model)
+            return CustomAppConfig.langchain_default_concept(embeddings)
+
+        elif embedding_function == EmbeddingFunctions.GPT4ALL:
+            # Note: We could use langchains GPT4ALL embedding, but it's not available in all versions.
+            from chromadb.utils import embedding_functions
+
+            return embedding_functions.SentenceTransformerEmbeddingFunction(model_name=model)

embedchain/config/apps/OpenSourceAppConfig.py

@@ -0,0 +1,38 @@
+from chromadb.utils import embedding_functions
+
+from .BaseAppConfig import BaseAppConfig
+
+
+class OpenSourceAppConfig(BaseAppConfig):
+    """
+    Config to initialize an embedchain custom `OpenSourceApp` instance, with extra config options.
+    """
+
+    def __init__(self, log_level=None, host=None, port=None, id=None, model=None):
+        """
+        :param log_level: Optional. (String) Debug level
+        ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'].
+        :param id: Optional. ID of the app. Document metadata will have this id.
+        :param host: Optional. Hostname for the database server.
+        :param port: Optional. Port for the database server.
+        :param model: Optional. GPT4ALL uses the model to instantiate the class.
+        So unlike `App`, it has to be provided before querying.
+        """
+        self.model = model or "orca-mini-3b.ggmlv3.q4_0.bin"
+
+        super().__init__(
+            log_level=log_level,
+            embedding_fn=OpenSourceAppConfig.default_embedding_function(),
+            host=host,
+            port=port,
+            id=id,
+        )
+
+    @staticmethod
+    def default_embedding_function():
+        """
+        Sets embedding function to default (`all-MiniLM-L6-v2`).
+
+        :returns: The default embedding function
+        """
+        return embedding_functions.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2")

embedchain/data_formatter/data_formatter.py

@@ -58,17 +58,20 @@ class DataFormatter:
         :return: The chunker for the given data type.
         :raises ValueError: If an unsupported data type is provided.
         """
-        chunkers = {
-            "youtube_video": YoutubeVideoChunker(config),
-            "pdf_file": PdfFileChunker(config),
-            "web_page": WebPageChunker(config),
-            "qna_pair": QnaPairChunker(config),
-            "text": TextChunker(config),
-            "docx": DocxFileChunker(config),
-            "sitemap": WebPageChunker(config),
-            "docs_site": DocsSiteChunker(config),
+        chunker_classes = {
+            "youtube_video": YoutubeVideoChunker,
+            "pdf_file": PdfFileChunker,
+            "web_page": WebPageChunker,
+            "qna_pair": QnaPairChunker,
+            "text": TextChunker,
+            "docx": DocxFileChunker,
+            "sitemap": WebPageChunker,
+            "docs_site": DocsSiteChunker,
         }
-        if data_type in chunkers:
-            return chunkers[data_type]
+        if data_type in chunker_classes:
+            chunker_class = chunker_classes[data_type]
+            chunker = chunker_class(config)
+            chunker.set_data_type(data_type)
+            return chunker
         else:
             raise ValueError(f"Unsupported data type: {data_type}")

embedchain/embedchain.py

@@ -1,19 +1,16 @@
 import logging
 import os
-from string import Template
 
-import openai
-from chromadb.utils import embedding_functions
+from chromadb.errors import InvalidDimensionException
 from dotenv import load_dotenv
 from langchain.docstore.document import Document
 from langchain.memory import ConversationBufferMemory
 
-from embedchain.config import AddConfig, ChatConfig, InitConfig, QueryConfig
-from embedchain.config.QueryConfig import DOCS_SITE_PROMPT_TEMPLATE, DEFAULT_PROMPT, DEFAULT_PROMPT_WITH_HISTORY
+from embedchain.config import AddConfig, ChatConfig, QueryConfig
+from embedchain.config.apps.BaseAppConfig import BaseAppConfig
+from embedchain.config.QueryConfig import DOCS_SITE_PROMPT_TEMPLATE
 from embedchain.data_formatter import DataFormatter
 
-gpt4all_model = None
-
 load_dotenv()
 
 ABS_PATH = os.getcwd()
@@ -23,12 +20,12 @@ memory = ConversationBufferMemory()
 
 
 class EmbedChain:
-    def __init__(self, config: InitConfig):
+    def __init__(self, config: BaseAppConfig):
         """
         Initializes the EmbedChain instance, sets up a vector DB client and
         creates a collection.
 
-        :param config: InitConfig instance to load as configuration.
+        :param config: BaseAppConfig instance to load as configuration.
         """
 
         self.config = config
@@ -139,7 +136,10 @@ class EmbedChain:
             )
         ]
 
-    def get_llm_model_answer(self, prompt):
+    def get_llm_model_answer(self):
+        """
+        Usually implemented by child class
+        """
         raise NotImplementedError
 
     def retrieve_from_database(self, input_query, config: QueryConfig):
@@ -151,14 +151,21 @@ class EmbedChain:
         :param config: The query configuration.
         :return: The content of the document that matched your query.
         """
-        where = {"app_id": self.config.id} if self.config.id is not None else {}  # optional filter
-        result = self.collection.query(
-            query_texts=[
-                input_query,
-            ],
-            n_results=config.number_documents,
-            where=where,
-        )
+        try:
+            where = {"app_id": self.config.id} if self.config.id is not None else {}  # optional filter
+            result = self.collection.query(
+                query_texts=[
+                    input_query,
+                ],
+                n_results=config.number_documents,
+                where=where,
+            )
+        except InvalidDimensionException as e:
+            raise InvalidDimensionException(
+                e.message()
+                + ". This is commonly a side-effect when an embedding function, different from the one used to add the embeddings, is used to retrieve an embedding from the database."  # noqa E501
+            ) from None
+
         results_formatted = self._format_result(result)
         contents = [result[0].page_content for result in results_formatted]
         return contents
@@ -206,7 +213,7 @@ class EmbedChain:
         logging.info(f"Access search to get answers for {input_query}")
         return search.run(input_query)
 
-    def query(self, input_query, config: QueryConfig = None):
+    def query(self, input_query, config: QueryConfig = None, dry_run=False):
         """
         Queries the vector database based on the given input query.
         Gets relevant doc based on the query and then passes it to an
@@ -215,6 +222,12 @@ class EmbedChain:
         :param input_query: The query to use.
         :param config: Optional. The `QueryConfig` instance to use as
         configuration options.
+        :param dry_run: Optional. A dry run does everything except send the resulting prompt to
+        the LLM. The purpose is to test the prompt, not the response.
+        You can use it to test your prompt, including the context provided
+        by the vector database's doc retrieval.
+        The only thing the dry run does not consider is the cut-off due to
+        the `max_tokens` parameter.
         :return: The answer to the query.
         """
         if config is None:
@@ -229,6 +242,9 @@ class EmbedChain:
         prompt = self.generate_prompt(input_query, contexts, config, **k)
         logging.info(f"Prompt: {prompt}")
 
+        if dry_run:
+            return prompt
+
         answer = self.get_answer_from_llm(prompt, config)
 
         if isinstance(answer, str):
@@ -244,7 +260,7 @@ class EmbedChain:
             yield chunk
         logging.info(f"Answer: {streamed_answer}")
 
-    def chat(self, input_query, config: ChatConfig = None):
+    def chat(self, input_query, config: ChatConfig = None, dry_run=False):
         """
         Queries the vector database on the given input query.
         Gets relevant doc based on the query and then passes it to an
@@ -254,6 +270,12 @@ class EmbedChain:
         :param input_query: The query to use.
         :param config: Optional. The `ChatConfig` instance to use as
         configuration options.
+        :param dry_run: Optional. A dry run does everything except send the resulting prompt to
+        the LLM. The purpose is to test the prompt, not the response.
+        You can use it to test your prompt, including the context provided
+        by the vector database's doc retrieval.
+        The only thing the dry run does not consider is the cut-off due to
+        the `max_tokens` parameter.
         :return: The answer to the query.
         """
         if config is None:
@@ -274,6 +296,10 @@ class EmbedChain:
 
         prompt = self.generate_prompt(input_query, contexts, config, **k)
         logging.info(f"Prompt: {prompt}")
+
+        if dry_run:
+            return prompt
+
         answer = self.get_answer_from_llm(prompt, config)
 
         memory.chat_memory.add_user_message(input_query)
@@ -294,27 +320,6 @@ class EmbedChain:
         memory.chat_memory.add_ai_message(streamed_answer)
         logging.info(f"Answer: {streamed_answer}")
 
-    def dry_run(self, input_query, config: QueryConfig = None):
-        """
-        A dry run does everything except send the resulting prompt to
-        the LLM. The purpose is to test the prompt, not the response.
-        You can use it to test your prompt, including the context provided
-        by the vector database's doc retrieval.
-        The only thing the dry run does not consider is the cut-off due to
-        the `max_tokens` parameter.
-
-        :param input_query: The query to use.
-        :param config: Optional. The `QueryConfig` instance to use as
-        configuration options.
-        :return: The prompt that would be sent to the LLM
-        """
-        if config is None:
-            config = QueryConfig()
-        contexts = self.retrieve_from_database(input_query, config)
-        prompt = self.generate_prompt(input_query, contexts, config)
-        logging.info(f"Prompt: {prompt}")
-        return prompt
-
     def count(self):
         """
         Count the number of embeddings.
@@ -329,152 +334,3 @@ class EmbedChain:
         `App` has to be reinitialized after using this method.
         """
         self.db_client.reset()
-
-
-class App(EmbedChain):
-    """
-    The EmbedChain app.
-    Has two functions: add and query.
-
-    adds(data_type, url): adds the data from the given URL to the vector db.
-    query(query): finds answer to the given query using vector database and LLM.
-    dry_run(query): test your prompt without consuming tokens.
-    """
-
-    def __init__(self, config: InitConfig = None):
-        """
-        :param config: InitConfig instance to load as configuration. Optional.
-        """
-        if config is None:
-            config = InitConfig()
-
-        if not config.ef:
-            config._set_embedding_function_to_default()
-
-        if not config.db:
-            config._set_db_to_default()
-
-        super().__init__(config)
-
-    def get_llm_model_answer(self, prompt, config: ChatConfig):
-        messages = []
-        messages.append({"role": "user", "content": prompt})
-        response = openai.ChatCompletion.create(
-            model=config.model,
-            messages=messages,
-            temperature=config.temperature,
-            max_tokens=config.max_tokens,
-            top_p=config.top_p,
-            stream=config.stream,
-        )
-
-        if config.stream:
-            return self._stream_llm_model_response(response)
-        else:
-            return response["choices"][0]["message"]["content"]
-
-    def _stream_llm_model_response(self, response):
-        """
-        This is a generator for streaming response from the OpenAI completions API
-        """
-        for line in response:
-            chunk = line["choices"][0].get("delta", {}).get("content", "")
-            yield chunk
-
-
-class OpenSourceApp(EmbedChain):
-    """
-    The OpenSource app.
-    Same as App, but uses an open source embedding model and LLM.
-
-    Has two function: add and query.
-
-    adds(data_type, url): adds the data from the given URL to the vector db.
-    query(query): finds answer to the given query using vector database and LLM.
-    """
-
-    def __init__(self, config: InitConfig = None):
-        """
-        :param config: InitConfig instance to load as configuration. Optional.
-        `ef` defaults to open source.
-        """
-        print("Loading open source embedding model. This may take some time...")  # noqa:E501
-        if not config:
-            config = InitConfig()
-
-        if not config.ef:
-            config._set_embedding_function(
-                embedding_functions.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2")
-            )
-
-        if not config.db:
-            config._set_db_to_default()
-
-        print("Successfully loaded open source embedding model.")
-        super().__init__(config)
-
-    def get_llm_model_answer(self, prompt, config: ChatConfig):
-        from gpt4all import GPT4All
-
-        global gpt4all_model
-        if gpt4all_model is None:
-            gpt4all_model = GPT4All("orca-mini-3b.ggmlv3.q4_0.bin")
-        response = gpt4all_model.generate(prompt=prompt, streaming=config.stream)
-        return response
-
-
-class EmbedChainPersonApp:
-    """
-    Base class to create a person bot.
-    This bot behaves and speaks like a person.
-
-    :param person: name of the person, better if its a well known person.
-    :param config: InitConfig instance to load as configuration.
-    """
-
-    def __init__(self, person, config: InitConfig = None):
-        self.person = person
-        self.person_prompt = f"You are {person}. Whatever you say, you will always say in {person} style."  # noqa:E501
-        if config is None:
-            config = InitConfig()
-        super().__init__(config)
-
-
-class PersonApp(EmbedChainPersonApp, App):
-    """
-    The Person app.
-    Extends functionality from EmbedChainPersonApp and App
-    """
-
-    def query(self, input_query, config: QueryConfig = None):
-        self.template = Template(self.person_prompt + " " + DEFAULT_PROMPT)
-        query_config = QueryConfig(
-            template=self.template,
-        )
-        return super().query(input_query, query_config)
-
-    def chat(self, input_query, config: ChatConfig = None):
-        self.template = Template(self.person_prompt + " " + DEFAULT_PROMPT_WITH_HISTORY)
-        chat_config = ChatConfig(
-            template=self.template,
-        )
-        return super().chat(input_query, chat_config)
-
-
-class PersonOpenSourceApp(EmbedChainPersonApp, OpenSourceApp):
-    """
-    The Person app.
-    Extends functionality from EmbedChainPersonApp and OpenSourceApp
-    """
-
-    def query(self, input_query, config: QueryConfig = None):
-        query_config = QueryConfig(
-            template=self.template,
-        )
-        return super().query(input_query, query_config)
-
-    def chat(self, input_query, config: ChatConfig = None):
-        chat_config = ChatConfig(
-            template=self.template,
-        )
-        return super().chat(input_query, chat_config)

embedchain/loaders/sitemap.py

@@ -5,6 +5,7 @@ from bs4 import BeautifulSoup
 from bs4.builder import ParserRejectedMarkup
 
 from embedchain.loaders.web_page import WebPageLoader
+from embedchain.utils import is_readable
 
 
 class SitemapLoader:
@@ -20,11 +21,20 @@ class SitemapLoader:
         response.raise_for_status()
 
         soup = BeautifulSoup(response.text, "xml")
-        links = [link.text for link in soup.find_all("loc")]
+
+        links = [link.text for link in soup.find_all("loc") if link.parent.name == "url"]
+        if len(links) == 0:
+            # Get all <loc> tags as a fallback. This might include images.
+            links = [link.text for link in soup.find_all("loc")]
+
         for link in links:
             try:
                 each_load_data = web_page_loader.load_data(link)
-                output.append(each_load_data)
+
+                if is_readable(each_load_data[0].get("content")):
+                    output.append(each_load_data)
+                else:
+                    logging.warning(f"Page is not readable (too many invalid characters): {link}")
             except ParserRejectedMarkup as e:
                 logging.error(f"Failed to parse {link}: {e}")
         return [data[0] for data in output]

embedchain/models/EmbeddingFunctions.py

@@ -0,0 +1,8 @@
+from enum import Enum
+
+
+class EmbeddingFunctions(Enum):
+    OPENAI = "OPENAI"
+    HUGGING_FACE = "HUGGING_FACE"
+    VERTEX_AI = "VERTEX_AI"
+    GPT4ALL = "GPT4ALL"

embedchain/models/Providers.py

@@ -0,0 +1,8 @@
+from enum import Enum
+
+
+class Providers(Enum):
+    OPENAI = "OPENAI"
+    ANTHROPHIC = "ANTHPROPIC"
+    VERTEX_AI = "VERTEX_AI"
+    GPT4ALL = "GPT4ALL"

embedchain/models/__init__.py

@@ -0,0 +1,2 @@
+from .EmbeddingFunctions import EmbeddingFunctions  # noqa: F401
+from .Providers import Providers  # noqa: F401

embedchain/utils.py

@@ -1,4 +1,5 @@
 import re
+import string
 
 
 def clean_string(text):
@@ -33,3 +34,14 @@ def clean_string(text):
     cleaned_text = re.sub(r"([^\w\s])\1*", r"\1", cleaned_text)
 
     return cleaned_text
+
+
+def is_readable(s):
+    """
+    Heuristic to determine if a string is "readable" (mostly contains printable characters and forms meaningful words)
+
+    :param s: string
+    :return: True if the string is more than 95% printable.
+    """
+    printable_ratio = sum(c in string.printable for c in s) / len(s)
+    return printable_ratio > 0.95  # 95% of characters are printable

embedchain/vectordb/chroma_db.py

@@ -8,8 +8,11 @@ from embedchain.vectordb.base_vector_db import BaseVectorDB
 class ChromaDB(BaseVectorDB):
     """Vector database using ChromaDB."""
 
-    def __init__(self, db_dir=None, ef=None, host=None, port=None):
-        self.ef = ef
+    def __init__(self, db_dir=None, embedding_fn=None, host=None, port=None):
+        self.embedding_fn = embedding_fn
+
+        if not hasattr(embedding_fn, "__call__"):
+            raise ValueError("Embedding function is not a function")
 
         if host and port:
             logging.info(f"Connecting to ChromaDB server: {host}:{port}")
@@ -36,5 +39,5 @@ class ChromaDB(BaseVectorDB):
         """Get or create the collection."""
         return self.client.get_or_create_collection(
             "embedchain_store",
-            embedding_function=self.ef,
+            embedding_function=self.embedding_fn,
         )

notebooks/embedchain-chromadb-server.ipynb

@@ -34,13 +34,13 @@
    "source": [
     "import os\n",
     "from embedchain import App\n",
-    "from embedchain.config import InitConfig\n",
+    "from embedchain.config import AppConfig\n",
     "\n",
     "\n",
     "chromadb_host = \"localhost\"\n",
     "chromadb_port = 8000\n",
     "\n",
-    "config = InitConfig(host=chromadb_host, port=chromadb_port)\n",
+    "config = AppConfig(host=chromadb_host, port=chromadb_port)\n",
     "elon_bot = App(config)"
    ]
   },

pyproject.toml

@@ -101,6 +101,7 @@ pytest = "^7.3.1"
 pytest-mock = "^3.10.0"
 pytest-env = "^0.8.1"
 click = "^8.1.3"
+isort = "^5.12.0"
 
 [tool.poetry.extras]
 streamlit = ["streamlit"]

setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
 
 setuptools.setup(
     name="embedchain",
-    version="0.0.24",
+    version="0.0.26",
     author="Taranjeet Singh",
     author_email="reachtotj@gmail.com",
     description="embedchain is a framework to easily create LLM powered bots over any dataset",  # noqa:E501
@@ -34,6 +34,8 @@ setuptools.setup(
         "sentence_transformers",
         "docx2txt",
         "pydantic==1.10.8",
+        "replicate==0.9.0",
+        "duckduckgo-search==3.8.4",
     ],
     extras_require={"dev": ["black", "ruff", "isort", "pytest"]},
 )

tests/embedchain/test_dryrun.py

@@ -1,52 +0,0 @@
-import os
-import unittest
-from string import Template
-from unittest.mock import patch
-
-from embedchain import App
-from embedchain.embedchain import QueryConfig
-
-
-class TestApp(unittest.TestCase):
-    os.environ["OPENAI_API_KEY"] = "test_key"
-
-    def setUp(self):
-        self.app = App()
-
-    @patch("logging.info")
-    def test_query_logs_same_prompt_as_dry_run(self, mock_logging_info):
-        """
-        Test that the 'query' method logs the same prompt as the 'dry_run' method.
-        This is the only way I found to test the prompt in query, that's not returned.
-        """
-        with patch.object(self.app, "retrieve_from_database") as mock_retrieve:
-            mock_retrieve.return_value = ["Test context"]
-            input_query = "Test query"
-            config = QueryConfig(
-                number_documents=3,
-                template=Template("Question: $query, context: $context, history: $history"),
-                history=["Past context 1", "Past context 2"],
-            )
-
-            with patch.object(self.app, "get_answer_from_llm"):
-                self.app.dry_run(input_query, config)
-                self.app.query(input_query, config)
-
-            # Access the log messages captured during the execution
-            logged_messages = [call[0][0] for call in mock_logging_info.call_args_list]
-
-            # Extract the prompts from the log messages
-            dry_run_prompt = self.extract_prompt(logged_messages[0])
-            query_prompt = self.extract_prompt(logged_messages[1])
-
-            # Perform assertions on the prompts
-            self.assertEqual(dry_run_prompt, query_prompt)
-
-    def extract_prompt(self, log_message):
-        """
-        Extracts the prompt value from the log message.
-        Adjust this method based on the log message format in your implementation.
-        """
-        # Modify this logic based on your log message format
-        prefix = "Prompt: "
-        return log_message.split(prefix, 1)[1]

tests/embedchain/test_embedchain.py

@@ -3,7 +3,7 @@ import unittest
 from unittest.mock import patch
 
 from embedchain import App
-from embedchain.config import InitConfig
+from embedchain.config import AppConfig
 
 
 class TestChromaDbHostsLoglevel(unittest.TestCase):
@@ -25,7 +25,7 @@ class TestChromaDbHostsLoglevel(unittest.TestCase):
         """
         Test if the `App` instance is initialized without a config that does not contain default hosts and ports.
         """
-        config = InitConfig(log_level="DEBUG")
+        config = AppConfig(log_level="DEBUG")
 
         app = App(config)
 

tests/vectordb/test_chroma_db.py

@@ -4,7 +4,7 @@ import unittest
 from unittest.mock import patch
 
 from embedchain import App
-from embedchain.config import InitConfig
+from embedchain.config import AppConfig
 from embedchain.vectordb.chroma_db import ChromaDB, chromadb
 
 
@@ -17,7 +17,7 @@ class TestChromaDbHosts(unittest.TestCase):
         port = "1234"
 
         with patch.object(chromadb, "Client") as mock_client:
-            _db = ChromaDB(host=host, port=port)
+            _db = ChromaDB(host=host, port=port, embedding_fn=len)
 
         expected_settings = chromadb.config.Settings(
             chroma_api_impl="rest",
@@ -38,7 +38,7 @@ class TestChromaDbHostsInit(unittest.TestCase):
         host = "test-host"
         port = "1234"
 
-        config = InitConfig(host=host, port=port)
+        config = AppConfig(host=host, port=port)
 
         _app = App(config)
 
@@ -65,7 +65,7 @@ class TestChromaDbHostsLoglevel(unittest.TestCase):
         """
         Test if the `App` instance is initialized without a config that does not contain default hosts and ports.
         """
-        config = InitConfig(log_level="DEBUG")
+        config = AppConfig(log_level="DEBUG")
 
         _app = App(config)