Bump package version to 0.1.27 (#994 )

Co-authored-by: Deven Patel <deven298@yahoo.com>
[Improvements] Package improvements (#993 )
2023-12-06 00:03:13 -08:00 · 2023-12-05 23:42:45 -08:00 · 2023-12-05 19:04:20 -08:00 · 2023-12-05 17:56:24 -08:00 · 2023-12-05 16:52:48 -08:00 · 2023-12-05 16:48:00 -08:00
154 changed files with 2839 additions and 891 deletions
@@ -3,7 +3,15 @@ name: ci
 on:
  push:
    branches: [main]
+    paths:
+      - 'embedchain/**'
+      - 'tests/**'
+      - 'examples/**'
  pull_request:
+    paths:
+      - 'embedchain/**'
+      - 'tests/**'
+      - 'examples/**'

 jobs:
  build:
@@ -23,4 +23,3 @@ embedder:
  provider: openai
  config:
    model: 'text-embedding-ada-002'
-    deployment_name: 'test-deployment'
@@ -0,0 +1,8 @@
+llm:
+  provider: openai
+  config:
+    model: 'gpt-4'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 1
+    stream: false
@@ -6,15 +6,16 @@ Embedchain is made to work out of the box. However, for advanced users we're als

 You can configure different components of your app (`llm`, `embedding model`, or `vector database`) through a simple yaml configuration that Embedchain offers. Here is a generic full-stack example of the yaml config:

-```yaml
+
+<Tip>
+Embedchain applications are configurable using YAML file, JSON file or by directly passing the config dictionary.
+</Tip>
+
+<CodeGroup>
+```yaml config.yaml
 app:
  config:
-    id: 'full-stack-app'
-
-chunker:
-  chunk_size: 100
-  chunk_overlap: 20
-  length_function: 'len'
+    name: 'full-stack-app'

 llm:
  provider: openai
@@ -47,38 +48,138 @@ embedder:
  provider: openai
  config:
    model: 'text-embedding-ada-002'
+
+chunker:
+  chunk_size: 2000
+  chunk_overlap: 100
+  length_function: 'len'
 ```

+```json config.json
+{
+  "app": {
+    "config": {
+      "name": "full-stack-app"
+    }
+  },
+  "llm": {
+    "provider": "openai",
+    "config": {
+      "model": "gpt-3.5-turbo",
+      "temperature": 0.5,
+      "max_tokens": 1000,
+      "top_p": 1,
+      "stream": false,
+      "template": "Use the following pieces of context to answer the query at the end.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n$context\n\nQuery: $query\n\nHelpful Answer:",
+      "system_prompt": "Act as William Shakespeare. Answer the following questions in the style of William Shakespeare."
+    }
+  },
+  "vectordb": {
+    "provider": "chroma",
+    "config": {
+      "collection_name": "full-stack-app",
+      "dir": "db",
+      "allow_reset": true
+    }
+  },
+  "embedder": {
+    "provider": "openai",
+    "config": {
+      "model": "text-embedding-ada-002"
+    }
+  },
+  "chunker": {
+    "chunk_size": 2000,
+    "chunk_overlap": 100,
+    "length_function": "len"
+  }
+}
+```
+
+```python config.py
+config = {
+    'app': {
+        'config': {
+            'name': 'full-stack-app'
+        }
+    },
+    'llm': {
+        'provider': 'openai',
+        'config': {
+            'model': 'gpt-3.5-turbo',
+            'temperature': 0.5,
+            'max_tokens': 1000,
+            'top_p': 1,
+            'stream': False,
+            'template': (
+                "Use the following pieces of context to answer the query at the end.\n"
+                "If you don't know the answer, just say that you don't know, don't try to make up an answer.\n"
+                "$context\n\nQuery: $query\n\nHelpful Answer:"
+            ),
+            'system_prompt': (
+                "Act as William Shakespeare. Answer the following questions in the style of William Shakespeare."
+            )
+        }
+    },
+    'vectordb': {
+        'provider': 'chroma',
+        'config': {
+            'collection_name': 'full-stack-app',
+            'dir': 'db',
+            'allow_reset': True
+        }
+    },
+    'embedder': {
+        'provider': 'openai',
+        'config': {
+            'model': 'text-embedding-ada-002'
+        }
+    },
+    'chunker': {
+        'chunk_size': 2000,
+        'chunk_overlap': 100,
+        'length_function': 'len'
+    }
+}
+```
+</CodeGroup>
+
 Alright, let's dive into what each key means in the yaml config above:

 1. `app` Section:
    - `config`:
-        - `id` (String): The ID or name of your full-stack application.
-2. `chunker` Section:
-    - `chunk_size` (Integer): The size of each chunk of text that is sent to the language model.
-    - `chunk_overlap` (Integer): The amount of overlap between each chunk of text.
-    - `length_function` (String): The function used to calculate the length of each chunk of text. In this case, it's set to 'len'. You can also use any function import directly as a string here.
-3. `llm` Section:
+        - `name` (String): The name of your full-stack application.
+        - `id` (String): The id of your full-stack application.
+        <Note>Only use this to reload already created apps. We recommend users to not create their own ids.</Note>
+        - `collect_metrics` (Boolean): Indicates whether metrics should be collected for the app, defaults to `True`
+        - `log_level` (String): The log level for the app, defaults to `WARNING`
+2. `llm` Section:
    - `provider` (String): The provider for the language model, which is set to 'openai'. You can find the full list of llm providers in [our docs](/components/llms).
-    - `model` (String): The specific model being used, 'gpt-3.5-turbo'.
    - `config`:
+        - `model` (String): The specific model being used, 'gpt-3.5-turbo'.
        - `temperature` (Float): Controls the randomness of the model's output. A higher value (closer to 1) makes the output more random.
        - `max_tokens` (Integer): Controls how many tokens are used in the response.
        - `top_p` (Float): Controls the diversity of word selection. A higher value (closer to 1) makes word selection more diverse.
        - `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
        - `template` (String): A custom template for the prompt that the model uses to generate responses.
        - `system_prompt` (String): A system prompt for the model to follow when generating responses, in this case, it's set to the style of William Shakespeare.
-4. `vectordb` Section:
+        -  `stream` (Boolean): Controls if the response is streamed back to the user (set to false).
+        - `number_documents` (Integer): Number of documents to pull from the vectordb as context, defaults to 1
+3. `vectordb` Section:
    - `provider` (String): The provider for the vector database, set to 'chroma'. You can find the full list of vector database providers in [our docs](/components/vector-databases).
    - `config`:
-        - `collection_name` (String): The initial collection name for the database, set to 'full-stack-app'.
-        - `dir` (String): The directory for the database, set to 'db'.
-        - `allow_reset` (Boolean): Indicates whether resetting the database is allowed, set to true.
-5. `embedder` Section:
+        - `collection_name` (String): The initial collection name for the vectordb, set to 'full-stack-app'.
+        - `dir` (String): The directory for the local database, set to 'db'.
+        - `allow_reset` (Boolean): Indicates whether resetting the vectordb is allowed, set to true.
+    <Note>We recommend you to checkout vectordb specific config [here](https://docs.embedchain.ai/components/vector-databases)</Note>
+4. `embedder` Section:
    - `provider` (String): The provider for the embedder, set to 'openai'. You can find the full list of embedding model providers in [our docs](/components/embedding-models).
    - `config`:
        - `model` (String): The specific model used for text embedding, 'text-embedding-ada-002'.
-
+5. `chunker` Section:
+    - `chunk_size` (Integer): The size of each chunk of text that is sent to the language model.
+    - `chunk_overlap` (Integer): The amount of overlap between each chunk of text.
+    - `length_function` (String): The function used to calculate the length of each chunk of text. In this case, it's set to 'len'. You can also use any function import directly as a string here.
 If you have questions about the configuration above, please feel free to reach out to us using one of the following methods:

 <Snippet file="get-help.mdx" />
@@ -29,7 +29,7 @@ from embedchain import Pipeline as App
 os.environ['OPENAI_API_KEY'] = 'xxx'

 # load embedding model configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")

 app.add("https://en.wikipedia.org/wiki/OpenAI")
 app.query("What is OpenAI?")
@@ -55,11 +55,11 @@ import os
 from embedchain import Pipeline as App

 os.environ["OPENAI_API_TYPE"] = "azure"
-os.environ["OPENAI_API_BASE"] = "https://xxx.openai.azure.com/"
-os.environ["OPENAI_API_KEY"] = "xxx"
+os.environ["AZURE_OPENAI_ENDPOINT"] = "https://xxx.openai.azure.com/"
+os.environ["AZURE_OPENAI_API_KEY"] = "xxx"
 os.environ["OPENAI_API_VERSION"] = "xxx"

-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -93,7 +93,7 @@ GPT4All supports generating high quality embeddings of arbitrary length document
 from embedchain import Pipeline as App

 # load embedding model configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -122,7 +122,7 @@ Hugging Face supports generating embeddings of arbitrary length documents of tex
 from embedchain import Pipeline as App

 # load embedding model configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -153,7 +153,7 @@ Embedchain supports Google's VertexAI embeddings model through a simple interfac
 from embedchain import Pipeline as App

 # load embedding model configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -46,7 +46,7 @@ from embedchain import Pipeline as App
 os.environ['OPENAI_API_KEY'] = 'xxx'

 # load llm configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -78,7 +78,7 @@ os.environ["OPENAI_API_BASE"] = "https://xxx.openai.azure.com/"
 os.environ["OPENAI_API_KEY"] = "xxx"
 os.environ["OPENAI_API_VERSION"] = "xxx"

-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -115,7 +115,7 @@ from embedchain import Pipeline as App
 os.environ["ANTHROPIC_API_KEY"] = "xxx"

 # load llm configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -152,7 +152,7 @@ from embedchain import Pipeline as App
 os.environ["COHERE_API_KEY"] = "xxx"

 # load llm configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -183,7 +183,7 @@ GPT4all is a free-to-use, locally running, privacy-aware chatbot. No GPU or inte
 from embedchain import Pipeline as App

 # load llm configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -216,7 +216,7 @@ from embedchain import Pipeline as App

 os.environ["JINACHAT_API_KEY"] = "xxx"
 # load llm configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -237,7 +237,7 @@ llm:
 Install related dependencies using the following command:

 ```bash
-pip install --upgrade 'embedchain[huggingface_hub]'
+pip install --upgrade 'embedchain[huggingface-hub]'
 ```

 First, set `HUGGINGFACE_ACCESS_TOKEN` in environment variable which you can obtain from [their platform](https://huggingface.co/settings/tokens).
@@ -253,7 +253,7 @@ from embedchain import Pipeline as App
 os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "xxx"

 # load llm configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -283,7 +283,7 @@ from embedchain import Pipeline as App
 os.environ["REPLICATE_API_TOKEN"] = "xxx"

 # load llm configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -308,7 +308,7 @@ Setup Google Cloud Platform application credentials by following the instruction
 from embedchain import Pipeline as App

 # load llm configuration from config.yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -25,7 +25,7 @@ Utilizing a vector database alongside Embedchain is a seamless process. All you
 from embedchain import Pipeline as App

 # load chroma configuration from yaml file
-app = App.from_config(yaml_path="config1.yaml")
+app = App.from_config(config_path="config1.yaml")
 ```

 ```yaml config1.yaml
@@ -64,7 +64,7 @@ pip install --upgrade 'embedchain[elasticsearch]'
 from embedchain import Pipeline as App

 # load elasticsearch configuration from yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -73,8 +73,11 @@ vectordb:
  config:
    collection_name: 'es-index'
    es_url: http://localhost:9200
-    allow_reset: true
+    http_auth:
+      - admin
+      - admin
    api_key: xxx
+    verify_certs: false
 ```
 </CodeGroup>

@@ -92,19 +95,19 @@ pip install --upgrade 'embedchain[opensearch]'
 from embedchain import Pipeline as App

 # load opensearch configuration from yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
 vectordb:
  provider: opensearch
  config:
+    collection_name: 'my-app'
    opensearch_url: 'https://localhost:9200'
    http_auth:
      - admin
      - admin
    vector_dimension: 1536
-    collection_name: 'my-app'
    use_ssl: false
    verify_certs: false
 ```
@@ -131,7 +134,7 @@ os.environ['ZILLIZ_CLOUD_URI'] = 'https://xxx.zillizcloud.com'
 os.environ['ZILLIZ_CLOUD_TOKEN'] = 'xxx'

 # load zilliz configuration from yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -167,7 +170,7 @@ In order to use Pinecone as vector database, set the environment variables `PINE
 from embedchain import Pipeline as App

 # load pinecone configuration from yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -190,7 +193,7 @@ In order to use Qdrant as a vector database, set the environment variables `QDRA
 from embedchain import Pipeline as App

 # load qdrant configuration from yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -210,7 +213,7 @@ In order to use Weaviate as a vector database, set the environment variables `WE
 from embedchain import Pipeline as App

 # load weaviate configuration from yaml file
-app = App.from_config(yaml_path="config.yaml")
+app = App.from_config(config_path="config.yaml")
 ```

 ```yaml config.yaml
@@ -0,0 +1,41 @@
+---
+title: '⚙️ Custom'
+---
+
+When we say "custom", we mean that you can customize the loader and chunker to your needs. This is done by passing a custom loader and chunker to the `add` method.
+
+```python
+from embedchain import Pipeline as App
+import your_loader
+import your_chunker
+
+app = App()
+loader = your_loader()
+chunker = your_chunker()
+
+app.add("source", data_type="custom", loader=loader, chunker=chunker)
+```
+
+<Note>
+    The custom loader and chunker must be a class that inherits from the [`BaseLoader`](https://github.com/embedchain/embedchain/blob/main/embedchain/loaders/base_loader.py) and [`BaseChunker`](https://github.com/embedchain/embedchain/blob/main/embedchain/chunkers/base_chunker.py) classes respectively.
+</Note>
+
+<Note>
+    If the `data_type` is not a valid data type, the `add` method will fallback to the `custom` data type and expect a custom loader and chunker to be passed by the user.
+</Note>
+
+Example:
+
+```python
+from embedchain import Pipeline as App
+from embedchain.loaders.github import GithubLoader
+
+app = App()
+
+loader = GithubLoader(config={"token": "ghp_xxx"})
+
+app.add("repo:embedchain/embedchain type:repo", data_type="github", loader=loader)
+
+app.query("What is Embedchain?")
+# Answer: Embedchain is a Data Platform for Large Language Models (LLMs). It allows users to seamlessly load, index, retrieve, and sync unstructured data in order to build dynamic, LLM-powered applications. There is also a JavaScript implementation called embedchain-js available on GitHub.
+```
@@ -50,3 +50,15 @@ from embedchain import Pipeline as App
 naval_chat_bot = App()
 print(naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?"))
 ```
+
+## Resetting an app and vector database
+
+You can reset the app by simply calling the `reset` method. This will delete the vector database and all other app related files.
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+app.add("https://www.youtube.com/watch?v=3qHkcs3kG44")
+app.reset()
+```
@@ -0,0 +1,28 @@
+---
+title: "💬 Discord"
+---
+
+To add any Discord channel messages to your app, just add the `channel_id` as the source and set the `data_type` to `discord`.
+
+<Note>
+    This loader requires a Discord bot token with read messages access.
+    To obtain the token, follow the instructions provided in this tutorial: 
+    <a href="https://www.writebots.com/discord-bot-token/">How to Get a Discord Bot Token?</a>.
+</Note>
+
+```python
+import os
+from embedchain import Pipeline as App
+
+# add your discord "BOT" token
+os.environ["DISCORD_TOKEN"] = "xxx"
+
+app = App()
+
+app.add("1177296711023075338", data_type="discord")
+
+response = app.query("What is Joe saying about Elon Musk?")
+
+print(response)
+# Answer: Joe is saying "Elon Musk is a genius".
+```
@@ -0,0 +1,44 @@
+---
+title: '🗨️ Discourse'
+---
+
+You can now easily load data from your community built with [Discourse](https://discourse.org/).
+
+## Example
+
+1. Setup the Discourse Loader with your community url.
+```Python
+from embedchain.loaders.discourse import DiscourseLoader
+
+dicourse_loader = DiscourseLoader(config={"domain": "https://community.openai.com"})
+```
+
+2. Once you setup the loader, you can create an app and load data using the above discourse loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+
+app = App()
+
+app.add("openai after:2023-10-1", data_type="discourse", loader=dicourse_loader)
+
+question = "Where can I find the OpenAI API status page?"
+app.query(question)
+# Answer: You can find the OpenAI API status page at https:/status.openai.com/.
+```
+
+NOTE: The `add` function of the app will accept any executable search query to load data. Refer [Discourse API Docs](https://docs.discourse.org/#tag/Search) to learn more about search queries.
+
+3. We automatically create a chunker to chunk your discourse data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+
+from embedchain.chunkers.discourse import DiscourseChunker
+from embedchain.config.add_config import ChunkerConfig
+
+discourse_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+discourse_chunker = DiscourseChunker(config=discourse_chunker_config)
+
+app.add("openai", data_type='discourse', loader=dicourse_loader, chunker=discourse_chunker)
+```
@@ -1,5 +1,5 @@
 ---
-title: '📚🌐 Code documentation'
+title: '📚 Code documentation'
 ---

 To add any code documentation website as a loader, use the data_type as `docs_site`. Eg:
@@ -0,0 +1,50 @@
+---
+title: 📝 Github
+---
+
+1. Setup the Github loader by configuring the Github account with username and personal access token (PAT). Check out [this](https://docs.github.com/en/enterprise-server@3.6/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token) link to learn how to create a PAT.
+```Python
+from embedchain.loaders.github import GithubLoader
+
+loader = GithubLoader(
+    config={
+        "token":"ghp_xxxx"
+        }
+    )
+```
+
+2. Once you setup the loader, you can create an app and load data using the above Github loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+os.environ["OPENAI_API_KEY"] = "sk-xxxx"
+
+app = App()
+
+app.add("repo:embedchain/embedchain type:repo", data_type="github", loader=loader)
+
+response = app.query("What is Embedchain?")
+# Answer: Embedchain is a Data Platform for Large Language Models (LLMs). It allows users to seamlessly load, index, retrieve, and sync unstructured data in order to build dynamic, LLM-powered applications. There is also a JavaScript implementation called embedchain-js available on GitHub.
+```
+The `add` function of the app will accept any valid github query with qualifiers. It only supports loading github code, repository, issues and pull-requests.
+<Note>
+You must provide qualifiers `type:` and `repo:` in the query. The `type:` qualifier can be a combination of `code`, `repo`, `pr`, `issue`. The `repo:` qualifier must be a valid github repository name.
+</Note>
+
+<Card title="Valid queries" icon="lightbulb" iconType="duotone" color="#ca8b04">
+    - `repo:embedchain/embedchain type:repo` - to load the repository
+    - `repo:embedchain/embedchain type:issue,pr` - to load the issues and pull-requests of the repository
+    - `repo:embedchain/embedchain type:issue state:closed` - to load the closed issues of the repository
+</Card>
+
+3. We automatically create a chunker to chunk your GitHub data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+from embedchain.chunkers.common_chunker import CommonChunker
+from embedchain.config.add_config import ChunkerConfig
+
+github_chunker_config = ChunkerConfig(chunk_size=2000, chunk_overlap=0, length_function=len)
+github_chunker = CommonChunker(config=github_chunker_config)
+
+app.add(load_query, data_type="github", loader=loader, chunker=github_chunker)
+```
@@ -0,0 +1,47 @@
+---
+title: '🐬 MySQL'
+---
+
+1. Setup the MySQL loader by configuring the SQL db.
+```Python
+from embedchain.loaders.mysql import MySQLLoader
+
+config = {
+    "host": "host",
+    "port": "port",
+    "database": "database",
+    "user": "username",
+    "password": "password",
+}
+
+mysql_loader = MySQLLoader(config=config)
+```
+
+For more details on how to setup with valid config, check MySQL [documentation](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html).
+
+2. Once you setup the loader, you can create an app and load data using the above MySQL loader
+```Python
+from embedchain.pipeline import Pipeline as App
+
+app = App()
+
+app.add("SELECT * FROM table_name;", data_type='mysql', loader=mysql_loader)
+# Adds `(1, 'What is your net worth, Elon Musk?', "As of October 2023, Elon Musk's net worth is $255.2 billion.")`
+
+response = app.query(question)
+# Answer: As of October 2023, Elon Musk's net worth is $255.2 billion.
+```
+
+NOTE: The `add` function of the app will accept any executable query to load data. DO NOT pass the `CREATE`, `INSERT` queries in `add` function.
+
+3. We automatically create a chunker to chunk your SQL data, however if you wish to provide your own chunker class. Here is how you can do that:
+``Python
+
+from embedchain.chunkers.mysql import MySQLChunker
+from embedchain.config.add_config import ChunkerConfig
+
+mysql_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+mysql_chunker = MySQLChunker(config=mysql_chunker_config)
+
+app.add("SELECT * FROM table_name;", data_type='mysql', loader=mysql_loader, chunker=mysql_chunker)
+```
@@ -5,22 +5,28 @@ title: Overview
 Embedchain comes with built-in support for various data sources. We handle the complexity of loading unstructured data from these data sources, allowing you to easily customize your app through a user-friendly interface.

 <CardGroup cols={4}>
-  <Card title="📊 csv" href="/data-sources/csv"></Card>
+  <Card title="📊 CSV" href="/data-sources/csv"></Card>
  <Card title="📃 JSON" href="/data-sources/json"></Card>
-  <Card title="📚🌐 docs site" href="/data-sources/docs-site"></Card>
+  <Card title="📚 docs site" href="/data-sources/docs-site"></Card>
  <Card title="📄 docx" href="/data-sources/docx"></Card>
  <Card title="📝 mdx" href="/data-sources/mdx"></Card>
-  <Card title="📓 notion" href="/data-sources/notion"></Card>
-  <Card title="📰 pdf" href="/data-sources/pdf-file"></Card>
+  <Card title="📓 Notion" href="/data-sources/notion"></Card>
+  <Card title="📰 PDF" href="/data-sources/pdf-file"></Card>
  <Card title="❓💬 q&a pair" href="/data-sources/qna"></Card>
  <Card title="🗺️ sitemap" href="/data-sources/sitemap"></Card>
  <Card title="📝 text" href="/data-sources/text"></Card>
-  <Card title="🌐📄 web page" href="/data-sources/web-page"></Card>
+  <Card title="🌐 web page" href="/data-sources/web-page"></Card>
  <Card title="🧾 xml" href="/data-sources/xml"></Card>
  <Card title="🙌 OpenAPI" href="/data-sources/openapi"></Card>
-  <Card title="🎥📺 youtube video" href="/data-sources/youtube-video"></Card>
+  <Card title="📺 Youtube" href="/data-sources/youtube-video"></Card>
  <Card title="📬 Gmail" href="/data-sources/gmail"></Card>
  <Card title="🐘 Postgres" href="/data-sources/postgres"></Card>
+  <Card title="🐬 MySQL" href="/data-sources/mysql"></Card>
+  <Card title="🤖 Slack" href="/data-sources/slack"></Card>
+  <Card title="🗨️ Discourse" href="/data-sources/discourse"></Card>
+  <Card title="💬 Discord" href="/data-sources/discord"></Card>
+  <Card title="📝 Github" href="/data-sources/github"></Card>
+  <Card title="⚙️ Custom" href="/data-sources/custom"></Card>
 </CardGroup>

 <br/ >
@@ -0,0 +1,54 @@
+---
+title: '🤖 Slack'
+---
+
+## Pre-requisite
+- Download required packages by running `pip install --upgrade "embedchain[slack]"`.
+- Configure your slack bot token as environment variable `SLACK_USER_TOKEN`.
+    - Find your user token on your [Slack Account](https://api.slack.com/authentication/token-types)
+    - Make sure your slack user token includes [search](https://api.slack.com/scopes/search:read) scope.
+
+## Example
+1. Setup the Slack loader by configuring the Slack Webclient.
+```Python
+from embedchain.loaders.slack import SlackLoader
+
+os.environ["SLACK_USER_TOKEN"] = "xoxp-*"
+
+loader = SlackLoader()
+
+"""
+config = {
+    'base_url': slack_app_url,
+    'headers': web_headers,
+    'team_id': slack_team_id,
+}
+
+loader = SlackLoader(config)
+"""
+```
+
+NOTE: you can also pass the `config` with `base_url`, `headers`, `team_id` to setup your SlackLoader.
+
+2. Once you setup the loader, you can create an app and load data using the above slack loader
+```Python
+import os
+from embedchain.pipeline import Pipeline as App
+
+app = App()
+
+app.add("in:random", data_type="slack", loader=loader)
+question = "Which bots are available in the slack workspace's random channel?"
+# Answer: The available bot in the slack workspace's random channel is the Embedchain bot.
+```
+
+3. We automatically create a chunker to chunk your slack data, however if you wish to provide your own chunker class. Here is how you can do that:
+```Python
+from embedchain.chunkers.slack import SlackChunker
+from embedchain.config.add_config import ChunkerConfig
+
+slack_chunker_config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+slack_chunker = SlackChunker(config=slack_chunker_config)
+
+app.add(slack_chunker, data_type="slack", loader=loader, chunker=slack_chunker)
+```
@@ -0,0 +1,16 @@
+---
+title: "📝 Substack"
+---
+
+To add any Substack data sources to your app, just add the sitemap.xml of that url as the source and set the data_type to `substack`.
+
+```python
+from embedchain import Pipeline as App
+
+app = App()
+
+# source: for any substack just add the sitemap.xml url
+app.add('https://www.lennysnewsletter.com/sitemap.xml', data_type='substack')
+app.query("Who is Brian Chesky?")
+# Answer: Brian Chesky is the co-founder and CEO of Airbnb.
+```
@@ -1,5 +1,5 @@
 ---
-title: '🌐📄 Web page'
+title: '🌐 Web page'
 ---

 To add any web page, use the data_type as `web_page`. Eg:
@@ -1,5 +1,5 @@
 ---
-title: '🎥📺 Youtube video'
+title: '📺 Youtube'
 ---


@@ -15,8 +15,21 @@ channels:read
 chat:write
 ```
 5. Now select the option `Install to Workspace` and after it's done, copy the `Bot User OAuth Token` and set it in your secrets as `SLACK_BOT_TOKEN`.
-6. Run your bot now with `python3 -m embedchain.bots.slack`
-7. Expose your bot to the internet. Default port is `5000`, which can be changed by adding `port --8080` to the startup command. You can use your machine's public IP or DNS. Otherwise, employ a proxy server like [ngrok](https://ngrok.com/) to make your local bot accessible.
+6. Run your bot now,
+<Tabs>
+    <Tab title="docker">
+        ```bash
+        docker run --name slack-bot -e OPENAI_API_KEY=sk-xxx -e SLACK_BOT_TOKEN=xxx -p 8000:8000 embedchain/slack-bot
+        ```
+    </Tab>
+    <Tab title="python">
+        ```bash
+        pip install --upgrade "embedchain[slack]"
+        python3 -m embedchain.bots.slack --port 8000
+        ```
+</Tab>
+</Tabs>
+7. Expose your bot to the internet. You can use your machine's public IP or DNS. Otherwise, employ a proxy server like [ngrok](https://ngrok.com/) to make your local bot accessible.
 8. On the Slack API website go to `Event Subscriptions` on the left Sidebar and turn on `Enable Events`.
 9. In `Request URL`, enter your server or ngrok address.
 10. After it gets verified, click on `Subscribe to bot events`, add `message.channels` Bot User Event and click on `Save Changes`.
@@ -1,8 +1,15 @@
 ---
 title: 🔎 Examples
-description: 'Collection of Google colab notebook and Replit links for users'
 ---

+# Explore awesome apps
+
+Check out the remarkable work accomplished using [Embedchain](https://app.embedchain.ai/custom-gpts/).
+
+## Collection of Google colab notebook and Replit links for users
+
+Get started with Embedchain by trying out the examples below. You can run the examples in your browser using Google Colab or Replit.
+
 <table>
  <thead>
    <tr>
@@ -2,13 +2,36 @@
 title: ❓ FAQs
 description: 'Collections of all the frequently asked questions'
 ---
-
-#### Does Embedchain support OpenAI's Assistant APIs?
-
+<AccordionGroup>
+<Accordion title="Does Embedchain support OpenAI's Assistant APIs?">
 Yes, it does. Please refer to the [OpenAI Assistant docs page](/get-started/openai-assistant).
+</Accordion>
+<Accordion title="How to use MistralAI language model?">
+Use the model provided on huggingface: `mistralai/Mistral-7B-v0.1`
+<CodeGroup>
+```python main.py
+import os
+from embedchain import Pipeline as App

-#### How to use `gpt-4-turbo` model released on OpenAI DevDay?
+os.environ["OPENAI_API_KEY"] = "sk-xxx"
+os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "hf_your_token"

+app = App.from_config("huggingface.yaml")
+```
+```yaml huggingface.yaml
+llm:
+  provider: huggingface
+  config:
+    model: 'mistralai/Mistral-7B-v0.1'
+    temperature: 0.5
+    max_tokens: 1000
+    top_p: 0.5
+    stream: false
+```
+</CodeGroup>
+</Accordion>
+<Accordion title="How to use ChatGPT 4 turbo model released on OpenAI DevDay?">
+Use the model `gpt-4-turbo` provided my openai.
 <CodeGroup>

 ```python main.py
@@ -18,7 +41,7 @@ from embedchain import Pipeline as App
 os.environ['OPENAI_API_KEY'] = 'xxx'

 # load llm configuration from gpt4_turbo.yaml file
-app = App.from_config(yaml_path="gpt4_turbo.yaml")
+app = App.from_config(config_path="gpt4_turbo.yaml")
 ```

 ```yaml gpt4_turbo.yaml
@@ -31,12 +54,9 @@ llm:
    top_p: 1
    stream: false
 ```
-
 </CodeGroup>
-
-
-#### How to use GPT-4 as the LLM model?
-
+</Accordion>
+<Accordion title="How to use GPT-4 as the LLM model?">
 <CodeGroup>

 ```python main.py
@@ -46,7 +66,7 @@ from embedchain import Pipeline as App
 os.environ['OPENAI_API_KEY'] = 'xxx'

 # load llm configuration from gpt4.yaml file
-app = App.from_config(yaml_path="gpt4.yaml")
+app = App.from_config(config_path="gpt4.yaml")
 ```

 ```yaml gpt4.yaml
@@ -61,9 +81,8 @@ llm:
 ```

 </CodeGroup>
-
-#### I don't have OpenAI credits. How can I use some open source model?
-
+</Accordion>
+<Accordion title="I don't have OpenAI credits. How can I use some open source model?">
 <CodeGroup>

 ```python main.py
@@ -73,7 +92,7 @@ from embedchain import Pipeline as App
 os.environ['OPENAI_API_KEY'] = 'xxx'

 # load llm configuration from opensource.yaml file
-app = App.from_config(yaml_path="opensource.yaml")
+app = App.from_config(config_path="opensource.yaml")
 ```

 ```yaml opensource.yaml
@@ -93,8 +112,10 @@ embedder:
 ```
 </CodeGroup>

-#### How to contact support?
+</Accordion>
+</AccordionGroup>

+#### Need more help?
 If docs aren't sufficient, please feel free to reach out to us using one of the following methods:

 <Snippet file="get-help.mdx" />
@@ -105,7 +105,7 @@ app.deploy()
 # ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
 ```

-## 🚀 How it works?
+## 🛠️ How it works?

 Embedchain abstracts out the following steps from you to easily create LLM powered apps:

@@ -129,3 +129,5 @@ The process of loading the dataset and querying involves multiple steps, each wi
 - How should I find similar documents for a query? Which ranking model should I use?

 Embedchain takes care of all these nuances and provides a simple interface to create apps on any data.
+
+## [🚀 Get started](https://docs.embedchain.ai/get-started/quickstart)
@@ -12,74 +12,85 @@ pip install embedchain
 ```

 <Tip>
-Embedchain now supports OpenAI's latest `gpt-4-turbo` model. Checkout the [docs here](/get-started/faq#how-to-use-gpt-4-turbo-model-released-on-openai-devday) on how to use it.
+Embedchain now supports OpenAI's latest `gpt-4-turbo` model. Checkout the [FAQs](/get-started/faq#how-to-use-gpt-4-turbo-model-released-on-openai-devday).
 </Tip>

 Creating an app involves 3 steps:

 <Steps>
  <Step title="⚙️ Import app instance">
-```python
-from embedchain import Pipeline as App
-app = App()
-```
+    ```python
+    from embedchain import Pipeline as App
+    app = App()
+    ```
+    <Accordion title="Customize your app by a simple YAML config" icon="gear-complex">
+      Embedchain provides a wide range of options to customize your app. You can customize the model, data sources, and much more.
+      Explore the custom configurations [here](https://docs.embedchain.ai/advanced/configuration).
+      <CodeGroup>
+      ```python yaml_app.py
+      from embedchain import Pipeline as App
+      app = App.from_config(config_path="config.yaml")
+      ```
+      ```python json_app.py
+      from embedchain import Pipeline as App
+      app = App.from_config(config_path="config.json")
+      ```
+      ```python app.py
+      from embedchain import Pipeline as App
+      config = {} # Add your config here
+      app = App.from_config(config=config)
+      ```
+      </CodeGroup>
+    </Accordion>
  </Step>
  <Step title="🗃️ Add data sources">
-```python
-# Add different data sources
-app.add("https://en.wikipedia.org/wiki/Elon_Musk")
-app.add("https://www.forbes.com/profile/elon-musk")
-# You can also add local data sources such as pdf, csv files etc.
-# app.add("/path/to/file.pdf")
-```
+    ```python
+    app.add("https://en.wikipedia.org/wiki/Elon_Musk")
+    app.add("https://www.forbes.com/profile/elon-musk")
+    # app.add("path/to/file/elon_musk.pdf")
+    ```
+    <Accordion title="Embedchain supports adding data from many data sources." icon="files">
+      Embedchain supports adding data from many data sources including web pages, PDFs, databases, and more.
+      Explore the list of supported [data sources](https://docs.embedchain.ai/data-sources/overview).
+    </Accordion>
  </Step>
-  <Step title="💬 Query or chat or search context on your data">
-```python
-app.query("What is the net worth of Elon Musk today?")
-# Answer: The net worth of Elon Musk today is $258.7 billion.
-```
+  <Step title="💬 Ask questions, chat, or search through your data with ease">
+    ```python
+    app.query("What is the net worth of Elon Musk today?")
+    # Answer: The net worth of Elon Musk today is $258.7 billion.
+    ```
+    <Accordion title="Want to chat with your app?" icon="face-thinking">
+      Embedchain provides a wide range of features to interact with your app. You can chat with your app, ask questions, search through your data, and much more.
+      ```python
+      app.chat("How many companies does Elon Musk run? Name those")
+      # Answer: Elon Musk runs 3 companies: Tesla, SpaceX, and Neuralink.
+      app.chat("What is his net worth today?")
+      # Answer: The net worth of Elon Musk today is $258.7 billion.
+      ```
+      To learn about other features, click [here](https://docs.embedchain.ai/get-started/introduction)
+    </Accordion>
  </Step>
-  <Step title="🚀 (Optional) Deploy your pipeline to Embedchain Platform">
-```python
-app.deploy()
-# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
-# ec-xxxxxx
+  <Step title="🚀 Seamlessly launch your App on the Embedchain Platform!">
+    ```python
+    app.deploy()
+    # 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+    # ec-xxxxxx

-# 🛠️ Creating pipeline on the platform...
-# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+    # 🛠️ Creating pipeline on the platform...
+    # 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx

-# 🛠️ Adding data to your pipeline...
-# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
-```
+    # 🛠️ Adding data to your pipeline...
+    # ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
+    ```
+    <Accordion title="Share your app with others" icon="laptop-mobile">
+      You can now share your app with others from our platform.
+      Access your app on our [platform](https://app.embedchain.ai/).
+    </Accordion>
  </Step>
 </Steps>

-Putting it together, you can run your first app using the following code. Make sure to set the `OPENAI_API_KEY` 🔑 environment variable in the code.
+Putting it together, you can run your first app using the following Google Colab. Make sure to set the `OPENAI_API_KEY` 🔑 environment variable in the code.

-```python
-import os
-from embedchain import Pipeline as App
-
-os.environ["OPENAI_API_KEY"] = "xxx"
-app = App()
-
-# Add different data sources
-app.add("https://en.wikipedia.org/wiki/Elon_Musk")
-app.add("https://www.forbes.com/profile/elon-musk")
-# You can also add local data sources such as pdf, csv files etc.
-# app.add("/path/to/file.pdf")
-
-response = app.query("What is the net worth of Elon Musk today?")
-print(response)
-# Answer: The net worth of Elon Musk today is $258.7 billion.
-
-app.deploy()
-# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
-# ec-xxxxxx
-
-# 🛠️ Creating pipeline on the platform...
-# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
-
-# 🛠️ Adding data to your pipeline...
-# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
-```
+<a href="https://colab.research.google.com/drive/17ON1LPonnXAtLaZEebnOktstB_1cJJmh?usp=sharing">
+    <img src="https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667" alt="Open in Colab" />
+</a>
@@ -87,7 +87,10 @@
            "data-sources/text",
            "data-sources/web-page",
            "data-sources/openapi",
-            "data-sources/youtube-video"
+            "data-sources/youtube-video",
+            "data-sources/discourse",
+            "data-sources/substack",
+            "data-sources/discord"
          ]
        },
        "data-sources/data-type-handling"
@@ -6,3 +6,6 @@ from embedchain.apps.app import App  # noqa: F401
 from embedchain.client import Client  # noqa: F401
 from embedchain.pipeline import Pipeline  # noqa: F401
 from embedchain.vectordb.chroma import ChromaDB  # noqa: F401
+
+# Setup the user directory if doesn't exist already
+Client.setup_dir()
@@ -2,7 +2,6 @@ from typing import Optional

 import yaml

-from embedchain.client import Client
 from embedchain.config import (AppConfig, BaseEmbedderConfig, BaseLlmConfig,
                               ChunkerConfig)
 from embedchain.config.vectordb.base import BaseVectorDbConfig
@@ -10,10 +9,10 @@ from embedchain.embedchain import EmbedChain
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.embedder.openai import OpenAIEmbedder
 from embedchain.factory import EmbedderFactory, LlmFactory, VectorDBFactory
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm
 from embedchain.llm.openai import OpenAILlm
-from embedchain.utils import validate_yaml_config
+from embedchain.utils import validate_config
 from embedchain.vectordb.base import BaseVectorDB
 from embedchain.vectordb.chroma import ChromaDB

@@ -68,9 +67,6 @@ class App(EmbedChain):
        :type system_prompt: Optional[str], optional
        :raises TypeError: LLM, database or embedder or their config is not a valid class instance.
        """
-        # Setup user directory if it doesn't exist already
-        Client.setup_dir()
-
        # Type check configs
        if config and not isinstance(config, AppConfig):
            raise TypeError(
@@ -138,7 +134,7 @@ class App(EmbedChain):
            config_data = yaml.safe_load(file)

        try:
-            validate_yaml_config(config_data)
+            validate_config(config_data)
        except Exception as e:
            raise Exception(f"❌ Error occurred while validating the YAML config. Error: {str(e)}")

@@ -3,8 +3,8 @@ from typing import Any
 from embedchain import Pipeline as App
 from embedchain.config import AddConfig, BaseLlmConfig, PipelineConfig
 from embedchain.embedder.openai import OpenAIEmbedder
-from embedchain.helper.json_serializable import (JSONSerializable,
-                                                 register_deserializable)
+from embedchain.helpers.json_serializable import (JSONSerializable,
+                                                  register_deserializable)
 from embedchain.llm.openai import OpenAILlm
 from embedchain.vectordb.chroma import ChromaDB

@@ -2,7 +2,7 @@ import argparse
 import logging
 import os

-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable

 from .base import BaseBot

@@ -3,7 +3,7 @@ import logging
 import os
 from typing import List, Optional

-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable

 from .base import BaseBot

@@ -5,7 +5,7 @@ import signal
 import sys

 from embedchain import App
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable

 from .base import BaseBot

@@ -4,7 +4,7 @@ import logging
 import signal
 import sys

-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable

 from .base import BaseBot

@@ -1,6 +1,6 @@
 import hashlib

-from embedchain.helper.json_serializable import JSONSerializable
+from embedchain.helpers.json_serializable import JSONSerializable
 from embedchain.models.data_type import DataType


@@ -41,7 +41,6 @@ class BaseChunker(JSONSerializable):
            url = meta_data["url"]

            chunks = self.get_chunks(content)
-
            for chunk in chunks:
                chunk_id = hashlib.sha256((chunk + url).encode()).hexdigest()
                chunk_id = f"{app_id}--{chunk_id}" if app_id is not None else chunk_id
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helpers.json_serializable import register_deserializable
+
+
+@register_deserializable
+class CommonChunker(BaseChunker):
+    """Common chunker for all loaders."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=2000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helpers.json_serializable import register_deserializable
+
+
+@register_deserializable
+class DiscourseChunker(BaseChunker):
+    """Chunker for discourse."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helpers.json_serializable import register_deserializable
+
+
+@register_deserializable
+class MySQLChunker(BaseChunker):
+    """Chunker for json."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helpers.json_serializable import register_deserializable
+
+
+@register_deserializable
+class SlackChunker(BaseChunker):
+    """Chunker for postgres."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helpers.json_serializable import register_deserializable
+
+
+@register_deserializable
+class SubstackChunker(BaseChunker):
+    """Chunker for Substack."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -4,7 +4,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config.add_config import ChunkerConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -3,7 +3,7 @@ from importlib import import_module
 from typing import Callable, Optional

 from embedchain.config.base_config import BaseConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -1,6 +1,6 @@
 from typing import Optional

-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable

 from .base_app_config import BaseAppConfig

@@ -2,7 +2,7 @@ import logging
 from typing import Optional

 from embedchain.config.base_config import BaseConfig
-from embedchain.helper.json_serializable import JSONSerializable
+from embedchain.helpers.json_serializable import JSONSerializable
 from embedchain.vectordb.base import BaseVectorDB


@@ -1,6 +1,6 @@
 from typing import Any, Dict

-from embedchain.helper.json_serializable import JSONSerializable
+from embedchain.helpers.json_serializable import JSONSerializable


 class BaseConfig(JSONSerializable):
@@ -1,6 +1,6 @@
 from typing import Optional

-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -1,9 +1,9 @@
 import re
 from string import Template
-from typing import Any, Dict, Optional
+from typing import Any, Dict, List, Optional

 from embedchain.config.base_config import BaseConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable

 DEFAULT_PROMPT = """
  Use the following pieces of context to answer the query at the end.
@@ -68,6 +68,7 @@ class BaseLlmConfig(BaseConfig):
        system_prompt: Optional[str] = None,
        where: Dict[str, Any] = None,
        query_type: Optional[str] = None,
+        callbacks: Optional[List] = None,
    ):
        """
        Initializes a configuration class instance for the LLM.
@@ -98,6 +99,8 @@ class BaseLlmConfig(BaseConfig):
        :type system_prompt: Optional[str], optional
        :param where: A dictionary of key-value pairs to filter the database results., defaults to None
        :type where: Dict[str, Any], optional
+        :param callbacks: Langchain callback functions to use, defaults to None
+        :type callbacks: Optional[List], optional
        :raises ValueError: If the template is not valid as template should
        contain $context and $query (and optionally $history)
        :raises ValueError: Stream is not boolean
@@ -113,6 +116,7 @@ class BaseLlmConfig(BaseConfig):
        self.deployment_name = deployment_name
        self.system_prompt = system_prompt
        self.query_type = query_type
+        self.callbacks = callbacks

        if type(template) is str:
            template = Template(template)
@@ -1,6 +1,6 @@
 from typing import Optional

-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable

 from .apps.base_app_config import BaseAppConfig

@@ -1,7 +1,7 @@
 from typing import Optional

 from embedchain.config.vectordb.base import BaseVectorDbConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -2,7 +2,7 @@ import os
 from typing import Dict, List, Optional, Union

 from embedchain.config.vectordb.base import BaseVectorDbConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -1,7 +1,7 @@
 from typing import Dict, Optional, Tuple

 from embedchain.config.vectordb.base import BaseVectorDbConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -1,7 +1,7 @@
 from typing import Dict, Optional

 from embedchain.config.vectordb.base import BaseVectorDbConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -1,7 +1,7 @@
 from typing import Dict, Optional

 from embedchain.config.vectordb.base import BaseVectorDbConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -1,7 +1,7 @@
 from typing import Dict, Optional

 from embedchain.config.vectordb.base import BaseVectorDbConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -2,7 +2,7 @@ import os
 from typing import Optional

 from embedchain.config.vectordb.base import BaseVectorDbConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable


@register_deserializable
@@ -1,10 +1,10 @@
 from importlib import import_module
-from typing import Any, Dict
+from typing import Optional

 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.config import AddConfig
 from embedchain.config.add_config import ChunkerConfig, LoaderConfig
-from embedchain.helper.json_serializable import JSONSerializable
+from embedchain.helpers.json_serializable import JSONSerializable
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.models.data_type import DataType

@@ -16,7 +16,13 @@ class DataFormatter(JSONSerializable):
    .add or .add_local method call
    """

-    def __init__(self, data_type: DataType, config: AddConfig, kwargs: Dict[str, Any]):
+    def __init__(
+        self,
+        data_type: DataType,
+        config: AddConfig,
+        loader: Optional[BaseLoader] = None,
+        chunker: Optional[BaseChunker] = None,
+    ):
        """
        Initialize a dataformatter, set data type and chunker based on datatype.

@@ -25,15 +31,15 @@ class DataFormatter(JSONSerializable):
        :param config: AddConfig instance with nested loader and chunker config attributes.
        :type config: AddConfig
        """
-        self.loader = self._get_loader(data_type=data_type, config=config.loader, kwargs=kwargs)
-        self.chunker = self._get_chunker(data_type=data_type, config=config.chunker, kwargs=kwargs)
+        self.loader = self._get_loader(data_type=data_type, config=config.loader, loader=loader)
+        self.chunker = self._get_chunker(data_type=data_type, config=config.chunker, chunker=chunker)

    def _lazy_load(self, module_path: str):
        module_path, class_name = module_path.rsplit(".", 1)
        module = import_module(module_path)
        return getattr(module, class_name)

-    def _get_loader(self, data_type: DataType, config: LoaderConfig, kwargs: Dict[str, Any]) -> BaseLoader:
+    def _get_loader(self, data_type: DataType, config: LoaderConfig, loader: Optional[BaseLoader]) -> BaseLoader:
        """
        Returns the appropriate data loader for the given data type.

@@ -63,21 +69,18 @@ class DataFormatter(JSONSerializable):
            DataType.OPENAPI: "embedchain.loaders.openapi.OpenAPILoader",
            DataType.GMAIL: "embedchain.loaders.gmail.GmailLoader",
            DataType.NOTION: "embedchain.loaders.notion.NotionLoader",
+            DataType.SUBSTACK: "embedchain.loaders.substack.SubstackLoader",
+            DataType.YOUTUBE_CHANNEL: "embedchain.loaders.youtube_channel.YoutubeChannelLoader",
+            DataType.DISCORD: "embedchain.loaders.discord.DiscordLoader",
        }

-        custom_loaders = set(
-            [
-                DataType.POSTGRES,
-            ]
-        )
-
-        if data_type in loaders:
+        if data_type == DataType.CUSTOM or loader is not None:
+            loader_class: type = loader
+            if loader_class:
+                return loader_class
+        elif data_type in loaders:
            loader_class: type = self._lazy_load(loaders[data_type])
            return loader_class()
-        elif data_type in custom_loaders:
-            loader_class: type = kwargs.get("loader", None)
-            if loader_class is not None:
-                return loader_class

        raise ValueError(
            f"Cant find the loader for {data_type}.\
@@ -85,7 +88,7 @@ class DataFormatter(JSONSerializable):
                        check `https://docs.embedchain.ai/data-sources/overview`."
        )

-    def _get_chunker(self, data_type: DataType, config: ChunkerConfig, kwargs: Dict[str, Any]) -> BaseChunker:
+    def _get_chunker(self, data_type: DataType, config: ChunkerConfig, chunker: Optional[BaseChunker]) -> BaseChunker:
        """Returns the appropriate chunker for the given data type (updated for lazy loading)."""
        chunker_classes = {
            DataType.YOUTUBE_VIDEO: "embedchain.chunkers.youtube_video.YoutubeVideoChunker",
@@ -105,21 +108,22 @@ class DataFormatter(JSONSerializable):
            DataType.OPENAPI: "embedchain.chunkers.openapi.OpenAPIChunker",
            DataType.GMAIL: "embedchain.chunkers.gmail.GmailChunker",
            DataType.NOTION: "embedchain.chunkers.notion.NotionChunker",
-            DataType.POSTGRES: "embedchain.chunkers.postgres.PostgresChunker",
+            DataType.SUBSTACK: "embedchain.chunkers.substack.SubstackChunker",
+            DataType.YOUTUBE_CHANNEL: "embedchain.chunkers.common_chunker.CommonChunker",
+            DataType.DISCORD: "embedchain.chunkers.common_chunker.CommonChunker",
+            DataType.CUSTOM: "embedchain.chunkers.common_chunker.CommonChunker",
        }

-        if data_type in chunker_classes:
-            if "chunker" in kwargs:
-                chunker_class = kwargs.get("chunker")
-            else:
-                chunker_class = self._lazy_load(chunker_classes[data_type])
-
+        if chunker is not None:
+            return chunker
+        elif data_type in chunker_classes:
+            chunker_class = self._lazy_load(chunker_classes[data_type])
            chunker = chunker_class(config)
            chunker.set_data_type(data_type)
            return chunker
-        else:
-            raise ValueError(
-                f"Cant find the chunker for {data_type}.\
-                    We recommend to pass the chunker to use data_type: {data_type},\
-                        check `https://docs.embedchain.ai/data-sources/overview`."
-            )
+
+        raise ValueError(
+            f"Cant find the chunker for {data_type}.\
+                We recommend to pass the chunker to use data_type: {data_type},\
+                    check `https://docs.embedchain.ai/data-sources/overview`."
+        )
@@ -13,7 +13,7 @@ from embedchain.config.apps.base_app_config import BaseAppConfig
 from embedchain.constants import SQLITE_PATH
 from embedchain.data_formatter import DataFormatter
 from embedchain.embedder.base import BaseEmbedder
-from embedchain.helper.json_serializable import JSONSerializable
+from embedchain.helpers.json_serializable import JSONSerializable
 from embedchain.llm.base import BaseLlm
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.models.data_type import (DataType, DirectDataType,
@@ -133,7 +133,9 @@ class EmbedChain(JSONSerializable):
        metadata: Optional[Dict[str, Any]] = None,
        config: Optional[AddConfig] = None,
        dry_run=False,
-        **kwargs: Dict[str, Any],
+        loader: Optional[BaseLoader] = None,
+        chunker: Optional[BaseChunker] = None,
+        **kwargs: Optional[Dict[str, Any]],
    ):
        """
        Adds the data from the given URL to the vector db.
@@ -178,10 +180,10 @@ class EmbedChain(JSONSerializable):
            try:
                data_type = DataType(data_type)
            except ValueError:
-                raise ValueError(
-                    f"Invalid data_type: '{data_type}'.",
-                    f"Please use one of the following: {[data_type.value for data_type in DataType]}",
-                ) from None
+                logging.info(
+                    f"Invalid data_type: '{data_type}', using `custom` instead.\n Check docs to pass the valid data type: `https://docs.embedchain.ai/data-sources/overview`"  # noqa: E501
+                )
+                data_type = DataType.CUSTOM

        if not data_type:
            data_type = detect_datatype(source)
@@ -190,21 +192,11 @@ class EmbedChain(JSONSerializable):
        hash_object = hashlib.md5(str(source).encode("utf-8"))
        source_hash = hash_object.hexdigest()

-        # Check if the data hash already exists, if so, skip the addition
-        self.cursor.execute(
-            "SELECT 1 FROM data_sources WHERE hash = ? AND pipeline_id = ?", (source_hash, self.config.id)
-        )
-        existing_data = self.cursor.fetchone()
-
-        if existing_data:
-            print(f"Data with hash {source_hash} already exists. Skipping addition.")
-            return source_hash
-
        self.user_asks.append([source, data_type.value, metadata])

-        data_formatter = DataFormatter(data_type, config, kwargs)
-        documents, metadatas, _ids, new_chunks = self.load_and_embed(
-            data_formatter.loader, data_formatter.chunker, source, metadata, source_hash, dry_run
+        data_formatter = DataFormatter(data_type, config, loader, chunker)
+        documents, metadatas, _ids, new_chunks = self._load_and_embed(
+            data_formatter.loader, data_formatter.chunker, source, metadata, source_hash, dry_run, **kwargs
        )
        if data_type in {DataType.DOCS_SITE}:
            self.is_docs_site_instance = True
@@ -212,7 +204,7 @@ class EmbedChain(JSONSerializable):
        # Insert the data into the 'data' table
        self.cursor.execute(
            """
-            INSERT INTO data_sources (hash, pipeline_id, type, value, metadata)
+            INSERT OR REPLACE INTO data_sources (hash, pipeline_id, type, value, metadata)
            VALUES (?, ?, ?, ?, ?)
        """,
            (source_hash, self.config.id, data_type.value, str(source), json.dumps(metadata)),
@@ -248,7 +240,7 @@ class EmbedChain(JSONSerializable):
        data_type: Optional[DataType] = None,
        metadata: Optional[Dict[str, Any]] = None,
        config: Optional[AddConfig] = None,
-        **kwargs: Dict[str, Any],
+        **kwargs: Optional[Dict[str, Any]],
    ):
        """
        Adds the data from the given URL to the vector db.
@@ -279,7 +271,7 @@ class EmbedChain(JSONSerializable):
            data_type=data_type,
            metadata=metadata,
            config=config,
-            kwargs=kwargs,
+            **kwargs,
        )

    def _get_existing_doc_id(self, chunker: BaseChunker, src: Any):
@@ -340,7 +332,7 @@ class EmbedChain(JSONSerializable):
                "When it should be  DirectDataType, IndirectDataType or SpecialDataType."
            )

-    def load_and_embed(
+    def _load_and_embed(
        self,
        loader: BaseLoader,
        chunker: BaseChunker,
@@ -348,6 +340,7 @@ class EmbedChain(JSONSerializable):
        metadata: Optional[Dict[str, Any]] = None,
        source_hash: Optional[str] = None,
        dry_run=False,
+        **kwargs: Optional[Dict[str, Any]],
    ):
        """
        Loads the data from the given URL, chunks it, and adds it to database.
@@ -441,6 +434,7 @@ class EmbedChain(JSONSerializable):
            metadatas=metadatas,
            ids=ids,
            skip_embedding=(chunker.data_type == DataType.IMAGES),
+            **kwargs,
        )
        count_new_chunks = self.db.count() - chunks_before_addition

@@ -457,8 +451,13 @@ class EmbedChain(JSONSerializable):
            )
        ]

-    def retrieve_from_database(
-        self, input_query: str, config: Optional[BaseLlmConfig] = None, where=None, citations: bool = False
+    def _retrieve_from_database(
+        self,
+        input_query: str,
+        config: Optional[BaseLlmConfig] = None,
+        where=None,
+        citations: bool = False,
+        **kwargs: Optional[Dict[str, Any]],
    ) -> Union[List[Tuple[str, str, str]], List[str]]:
        """
        Queries the vector database based on the given input query.
@@ -502,6 +501,7 @@ class EmbedChain(JSONSerializable):
            where=where,
            skip_embedding=(hasattr(config, "query_type") and config.query_type == "Images"),
            citations=citations,
+            **kwargs,
        )

        return contexts
@@ -537,7 +537,10 @@ class EmbedChain(JSONSerializable):
        :rtype: str, if citations is False, otherwise Tuple[str,List[Tuple[str,str,str]]]
        """
        citations = kwargs.get("citations", False)
-        contexts = self.retrieve_from_database(input_query=input_query, config=config, where=where, citations=citations)
+        db_kwargs = {key: value for key, value in kwargs.items() if key != "citations"}
+        contexts = self._retrieve_from_database(
+            input_query=input_query, config=config, where=where, citations=citations, **db_kwargs
+        )
        if citations and len(contexts) > 0 and isinstance(contexts[0], tuple):
            contexts_data_for_llm_query = list(map(lambda x: x[0], contexts))
        else:
@@ -587,8 +590,14 @@ class EmbedChain(JSONSerializable):
        or the dry run result
        :rtype: str, if citations is False, otherwise Tuple[str,List[Tuple[str,str,str]]]
        """
-        citations = kwargs.get("citations", False)
-        contexts = self.retrieve_from_database(input_query=input_query, config=config, where=where, citations=citations)
+        if "citations" in kwargs:
+            citations = kwargs.pop("citations")
+        else:
+            citations = False
+
+        contexts = self._retrieve_from_database(
+            input_query=input_query, config=config, where=where, citations=citations, **kwargs
+        )
        if citations and len(contexts) > 0 and isinstance(contexts[0], tuple):
            contexts_data_for_llm_query = list(map(lambda x: x[0], contexts))
        else:
@@ -3,12 +3,20 @@ from typing import Any, Callable, Optional
 from embedchain.config.embedder.base import BaseEmbedderConfig

 try:
-    from chromadb.api.types import Documents, Embeddings
+    from chromadb.api.types import Embeddable, EmbeddingFunction, Embeddings
 except RuntimeError:
    from embedchain.utils import use_pysqlite3

    use_pysqlite3()
-    from chromadb.api.types import Documents, Embeddings
+    from chromadb.api.types import Embeddable, EmbeddingFunction, Embeddings
+
+
+class EmbeddingFunc(EmbeddingFunction):
+    def __init__(self, embedding_fn: Callable[[list[str]], list[str]]):
+        self.embedding_fn = embedding_fn
+
+    def __call__(self, input: Embeddable) -> Embeddings:
+        return self.embedding_fn(input)


 class BaseEmbedder:
@@ -66,7 +74,4 @@ class BaseEmbedder:
        :rtype: Callable
        """

-        def embed_function(texts: Documents) -> Embeddings:
-            return embeddings.embed_documents(texts)
-
-        return embed_function
+        return EmbeddingFunc(embeddings.embed_documents)
@@ -1,104 +0,0 @@
-"""
-Note that this file is copied from Chroma repository. We will remove this file once the fix in
-ChromaDB's repository.
-"""
-
-from typing import Optional
-
-from chromadb.api.types import Documents, Embeddings
-
-
-class OpenAIEmbeddingFunction:
-    def __init__(
-        self,
-        api_key: Optional[str] = None,
-        model_name: str = "text-embedding-ada-002",
-        organization_id: Optional[str] = None,
-        api_base: Optional[str] = None,
-        api_type: Optional[str] = None,
-        api_version: Optional[str] = None,
-        deployment_id: Optional[str] = None,
-    ):
-        """
-        Initialize the OpenAIEmbeddingFunction.
-        Args:
-            api_key (str, optional): Your API key for the OpenAI API. If not
-                provided, it will raise an error to provide an OpenAI API key.
-            organization_id(str, optional): The OpenAI organization ID if applicable
-            model_name (str, optional): The name of the model to use for text
-                embeddings. Defaults to "text-embedding-ada-002".
-            api_base (str, optional): The base path for the API. If not provided,
-                it will use the base path for the OpenAI API. This can be used to
-                point to a different deployment, such as an Azure deployment.
-            api_type (str, optional): The type of the API deployment. This can be
-                used to specify a different deployment, such as 'azure'. If not
-                provided, it will use the default OpenAI deployment.
-            api_version (str, optional): The api version for the API. If not provided,
-                it will use the api version for the OpenAI API. This can be used to
-                point to a different deployment, such as an Azure deployment.
-            deployment_id (str, optional): Deployment ID for Azure OpenAI.
-
-        """
-        try:
-            import openai
-        except ImportError:
-            raise ValueError("The openai python package is not installed. Please install it with `pip install openai`")
-
-        if api_key is not None:
-            openai.api_key = api_key
-        # If the api key is still not set, raise an error
-        elif openai.api_key is None:
-            raise ValueError(
-                "Please provide an OpenAI API key. You can get one at https://platform.openai.com/account/api-keys"
-            )
-
-        if api_base is not None:
-            openai.api_base = api_base
-
-        if api_version is not None:
-            openai.api_version = api_version
-
-        self._api_type = api_type
-        if api_type is not None:
-            openai.api_type = api_type
-
-        if organization_id is not None:
-            openai.organization = organization_id
-
-        self._v1 = openai.__version__.startswith("1.")
-        if self._v1:
-            if api_type == "azure":
-                self._client = openai.AzureOpenAI(
-                    api_key=api_key, api_version=api_version, azure_endpoint=api_base
-                ).embeddings
-            else:
-                self._client = openai.OpenAI(api_key=api_key, base_url=api_base).embeddings
-        else:
-            self._client = openai.Embedding
-        self._model_name = model_name
-        self._deployment_id = deployment_id
-
-    def __call__(self, input: Documents) -> Embeddings:
-        # replace newlines, which can negatively affect performance.
-        input = [t.replace("\n", " ") for t in input]
-
-        # Call the OpenAI Embedding API
-        if self._v1:
-            embeddings = self._client.create(input=input, model=self._deployment_id or self._model_name).data
-
-            # Sort resulting embeddings by index
-            sorted_embeddings = sorted(embeddings, key=lambda e: e.index)  # type: ignore
-
-            # Return just the embeddings
-            return [result.embedding for result in sorted_embeddings]
-        else:
-            if self._api_type == "azure":
-                embeddings = self._client.create(input=input, engine=self._deployment_id or self._model_name)["data"]
-            else:
-                embeddings = self._client.create(input=input, model=self._model_name)["data"]
-
-            # Sort resulting embeddings by index
-            sorted_embeddings = sorted(embeddings, key=lambda e: e["index"])  # type: ignore
-
-            # Return just the embeddings
-            return [result["embedding"] for result in sorted_embeddings]
@@ -1,23 +1,23 @@
 import os
 from typing import Optional

-from langchain.embeddings import OpenAIEmbeddings
+from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
+from langchain.embeddings import AzureOpenAIEmbeddings

 from embedchain.config import BaseEmbedderConfig
 from embedchain.embedder.base import BaseEmbedder
 from embedchain.models import VectorDimensions

-from .chroma_embeddings import OpenAIEmbeddingFunction
-

 class OpenAIEmbedder(BaseEmbedder):
    def __init__(self, config: Optional[BaseEmbedderConfig] = None):
        super().__init__(config=config)
+
        if self.config.model is None:
            self.config.model = "text-embedding-ada-002"

        if self.config.deployment_name:
-            embeddings = OpenAIEmbeddings(deployment=self.config.deployment_name)
+            embeddings = AzureOpenAIEmbeddings(deployment=self.config.deployment_name)
            embedding_fn = BaseEmbedder._langchain_default_concept(embeddings)
        else:
            if os.getenv("OPENAI_API_KEY") is None and os.getenv("OPENAI_ORGANIZATION") is None:
@@ -0,0 +1,73 @@
+import queue
+from typing import Any, Dict, List, Union
+
+from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
+from langchain.schema import LLMResult
+
+STOP_ITEM = "[END]"
+"""
+This is a special item that is used to signal the end of the stream.
+"""
+
+
+class StreamingStdOutCallbackHandlerYield(StreamingStdOutCallbackHandler):
+    """
+    This is a callback handler that yields the tokens as they are generated.
+    For a usage example, see the :func:`generate` function below.
+    """
+
+    q: queue.Queue
+    """
+    The queue to write the tokens to as they are generated.
+    """
+
+    def __init__(self, q: queue.Queue) -> None:
+        """
+        Initialize the callback handler.
+        q: The queue to write the tokens to as they are generated.
+        """
+        super().__init__()
+        self.q = q
+
+    def on_llm_start(self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) -> None:
+        """Run when LLM starts running."""
+        with self.q.mutex:
+            self.q.queue.clear()
+
+    def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
+        """Run on new LLM token. Only available when streaming is enabled."""
+        self.q.put(token)
+
+    def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
+        """Run when LLM ends running."""
+        self.q.put(STOP_ITEM)
+
+    def on_llm_error(self, error: Union[Exception, KeyboardInterrupt], **kwargs: Any) -> None:
+        """Run when LLM errors."""
+        self.q.put("%s: %s" % (type(error).__name__, str(error)))
+        self.q.put(STOP_ITEM)
+
+
+def generate(rq: queue.Queue):
+    """
+    This is a generator that yields the items in the queue until it reaches the stop item.
+
+    Usage example:
+    ```
+    def askQuestion(callback_fn: StreamingStdOutCallbackHandlerYield):
+        llm = OpenAI(streaming=True, callbacks=[callback_fn])
+        return llm(prompt="Write a poem about a tree.")
+
+    @app.route("/", methods=["GET"])
+    def generate_output():
+        q = Queue()
+        callback_fn = StreamingStdOutCallbackHandlerYield(q)
+        threading.Thread(target=askQuestion, args=(callback_fn,)).start()
+        return Response(generate(q), mimetype="text/event-stream")
+    ```
+    """
+    while True:
+        result: str = rq.get()
+        if result == STOP_ITEM or result is None:
+            break
+        yield result
@@ -33,7 +33,7 @@ def register_deserializable(cls: Type[T]) -> Type[T]:
    Returns:
        Type: The same class, after registration.
    """
-    JSONSerializable.register_class_as_deserializable(cls)
+    JSONSerializable._register_class_as_deserializable(cls)
    return cls


@@ -183,7 +183,7 @@ class JSONSerializable:
            return cls.deserialize(json_str)

    @classmethod
-    def register_class_as_deserializable(cls, target_class: Type[T]) -> None:
+    def _register_class_as_deserializable(cls, target_class: Type[T]) -> None:
        """
        Register a class as deserializable. This is a classmethod and globally shared.

@@ -3,7 +3,7 @@ import os
 from typing import Optional

 from embedchain.config import BaseLlmConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm


@@ -2,7 +2,7 @@ import logging
 from typing import Optional

 from embedchain.config import BaseLlmConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm


@@ -7,7 +7,7 @@ from embedchain.config import BaseLlmConfig
 from embedchain.config.llm.base import (DEFAULT_PROMPT,
                                        DEFAULT_PROMPT_WITH_HISTORY_TEMPLATE,
                                        DOCS_SITE_PROMPT_TEMPLATE)
-from embedchain.helper.json_serializable import JSONSerializable
+from embedchain.helpers.json_serializable import JSONSerializable
 from embedchain.memory.base import ECChatMemory
 from embedchain.memory.message import ChatMessage

@@ -5,7 +5,7 @@ from typing import Optional
 from langchain.llms import Cohere

 from embedchain.config import BaseLlmConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm


@@ -4,7 +4,7 @@ from langchain.callbacks.stdout import StdOutCallbackHandler
 from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

 from embedchain.config import BaseLlmConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm


@@ -1,11 +1,12 @@
 import importlib
+import logging
 import os
 from typing import Optional

 from langchain.llms import HuggingFaceHub

 from embedchain.config import BaseLlmConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm


@@ -20,7 +21,7 @@ class HuggingFaceLlm(BaseLlm):
        except ModuleNotFoundError:
            raise ModuleNotFoundError(
                "The required dependencies for HuggingFaceHub are not installed."
-                'Please install with `pip install --upgrade "embedchain[huggingface_hub]"`'
+                'Please install with `pip install --upgrade "embedchain[huggingface-hub]"`'
            ) from None

        super().__init__(config=config)
@@ -42,9 +43,11 @@ class HuggingFaceLlm(BaseLlm):
        else:
            raise ValueError("`top_p` must be > 0.0 and < 1.0")

+        model = config.model or "google/flan-t5-xxl"
+        logging.info(f"Using HuggingFaceHub with model {model}")
        llm = HuggingFaceHub(
            huggingfacehub_api_token=os.environ["HUGGINGFACE_ACCESS_TOKEN"],
-            repo_id=config.model or "google/flan-t5-xxl",
+            repo_id=model,
            model_kwargs=model_kwargs,
        )

@@ -5,7 +5,7 @@ from langchain.chat_models import JinaChat
 from langchain.schema import HumanMessage, SystemMessage

 from embedchain.config import BaseLlmConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm


@@ -5,7 +5,7 @@ from typing import Optional
 from langchain.llms import Replicate

 from embedchain.config import BaseLlmConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm


@@ -4,7 +4,7 @@ from langchain.chat_models import ChatOpenAI
 from langchain.schema import HumanMessage, SystemMessage

 from embedchain.config import BaseLlmConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm


@@ -34,7 +34,8 @@ class OpenAILlm(BaseLlm):
            from langchain.callbacks.streaming_stdout import \
                StreamingStdOutCallbackHandler

-            chat = ChatOpenAI(**kwargs, streaming=config.stream, callbacks=[StreamingStdOutCallbackHandler()])
+            callbacks = config.callbacks if config.callbacks else [StreamingStdOutCallbackHandler()]
+            chat = ChatOpenAI(**kwargs, streaming=config.stream, callbacks=callbacks)
        else:
            chat = ChatOpenAI(**kwargs)
        return chat(messages).content
@@ -3,7 +3,7 @@ import logging
 from typing import Optional

 from embedchain.config import BaseLlmConfig
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.llm.base import BaseLlm


@@ -1,4 +1,4 @@
-from embedchain.helper.json_serializable import JSONSerializable
+from embedchain.helpers.json_serializable import JSONSerializable


 class BaseLoader(JSONSerializable):
@@ -0,0 +1,150 @@
+import hashlib
+import logging
+import os
+
+from embedchain.helpers.json_serializable import register_deserializable
+from embedchain.loaders.base_loader import BaseLoader
+
+
+@register_deserializable
+class DiscordLoader(BaseLoader):
+    """
+    Load data from a Discord Channel ID.
+    """
+
+    def __init__(self):
+        if not os.environ.get("DISCORD_TOKEN"):
+            raise ValueError("DISCORD_TOKEN is not set")
+
+        self.token = os.environ.get("DISCORD_TOKEN")
+
+    @staticmethod
+    def _format_message(message):
+        return {
+            "message_id": message.id,
+            "content": message.content,
+            "author": {
+                "id": message.author.id,
+                "name": message.author.name,
+                "discriminator": message.author.discriminator,
+            },
+            "created_at": message.created_at.isoformat(),
+            "attachments": [
+                {
+                    "id": attachment.id,
+                    "filename": attachment.filename,
+                    "size": attachment.size,
+                    "url": attachment.url,
+                    "proxy_url": attachment.proxy_url,
+                    "height": attachment.height,
+                    "width": attachment.width,
+                }
+                for attachment in message.attachments
+            ],
+            "embeds": [
+                {
+                    "title": embed.title,
+                    "type": embed.type,
+                    "description": embed.description,
+                    "url": embed.url,
+                    "timestamp": embed.timestamp.isoformat(),
+                    "color": embed.color,
+                    "footer": {
+                        "text": embed.footer.text,
+                        "icon_url": embed.footer.icon_url,
+                        "proxy_icon_url": embed.footer.proxy_icon_url,
+                    },
+                    "image": {
+                        "url": embed.image.url,
+                        "proxy_url": embed.image.proxy_url,
+                        "height": embed.image.height,
+                        "width": embed.image.width,
+                    },
+                    "thumbnail": {
+                        "url": embed.thumbnail.url,
+                        "proxy_url": embed.thumbnail.proxy_url,
+                        "height": embed.thumbnail.height,
+                        "width": embed.thumbnail.width,
+                    },
+                    "video": {
+                        "url": embed.video.url,
+                        "height": embed.video.height,
+                        "width": embed.video.width,
+                    },
+                    "provider": {
+                        "name": embed.provider.name,
+                        "url": embed.provider.url,
+                    },
+                    "author": {
+                        "name": embed.author.name,
+                        "url": embed.author.url,
+                        "icon_url": embed.author.icon_url,
+                        "proxy_icon_url": embed.author.proxy_icon_url,
+                    },
+                    "fields": [
+                        {
+                            "name": field.name,
+                            "value": field.value,
+                            "inline": field.inline,
+                        }
+                        for field in embed.fields
+                    ],
+                }
+                for embed in message.embeds
+            ],
+        }
+
+    def load_data(self, channel_id: str):
+        """Load data from a Discord Channel ID."""
+        import discord
+
+        messages = []
+
+        class DiscordClient(discord.Client):
+            async def on_ready(self) -> None:
+                logging.info("Logged on as {0}!".format(self.user))
+                try:
+                    channel = self.get_channel(int(channel_id))
+                    if not isinstance(channel, discord.TextChannel):
+                        raise ValueError(
+                            f"Channel {channel_id} is not a text channel. " "Only text channels are supported for now."
+                        )
+                    threads = {}
+
+                    for thread in channel.threads:
+                        threads[thread.id] = thread
+
+                    async for message in channel.history(limit=None):
+                        messages.append(DiscordLoader._format_message(message))
+                        if message.id in threads:
+                            async for thread_message in threads[message.id].history(limit=None):
+                                messages.append(DiscordLoader._format_message(thread_message))
+
+                except Exception as e:
+                    logging.error(e)
+                    await self.close()
+                finally:
+                    await self.close()
+
+        intents = discord.Intents.default()
+        intents.message_content = True
+        client = DiscordClient(intents=intents)
+        client.run(self.token)
+
+        meta_data = {
+            "url": channel_id,
+        }
+
+        messages = str(messages)
+
+        doc_id = hashlib.sha256((messages + channel_id).encode()).hexdigest()
+
+        return {
+            "doc_id": doc_id,
+            "data": [
+                {
+                    "content": messages,
+                    "meta_data": meta_data,
+                }
+            ],
+        }
@@ -0,0 +1,77 @@
+import hashlib
+import logging
+import time
+from typing import Any, Dict, Optional
+
+import requests
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+
+class DiscourseLoader(BaseLoader):
+    def __init__(self, config: Optional[Dict[str, Any]] = None):
+        super().__init__()
+        if not config:
+            raise ValueError(
+                "DiscourseLoader requires a config. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+            )
+
+        self.domain = config.get("domain")
+        if not self.domain:
+            raise ValueError(
+                "DiscourseLoader requires a domain. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+            )
+
+    def _check_query(self, query):
+        if not query or not isinstance(query, str):
+            raise ValueError(
+                "DiscourseLoader requires a query. Check the documentation for the correct format - `https://docs.embedchain.ai/data-sources/discourse`"  # noqa: E501
+            )
+
+    def _load_post(self, post_id):
+        post_url = f"{self.domain}posts/{post_id}.json"
+        response = requests.get(post_url)
+        try:
+            response.raise_for_status()
+        except Exception as e:
+            logging.error(f"Failed to load post {post_id}: {e}")
+            return
+        response_data = response.json()
+        post_contents = clean_string(response_data.get("raw"))
+        meta_data = {
+            "url": post_url,
+            "created_at": response_data.get("created_at", ""),
+            "username": response_data.get("username", ""),
+            "topic_slug": response_data.get("topic_slug", ""),
+            "score": response_data.get("score", ""),
+        }
+        data = {
+            "content": post_contents,
+            "meta_data": meta_data,
+        }
+        return data
+
+    def load_data(self, query):
+        self._check_query(query)
+        data = []
+        data_contents = []
+        logging.info(f"Searching data on discourse url: {self.domain}, for query: {query}")
+        search_url = f"{self.domain}search.json?q={query}"
+        response = requests.get(search_url)
+        try:
+            response.raise_for_status()
+        except Exception as e:
+            raise ValueError(f"Failed to search query {query}: {e}")
+        response_data = response.json()
+        post_ids = response_data.get("grouped_search_result").get("post_ids")
+        for id in post_ids:
+            post_data = self._load_post(id)
+            if post_data:
+                data.append(post_data)
+                data_contents.append(post_data.get("content"))
+            # Sleep for 0.4 sec, to avoid rate limiting. Check `https://meta.discourse.org/t/api-rate-limits/208405/6`
+            time.sleep(0.4)
+        doc_id = hashlib.sha256((query + ", ".join(data_contents)).encode()).hexdigest()
+        response_data = {"doc_id": doc_id, "data": data}
+        return response_data
@@ -12,7 +12,7 @@ except ImportError:
    ) from None


-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader


@@ -6,7 +6,7 @@ except ImportError:
    raise ImportError(
        'Docx file requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'
    ) from None
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader


@@ -0,0 +1,297 @@
+import concurrent.futures
+import hashlib
+import logging
+import os
+import re
+import shlex
+from typing import Any, Dict, Optional
+
+from tqdm import tqdm
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+GITHUB_URL = "https://github.com"
+GITHUB_API_URL = "https://api.github.com"
+
+VALID_SEARCH_TYPES = set(["code", "repo", "pr", "issue", "discussion"])
+
+
+class GithubLoader(BaseLoader):
+    """Load data from github search query."""
+
+    def __init__(self, config: Optional[Dict[str, Any]] = None):
+        super().__init__()
+        if not config:
+            raise ValueError(
+                "GithubLoader requires a personal access token to use github api. Check - `https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic`"  # noqa: E501
+            )
+
+        try:
+            from github import Github
+        except ImportError as e:
+            raise ValueError(
+                "GithubLoader requires extra dependencies. Install with `pip install --upgrade 'embedchain[github]'`"
+            ) from e
+
+        self.config = config
+        token = config.get("token")
+        if not token:
+            raise ValueError(
+                "GithubLoader requires a personal access token to use github api. Check - `https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic`"  # noqa: E501
+            )
+
+        try:
+            self.client = Github(token)
+        except Exception as e:
+            logging.error(f"GithubLoader failed to initialize client: {e}")
+            self.client = None
+
+    def _github_search_code(self, query: str):
+        """Search github code."""
+        data = []
+        results = self.client.search_code(query)
+        for result in tqdm(results, total=results.totalCount, desc="Loading code files from github"):
+            url = result.html_url
+            logging.info(f"Added data from url: {url}")
+            content = result.decoded_content.decode("utf-8")
+            metadata = {
+                "url": url,
+            }
+            data.append(
+                {
+                    "content": clean_string(content),
+                    "meta_data": metadata,
+                }
+            )
+        return data
+
+    def _get_github_repo_data(self, repo_url: str):
+        local_hash = hashlib.sha256(repo_url.encode()).hexdigest()
+        local_path = f"/tmp/{local_hash}"
+        data = []
+
+        def _get_repo_tree(repo_url: str, local_path: str):
+            try:
+                from git import Repo
+            except ImportError as e:
+                raise ValueError(
+                    "GithubLoader requires extra dependencies. Install with `pip install --upgrade 'embedchain[github]'`"  # noqa: E501
+                ) from e
+
+            if os.path.exists(local_path):
+                logging.info("Repository already exists. Fetching updates...")
+                repo = Repo(local_path)
+                logging.info("Fetch completed.")
+            else:
+                logging.info("Cloning repository...")
+                Repo.clone_from(repo_url, local_path)
+                repo = Repo.clone_from(repo_url, local_path)
+                logging.info("Clone completed.")
+            return repo.head.commit.tree
+
+        def _get_repo_tree_contents(repo_path, tree, progress_bar):
+            for subtree in tree:
+                if subtree.type == "tree":
+                    _get_repo_tree_contents(repo_path, subtree, progress_bar)
+                else:
+                    assert subtree.type == "blob"
+                    try:
+                        contents = subtree.data_stream.read().decode("utf-8")
+                    except Exception:
+                        logging.warning(f"Failed to read file: {subtree.path}")
+                        progress_bar.update(1) if progress_bar else None
+                        continue
+
+                    url = f"{repo_url.rstrip('.git')}/blob/main/{subtree.path}"
+                    data.append(
+                        {
+                            "content": clean_string(contents),
+                            "meta_data": {
+                                "url": url,
+                            },
+                        }
+                    )
+                if progress_bar is not None:
+                    progress_bar.update(1)
+
+        repo_tree = _get_repo_tree(repo_url, local_path)
+        tree_list = list(repo_tree.traverse())
+        with tqdm(total=len(tree_list), desc="Loading files:", unit="item") as progress_bar:
+            _get_repo_tree_contents(local_path, repo_tree, progress_bar)
+
+        return data
+
+    def _github_search_repo(self, query: str):
+        """Search github repo."""
+        data = []
+        logging.info(f"Searching github repos with query: {query}")
+        results = self.client.search_repositories(query)
+        # Add repo urls and descriptions
+        urls = list(map(lambda x: x.html_url, results))
+        discriptions = list(map(lambda x: x.description, results))
+        data.append(
+            {
+                "content": clean_string(desc),
+                "meta_data": {
+                    "url": url,
+                },
+            }
+            for url, desc in zip(urls, discriptions)
+        )
+
+        # Add repo contents
+        for result in results:
+            clone_url = result.clone_url
+            logging.info(f"Cloning repository: {clone_url}")
+            data = self._get_github_repo_data(clone_url)
+        return data
+
+    def _github_search_issues_and_pr(self, query: str, type: str):
+        """Search github issues and PRs."""
+        data = []
+
+        query = f"{query} is:{type}"
+        logging.info(f"Searching github for query: {query}")
+
+        results = self.client.search_issues(query)
+
+        logging.info(f"Total results: {results.totalCount}")
+        for result in tqdm(results, total=results.totalCount, desc=f"Loading {type} from github"):
+            url = result.html_url
+            title = result.title
+            body = result.body
+            if not body:
+                logging.warn(f"Skipping issue because empty content for: {url}")
+                continue
+            labels = " ".join([label.name for label in result.labels])
+            issue_comments = result.get_comments()
+            comments = []
+            comments_created_at = []
+            for comment in issue_comments:
+                comments_created_at.append(str(comment.created_at))
+                comments.append(f"{comment.user.name}:{comment.body}")
+            content = "\n".join([title, labels, body, *comments])
+            metadata = {
+                "url": url,
+                "created_at": str(result.created_at),
+                "comments_created_at": " ".join(comments_created_at),
+            }
+            data.append(
+                {
+                    "content": clean_string(content),
+                    "meta_data": metadata,
+                }
+            )
+        return data
+
+    # need to test more for discussion
+    def _github_search_discussions(self, query: str):
+        """Search github discussions."""
+        data = []
+
+        query = f"{query} is:discussion"
+        logging.info(f"Searching github repo for query: {query}")
+        repos_results = self.client.search_repositories(query)
+        logging.info(f"Total repos found: {repos_results.totalCount}")
+        for repo_result in tqdm(repos_results, total=repos_results.totalCount, desc="Loading discussions from github"):
+            teams = repo_result.get_teams()
+            for team in teams:
+                team_discussions = team.get_discussions()
+                for discussion in team_discussions:
+                    url = discussion.html_url
+                    title = discussion.title
+                    body = discussion.body
+                    if not body:
+                        logging.warn(f"Skipping discussion because empty content for: {url}")
+                        continue
+                    comments = []
+                    comments_created_at = []
+                    print("Discussion comments: ", discussion.comments_url)
+                    content = "\n".join([title, body, *comments])
+                    metadata = {
+                        "url": url,
+                        "created_at": str(discussion.created_at),
+                        "comments_created_at": " ".join(comments_created_at),
+                    }
+                    data.append(
+                        {
+                            "content": clean_string(content),
+                            "meta_data": metadata,
+                        }
+                    )
+        return data
+
+    def _search_github_data(self, search_type: str, query: str):
+        """Search github data."""
+        if search_type == "code":
+            data = self._github_search_code(query)
+        elif search_type == "repo":
+            data = self._github_search_repo(query)
+        elif search_type == "issue":
+            data = self._github_search_issues_and_pr(query, search_type)
+        elif search_type == "pr":
+            data = self._github_search_issues_and_pr(query, search_type)
+        elif search_type == "discussion":
+            raise ValueError("GithubLoader does not support searching discussions yet.")
+
+        return data
+
+    def _get_valid_github_query(self, query: str):
+        """Check if query is valid and return search types and valid github query."""
+        query_terms = shlex.split(query)
+        # query must provide repo to load data from
+        if len(query_terms) < 1 or "repo:" not in query:
+            raise ValueError(
+                "GithubLoader requires a search query with `repo:` term. Refer docs - `https://docs.embedchain.ai/data-sources/github`"  # noqa: E501
+            )
+
+        github_query = []
+        types = set()
+        type_pattern = r"type:([a-zA-Z,]+)"
+        for term in query_terms:
+            term_match = re.search(type_pattern, term)
+            if term_match:
+                search_types = term_match.group(1).split(",")
+                types.update(search_types)
+            else:
+                github_query.append(term)
+
+        # query must provide search type
+        if len(types) == 0:
+            raise ValueError(
+                "GithubLoader requires a search query with `type:` term. Refer docs - `https://docs.embedchain.ai/data-sources/github`"  # noqa: E501
+            )
+
+        for search_type in search_types:
+            if search_type not in VALID_SEARCH_TYPES:
+                raise ValueError(
+                    f"Invalid search type: {search_type}. Valid types are: {', '.join(VALID_SEARCH_TYPES)}"
+                )
+
+        query = " ".join(github_query)
+
+        return types, query
+
+    def load_data(self, search_query: str, max_results: int = 1000):
+        """Load data from github search query."""
+
+        if not self.client:
+            raise ValueError(
+                "GithubLoader client is not initialized, data will not be loaded. Refer docs - `https://docs.embedchain.ai/data-sources/github`"  # noqa: E501
+            )
+
+        search_types, query = self._get_valid_github_query(search_query)
+        logging.info(f"Searching github for query: {query}, with types: {', '.join(search_types)}")
+
+        data = []
+
+        with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
+            futures_map = executor.map(self._search_github_data, search_types, [query] * len(search_types))
+            for search_data in tqdm(futures_map, total=len(search_types), desc="Searching data from github"):
+                data.extend(search_data)
+
+        return {
+            "doc_id": hashlib.sha256(query.encode()).hexdigest(),
+            "data": data,
+        }
@@ -1,6 +1,6 @@
 import hashlib

-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader


@@ -1,6 +1,6 @@
 import hashlib

-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader


@@ -1,6 +1,6 @@
 import hashlib

-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader


@@ -0,0 +1,64 @@
+import hashlib
+import logging
+from typing import Any, Dict, Optional
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+
+class MySQLLoader(BaseLoader):
+    def __init__(self, config: Optional[Dict[str, Any]]):
+        super().__init__()
+        if not config:
+            raise ValueError(
+                f"Invalid sql config: {config}.",
+                "Provide the correct config, refer `https://docs.embedchain.ai/data-sources/mysql`.",
+            )
+
+        self.config = config
+        self.connection = None
+        self.cursor = None
+        self._setup_loader(config=config)
+
+    def _setup_loader(self, config: Dict[str, Any]):
+        try:
+            import mysql.connector as sqlconnector
+        except ImportError as e:
+            raise ImportError(
+                "Unable to import required packages for MySQL loader. Run `pip install --upgrade 'embedchain[mysql]'`."  # noqa: E501
+            ) from e
+
+        try:
+            self.connection = sqlconnector.connection.MySQLConnection(**config)
+            self.cursor = self.connection.cursor()
+        except (sqlconnector.Error, IOError) as err:
+            logging.info(f"Connection failed: {err}")
+            raise ValueError(
+                f"Unable to connect with the given config: {config}.",
+                "Please provide the correct configuration to load data from you MySQL DB. \
+                    Refer `https://docs.embedchain.ai/data-sources/mysql`.",
+            )
+
+    def _check_query(self, query):
+        if not isinstance(query, str):
+            raise ValueError(
+                f"Invalid mysql query: {query}",
+                "Provide the valid query to add from mysql, \
+                    make sure you are following `https://docs.embedchain.ai/data-sources/mysql`",
+            )
+
+    def load_data(self, query):
+        self._check_query(query=query)
+        data = []
+        data_content = []
+        self.cursor.execute(query)
+        rows = self.cursor.fetchall()
+        for row in rows:
+            doc_content = clean_string(str(row))
+            data.append({"content": doc_content, "meta_data": {"url": query}})
+            data_content.append(doc_content)
+        doc_id = hashlib.sha256((query + ", ".join(data_content)).encode()).hexdigest()
+        return {
+            "doc_id": doc_id,
+            "data": data,
+        }
@@ -10,7 +10,7 @@ except ImportError:
    ) from None


-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils import clean_string

@@ -6,7 +6,7 @@ except ImportError:
    raise ImportError(
        'PDF File requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'
    ) from None
-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.utils import clean_string

@@ -40,9 +40,7 @@ class PostgresLoader(BaseLoader):
    def _check_query(self, query):
        if not isinstance(query, str):
            raise ValueError(
-                f"Invalid postgres query: {query}",
-                "Provide the valid source to add from postgres, \
-                    make sure you are following `https://docs.embedchain.ai/data-sources/postgres`",
+                f"Invalid postgres query: {query}. Provide the valid source to add from postgres, make sure you are following `https://docs.embedchain.ai/data-sources/postgres`",  # noqa:E501
            )

    def load_data(self, query):
@@ -54,7 +52,7 @@ class PostgresLoader(BaseLoader):
            results = self.cursor.fetchall()
            for result in results:
                doc_content = str(result)
-                data.append({"content": doc_content, "meta_data": {"url": f"postgres_query-({query})"}})
+                data.append({"content": doc_content, "meta_data": {"url": query}})
                data_content.append(doc_content)
            doc_id = hashlib.sha256((query + ", ".join(data_content)).encode()).hexdigest()
            return {
@@ -1,7 +1,10 @@
+import concurrent.futures
 import hashlib
 import logging
+from urllib.parse import urlparse

 import requests
+from tqdm import tqdm

 try:
    from bs4 import BeautifulSoup
@@ -11,41 +14,55 @@ except ImportError:
        'Sitemap requires extra dependencies. Install with `pip install --upgrade "embedchain[dataloaders]"`'
    ) from None

-from embedchain.helper.json_serializable import register_deserializable
+from embedchain.helpers.json_serializable import register_deserializable
 from embedchain.loaders.base_loader import BaseLoader
 from embedchain.loaders.web_page import WebPageLoader
-from embedchain.utils import is_readable


@register_deserializable
 class SitemapLoader(BaseLoader):
+    """
+    This method takes a sitemap URL as input and retrieves
+    all the URLs to use the WebPageLoader to load content
+    of each page.
+    """
+
    def load_data(self, sitemap_url):
-        """
-        This method takes a sitemap URL as input and retrieves
-        all the URLs to use the WebPageLoader to load content
-        of each page.
-        """
        output = []
        web_page_loader = WebPageLoader()
-        response = requests.get(sitemap_url)
-        response.raise_for_status()

-        soup = BeautifulSoup(response.text, "xml")
+        if urlparse(sitemap_url).scheme not in ["file", "http", "https"]:
+            raise ValueError("Not a valid URL.")

+        if urlparse(sitemap_url).scheme in ["http", "https"]:
+            response = requests.get(sitemap_url)
+            response.raise_for_status()
+        else:
+            with open(sitemap_url, "r") as file:
+                soup = BeautifulSoup(file, "xml")
        links = [link.text for link in soup.find_all("loc") if link.parent.name == "url"]
        if len(links) == 0:
-            # Get all <loc> tags as a fallback. This might include images.
            links = [link.text for link in soup.find_all("loc")]

        doc_id = hashlib.sha256((" ".join(links) + sitemap_url).encode()).hexdigest()

-        for link in links:
+        def load_web_page(link):
            try:
-                each_load_data = web_page_loader.load_data(link)
-                if is_readable(each_load_data.get("data")[0].get("content")):
-                    output.append(each_load_data.get("data"))
-                else:
-                    logging.warning(f"Page is not readable (too many invalid characters): {link}")
+                loader_data = web_page_loader.load_data(link)
+                return loader_data.get("data")
            except ParserRejectedMarkup as e:
                logging.error(f"Failed to parse {link}: {e}")
-        return {"doc_id": doc_id, "data": [data[0] for data in output]}
+            return None
+
+        with concurrent.futures.ThreadPoolExecutor() as executor:
+            future_to_link = {executor.submit(load_web_page, link): link for link in links}
+            for future in tqdm(concurrent.futures.as_completed(future_to_link), total=len(links), desc="Loading pages"):
+                link = future_to_link[future]
+                try:
+                    data = future.result()
+                    if data:
+                        output.extend(data)
+                except Exception as e:
+                    logging.error(f"Error loading page {link}: {e}")
+
+        return {"doc_id": doc_id, "data": output}
@@ -0,0 +1,108 @@
+import hashlib
+import logging
+import os
+import ssl
+from typing import Any, Dict, Optional
+
+import certifi
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+SLACK_API_BASE_URL = "https://www.slack.com/api/"
+
+
+class SlackLoader(BaseLoader):
+    def __init__(self, config: Optional[Dict[str, Any]] = None):
+        super().__init__()
+
+        if config is not None:
+            self.config = config
+        else:
+            self.config = {"base_url": SLACK_API_BASE_URL}
+
+        self.client = None
+        self._setup_loader(self.config)
+
+    def _setup_loader(self, config: Dict[str, Any]):
+        try:
+            from slack_sdk import WebClient
+        except ImportError as e:
+            raise ImportError(
+                "Slack loader requires extra dependencies. \
+                Install with `pip install --upgrade embedchain[slack]`"
+            ) from e
+
+        if os.getenv("SLACK_USER_TOKEN") is None:
+            raise ValueError(
+                "SLACK_USER_TOKEN environment variables not provided. Check `https://docs.embedchain.ai/data-sources/slack` to learn more."  # noqa:E501
+            )
+
+        logging.info(f"Creating Slack Loader with config: {config}")
+        # get slack client config params
+        slack_bot_token = os.getenv("SLACK_USER_TOKEN")
+        ssl_cert = ssl.create_default_context(cafile=certifi.where())
+        base_url = config.get("base_url", SLACK_API_BASE_URL)
+        headers = config.get("headers")
+        # for Org-Wide App
+        team_id = config.get("team_id")
+
+        self.client = WebClient(
+            token=slack_bot_token,
+            base_url=base_url,
+            ssl=ssl_cert,
+            headers=headers,
+            team_id=team_id,
+        )
+        logging.info("Slack Loader setup successful!")
+
+    def _check_query(self, query):
+        if not isinstance(query, str):
+            raise ValueError(
+                f"Invalid query passed to Slack loader, found: {query}. Check `https://docs.embedchain.ai/data-sources/slack` to learn more."  # noqa:E501
+            )
+
+    def load_data(self, query):
+        self._check_query(query)
+        try:
+            data = []
+            data_content = []
+
+            logging.info(f"Searching slack conversations for query: {query}")
+            results = self.client.search_messages(
+                query=query,
+                sort="timestamp",
+                sort_dir="desc",
+                count=1000,
+            )
+
+            messages = results.get("messages")
+            num_message = results.get("total")
+            logging.info(f"Found {num_message} messages for query: {query}")
+
+            matches = messages.get("matches", [])
+            for message in matches:
+                url = message.get("permalink")
+                text = message.get("text")
+                content = clean_string(text)
+
+                message_meta_data_keys = ["channel", "iid", "team", "ts", "type", "user", "username"]
+                meta_data = message.fromkeys(message_meta_data_keys, "")
+                meta_data.update({"url": url})
+                data.append(
+                    {
+                        "content": content,
+                        "meta_data": meta_data,
+                    }
+                )
+                data_content.append(content)
+            doc_id = hashlib.md5((query + ", ".join(data_content)).encode()).hexdigest()
+            return {
+                "doc_id": doc_id,
+                "data": data,
+            }
+        except Exception as e:
+            logging.warning(f"Error in loading slack data: {e}")
+            raise ValueError(
+                f"Error in loading slack data: {e}. Check `https://docs.embedchain.ai/data-sources/slack` to learn more."  # noqa:E501
+            ) from e
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Deven Patel GitHub Deven Patel	a4831d6ed9	Bump package version to 0.1.27 (#994 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-12-06 00:03:13 -08:00
Deven Patel GitHub Deven Patel	51b4966801	[Improvements] Package improvements (#993 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-12-05 23:42:45 -08:00
atkinshandGitHub	1d4e00ccef	local file path support for sitemap loader (#992 )	2023-12-05 19:04:20 -08:00
Deven Patel GitHub Deven Patel	c9fbc2e7d6	fix sitemap loader (#986 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-12-05 17:56:24 -08:00
Deshraj YadavandGitHub	fa34788df6	Bump package version to 0.1.26 (#991 )	2023-12-05 16:52:48 -08:00
Deven Patel GitHub Deven Patel	0f4f220119	Package improvements (#989 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-12-05 16:48:00 -08:00
Deven PatelandGitHub	512cfc9466	[Improvement] customize add method (#988 )	2023-12-05 00:55:33 -08:00
Deshraj YadavandGitHub	541b1cb7c7	Update version to 0.1.25 (#985 )	2023-12-03 15:15:52 -08:00
Deven PatelandGitHub	36af1a7615	[Improvement] improve github loader (#984 )	2023-12-01 11:24:13 -08:00
Deshraj YadavandGitHub	b02e8feeda	[BugFix] Fix issue of chunks not getting embedded in opensearch index (#983 )	2023-11-29 21:56:05 -08:00
Deven PatelandGitHub	406c46e7f4	[Improvements] Add support for creating app from YAML string config (#980 )	2023-11-29 12:25:30 -08:00
Sidharth MohantyandGitHub	e35eaf1bfc	Improve deps installation by converting them to one liner (#967 )	2023-11-29 10:08:34 -08:00
Sidharth MohantyandGitHub	38426a7af1	Discord loader (#976 )	2023-11-29 10:07:05 -08:00
Deshraj YadavandGitHub	141a23fb1e	[BugFix] Skip checking thread when making sqlite connection (#978 )	2023-11-26 15:44:06 -08:00
Sidharth MohantyandGitHub	bb28569abf	Update workflow to run when required (#941 )	2023-11-24 09:29:31 -08:00
Deshraj YadavandGitHub	1df46b2bb3	[Bug fix] Fix issue of missing user directory (#975 )	2023-11-24 09:26:59 -08:00
Deshraj YadavandGitHub	58f72e1ffe	Update Azure OpenAI embedding model docs (#974 )	2023-11-23 01:45:46 -08:00
Deshraj YadavandGitHub	33409140b4	[Bug fix] Fix Azure OpenAI related issue (#973 )	2023-11-23 01:40:54 -08:00
Deshraj YadavandGitHub	f6b80e01a1	[Feature] Add support for custom streaming callback (#971 )	2023-11-22 01:06:33 -08:00
Sidharth MohantyandGitHub	798d3fcc5a	Update version to 0.1.18 (#970 )	2023-11-21 10:01:09 -08:00
Sidharth MohantyandGitHub	85f3ac428b	Update embedding_fn signature to newest chroma db's (#969 )	2023-11-21 09:42:11 -08:00
Deshraj Yadav GitHub Deven Patel	9fcf2130b5	[Feature] Improve github and youtube channel loader (#966 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-17 18:25:14 -08:00
Taranjeet SinghandGitHub	51df00729e	Import beautifulsoup pacakge lazily. (#964 )	2023-11-17 18:19:08 -08:00
Deven PatelandGitHub	023a61446f	[Feature] Improve GitHub loader (#962 )	2023-11-16 22:06:36 -08:00
Deshraj YadavandGitHub	e0b73e6a5a	[Loaders] Improve web page and sitemap loader usability (#961 )	2023-11-16 16:01:43 -08:00
Deven PatelandGitHub	28460f725c	[Bugfix] fix poetry lock (#960 )	2023-11-16 13:30:38 -08:00
Deshraj YadavandGitHub	c93e49d2b8	[Bug fix] Update sleep time for substack loader and version bump (#958 )	2023-11-15 19:35:30 -08:00
Deven Patel GitHub Deven Patel Deshraj Yadav	07fb6bee54	[Features] Add Github and Youtube Channel loaders (#957 ) Co-authored-by: Deven Patel <deven298@yahoo.com> Co-authored-by: Deshraj Yadav <deshrajdry@gmail.com>	2023-11-15 19:17:42 -08:00
Deshraj YadavandGitHub	3fa7db8420	Bump version to 0.1.13 (#956 )	2023-11-15 18:42:48 -08:00
Deven Patel GitHub Deven Patel	c14bd7b73b	[Improvement] fix discourse loader to avoid rate limit (#953 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-15 15:33:16 -08:00
Sidharth MohantyandGitHub	5201beaab0	Bump version to `0.1.12` (#951 )	2023-11-15 09:33:26 -08:00
Sidharth MohantyandGitHub	122313d8a5	[New] Substack loader (#949 )	2023-11-14 21:52:15 -08:00
Deven Patel GitHub Deven Patel	82fd595306	[Improvements] improve package ux (#950 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-14 17:53:43 -08:00
Deven Patel GitHub Deven Patel	95c0d47236	[Feature] Discourse Loader (#948 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-13 16:39:11 -08:00
Deven Patel GitHub Deven Patel Deshraj Yadav	919cc74e94	[Feature] Add MySQL Loader (#920 ) Co-authored-by: Deven Patel <deven298@yahoo.com> Co-authored-by: Deshraj Yadav <deshrajdry@gmail.com>	2023-11-13 13:21:36 -08:00
Deshraj YadavandGitHub	d839991acb	[Docs] Add back sitemap loader docstring (#947 )	2023-11-13 13:08:09 -08:00
Deven Patel GitHub Deven Patel	539286aafd	[Feature] Add Slack Loader (#932 ) Co-authored-by: Deven Patel <deven298@yahoo.com>	2023-11-13 13:06:01 -08:00
Sidharth MohantyandGitHub	23522b7b55	Add slack_bot docker image (#933 )	2023-11-13 13:04:04 -08:00
UnMonsieurandGitHub	bf3fac56e4	Refactor: Make it clear what methods are private (#946 )	2023-11-13 13:00:13 -08:00
Deshraj Yadav	a5bf8e9075	[Improvement] Parallelize loading of sitemap urls	2023-11-13 12:53:34 -08:00
Deshraj YadavandGitHub	1d31b8f7e4	[Bugfix] Fix issue of "unable to open database file" (#945 )	2023-11-13 12:37:00 -08:00
Deshraj YadavandGitHub	b144c7dccc	[Bugfix] Fix hugging face command (#944 )	2023-11-13 12:17:27 -08:00