[Data loader] Make json data loader work for other languages (#897)

Co-authored-by: Deven Patel <deven298@yahoo.com>

README.md

@@ -64,7 +64,7 @@ For example, you can use Embedchain to create an Elon Musk bot using the followi
 
 ```python
 import os
-from embedchain import App
+from embedchain import Pipeline as App
 
 # Create a bot instance
 os.environ["OPENAI_API_KEY"] = "YOUR API KEY"
@@ -78,6 +78,17 @@ elon_bot.add("https://www.youtube.com/watch?v=RcYjXbSJBN8")
 # Query the bot
 elon_bot.query("How many companies does Elon Musk run and name those?")
 # Answer: Elon Musk currently runs several companies. As of my knowledge, he is the CEO and lead designer of SpaceX, the CEO and product architect of Tesla, Inc., the CEO and founder of Neuralink, and the CEO and founder of The Boring Company. However, please note that this information may change over time, so it's always good to verify the latest updates.
+
+# (Optional): Deploy app to Embedchain Platform
+app.deploy()
+# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+# ec-xxxxxx
+
+# 🛠️ Creating pipeline on the platform...
+# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+
+# 🛠️ Adding data to your pipeline...
+# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
 ```
 
 ## Examples

docs/get-started/introduction.mdx

@@ -3,30 +3,106 @@ title: 📚 Introduction
 description: '📝 Embedchain is a Data Platform for LLMs - load, index, retrieve, and sync any unstructured data'
 ---
 
-## 🤔 What is Embedchain?
+## 🌐 What is Embedchain?
 
-Embedchain abstracts the entire process of loading data, chunking it, creating embeddings, and storing it in a vector database.
+Embedchain simplifies data handling by automatically processing unstructured data, breaking it into chunks, generating embeddings, and storing it in a vector database.
 
-You can add data from different data sources using the `.add()` method. Then, simply use the `.query()` method to find answers from the added datasets.
+Through various APIs, you can obtain contextual information for queries, find answers to specific questions, and engage in chat conversations using your data.
+## 🔍 Search
 
-If you want to create a Naval Ravikant bot with a YouTube video, a book in PDF format, two blog posts, and a question and answer pair, all you need to do is add the respective links. Embedchain will take care of the rest, creating a bot for you.
+Embedchain lets you get most relevant context by doing semantic search over your data sources for a provided query. See the example below:
 
 ```python
 from embedchain import Pipeline as App
 
-naval_bot = App()
+# Initialize app
-# Add online data
+app = App()
-naval_bot.add("https://www.youtube.com/watch?v=3qHkcs3kG44")
-naval_bot.add("https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf")
-naval_bot.add("https://nav.al/feedback")
-naval_bot.add("https://nav.al/agi")
-naval_bot.add("The Meanings of Life", 'text', metadata={'chapter': 'philosphy'})
 
-# Add local resources
+# Add data source
-naval_bot.add(("Who is Naval Ravikant?", "Naval Ravikant is an Indian-American entrepreneur and investor."))
+app.add("https://www.forbes.com/profile/elon-musk")
 
-naval_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?")
+# Get relevant context using semantic search
-# Answer: Naval argues that humans possess the unique capacity to understand explanations or concepts to the maximum extent possible in this physical reality.
+context = app.search("What is the net worth of Elon?", num_documents=2)
+print(context)
+# Context:
+# [
+#     {
+#         'context': 'Elon Musk PROFILEElon MuskCEO, Tesla$221.9BReal Time Net Worthas of 10/29/23Reflects change since 5 pm ET of prior trading day. 1 in the world todayPhoto by Martin Schoeller for ForbesAbout Elon MuskElon Musk cofounded six companies, including electric car maker Tesla, rocket producer SpaceX and tunneling startup Boring Company.He owns about 21% of Tesla between stock and options, but has pledged more than half his shares as collateral for personal loans of up to $3.5 billion.SpaceX, founded in',
+#         'source': 'https://www.forbes.com/profile/elon-musk',
+#         'document_id': 'some_document_id'
+#     },
+#     {
+#         'context': 'company, which is now called X.Wealth HistoryHOVER TO REVEAL NET WORTH BY YEARForbes Lists 1Forbes 400 (2023)The Richest Person In Every State (2023) 2Billionaires (2023) 1Innovative Leaders (2019) 25Powerful People (2018) 12Richest In Tech (2017)Global Game Changers (2016)More ListsPersonal StatsAge52Source of WealthTesla, SpaceX, Self MadeSelf-Made Score8Philanthropy Score1ResidenceAustin, TexasCitizenshipUnited StatesMarital StatusSingleChildren11EducationBachelor of Arts/Science, University',
+#         'source': 'https://www.forbes.com/profile/elon-musk',
+#         'document_id': 'some_document_id'
+#     }
+# ]
+```
+
+## ❓Query
+
+Embedchain empowers developers to ask questions and receive relevant answers through a user-friendly query API. Refer to the following example to learn how to utilize the query API:
+
+```python
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Get relevant answer for your query
+answer = app.query("What is the net worth of Elon?")
+print(answer)
+# Answer: The net worth of Elon Musk is $221.9 billion.
+```
+
+## 💬 Chat
+
+Embedchain allows easy chatting over your data sources using a user-friendly chat API. Check out the example below to understand how to use the chat API:
+
+```python
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Chat on your data using `.chat()`
+answer = app.chat("How much did Elon pay for Twitter?")
+print(answer)
+# Answer: Elon Musk paid $44 billion for Twitter.
+```
+
+## 🚀 Deploy
+
+Embedchain enables developers to deploy their LLM-powered apps in production using the Embedchain platform. The platform offers free access to context on your data through its REST API. Once the pipeline is deployed, you can update your data sources anytime after deployment.
+
+See the example below on how to use the deploy API:
+
+```python
+from embedchain import Pipeline as App
+
+# Initialize app
+app = App()
+
+# Add data source
+app.add("https://www.forbes.com/profile/elon-musk")
+
+# Deploy your pipeline to Embedchain Platform
+app.deploy()
+
+# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+# ec-xxxxxx
+
+# 🛠️ Creating pipeline on the platform...
+# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+
+# 🛠️ Adding data to your pipeline...
+# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
 ```
 
 ## 🚀 How it works?

docs/get-started/quickstart.mdx

@@ -33,6 +33,19 @@ app.add("https://www.forbes.com/profile/elon-musk")
 ```python
 app.query("What is the net worth of Elon Musk today?")
 # Answer: The net worth of Elon Musk today is $258.7 billion.
+```
+  </Step>
+  <Step title="🚀 (Optional) Deploy your pipeline to Embedchain Platform">
+```python
+app.deploy()
+# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+# ec-xxxxxx
+
+# 🛠️ Creating pipeline on the platform...
+# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+
+# 🛠️ Adding data to your pipeline...
+# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
 ```
   </Step>
 </Steps>
@@ -55,4 +68,14 @@ app.add("https://www.forbes.com/profile/elon-musk")
 response = app.query("What is the net worth of Elon Musk today?")
 print(response)
 # Answer: The net worth of Elon Musk today is $258.7 billion.
+
+app.deploy()
+# 🔑 Enter your Embedchain API key. You can find the API key at https://app.embedchain.ai/settings/keys/
+# ec-xxxxxx
+
+# 🛠️ Creating pipeline on the platform...
+# 🎉🎉🎉 Pipeline created successfully! View your pipeline: https://app.embedchain.ai/pipelines/xxxxx
+
+# 🛠️ Adding data to your pipeline...
+# ✅ Data of type: web_page, value: https://www.forbes.com/profile/elon-musk added successfully.
 ```

docs/mint.json

@@ -7,9 +7,20 @@
   },
   "favicon": "/favicon.png",
   "colors": {
-    "primary": "#12A7D3",
+    "primary": "#2B48EE",
-    "light": "#81D7F7",
+    "light": "#2B48EE",
-    "dark": "#004E7A"
+    "dark": "#2B48EE",
+    "background": {
+      "dark": "#020415"
+    }
+  },
+  "metadata": {
+    "og:image": "/images/og.png",
+    "twitter:site": "@embedchain"
+  },
+  "topAnchor": {
+    "name": "Documentation",
+    "icon": "book-open"
   },
   "topbarLinks": [
     {
@@ -26,8 +37,11 @@
     }
   ],
   "topbarCtaButton": {
-    "name": "GitHub",
+    "name": "Get started",
-    "url": "https://embedchain.ai"
+    "url": "https://app.embedchain.ai"
+  },
+  "primaryTab": {
+    "name": "Docs"
   },
   "navigation": [
     {
@@ -100,7 +114,6 @@
     }
 
   ],
-
   "footerSocials": {
     "website": "https://embedchain.ai",
     "github": "https://github.com/embedchain/embedchain",
@@ -109,7 +122,19 @@
     "twitter": "https://twitter.com/embedchain",
     "linkedin": "https://www.linkedin.com/company/embedchain"
   },
-  "backgroundImage": "/background.png",
   "isWhiteLabeled": true,
-  "feedback.thumbsRating": true
+  "analytics": {
+    "posthog": {
+      "apiKey": "phc_PHQDA5KwztijnSojsxJ2c1DuJd52QCzJzT2xnSGvjN2",
+      "apiHost": "https://app.embedchain.ai/ingest"
+    }
+  },
+  "feedback": {
+    "suggestEdit": true,
+    "raiseIssue": true,
+    "thumbsRating": true
+  },
+  "search": {
+    "prompt": "✨ Search embedchain docs..."
+  }
 }

embedchain/data_formatter/data_formatter.py

@@ -1,41 +1,10 @@
+from importlib import import_module
+
 from embedchain.chunkers.base_chunker import BaseChunker
-from embedchain.chunkers.docs_site import DocsSiteChunker
-from embedchain.chunkers.docx_file import DocxFileChunker
-from embedchain.chunkers.gmail import GmailChunker
-from embedchain.chunkers.images import ImagesChunker
-from embedchain.chunkers.json import JSONChunker
-from embedchain.chunkers.mdx import MdxChunker
-from embedchain.chunkers.notion import NotionChunker
-from embedchain.chunkers.openapi import OpenAPIChunker
-from embedchain.chunkers.pdf_file import PdfFileChunker
-from embedchain.chunkers.qna_pair import QnaPairChunker
-from embedchain.chunkers.sitemap import SitemapChunker
-from embedchain.chunkers.table import TableChunker
-from embedchain.chunkers.text import TextChunker
-from embedchain.chunkers.unstructured_file import UnstructuredFileChunker
-from embedchain.chunkers.web_page import WebPageChunker
-from embedchain.chunkers.xml import XmlChunker
-from embedchain.chunkers.youtube_video import YoutubeVideoChunker
 from embedchain.config import AddConfig
 from embedchain.config.add_config import ChunkerConfig, LoaderConfig
 from embedchain.helper.json_serializable import JSONSerializable
 from embedchain.loaders.base_loader import BaseLoader
-from embedchain.loaders.csv import CsvLoader
-from embedchain.loaders.docs_site_loader import DocsSiteLoader
-from embedchain.loaders.docx_file import DocxFileLoader
-from embedchain.loaders.gmail import GmailLoader
-from embedchain.loaders.images import ImagesLoader
-from embedchain.loaders.json import JSONLoader
-from embedchain.loaders.local_qna_pair import LocalQnaPairLoader
-from embedchain.loaders.local_text import LocalTextLoader
-from embedchain.loaders.mdx import MdxLoader
-from embedchain.loaders.openapi import OpenAPILoader
-from embedchain.loaders.pdf_file import PdfFileLoader
-from embedchain.loaders.sitemap import SitemapLoader
-from embedchain.loaders.unstructured_file import UnstructuredLoader
-from embedchain.loaders.web_page import WebPageLoader
-from embedchain.loaders.xml import XmlLoader
-from embedchain.loaders.youtube_video import YoutubeVideoLoader
 from embedchain.models.data_type import DataType
 
 
@@ -58,6 +27,11 @@ class DataFormatter(JSONSerializable):
         self.loader = self._get_loader(data_type=data_type, config=config.loader)
         self.chunker = self._get_chunker(data_type=data_type, config=config.chunker)
 
+    def _lazy_load(self, module_path: str):
+        module_path, class_name = module_path.rsplit(".", 1)
+        module = import_module(module_path)
+        return getattr(module, class_name)
+
     def _get_loader(self, data_type: DataType, config: LoaderConfig) -> BaseLoader:
         """
         Returns the appropriate data loader for the given data type.
@@ -71,71 +45,55 @@ class DataFormatter(JSONSerializable):
         :rtype: BaseLoader
         """
         loaders = {
-            DataType.YOUTUBE_VIDEO: YoutubeVideoLoader,
+            DataType.YOUTUBE_VIDEO: "embedchain.loaders.youtube_video.YoutubeVideoLoader",
-            DataType.PDF_FILE: PdfFileLoader,
+            DataType.PDF_FILE: "embedchain.loaders.pdf_file.PdfFileLoader",
-            DataType.WEB_PAGE: WebPageLoader,
+            DataType.WEB_PAGE: "embedchain.loaders.web_page.WebPageLoader",
-            DataType.QNA_PAIR: LocalQnaPairLoader,
+            DataType.QNA_PAIR: "embedchain.loaders.local_qna_pair.LocalQnaPairLoader",
-            DataType.TEXT: LocalTextLoader,
+            DataType.TEXT: "embedchain.loaders.local_text.LocalTextLoader",
-            DataType.DOCX: DocxFileLoader,
+            DataType.DOCX: "embedchain.loaders.docx_file.DocxFileLoader",
-            DataType.SITEMAP: SitemapLoader,
+            DataType.SITEMAP: "embedchain.loaders.sitemap.SitemapLoader",
-            DataType.XML: XmlLoader,
+            DataType.XML: "embedchain.loaders.xml.XmlLoader",
-            DataType.DOCS_SITE: DocsSiteLoader,
+            DataType.DOCS_SITE: "embedchain.loaders.docs_site_loader.DocsSiteLoader",
-            DataType.CSV: CsvLoader,
+            DataType.CSV: "embedchain.loaders.csv.CsvLoader",
-            DataType.MDX: MdxLoader,
+            DataType.MDX: "embedchain.loaders.mdx.MdxLoader",
-            DataType.IMAGES: ImagesLoader,
+            DataType.IMAGES: "embedchain.loaders.images.ImagesLoader",
-            DataType.UNSTRUCTURED: UnstructuredLoader,
+            DataType.UNSTRUCTURED: "embedchain.loaders.unstructured_file.UnstructuredLoader",
-            DataType.JSON: JSONLoader,
+            DataType.JSON: "embedchain.loaders.json.JSONLoader",
-            DataType.OPENAPI: OpenAPILoader,
+            DataType.OPENAPI: "embedchain.loaders.openapi.OpenAPILoader",
-            DataType.GMAIL: GmailLoader,
+            DataType.GMAIL: "embedchain.loaders.gmail.GmailLoader",
+            DataType.NOTION: "embedchain.loaders.notion.NotionLoader",
         }
-        lazy_loaders = {DataType.NOTION}
         if data_type in loaders:
-            loader_class: type = loaders[data_type]
+            loader_class: type = self._lazy_load(loaders[data_type])
-            loader: BaseLoader = loader_class()
+            return loader_class()
-            return loader
-        elif data_type in lazy_loaders:
-            if data_type == DataType.NOTION:
-                from embedchain.loaders.notion import NotionLoader
-
-                return NotionLoader()
-            else:
-                raise ValueError(f"Unsupported data type: {data_type}")
         else:
             raise ValueError(f"Unsupported data type: {data_type}")
 
     def _get_chunker(self, data_type: DataType, config: ChunkerConfig) -> BaseChunker:
-        """Returns the appropriate chunker for the given data type.
+        """Returns the appropriate chunker for the given data type (updated for lazy loading)."""
-
-        :param data_type: The type of the data to chunk.
-        :type data_type: DataType
-        :param config: Config to initialize the chunker with.
-        :type config: ChunkerConfig
-        :raises ValueError: If an unsupported data type is provided.
-        :return: The chunker for the given data type.
-        :rtype: BaseChunker
-        """
         chunker_classes = {
-            DataType.YOUTUBE_VIDEO: YoutubeVideoChunker,
+            DataType.YOUTUBE_VIDEO: "embedchain.chunkers.youtube_video.YoutubeVideoChunker",
-            DataType.PDF_FILE: PdfFileChunker,
+            DataType.PDF_FILE: "embedchain.chunkers.pdf_file.PdfFileChunker",
-            DataType.WEB_PAGE: WebPageChunker,
+            DataType.WEB_PAGE: "embedchain.chunkers.web_page.WebPageChunker",
-            DataType.QNA_PAIR: QnaPairChunker,
+            DataType.QNA_PAIR: "embedchain.chunkers.qna_pair.QnaPairChunker",
-            DataType.TEXT: TextChunker,
+            DataType.TEXT: "embedchain.chunkers.text.TextChunker",
-            DataType.DOCX: DocxFileChunker,
+            DataType.DOCX: "embedchain.chunkers.docx_file.DocxFileChunker",
-            DataType.DOCS_SITE: DocsSiteChunker,
+            DataType.SITEMAP: "embedchain.chunkers.sitemap.SitemapChunker",
-            DataType.SITEMAP: SitemapChunker,
+            DataType.XML: "embedchain.chunkers.xml.XmlChunker",
-            DataType.NOTION: NotionChunker,
+            DataType.DOCS_SITE: "embedchain.chunkers.docs_site.DocsSiteChunker",
-            DataType.CSV: TableChunker,
+            DataType.CSV: "embedchain.chunkers.table.TableChunker",
-            DataType.MDX: MdxChunker,
+            DataType.MDX: "embedchain.chunkers.mdx.MdxChunker",
-            DataType.IMAGES: ImagesChunker,
+            DataType.IMAGES: "embedchain.chunkers.images.ImagesChunker",
-            DataType.XML: XmlChunker,
+            DataType.UNSTRUCTURED: "embedchain.chunkers.unstructured_file.UnstructuredFileChunker",
-            DataType.UNSTRUCTURED: UnstructuredFileChunker,
+            DataType.JSON: "embedchain.chunkers.json.JSONChunker",
-            DataType.JSON: JSONChunker,
+            DataType.OPENAPI: "embedchain.chunkers.openapi.OpenAPIChunker",
-            DataType.OPENAPI: OpenAPIChunker,
+            DataType.GMAIL: "embedchain.chunkers.gmail.GmailChunker",
-            DataType.GMAIL: GmailChunker,
+            DataType.NOTION: "embedchain.chunkers.notion.NotionChunker",
         }
+
         if data_type in chunker_classes:
-            chunker_class: type = chunker_classes[data_type]
+            chunker_class = self._lazy_load(chunker_classes[data_type])
-            chunker: BaseChunker = chunker_class(config)
+            chunker = chunker_class(config)
             chunker.set_data_type(data_type)
             return chunker
         else:

embedchain/embedchain.py

@@ -4,7 +4,7 @@ import logging
 import os
 import sqlite3
 from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Tuple, Union
 
 from dotenv import load_dotenv
 from langchain.docstore.document import Document
@@ -438,7 +438,9 @@ class EmbedChain(JSONSerializable):
             )
         ]
 
-    def retrieve_from_database(self, input_query: str, config: Optional[BaseLlmConfig] = None, where=None) -> List[str]:
+    def retrieve_from_database(
+        self, input_query: str, config: Optional[BaseLlmConfig] = None, where=None, citations: bool = False
+    ) -> Union[List[Tuple[str, str, str]], List[str]]:
         """
         Queries the vector database based on the given input query.
         Gets relevant doc based on the query
@@ -449,6 +451,8 @@ class EmbedChain(JSONSerializable):
         :type config: Optional[BaseLlmConfig], optional
         :param where: A dictionary of key-value pairs to filter the database results, defaults to None
         :type where: _type_, optional
+        :param citations: A boolean to indicate if db should fetch citation source
+        :type citations: bool
         :return: List of contents of the document that matched your query
         :rtype: List[str]
         """
@@ -478,14 +482,19 @@ class EmbedChain(JSONSerializable):
             n_results=query_config.number_documents,
             where=where,
             skip_embedding=(hasattr(config, "query_type") and config.query_type == "Images"),
+            citations=citations,
         )
 
-        if len(contexts) > 0 and isinstance(contexts[0], tuple):
-            contexts = list(map(lambda x: x[0], contexts))
-
         return contexts
 
-    def query(self, input_query: str, config: BaseLlmConfig = None, dry_run=False, where: Optional[Dict] = None) -> str:
+    def query(
+        self,
+        input_query: str,
+        config: BaseLlmConfig = None,
+        dry_run=False,
+        where: Optional[Dict] = None,
+        **kwargs: Dict[str, Any],
+    ) -> Union[Tuple[str, List[Tuple[str, str, str]]], str]:
         """
         Queries the vector database based on the given input query.
         Gets relevant doc based on the query and then passes it to an
@@ -501,15 +510,31 @@ class EmbedChain(JSONSerializable):
         :type dry_run: bool, optional
         :param where: A dictionary of key-value pairs to filter the database results., defaults to None
         :type where: Optional[Dict[str, str]], optional
-        :return: The answer to the query or the dry run result
+        :param kwargs: To read more params for the query function. Ex. we use citations boolean
-        :rtype: str
+        param to return context along with the answer
+        :type kwargs: Dict[str, Any]
+        :return: The answer to the query, with citations if the citation flag is True
+        or the dry run result
+        :rtype: str, if citations is False, otherwise Tuple[str,List[Tuple[str,str,str]]]
         """
-        contexts = self.retrieve_from_database(input_query=input_query, config=config, where=where)
+        citations = kwargs.get("citations", False)
-        answer = self.llm.query(input_query=input_query, contexts=contexts, config=config, dry_run=dry_run)
+        contexts = self.retrieve_from_database(input_query=input_query, config=config, where=where, citations=citations)
+        if citations and len(contexts) > 0 and isinstance(contexts[0], tuple):
+            contexts_data_for_llm_query = list(map(lambda x: x[0], contexts))
+        else:
+            contexts_data_for_llm_query = contexts
+
+        answer = self.llm.query(
+            input_query=input_query, contexts=contexts_data_for_llm_query, config=config, dry_run=dry_run
+        )
 
         # Send anonymous telemetry
         self.telemetry.capture(event_name="query", properties=self._telemetry_props)
-        return answer
+
+        if citations:
+            return answer, contexts
+        else:
+            return answer
 
     def chat(
         self,
@@ -517,6 +542,7 @@ class EmbedChain(JSONSerializable):
         config: Optional[BaseLlmConfig] = None,
         dry_run=False,
         where: Optional[Dict[str, str]] = None,
+        **kwargs: Dict[str, Any],
     ) -> str:
         """
         Queries the vector database on the given input query.
@@ -535,15 +561,31 @@ class EmbedChain(JSONSerializable):
         :type dry_run: bool, optional
         :param where: A dictionary of key-value pairs to filter the database results., defaults to None
         :type where: Optional[Dict[str, str]], optional
-        :return: The answer to the query or the dry run result
+        :param kwargs: To read more params for the query function. Ex. we use citations boolean
-        :rtype: str
+        param to return context along with the answer
+        :type kwargs: Dict[str, Any]
+        :return: The answer to the query, with citations if the citation flag is True
+        or the dry run result
+        :rtype: str, if citations is False, otherwise Tuple[str,List[Tuple[str,str,str]]]
         """
-        contexts = self.retrieve_from_database(input_query=input_query, config=config, where=where)
+        citations = kwargs.get("citations", False)
-        answer = self.llm.chat(input_query=input_query, contexts=contexts, config=config, dry_run=dry_run)
+        contexts = self.retrieve_from_database(input_query=input_query, config=config, where=where, citations=citations)
+        if citations and len(contexts) > 0 and isinstance(contexts[0], tuple):
+            contexts_data_for_llm_query = list(map(lambda x: x[0], contexts))
+        else:
+            contexts_data_for_llm_query = contexts
+
+        answer = self.llm.chat(
+            input_query=input_query, contexts=contexts_data_for_llm_query, config=config, dry_run=dry_run
+        )
+
         # Send anonymous telemetry
         self.telemetry.capture(event_name="chat", properties=self._telemetry_props)
 
-        return answer
+        if citations:
+            return answer, contexts
+        else:
+            return answer
 
     def set_collection_name(self, name: str):
         """

embedchain/loaders/json.py

@@ -1,24 +1,57 @@
 import hashlib
+import json
+import os
+import re
 
-from langchain.document_loaders.json_loader import \
+import requests
-    JSONLoader as LangchainJSONLoader
 
 from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
 
-langchain_json_jq_schema = 'to_entries | map("\(.key): \(.value|tostring)") | .[]'
+VALID_URL_PATTERN = "^https:\/\/[0-9A-z.]+.[0-9A-z.]+.[a-z]+\/.*\.json$"
 
 
 class JSONLoader(BaseLoader):
     @staticmethod
     def load_data(content):
         """Load a json file. Each data point is a key value pair."""
+        try:
+            from llama_hub.jsondata.base import \
+                JSONDataReader as LLHBUBJSONLoader
+        except ImportError:
+            raise Exception(
+                f"Couldn't import the required packages to load {content}, \
+                Do `pip install --upgrade 'embedchain[json]`"
+            )
+
+        loader = LLHBUBJSONLoader()
+
+        if not isinstance(content, str):
+            print(f"Invaid content input. Provide the correct path to the json file saved locally in {content}")
+
         data = []
         data_content = []
-        loader = LangchainJSONLoader(content, text_content=False, jq_schema=langchain_json_jq_schema)
+
-        docs = loader.load()
+        # Load json data from various sources. TODO: add support for dictionary
+        if os.path.isfile(content):
+            with open(content, "r", encoding="utf-8") as json_file:
+                json_data = json.load(json_file)
+        elif re.match(VALID_URL_PATTERN, content):
+            response = requests.get(content)
+            if response.status_code == 200:
+                json_data = response.json()
+            else:
+                raise ValueError(
+                    f"Loading data from the given url: {content} failed. \
+                    Make sure the url is working."
+                )
+        else:
+            raise ValueError(f"Invalid content to load json data from: {content}")
+
+        docs = loader.load_data(json_data)
         for doc in docs:
-            meta_data = doc.metadata
+            doc_content = clean_string(doc.text)
-            data.append({"content": doc.page_content, "meta_data": {"url": content, "row": meta_data["seq_num"]}})
+            data.append({"content": doc_content, "meta_data": {"url": content}})
-            data_content.append(doc.page_content)
+            data_content.append(doc_content)
         doc_id = hashlib.sha256((content + ", ".join(data_content)).encode()).hexdigest()
         return {"doc_id": doc_id, "data": data}

embedchain/pipeline.py

@@ -229,12 +229,23 @@ class Pipeline(EmbedChain):
         # TODO: Search will call the endpoint rather than fetching the data from the db itself when deploy=True.
         if self.id is None:
             where = {"app_id": self.local_id}
-            return self.db.query(
+            context = self.db.query(
                 query,
                 n_results=num_documents,
                 where=where,
                 skip_embedding=False,
+                citations=True,
             )
+            result = []
+            for c in context:
+                result.append(
+                    {
+                        "context": c[0],
+                        "source": c[1],
+                        "document_id": c[2],
+                    }
+                )
+            return result
         else:
             # Make API call to the backend to get the results
             NotImplementedError("Search is not implemented yet for the prod mode.")

embedchain/telemetry/posthog.py

@@ -17,7 +17,7 @@ logger = logging.getLogger(__name__)
 
 class AnonymousTelemetry:
     def __init__(self, host="https://app.posthog.com", enabled=True):
-        self.project_api_key = "phc_XnMmNHzwxE7PVHX4mD2r8K6nfxVM48a2sq2U3N1p2lO"
+        self.project_api_key = "phc_PHQDA5KwztijnSojsxJ2c1DuJd52QCzJzT2xnSGvjN2"
         self.host = host
         self.posthog = Posthog(project_api_key=self.project_api_key, host=self.host)
         self.user_id = self.get_user_id()

embedchain/vectordb/chroma.py

@@ -1,5 +1,5 @@
 import logging
-from typing import Any, Dict, List, Optional, Tuple
+from typing import Any, Dict, List, Optional, Tuple, Union
 
 from chromadb import Collection, QueryResult
 from langchain.docstore.document import Document
@@ -192,8 +192,13 @@ class ChromaDB(BaseVectorDB):
         ]
 
     def query(
-        self, input_query: List[str], n_results: int, where: Dict[str, any], skip_embedding: bool
+        self,
-    ) -> List[Tuple[str, str, str]]:
+        input_query: List[str],
+        n_results: int,
+        where: Dict[str, any],
+        skip_embedding: bool,
+        citations: bool = False,
+    ) -> Union[List[Tuple[str, str, str]], List[str]]:
         """
         Query contents from vector database based on vector similarity
 
@@ -205,9 +210,12 @@ class ChromaDB(BaseVectorDB):
         :type where: Dict[str, Any]
         :param skip_embedding: Optional. If True, then the input_query is assumed to be already embedded.
         :type skip_embedding: bool
+        :param citations: we use citations boolean param to return context along with the answer.
+        :type citations: bool, default is False.
         :raises InvalidDimensionException: Dimensions do not match.
-        :return: The content of the document that matched your query, url of the source, doc_id
+        :return: The content of the document that matched your query,
-        :rtype: List[Tuple[str,str,str]]
+        along with url of the source and doc_id (if citations flag is true)
+        :rtype: List[str], if citations=False, otherwise List[Tuple[str, str, str]]
         """
         try:
             if skip_embedding:
@@ -236,10 +244,13 @@ class ChromaDB(BaseVectorDB):
         contexts = []
         for result in results_formatted:
             context = result[0].page_content
-            metadata = result[0].metadata
+            if citations:
-            source = metadata["url"]
+                metadata = result[0].metadata
-            doc_id = metadata["doc_id"]
+                source = metadata["url"]
-            contexts.append((context, source, doc_id))
+                doc_id = metadata["doc_id"]
+                contexts.append((context, source, doc_id))
+            else:
+                contexts.append(context)
         return contexts
 
     def set_collection_name(self, name: str):

embedchain/vectordb/elasticsearch.py

@@ -1,5 +1,5 @@
 import logging
-from typing import Any, Dict, List, Optional, Tuple
+from typing import Any, Dict, List, Optional, Tuple, Union
 
 try:
     from elasticsearch import Elasticsearch
@@ -136,8 +136,13 @@ class ElasticsearchDB(BaseVectorDB):
         self.client.indices.refresh(index=self._get_index())
 
     def query(
-        self, input_query: List[str], n_results: int, where: Dict[str, any], skip_embedding: bool
+        self,
-    ) -> List[Tuple[str, str, str]]:
+        input_query: List[str],
+        n_results: int,
+        where: Dict[str, any],
+        skip_embedding: bool,
+        citations: bool = False,
+    ) -> Union[List[Tuple[str, str, str]], List[str]]:
         """
         query contents from vector data base based on vector similarity
 
@@ -150,8 +155,11 @@ class ElasticsearchDB(BaseVectorDB):
         :param skip_embedding: Optional. If True, then the input_query is assumed to be already embedded.
         :type skip_embedding: bool
         :return: The context of the document that matched your query, url of the source, doc_id
-
+        :param citations: we use citations boolean param to return context along with the answer.
-        :rtype: List[Tuple[str,str,str]]
+        :type citations: bool, default is False.
+        :return: The content of the document that matched your query,
+        along with url of the source and doc_id (if citations flag is true)
+        :rtype: List[str], if citations=False, otherwise List[Tuple[str, str, str]]
         """
         if skip_embedding:
             query_vector = input_query
@@ -175,14 +183,17 @@ class ElasticsearchDB(BaseVectorDB):
         _source = ["text", "metadata.url", "metadata.doc_id"]
         response = self.client.search(index=self._get_index(), query=query, _source=_source, size=n_results)
         docs = response["hits"]["hits"]
-        contents = []
+        contexts = []
         for doc in docs:
             context = doc["_source"]["text"]
-            metadata = doc["_source"]["metadata"]
+            if citations:
-            source = metadata["url"]
+                metadata = doc["_source"]["metadata"]
-            doc_id = metadata["doc_id"]
+                source = metadata["url"]
-            contents.append(tuple((context, source, doc_id)))
+                doc_id = metadata["doc_id"]
-        return contents
+                contexts.append(tuple((context, source, doc_id)))
+            else:
+                contexts.append(context)
+        return contexts
 
     def set_collection_name(self, name: str):
         """

embedchain/vectordb/opensearch.py

@@ -1,5 +1,5 @@
 import logging
-from typing import Dict, List, Optional, Set, Tuple
+from typing import Dict, List, Optional, Set, Tuple, Union
 
 try:
     from opensearchpy import OpenSearch
@@ -146,8 +146,13 @@ class OpenSearchDB(BaseVectorDB):
         self.client.indices.refresh(index=self._get_index())
 
     def query(
-        self, input_query: List[str], n_results: int, where: Dict[str, any], skip_embedding: bool
+        self,
-    ) -> List[Tuple[str, str, str]]:
+        input_query: List[str],
+        n_results: int,
+        where: Dict[str, any],
+        skip_embedding: bool,
+        citations: bool = False,
+    ) -> Union[List[Tuple[str, str, str]], List[str]]:
         """
         query contents from vector data base based on vector similarity
 
@@ -159,8 +164,11 @@ class OpenSearchDB(BaseVectorDB):
         :type where: Dict[str, any]
         :param skip_embedding: Optional. If True, then the input_query is assumed to be already embedded.
         :type skip_embedding: bool
-        :return: The content of the document that matched your query, url of the source, doc_id
+        :param citations: we use citations boolean param to return context along with the answer.
-        :rtype: List[Tuple[str,str,str]]
+        :type citations: bool, default is False.
+        :return: The content of the document that matched your query,
+        along with url of the source and doc_id (if citations flag is true)
+        :rtype: List[str], if citations=False, otherwise List[Tuple[str, str, str]]
         """
         # TODO(rupeshbansal, deshraj): Add support for skip embeddings here if already exists
         embeddings = OpenAIEmbeddings()
@@ -188,13 +196,16 @@ class OpenSearchDB(BaseVectorDB):
             k=n_results,
         )
 
-        contents = []
+        contexts = []
         for doc in docs:
             context = doc.page_content
-            source = doc.metadata["url"]
+            if citations:
-            doc_id = doc.metadata["doc_id"]
+                source = doc.metadata["url"]
-            contents.append(tuple((context, source, doc_id)))
+                doc_id = doc.metadata["doc_id"]
-        return contents
+                contexts.append(tuple((context, source, doc_id)))
+            else:
+                contexts.append(context)
+        return contexts
 
     def set_collection_name(self, name: str):
         """

embedchain/vectordb/pinecone.py

@@ -1,5 +1,5 @@
 import os
-from typing import Dict, List, Optional, Tuple
+from typing import Dict, List, Optional, Tuple, Union
 
 try:
     import pinecone
@@ -119,8 +119,13 @@ class PineconeDB(BaseVectorDB):
             self.client.upsert(docs[i : i + self.BATCH_SIZE])
 
     def query(
-        self, input_query: List[str], n_results: int, where: Dict[str, any], skip_embedding: bool
+        self,
-    ) -> List[Tuple[str, str, str]]:
+        input_query: List[str],
+        n_results: int,
+        where: Dict[str, any],
+        skip_embedding: bool,
+        citations: bool = False,
+    ) -> Union[List[Tuple[str, str, str]], List[str]]:
         """
         query contents from vector database based on vector similarity
         :param input_query: list of query string
@@ -131,22 +136,28 @@ class PineconeDB(BaseVectorDB):
         :type where: Dict[str, any]
         :param skip_embedding: Optional. if True, input_query is already embedded
         :type skip_embedding: bool
-        :return: The content of the document that matched your query, url of the source, doc_id
+        :param citations: we use citations boolean param to return context along with the answer.
-        :rtype: List[Tuple[str,str,str]]
+        :type citations: bool, default is False.
+        :return: The content of the document that matched your query,
+        along with url of the source and doc_id (if citations flag is true)
+        :rtype: List[str], if citations=False, otherwise List[Tuple[str, str, str]]
         """
         if not skip_embedding:
             query_vector = self.embedder.embedding_fn([input_query])[0]
         else:
             query_vector = input_query
         data = self.client.query(vector=query_vector, filter=where, top_k=n_results, include_metadata=True)
-        contents = []
+        contexts = []
         for doc in data["matches"]:
             metadata = doc["metadata"]
             context = metadata["text"]
-            source = metadata["url"]
+            if citations:
-            doc_id = metadata["doc_id"]
+                source = metadata["url"]
-            contents.append(tuple((context, source, doc_id)))
+                doc_id = metadata["doc_id"]
-        return contents
+                contexts.append(tuple((context, source, doc_id)))
+            else:
+                contexts.append(context)
+        return contexts
 
     def set_collection_name(self, name: str):
         """

embedchain/vectordb/qdrant.py

@@ -1,7 +1,7 @@
 import copy
 import os
 import uuid
-from typing import Dict, List, Optional, Tuple
+from typing import Dict, List, Optional, Tuple, Union
 
 try:
     from qdrant_client import QdrantClient
@@ -161,8 +161,13 @@ class QdrantDB(BaseVectorDB):
             )
 
     def query(
-        self, input_query: List[str], n_results: int, where: Dict[str, any], skip_embedding: bool
+        self,
-    ) -> List[Tuple[str, str, str]]:
+        input_query: List[str],
+        n_results: int,
+        where: Dict[str, any],
+        skip_embedding: bool,
+        citations: bool = False,
+    ) -> Union[List[Tuple[str, str, str]], List[str]]:
         """
         query contents from vector database based on vector similarity
         :param input_query: list of query string
@@ -174,8 +179,11 @@ class QdrantDB(BaseVectorDB):
         :param skip_embedding: A boolean flag indicating if the embedding for the documents to be added is to be
         generated or not
         :type skip_embedding: bool
-        :return: The context of the document that matched your query, url of the source, doc_id
+        :param citations: we use citations boolean param to return context along with the answer.
-        :rtype: List[Tuple[str,str,str]]
+        :type citations: bool, default is False.
+        :return: The content of the document that matched your query,
+        along with url of the source and doc_id (if citations flag is true)
+        :rtype: List[str], if citations=False, otherwise List[Tuple[str, str, str]]
         """
         if not skip_embedding:
             query_vector = self.embedder.embedding_fn([input_query])[0]
@@ -202,14 +210,17 @@ class QdrantDB(BaseVectorDB):
             limit=n_results,
         )
 
-        response = []
+        contexts = []
         for result in results:
             context = result.payload["text"]
-            metadata = result.payload["metadata"]
+            if citations:
-            source = metadata["url"]
+                metadata = result.payload["metadata"]
-            doc_id = metadata["doc_id"]
+                source = metadata["url"]
-            response.append(tuple((context, source, doc_id)))
+                doc_id = metadata["doc_id"]
-        return response
+                contexts.append(tuple((context, source, doc_id)))
+            else:
+                contexts.append(context)
+        return contexts
 
     def count(self) -> int:
         response = self.client.get_collection(collection_name=self.collection_name)

embedchain/vectordb/weaviate.py

@@ -1,6 +1,6 @@
 import copy
 import os
-from typing import Dict, List, Optional, Tuple
+from typing import Dict, List, Optional, Tuple, Union
 
 try:
     import weaviate
@@ -58,10 +58,14 @@ class WeaviateDB(BaseVectorDB):
             raise ValueError("Embedder not set. Please set an embedder with `set_embedder` before initialization.")
 
         self.index_name = self._get_index_name()
-        self.metadata_keys = {"data_type", "doc_id", "url", "hash", "app_id", "text"}
+        self.metadata_keys = {"data_type", "doc_id", "url", "hash", "app_id"}
         if not self.client.schema.exists(self.index_name):
             # id is a reserved field in Weaviate, hence we had to change the name of the id field to identifier
             # The none vectorizer is crucial as we have our own custom embedding function
+            """
+            TODO: wait for weaviate to add indexing on `object[]` data-type so that we can add filter while querying.
+            Once that is done, change `dataType` of "metadata" field to `object[]` and update the query below.
+            """
             class_obj = {
                 "classes": [
                     {
@@ -106,10 +110,6 @@ class WeaviateDB(BaseVectorDB):
                                 "name": "app_id",
                                 "dataType": ["text"],
                             },
-                            {
-                                "name": "text",
-                                "dataType": ["text"],
-                            },
                         ],
                     },
                 ]
@@ -195,8 +195,13 @@ class WeaviateDB(BaseVectorDB):
                 batch.add_reference(obj_uuid, self.index_name, "metadata", metadata_uuid, self.index_name + "_metadata")
 
     def query(
-        self, input_query: List[str], n_results: int, where: Dict[str, any], skip_embedding: bool
+        self,
-    ) -> List[Tuple[str, str, str]]:
+        input_query: List[str],
+        n_results: int,
+        where: Dict[str, any],
+        skip_embedding: bool,
+        citations: bool = False,
+    ) -> Union[List[Tuple[str, str, str]], List[str]]:
         """
         query contents from vector database based on vector similarity
         :param input_query: list of query string
@@ -208,15 +213,23 @@ class WeaviateDB(BaseVectorDB):
         :param skip_embedding: A boolean flag indicating if the embedding for the documents to be added is to be
         generated or not
         :type skip_embedding: bool
-        :return: The context of the document that matched your query, url of the source, doc_id
+        :param citations: we use citations boolean param to return context along with the answer.
-        :rtype: List[Tuple[str,str,str]]
+        :type citations: bool, default is False.
+        :return: The content of the document that matched your query,
+        along with url of the source and doc_id (if citations flag is true)
+        :rtype: List[str], if citations=False, otherwise List[Tuple[str, str, str]]
         """
         if not skip_embedding:
             query_vector = self.embedder.embedding_fn([input_query])[0]
         else:
             query_vector = input_query
+
         keys = set(where.keys() if where is not None else set())
         data_fields = ["text"]
+
+        if citations:
+            data_fields.append(weaviate.LinkTo("metadata", self.index_name + "_metadata", list(self.metadata_keys)))
+
         if len(keys.intersection(self.metadata_keys)) != 0:
             weaviate_where_operands = []
             for key in keys:
@@ -247,7 +260,18 @@ class WeaviateDB(BaseVectorDB):
                 .with_limit(n_results)
                 .do()
             )
-        contexts = results["data"]["Get"].get(self.index_name)
+
+        docs = results["data"]["Get"].get(self.index_name)
+        contexts = []
+        for doc in docs:
+            context = doc["text"]
+            if citations:
+                metadata = doc["metadata"][0]
+                source = metadata["url"]
+                doc_id = metadata["doc_id"]
+                contexts.append((context, source, doc_id))
+            else:
+                contexts.append(context)
         return contexts
 
     def set_collection_name(self, name: str):

embedchain/vectordb/zilliz.py

@@ -1,5 +1,5 @@
 import logging
-from typing import Dict, List, Optional, Tuple
+from typing import Dict, List, Optional, Tuple, Union
 
 from embedchain.config import ZillizDBConfig
 from embedchain.helper.json_serializable import register_deserializable
@@ -127,8 +127,13 @@ class ZillizVectorDB(BaseVectorDB):
         self.client.flush(self.config.collection_name)
 
     def query(
-        self, input_query: List[str], n_results: int, where: Dict[str, any], skip_embedding: bool
+        self,
-    ) -> List[Tuple[str, str, str]]:
+        input_query: List[str],
+        n_results: int,
+        where: Dict[str, any],
+        skip_embedding: bool,
+        citations: bool = False,
+    ) -> Union[List[Tuple[str, str, str]], List[str]]:
         """
         Query contents from vector data base based on vector similarity
 
@@ -139,8 +144,11 @@ class ZillizVectorDB(BaseVectorDB):
         :param where: to filter data
         :type where: str
         :raises InvalidDimensionException: Dimensions do not match.
-        :return: The context of the document that matched your query, url of the source, doc_id
+        :param citations: we use citations boolean param to return context along with the answer.
-        :rtype: List[Tuple[str,str,str]]
+        :type citations: bool, default is False.
+        :return: The content of the document that matched your query,
+        along with url of the source and doc_id (if citations flag is true)
+        :rtype: List[str], if citations=False, otherwise List[Tuple[str, str, str]]
         """
 
         if self.collection.is_empty:
@@ -170,14 +178,17 @@ class ZillizVectorDB(BaseVectorDB):
                 output_fields=output_fields,
             )
 
-        doc_list = []
+        contexts = []
         for query in query_result:
             data = query[0]["entity"]
             context = data["text"]
-            source = data["url"]
+            if citations:
-            doc_id = data["doc_id"]
+                source = data["url"]
-            doc_list.append(tuple((context, source, doc_id)))
+                doc_id = data["doc_id"]
-        return doc_list
+                contexts.append(tuple((context, source, doc_id)))
+            else:
+                contexts.append(context)
+        return contexts
 
     def count(self) -> int:
         """

examples/full_stack/frontend/public/icons/twitter.svg

@@ -1,24 +1,10 @@
-<svg
+<?xml version="1.0" encoding="utf-8"?>
-  fill="currentColor"
+<!-- Generator: Adobe Illustrator 27.5.0, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
-  version="1.1"
+<svg version="1.1" id="svg5" xmlns:svg="http://www.w3.org/2000/svg"
-  xmlns="http://www.w3.org/2000/svg"
+	 xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" viewBox="0 0 1668.56 1221.19"
-  xmlns:xlink="http://www.w3.org/1999/xlink"
+	 style="enable-background:new 0 0 1668.56 1221.19;" xml:space="preserve">
-  viewBox="0 0 512 512"
+<g id="layer1" transform="translate(52.390088,-25.058597)">
-  xml:space="preserve"
+	<path id="path1009" d="M283.94,167.31l386.39,516.64L281.5,1104h87.51l340.42-367.76L984.48,1104h297.8L874.15,558.3l361.92-390.99
->
+		h-87.51l-313.51,338.7l-253.31-338.7H283.94z M412.63,231.77h136.81l604.13,807.76h-136.81L412.63,231.77z"/>
-  <g id="SVGRepo_bgCarrier" stroke-width="0"></g>
+</g>
-  <g
+</svg>
-    id="SVGRepo_tracerCarrier"
-    stroke-linecap="round"
-    stroke-linejoin="round"
-  ></g>
-  <g id="SVGRepo_iconCarrier">
-    <g id="7935ec95c421cee6d86eb22ecd12f847">
-      <path
-        style="display: inline;"
-        d="M459.186,151.787c0.203,4.501,0.305,9.023,0.305,13.565 c0,138.542-105.461,298.285-298.274,298.285c-59.209,0-114.322-17.357-160.716-47.104c8.212,0.973,16.546,1.47,25.012,1.47 c49.121,0,94.318-16.759,130.209-44.884c-45.887-0.841-84.596-31.154-97.938-72.804c6.408,1.227,12.968,1.886,19.73,1.886 c9.55,0,18.816-1.287,27.617-3.68c-47.955-9.633-84.1-52.001-84.1-102.795c0-0.446,0-0.882,0.011-1.318 c14.133,7.847,30.294,12.562,47.488,13.109c-28.134-18.796-46.637-50.885-46.637-87.262c0-19.212,5.16-37.218,14.193-52.7 c51.707,63.426,128.941,105.156,216.072,109.536c-1.784-7.675-2.718-15.674-2.718-23.896c0-57.891,46.941-104.832,104.832-104.832 c30.173,0,57.404,12.734,76.525,33.102c23.887-4.694,46.313-13.423,66.569-25.438c-7.827,24.485-24.434,45.025-46.089,58.002 c21.209-2.535,41.426-8.171,60.222-16.505C497.448,118.542,479.666,137.004,459.186,151.787z"
-      > 
-      </path>
-    </g>
-  </g>
-</svg>

notebooks/anthropic.ipynb

@@ -34,6 +34,15 @@
         "!pip install embedchain"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -54,7 +63,7 @@
       "outputs": [],
       "source": [
         "import os\n",
-        "from embedchain import App\n",
+        "from embedchain import Pipeline as App\n",
         "\n",
         "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\"\n",
         "os.environ[\"ANTHROPIC_API_KEY\"] = \"xxx\""

notebooks/azure-openai.ipynb

@@ -26,6 +26,16 @@
     "!pip install embedchain"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "692ff37b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install embedchain[dataloaders]"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "ac982a56",
@@ -44,7 +54,7 @@
    "outputs": [],
    "source": [
     "import os\n",
-    "from embedchain import App\n",
+    "from embedchain import Pipeline as App\n",
     "\n",
     "os.environ[\"OPENAI_API_TYPE\"] = \"azure\"\n",
     "os.environ[\"OPENAI_API_BASE\"] = \"https://xxx.openai.azure.com/\"\n",

notebooks/chromadb.ipynb

@@ -1,84 +1,84 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
       "cell_type": "markdown",
-      "source": [
-        "## Cookbook for using ChromaDB with Embedchain"
-      ],
       "metadata": {
         "id": "b02n_zJ_hl3d"
-      }
+      },
+      "source": [
+        "## Cookbook for using ChromaDB with Embedchain"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-1: Install embedchain package"
-      ],
       "metadata": {
         "id": "gyJ6ui2vhtMY"
-      }
+      },
+      "source": [
+        "### Step-1: Install embedchain package"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "!pip install embedchain"
-      ],
       "metadata": {
         "id": "-NbXjAdlh0vJ"
       },
+      "outputs": [],
+      "source": [
+        "!pip install embedchain"
+      ]
+    },
+    {
+      "cell_type": "code",
       "execution_count": null,
-      "outputs": []
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "nGnpSYAAh2bQ"
+      },
       "source": [
         "### Step-2: Set OpenAI environment variables\n",
         "\n",
         "You can find this env variable on your [OpenAI dashboard](https://platform.openai.com/account/api-keys)."
-      ],
+      ]
-      "metadata": {
-        "id": "nGnpSYAAh2bQ"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "import os\n",
-        "from embedchain import App\n",
-        "\n",
-        "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\""
-      ],
       "metadata": {
         "id": "0fBdQ9GAiRvK"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "import os\n",
+        "from embedchain import Pipeline as App\n",
+        "\n",
+        "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\""
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-3: Define your Vector Database config"
-      ],
       "metadata": {
         "id": "Ns6RhPfbiitr"
-      }
+      },
+      "source": [
+        "### Step-3: Define your Vector Database config"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "S9CkxVjriotB"
+      },
+      "outputs": [],
       "source": [
         "config = \"\"\"\n",
         "vectordb:\n",
@@ -95,64 +95,64 @@
         "# Write the multi-line string to a YAML file\n",
         "with open('chromadb.yaml', 'w') as file:\n",
         "    file.write(config)"
-      ],
+      ]
-      "metadata": {
-        "id": "S9CkxVjriotB"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-4 Create embedchain app based on the config"
-      ],
       "metadata": {
         "id": "PGt6uPLIi1CS"
-      }
+      },
+      "source": [
+        "### Step-4 Create embedchain app based on the config"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "app = App.from_config(yaml_path=\"chromadb.yaml\")"
-      ],
       "metadata": {
         "id": "Amzxk3m-i3tD"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "app = App.from_config(yaml_path=\"chromadb.yaml\")"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-5: Add data sources to your app"
-      ],
       "metadata": {
         "id": "XNXv4yZwi7ef"
-      }
+      },
+      "source": [
+        "### Step-5: Add data sources to your app"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "app.add(\"https://www.forbes.com/profile/elon-musk\")"
-      ],
       "metadata": {
         "id": "Sn_0rx9QjIY9"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "app.add(\"https://www.forbes.com/profile/elon-musk\")"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-6: All set. Now start asking questions related to your data"
-      ],
       "metadata": {
         "id": "_7W6fDeAjMAP"
-      }
+      },
+      "source": [
+        "### Step-6: All set. Now start asking questions related to your data"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cvIK7dWRjN_f"
+      },
+      "outputs": [],
       "source": [
         "while(True):\n",
         "    question = input(\"Enter question: \")\n",
@@ -160,12 +160,21 @@
         "        break\n",
         "    answer = app.query(question)\n",
         "    print(answer)"
-      ],
+      ]
-      "metadata": {
-        "id": "cvIK7dWRjN_f"
-      },
-      "execution_count": null,
-      "outputs": []
     }
-  ]
+  ],
-}
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

notebooks/cohere.ipynb

@@ -33,6 +33,15 @@
         "!pip install embedchain"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -69,7 +78,7 @@
       "outputs": [],
       "source": [
         "import os\n",
-        "from embedchain import App\n",
+        "from embedchain import Pipeline as App\n",
         "\n",
         "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\"\n",
         "os.environ[\"COHERE_API_KEY\"] = \"xxx\""

notebooks/elasticsearch.ipynb

@@ -1,95 +1,95 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
       "cell_type": "markdown",
-      "source": [
-        "## Cookbook for using ElasticSearchDB with Embedchain"
-      ],
       "metadata": {
         "id": "b02n_zJ_hl3d"
-      }
+      },
+      "source": [
+        "## Cookbook for using ElasticSearchDB with Embedchain"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-1: Install embedchain package"
-      ],
       "metadata": {
         "id": "gyJ6ui2vhtMY"
-      }
+      },
+      "source": [
+        "### Step-1: Install embedchain package"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "!pip install embedchain"
-      ],
       "metadata": {
         "id": "-NbXjAdlh0vJ"
       },
+      "outputs": [],
+      "source": [
+        "!pip install embedchain"
+      ]
+    },
+    {
+      "cell_type": "code",
       "execution_count": null,
-      "outputs": []
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "nGnpSYAAh2bQ"
+      },
       "source": [
         "### Step-2: Set OpenAI environment variables and install the dependencies.\n",
         "\n",
         "You can find this env variable on your [OpenAI dashboard](https://platform.openai.com/account/api-keys). Now lets install the dependencies needed for Elasticsearch."
-      ],
+      ]
-      "metadata": {
-        "id": "nGnpSYAAh2bQ"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "!pip install --upgrade 'embedchain[elasticsearch]'"
-      ],
       "metadata": {
         "id": "-MUFRfxV7Jk7"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "!pip install --upgrade 'embedchain[elasticsearch]'"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "import os\n",
-        "from embedchain import App\n",
-        "\n",
-        "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\""
-      ],
       "metadata": {
         "id": "0fBdQ9GAiRvK"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "import os\n",
+        "from embedchain import Pipeline as App\n",
+        "\n",
+        "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\""
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-3: Define your Vector Database config"
-      ],
       "metadata": {
         "id": "Ns6RhPfbiitr"
-      }
+      },
+      "source": [
+        "### Step-3: Define your Vector Database config"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "S9CkxVjriotB"
+      },
+      "outputs": [],
       "source": [
         "config = \"\"\"\n",
         "vectordb:\n",
@@ -104,64 +104,64 @@
         "# Write the multi-line string to a YAML file\n",
         "with open('elasticsearch.yaml', 'w') as file:\n",
         "    file.write(config)"
-      ],
+      ]
-      "metadata": {
-        "id": "S9CkxVjriotB"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-4 Create embedchain app based on the config"
-      ],
       "metadata": {
         "id": "PGt6uPLIi1CS"
-      }
+      },
+      "source": [
+        "### Step-4 Create embedchain app based on the config"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "app = App.from_config(yaml_path=\"elasticsearch.yaml\")"
-      ],
       "metadata": {
         "id": "Amzxk3m-i3tD"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "app = App.from_config(yaml_path=\"elasticsearch.yaml\")"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-5: Add data sources to your app"
-      ],
       "metadata": {
         "id": "XNXv4yZwi7ef"
-      }
+      },
+      "source": [
+        "### Step-5: Add data sources to your app"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "app.add(\"https://www.forbes.com/profile/elon-musk\")"
-      ],
       "metadata": {
         "id": "Sn_0rx9QjIY9"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "app.add(\"https://www.forbes.com/profile/elon-musk\")"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-6: All set. Now start asking questions related to your data"
-      ],
       "metadata": {
         "id": "_7W6fDeAjMAP"
-      }
+      },
+      "source": [
+        "### Step-6: All set. Now start asking questions related to your data"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cvIK7dWRjN_f"
+      },
+      "outputs": [],
       "source": [
         "while(True):\n",
         "    question = input(\"Enter question: \")\n",
@@ -169,12 +169,21 @@
         "        break\n",
         "    answer = app.query(question)\n",
         "    print(answer)"
-      ],
+      ]
-      "metadata": {
-        "id": "cvIK7dWRjN_f"
-      },
-      "execution_count": null,
-      "outputs": []
     }
-  ]
+  ],
-}
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

notebooks/embedchain-chromadb-server.ipynb

@@ -33,7 +33,7 @@
    "outputs": [],
    "source": [
     "import os\n",
-    "from embedchain import App\n",
+    "from embedchain import Pipeline as App\n",
     "from embedchain.config import AppConfig\n",
     "\n",
     "\n",

notebooks/embedchain-docs-site-example.ipynb

@@ -7,7 +7,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from embedchain import App\n",
+    "from embedchain import Pipeline as App\n",
     "\n",
     "embedchain_docs_bot = App()"
    ]

notebooks/gpt4all.ipynb

@@ -33,6 +33,15 @@
         "!pip install embedchain"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -67,7 +76,7 @@
       },
       "outputs": [],
       "source": [
-        "from embedchain import App"
+        "from embedchain import Pipeline as App"
       ]
     },
     {

notebooks/hugging_face_hub.ipynb

@@ -34,6 +34,15 @@
         "!pip install embedchain"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -84,7 +93,7 @@
       "outputs": [],
       "source": [
         "import os\n",
-        "from embedchain import App\n",
+        "from embedchain import Pipeline as App\n",
         "\n",
         "os.environ[\"HUGGINGFACE_ACCESS_TOKEN\"] = \"hf_xxx\""
       ]

notebooks/jina.ipynb

@@ -34,6 +34,15 @@
         "!pip install embedchain"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -54,7 +63,7 @@
       "outputs": [],
       "source": [
         "import os\n",
-        "from embedchain import App\n",
+        "from embedchain import Pipeline as App\n",
         "\n",
         "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\"\n",
         "os.environ[\"JINACHAT_API_KEY\"] = \"xxx\""

notebooks/llama2.ipynb

@@ -33,6 +33,15 @@
         "!pip install embedchain"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -64,7 +73,7 @@
       "outputs": [],
       "source": [
         "import os\n",
-        "from embedchain import App\n",
+        "from embedchain import Pipeline as App\n",
         "\n",
         "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\"\n",
         "os.environ[\"REPLICATE_API_TOKEN\"] = \"xxx\""

notebooks/openai.ipynb

@@ -34,6 +34,15 @@
         "!pip install embedchain"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -54,7 +63,7 @@
       "outputs": [],
       "source": [
         "import os\n",
-        "from embedchain import App\n",
+        "from embedchain import Pipeline as App\n",
         "\n",
         "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\""
       ]

notebooks/opensearch.ipynb

@@ -1,95 +1,95 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": []
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
       "cell_type": "markdown",
-      "source": [
-        "## Cookbook for using OpenSearchDB with Embedchain"
-      ],
       "metadata": {
         "id": "b02n_zJ_hl3d"
-      }
+      },
+      "source": [
+        "## Cookbook for using OpenSearchDB with Embedchain"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-1: Install embedchain package"
-      ],
       "metadata": {
         "id": "gyJ6ui2vhtMY"
-      }
+      },
+      "source": [
+        "### Step-1: Install embedchain package"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "!pip install embedchain"
-      ],
       "metadata": {
         "id": "-NbXjAdlh0vJ"
       },
+      "outputs": [],
+      "source": [
+        "!pip install embedchain"
+      ]
+    },
+    {
+      "cell_type": "code",
       "execution_count": null,
-      "outputs": []
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "nGnpSYAAh2bQ"
+      },
       "source": [
         "### Step-2: Set OpenAI environment variables and install the dependencies.\n",
         "\n",
         "You can find this env variable on your [OpenAI dashboard](https://platform.openai.com/account/api-keys). Now lets install the dependencies needed for Opensearch."
-      ],
+      ]
-      "metadata": {
-        "id": "nGnpSYAAh2bQ"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "!pip install --upgrade 'embedchain[opensearch]'"
-      ],
       "metadata": {
         "id": "-MUFRfxV7Jk7"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "!pip install --upgrade 'embedchain[opensearch]'"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "import os\n",
-        "from embedchain import App\n",
-        "\n",
-        "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\""
-      ],
       "metadata": {
         "id": "0fBdQ9GAiRvK"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "import os\n",
+        "from embedchain import Pipeline as App\n",
+        "\n",
+        "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\""
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-3: Define your Vector Database config"
-      ],
       "metadata": {
         "id": "Ns6RhPfbiitr"
-      }
+      },
+      "source": [
+        "### Step-3: Define your Vector Database config"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "S9CkxVjriotB"
+      },
+      "outputs": [],
       "source": [
         "config = \"\"\"\n",
         "vectordb:\n",
@@ -108,64 +108,64 @@
         "# Write the multi-line string to a YAML file\n",
         "with open('opensearch.yaml', 'w') as file:\n",
         "    file.write(config)"
-      ],
+      ]
-      "metadata": {
-        "id": "S9CkxVjriotB"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-4 Create embedchain app based on the config"
-      ],
       "metadata": {
         "id": "PGt6uPLIi1CS"
-      }
+      },
+      "source": [
+        "### Step-4 Create embedchain app based on the config"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "app = App.from_config(yaml_path=\"opensearch.yaml\")"
-      ],
       "metadata": {
         "id": "Amzxk3m-i3tD"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "app = App.from_config(yaml_path=\"opensearch.yaml\")"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-5: Add data sources to your app"
-      ],
       "metadata": {
         "id": "XNXv4yZwi7ef"
-      }
+      },
+      "source": [
+        "### Step-5: Add data sources to your app"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "app.add(\"https://www.forbes.com/profile/elon-musk\")"
-      ],
       "metadata": {
         "id": "Sn_0rx9QjIY9"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "app.add(\"https://www.forbes.com/profile/elon-musk\")"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "### Step-6: All set. Now start asking questions related to your data"
-      ],
       "metadata": {
         "id": "_7W6fDeAjMAP"
-      }
+      },
+      "source": [
+        "### Step-6: All set. Now start asking questions related to your data"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cvIK7dWRjN_f"
+      },
+      "outputs": [],
       "source": [
         "while(True):\n",
         "    question = input(\"Enter question: \")\n",
@@ -173,12 +173,21 @@
         "        break\n",
         "    answer = app.query(question)\n",
         "    print(answer)"
-      ],
+      ]
-      "metadata": {
-        "id": "cvIK7dWRjN_f"
-      },
-      "execution_count": null,
-      "outputs": []
     }
-  ]
+  ],
-}
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

notebooks/pinecone.ipynb

@@ -29,6 +29,15 @@
         "!pip install embedchain"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -60,7 +69,7 @@
       "outputs": [],
       "source": [
         "import os\n",
-        "from embedchain import App\n",
+        "from embedchain import Pipeline as App\n",
         "\n",
         "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\"\n",
         "os.environ[\"PINECONE_API_KEY\"] = \"xxx\"\n",

notebooks/vertex_ai.ipynb

@@ -33,6 +33,15 @@
         "!pip install embedchain"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install embedchain[dataloaders]"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -64,7 +73,7 @@
       "outputs": [],
       "source": [
         "import os\n",
-        "from embedchain import App\n",
+        "from embedchain import Pipeline as App\n",
         "\n",
         "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\""
       ]

poetry.lock

@@ -2538,86 +2538,28 @@ files = [
 ]
 
 [[package]]
-name = "jq"
+name = "jsonpatch"
-version = "1.6.0"
+version = "1.33"
-description = "jq is a lightweight and flexible JSON processor."
+description = "Apply JSON-Patches (RFC 6902)"
-optional = true
+optional = false
-python-versions = ">=3.5"
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*"
 files = [
-    {file = "jq-1.6.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:5773851cfb9ec6525f362f5bf7f18adab5c1fd1f0161c3599264cd0118c799da"},
+    {file = "jsonpatch-1.33-py2.py3-none-any.whl", hash = "sha256:0ae28c0cd062bbd8b8ecc26d7d164fbbea9652a1a3693f3b956c1eae5145dade"},
-    {file = "jq-1.6.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a758df4eae767a21ebd8466dfd0066d99c9741d9f7fd4a7e1d5b5227e1924af7"},
+    {file = "jsonpatch-1.33.tar.gz", hash = "sha256:9fcd4009c41e6d12348b4a0ff2563ba56a2923a7dfee731d004e212e1ee5030c"},
-    {file = "jq-1.6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:15cf9dd3e7fb40d029f12f60cf418374c0b830a6ea6267dd285b48809069d6af"},
+]
-    {file = "jq-1.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c7e768cf5c25d703d944ef81c787d745da0eb266a97768f3003f91c4c828118d"},
+
-    {file = "jq-1.6.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:85a697b3cdc65e787f90faa1237caa44c117b6b2853f21263c3f0b16661b192c"},
+[package.dependencies]
-    {file = "jq-1.6.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:944e081c328501ddc0a22a8f08196df72afe7910ca11e1a1f21244410dbdd3b3"},
+jsonpointer = ">=1.9"
-    {file = "jq-1.6.0-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:09262d0e0cafb03acc968622e6450bb08abfb14c793bab47afd2732b47c655fd"},
+
-    {file = "jq-1.6.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:611f460f616f957d57e0da52ac6e1e6294b073c72a89651da5546a31347817bd"},
+[[package]]
-    {file = "jq-1.6.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:aba35b5cc07cd75202148e55f47ede3f4d0819b51c80f6d0c82a2ca47db07189"},
+name = "jsonpointer"
-    {file = "jq-1.6.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:ef5ddb76b03610df19a53583348aed3604f21d0ba6b583ee8d079e8df026cd47"},
+version = "2.4"
-    {file = "jq-1.6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:872f322ff7bfd7daff41b7e8248d414a88722df0e82d1027f3b091a438543e63"},
+description = "Identify specific nodes in a JSON document (RFC 6901)"
-    {file = "jq-1.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ca7a2982ff26f4620ac03099542a0230dabd8787af3f03ac93660598e26acbf0"},
+optional = false
-    {file = "jq-1.6.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:316affc6debf15eb05b7fd8e84ebf8993042b10b840e8d2a504659fb3ba07992"},
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*"
-    {file = "jq-1.6.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:9bc42ade4de77fe4370c0e8e105ef10ad1821ef74d61dcc70982178b9ecfdc72"},
+files = [
-    {file = "jq-1.6.0-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:02da59230912b886ed45489f3693ce75877f3e99c9e490c0a2dbcf0db397e0df"},
+    {file = "jsonpointer-2.4-py2.py3-none-any.whl", hash = "sha256:15d51bba20eea3165644553647711d150376234112651b4f1811022aecad7d7a"},
-    {file = "jq-1.6.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:7ea39f89aa469eb12145ddd686248916cd6d186647aa40b319af8444b1f45a2d"},
+    {file = "jsonpointer-2.4.tar.gz", hash = "sha256:585cee82b70211fa9e6043b7bb89db6e1aa49524340dde8ad6b63206ea689d88"},
-    {file = "jq-1.6.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:6e9016f5ba064fabc527adb609ebae1f27cac20c8e0da990abae1cfb12eca706"},
-    {file = "jq-1.6.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:022be104a548f7fbddf103ce749937956df9d37a4f2f1650396dacad73bce7ee"},
-    {file = "jq-1.6.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1d5a7f31f779e1aa3d165eaec237d74c7f5728227e81023a576c939ba3da34f8"},
-    {file = "jq-1.6.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5f1533a2a15c42be3368878b4031b12f30441246878e0b5f6bedfdd7828cdb1f"},
-    {file = "jq-1.6.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:8aa67a304e58aa85c550ec011a68754ae49abe227b37d63a351feef4eea4c7a7"},
-    {file = "jq-1.6.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:0893d1590cfa6facaf787cc6c28ac51e47d0d06a303613f84d4943ac0ca98e32"},
-    {file = "jq-1.6.0-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:63db80b4803905a4f4f6c87a17aa1816c530f6262bc795773ebe60f8ab259092"},
-    {file = "jq-1.6.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:e2c1f429e644cb962e846a6157b5352c3c556fbd0b22bba9fc2fea0710333369"},
-    {file = "jq-1.6.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:bcf574f28809ec63b8df6456fdd4a981751b7466851e80621993b4e9d3e3c8ee"},
-    {file = "jq-1.6.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:49dbe0f003b411ca52b5d0afaf09cad8e430a1011181c86f2ef720a0956f31c1"},
-    {file = "jq-1.6.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9f5a9c4185269a5faf395aa7ca086c7b02c9c8b448d542be3b899041d06e0970"},
-    {file = "jq-1.6.0-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:8265f3badcd125f234e55dfc02a078c5decdc6faafcd453fde04d4c0d2699886"},
-    {file = "jq-1.6.0-cp36-cp36m-musllinux_1_1_aarch64.whl", hash = "sha256:c6c39b53d000d2f7f9f6338061942b83c9034d04f3bc99acae0867d23c9e7127"},
-    {file = "jq-1.6.0-cp36-cp36m-musllinux_1_1_i686.whl", hash = "sha256:9897931ea7b9a46f8165ee69737ece4a2e6dbc8e10ececb81f459d51d71401df"},
-    {file = "jq-1.6.0-cp36-cp36m-musllinux_1_1_x86_64.whl", hash = "sha256:6312237159e88e92775ea497e0c739590528062d4074544aacf12a08d252f966"},
-    {file = "jq-1.6.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:aa786a60bdd1a3571f092a4021dd9abf6c46798530fa99f19ecf4f0fceaa7eaf"},
-    {file = "jq-1.6.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:22495573d8221320d3433e1aeded40132bd8e1726845629558bd73aaa66eef7b"},
-    {file = "jq-1.6.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:711eabc5d33ef3ec581e0744d9cff52f43896d84847a2692c287a0140a29c915"},
-    {file = "jq-1.6.0-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:57e75c1563d083b0424690b3c3ef2bb519e670770931fe633101ede16615d6ee"},
-    {file = "jq-1.6.0-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:c795f175b1a13bd716a0c180d062cc8e305271f47bbdb9eb0f0f62f7e4f5def4"},
-    {file = "jq-1.6.0-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:227b178b22a7f91ae88525810441791b1ca1fc71c86f03190911793be15cec3d"},
-    {file = "jq-1.6.0-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:780eb6383fbae12afa819ef676fc93e1548ae4b076c004a393af26a04b460742"},
-    {file = "jq-1.6.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:08ded6467f4ef89fec35b2bf310f210f8cd13fbd9d80e521500889edf8d22441"},
-    {file = "jq-1.6.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:49e44ed677713f4115bd5bf2dbae23baa4cd503be350e12a1c1f506b0687848f"},
-    {file = "jq-1.6.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:984f33862af285ad3e41e23179ac4795f1701822473e1a26bf87ff023e5a89ea"},
-    {file = "jq-1.6.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f42264fafc6166efb5611b5d4cb01058887d050a6c19334f6a3f8a13bb369df5"},
-    {file = "jq-1.6.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a67154f150aaf76cc1294032ed588436eb002097dd4fd1e283824bf753a05080"},
-    {file = "jq-1.6.0-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:1b3b95d5fd20e51f18a42647fdb52e5d8aaf150b7a666dd659cf282a2221ee3f"},
-    {file = "jq-1.6.0-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:3a8d98f72111043e75610cad7fa9ec5aec0b1ee2f7332dc7fd0f6603ea8144f8"},
-    {file = "jq-1.6.0-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:487483f10ae8f70e6acf7723f31b329736de4b421ce56b2f43b46d5cbd7337b0"},
-    {file = "jq-1.6.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:18a700f55b7ef83a1382edf0a48cb176b22bacd155e097375ef2345ff8621d97"},
-    {file = "jq-1.6.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:68aec8534ac3c4705e524b4ef54f66b8bdc867df9e0af2c3895e82c6774b5374"},
-    {file = "jq-1.6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b7a164748dbd03bb06d23bab7ead7ba7e5c4fcfebea7b082bdcd21d14136931e"},
-    {file = "jq-1.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aa22d24740276a8ce82411e4960ed2b5fab476230f913f9d9cf726f766a22208"},
-    {file = "jq-1.6.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:4c1a6fae1b74b3e0478e281eb6addedad7b32421221ac685e21c1d49af5e997f"},
-    {file = "jq-1.6.0-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:ce628546c22792b8870b9815086f65873ebb78d7bf617b5a16dd839adba36538"},
-    {file = "jq-1.6.0-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:7bb685f337cf5d4f4fe210c46220e31a7baec02a0ca0df3ace3dd4780328fc30"},
-    {file = "jq-1.6.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:bdbbc509a35ee6082d79c1f25eb97c08f1c59043d21e0772cd24baa909505899"},
-    {file = "jq-1.6.0-pp310-pypy310_pp73-macosx_10_9_x86_64.whl", hash = "sha256:1b332dfdf0d81fb7faf3d12aabf997565d7544bec9812e0ac5ee55e60ef4df8c"},
-    {file = "jq-1.6.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:3a4f6ef8c0bd19beae56074c50026665d66345d1908f050e5c442ceac2efe398"},
-    {file = "jq-1.6.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5184c2fcca40f8f2ab1b14662721accf68b4b5e772e2f5336fec24aa58fe235a"},
-    {file = "jq-1.6.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:689429fe1e07a2d6041daba2c21ced3a24895b2745326deb0c90ccab9386e116"},
-    {file = "jq-1.6.0-pp310-pypy310_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:8405d1c996c83711570f16aac32e3bf2c116d6fa4254a820276b87aed544d7e8"},
-    {file = "jq-1.6.0-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:138d56c7efc8bb162c1cfc3806bd6b4d779115943af36c9e3b8ca644dde856c2"},
-    {file = "jq-1.6.0-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fd28f8395687e45bba56dc771284ebb6492b02037f74f450176c102f3f4e86a3"},
-    {file = "jq-1.6.0-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b2c783288bf10e67aad321b58735e663f4975d7ddfbfb0a5bca8428eee283bde"},
-    {file = "jq-1.6.0-pp37-pypy37_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:206391ac5b2eb556720b94f0f131558cbf8d82d8cc7e0404e733eeef48bcd823"},
-    {file = "jq-1.6.0-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:35090fea1283402abc3a13b43261468162199d8b5dcdaba2d1029e557ed23070"},
-    {file = "jq-1.6.0-pp38-pypy38_pp73-macosx_11_0_arm64.whl", hash = "sha256:201c6384603aec87a744ad7b393cc4f1c58ece23d6e0a6c216a47bfcc405d231"},
-    {file = "jq-1.6.0-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a3d8b075351c29653f29a1fec5d31bc88aa198a0843c0a9550b9be74d8fab33b"},
-    {file = "jq-1.6.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:132e41f6e988c42b91c04b1b60dd8fa185a5c0681de5438ea1e6c64f5329768c"},
-    {file = "jq-1.6.0-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e1cb4751808b1d0dbddd37319e0c574fb0c3a29910d52ba35890b1343a1f1e59"},
-    {file = "jq-1.6.0-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:bd158911ed5f5c644f557ad94d6424c411560632a885eae47d105f290f0109cb"},
-    {file = "jq-1.6.0-pp39-pypy39_pp73-macosx_11_0_arm64.whl", hash = "sha256:64bc09ae6a9d9b82b78e15d142f90b816228bd3ee48833ddca3ff8c08e163fa7"},
-    {file = "jq-1.6.0-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f4eed167322662f4b7e65235723c54aa6879f6175b6f9b68bc24887549637ffb"},
-    {file = "jq-1.6.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:64bb4b305e2fabe5b5161b599bf934aceb0e0e7d3dd8f79246737ea91a2bc9ae"},
-    {file = "jq-1.6.0-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:165bfbe29bf73878d073edf75f384b7da8a9657ba0ab9fb1e5fe6be65ab7debb"},
-    {file = "jq-1.6.0.tar.gz", hash = "sha256:c7711f0c913a826a00990736efa6ffc285f8ef433414516bb14b7df971d6c1ea"},
 ]
 
 [[package]]
@@ -2735,20 +2677,22 @@ files = [
 
 [[package]]
 name = "langchain"
-version = "0.0.279"
+version = "0.0.303"
 description = "Building applications with LLMs through composability"
 optional = false
 python-versions = ">=3.8.1,<4.0"
 files = [
-    {file = "langchain-0.0.279-py3-none-any.whl", hash = "sha256:d91e65b3a210e9b52e42ed67c65d76caeaa3195ff097217f5ab2ca477bd3ae27"},
+    {file = "langchain-0.0.303-py3-none-any.whl", hash = "sha256:1745961f66b60bc3b513820a34c560dd37c4ba4b7499ba82545dc4816d0133bd"},
-    {file = "langchain-0.0.279.tar.gz", hash = "sha256:344e33d73c76ae35cdd8c3ba16c965bac29e0e889c93064718c5982002cbe30c"},
+    {file = "langchain-0.0.303.tar.gz", hash = "sha256:84d2727eb8b3b27a9d0aa0da9f05408c2564a4a923c7d5b154a16e488430e725"},
 ]
 
 [package.dependencies]
 aiohttp = ">=3.8.3,<4.0.0"
+anyio = "<4.0"
 async-timeout = {version = ">=4.0.0,<5.0.0", markers = "python_version < \"3.11\""}
-dataclasses-json = ">=0.5.7,<0.6.0"
+dataclasses-json = ">=0.5.7,<0.7"
-langsmith = ">=0.0.21,<0.1.0"
+jsonpatch = ">=1.33,<2.0"
+langsmith = ">=0.0.38,<0.1.0"
 numexpr = ">=2.8.4,<3.0.0"
 numpy = ">=1,<2"
 pydantic = ">=1,<3"
@@ -2764,7 +2708,7 @@ clarifai = ["clarifai (>=9.1.0)"]
 cohere = ["cohere (>=4,<5)"]
 docarray = ["docarray[hnswlib] (>=0.32.0,<0.33.0)"]
 embeddings = ["sentence-transformers (>=2,<3)"]
-extended-testing = ["amazon-textract-caller (<2)", "assemblyai (>=0.17.0,<0.18.0)", "atlassian-python-api (>=3.36.0,<4.0.0)", "beautifulsoup4 (>=4,<5)", "bibtexparser (>=1.4.0,<2.0.0)", "cassio (>=0.0.7,<0.0.8)", "chardet (>=5.1.0,<6.0.0)", "esprima (>=4.0.1,<5.0.0)", "faiss-cpu (>=1,<2)", "feedparser (>=6.0.10,<7.0.0)", "geopandas (>=0.13.1,<0.14.0)", "gitpython (>=3.1.32,<4.0.0)", "gql (>=3.4.1,<4.0.0)", "html2text (>=2020.1.16,<2021.0.0)", "jinja2 (>=3,<4)", "jq (>=1.4.1,<2.0.0)", "lxml (>=4.9.2,<5.0.0)", "markdownify (>=0.11.6,<0.12.0)", "mwparserfromhell (>=0.6.4,<0.7.0)", "mwxml (>=0.3.3,<0.4.0)", "newspaper3k (>=0.2.8,<0.3.0)", "openai (>=0,<1)", "openapi-schema-pydantic (>=1.2,<2.0)", "pandas (>=2.0.1,<3.0.0)", "pdfminer-six (>=20221105,<20221106)", "pgvector (>=0.1.6,<0.2.0)", "psychicapi (>=0.8.0,<0.9.0)", "py-trello (>=0.19.0,<0.20.0)", "pymupdf (>=1.22.3,<2.0.0)", "pypdf (>=3.4.0,<4.0.0)", "pypdfium2 (>=4.10.0,<5.0.0)", "pyspark (>=3.4.0,<4.0.0)", "rank-bm25 (>=0.2.2,<0.3.0)", "rapidfuzz (>=3.1.1,<4.0.0)", "requests-toolbelt (>=1.0.0,<2.0.0)", "scikit-learn (>=1.2.2,<2.0.0)", "sqlite-vss (>=0.1.2,<0.2.0)", "streamlit (>=1.18.0,<2.0.0)", "sympy (>=1.12,<2.0)", "telethon (>=1.28.5,<2.0.0)", "tqdm (>=4.48.0)", "xata (>=1.0.0a7,<2.0.0)", "xmltodict (>=0.13.0,<0.14.0)"]
+extended-testing = ["amazon-textract-caller (<2)", "assemblyai (>=0.17.0,<0.18.0)", "atlassian-python-api (>=3.36.0,<4.0.0)", "beautifulsoup4 (>=4,<5)", "bibtexparser (>=1.4.0,<2.0.0)", "cassio (>=0.1.0,<0.2.0)", "chardet (>=5.1.0,<6.0.0)", "dashvector (>=1.0.1,<2.0.0)", "esprima (>=4.0.1,<5.0.0)", "faiss-cpu (>=1,<2)", "feedparser (>=6.0.10,<7.0.0)", "geopandas (>=0.13.1,<0.14.0)", "gitpython (>=3.1.32,<4.0.0)", "gql (>=3.4.1,<4.0.0)", "html2text (>=2020.1.16,<2021.0.0)", "jinja2 (>=3,<4)", "jq (>=1.4.1,<2.0.0)", "lxml (>=4.9.2,<5.0.0)", "markdownify (>=0.11.6,<0.12.0)", "mwparserfromhell (>=0.6.4,<0.7.0)", "mwxml (>=0.3.3,<0.4.0)", "newspaper3k (>=0.2.8,<0.3.0)", "openai (>=0,<1)", "openapi-schema-pydantic (>=1.2,<2.0)", "pandas (>=2.0.1,<3.0.0)", "pdfminer-six (>=20221105,<20221106)", "pgvector (>=0.1.6,<0.2.0)", "psychicapi (>=0.8.0,<0.9.0)", "py-trello (>=0.19.0,<0.20.0)", "pymupdf (>=1.22.3,<2.0.0)", "pypdf (>=3.4.0,<4.0.0)", "pypdfium2 (>=4.10.0,<5.0.0)", "pyspark (>=3.4.0,<4.0.0)", "rank-bm25 (>=0.2.2,<0.3.0)", "rapidfuzz (>=3.1.1,<4.0.0)", "requests-toolbelt (>=1.0.0,<2.0.0)", "scikit-learn (>=1.2.2,<2.0.0)", "sqlite-vss (>=0.1.2,<0.2.0)", "streamlit (>=1.18.0,<2.0.0)", "sympy (>=1.12,<2.0)", "telethon (>=1.28.5,<2.0.0)", "timescale-vector (>=0.0.1,<0.0.2)", "tqdm (>=4.48.0)", "xata (>=1.0.0a7,<2.0.0)", "xmltodict (>=0.13.0,<0.14.0)"]
 javascript = ["esprima (>=4.0.1,<5.0.0)"]
 llms = ["clarifai (>=9.1.0)", "cohere (>=4,<5)", "huggingface_hub (>=0,<1)", "manifest-ml (>=0.0.1,<0.0.2)", "nlpcloud (>=1,<2)", "openai (>=0,<1)", "openlm (>=0.0.5,<0.0.6)", "torch (>=1,<3)", "transformers (>=4,<5)"]
 openai = ["openai (>=0,<1)", "tiktoken (>=0.3.2,<0.4.0)"]
@@ -7174,12 +7118,13 @@ testing = ["big-O", "jaraco.functools", "jaraco.itertools", "more-itertools", "p
 [extras]
 cohere = ["cohere"]
 community = ["llama-hub"]
-dataloaders = ["beautifulsoup4", "docx2txt", "duckduckgo-search", "jq", "jq", "pypdf", "pytube", "sentence-transformers", "unstructured"]
+dataloaders = ["beautifulsoup4", "docx2txt", "duckduckgo-search", "pypdf", "pytube", "sentence-transformers", "unstructured"]
 discord = ["discord"]
 elasticsearch = ["elasticsearch"]
 gmail = ["llama-hub", "requests"]
 huggingface-hub = ["huggingface_hub"]
 images = ["ftfy", "pillow", "regex", "torch", "torchvision"]
+json = ["llama-hub"]
 llama2 = ["replicate"]
 milvus = ["pymilvus"]
 opensearch = ["opensearch-py"]
@@ -7196,4 +7141,4 @@ whatsapp = ["flask", "twilio"]
 [metadata]
 lock-version = "2.0"
 python-versions = ">=3.9,<3.13"
-content-hash = "fad388baca92d2669530c2d928920e9f7abbe0ad74786b34dc915603b4a08a17"
+content-hash = "0b83ba3fd2485b3b4aa3c6a7534b214378d349538f7eb63c65768aafecdfad60"

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "embedchain"
-version = "0.0.83"
+version = "0.0.90"
 description = "Data platform for LLMs - Load, index, retrieve and sync any unstructured data"
 authors = [
     "Taranjeet Singh <taranjeet@embedchain.ai>",
@@ -90,7 +90,7 @@ color = true
 [tool.poetry.dependencies]
 python = ">=3.9,<3.13"
 python-dotenv = "^1.0.0"
-langchain = "^0.0.279"
+langchain = "^0.0.303"
 requests = "^2.31.0"
 openai = ">=0.28.0"
 chromadb = "^0.4.8"
@@ -128,7 +128,6 @@ huggingface_hub = { version = "^0.17.3", optional = true }
 pymilvus = { version = "2.3.1", optional = true }
 google-cloud-aiplatform = { version = "^1.26.1", optional = true }
 replicate = { version = "^0.15.4", optional = true }
-jq = { version = ">=1.6.0", optional = true}
 
 [tool.poetry.group.dev.dependencies]
 black = "^23.3.0"
@@ -166,12 +165,10 @@ dataloaders=[
     "beautifulsoup4",
     "docx2txt",
     "duckduckgo-search",
-    "jq",
     "pypdf",
     "pytube",
     "sentence-transformers",
     "unstructured",
-    "jq",
 ]
 vertexai = ["google-cloud-aiplatform"]
 llama2 = ["replicate"]
@@ -184,6 +181,7 @@ gmail = [
     "google-auth-httplib2",
     "google-api-core",
 ]
+json = ["llama-hub"]
 
 [tool.poetry.group.docs.dependencies]
 

tests/loaders/test_json.py

@@ -1,32 +1,91 @@
 import hashlib
-from unittest.mock import patch
 
-from langchain.docstore.document import Document
+import pytest
-from langchain.document_loaders.json_loader import \
+from llama_index.readers.schema.base import Document
-    JSONLoader as LangchainJSONLoader
 
 from embedchain.loaders.json import JSONLoader
 
 
-def test_load_data():
+def test_load_data(mocker):
-    mock_document = [
+    content = "temp.json"
-        Document(page_content="content1", metadata={"seq_num": 1}),
+
-        Document(page_content="content2", metadata={"seq_num": 2}),
+    mock_document = {
+        "doc_id": hashlib.sha256((content + ", ".join(["content1", "content2"])).encode()).hexdigest(),
+        "data": [
+            {"content": "content1", "meta_data": {"url": content}},
+            {"content": "content2", "meta_data": {"url": content}},
+        ],
+    }
+
+    mocker.patch("embedchain.loaders.json.JSONLoader.load_data", return_value=mock_document)
+
+    json_loader = JSONLoader()
+
+    result = json_loader.load_data(content)
+
+    assert "doc_id" in result
+    assert "data" in result
+
+    expected_data = [
+        {"content": "content1", "meta_data": {"url": content}},
+        {"content": "content2", "meta_data": {"url": content}},
     ]
-    with patch.object(LangchainJSONLoader, "load", return_value=mock_document):
-        content = "temp.json"
 
-        result = JSONLoader.load_data(content)
+    assert result["data"] == expected_data
 
-        assert "doc_id" in result
+    expected_doc_id = hashlib.sha256((content + ", ".join(["content1", "content2"])).encode()).hexdigest()
-        assert "data" in result
+    assert result["doc_id"] == expected_doc_id
 
-        expected_data = [
-            {"content": "content1", "meta_data": {"url": content, "row": 1}},
-            {"content": "content2", "meta_data": {"url": content, "row": 2}},
-        ]
 
-        assert result["data"] == expected_data
+def test_load_data_url(mocker):
+    content = "https://example.com/posts.json"
 
-        expected_doc_id = hashlib.sha256((content + ", ".join(["content1", "content2"])).encode()).hexdigest()
+    mocker.patch("os.path.isfile", return_value=False)  # Mocking os.path.isfile to simulate a URL case
-        assert result["doc_id"] == expected_doc_id
+    mocker.patch(
+        "llama_hub.jsondata.base.JSONDataReader.load_data",
+        return_value=[Document(text="content1"), Document(text="content2")],
+    )
+
+    mock_response = mocker.Mock()
+    mock_response.status_code = 200
+    mock_response.json.return_value = {"document1": "content1", "document2": "content2"}
+
+    mocker.patch("requests.get", return_value=mock_response)
+
+    result = JSONLoader.load_data(content)
+
+    assert "doc_id" in result
+    assert "data" in result
+
+    expected_data = [
+        {"content": "content1", "meta_data": {"url": content}},
+        {"content": "content2", "meta_data": {"url": content}},
+    ]
+
+    assert result["data"] == expected_data
+
+    expected_doc_id = hashlib.sha256((content + ", ".join(["content1", "content2"])).encode()).hexdigest()
+    assert result["doc_id"] == expected_doc_id
+
+
+def test_load_data_invalid_content(mocker):
+    mocker.patch("os.path.isfile", return_value=False)
+    mocker.patch("requests.get")
+
+    content = "123"
+
+    with pytest.raises(ValueError, match="Invalid content to load json data from"):
+        JSONLoader.load_data(content)
+
+
+def test_load_data_invalid_url(mocker):
+    mocker.patch("os.path.isfile", return_value=False)
+
+    mock_response = mocker.Mock()
+    mock_response.status_code = 404
+    mocker.patch("requests.get", return_value=mock_response)
+
+    content = "http://invalid-url.com/"
+
+    with pytest.raises(ValueError, match=f"Invalid content to load json data from: {content}"):
+        JSONLoader.load_data(content)

tests/telemetry/test_posthog.py

@@ -1,5 +1,6 @@
-import os
 import logging
+import os
+
 from embedchain.telemetry.posthog import AnonymousTelemetry
 
 
@@ -9,7 +10,7 @@ class TestAnonymousTelemetry:
         os.environ["EC_TELEMETRY"] = "true"
         mock_posthog = mocker.patch("embedchain.telemetry.posthog.Posthog")
         telemetry = AnonymousTelemetry()
-        assert telemetry.project_api_key == "phc_XnMmNHzwxE7PVHX4mD2r8K6nfxVM48a2sq2U3N1p2lO"
+        assert telemetry.project_api_key == "phc_PHQDA5KwztijnSojsxJ2c1DuJd52QCzJzT2xnSGvjN2"
         assert telemetry.host == "https://app.posthog.com"
         assert telemetry.enabled is True
         assert telemetry.user_id

tests/vectordb/test_chroma_db.py

@@ -163,10 +163,12 @@ def test_chroma_db_collection_add_with_skip_embedding(app_with_settings):
 
     assert data == expected_value
 
-    data = app_with_settings.db.query(input_query=[0, 0, 0], where={}, n_results=1, skip_embedding=True)
+    data_without_citations = app_with_settings.db.query(
-    expected_value = [("document", "url_1", "doc_id_1")]
+        input_query=[0, 0, 0], where={}, n_results=1, skip_embedding=True
+    )
+    expected_value_without_citations = ["document"]
+    assert data_without_citations == expected_value_without_citations
 
-    assert data == expected_value
     app_with_settings.db.reset()
 
 
@@ -326,8 +328,16 @@ def test_chroma_db_collection_query(app_with_settings):
 
     assert app_with_settings.db.count() == 2
 
-    data = app_with_settings.db.query(input_query=[0, 0, 0], where={}, n_results=2, skip_embedding=True)
+    data_without_citations = app_with_settings.db.query(
-    expected_value = [("document", "url_1", "doc_id_1"), ("document2", "url_2", "doc_id_2")]
+        input_query=[0, 0, 0], where={}, n_results=2, skip_embedding=True
+    )
+    expected_value_without_citations = ["document", "document2"]
+    assert data_without_citations == expected_value_without_citations
+
+    data_with_citations = app_with_settings.db.query(
+        input_query=[0, 0, 0], where={}, n_results=2, skip_embedding=True, citations=True
+    )
+    expected_value_with_citations = [("document", "url_1", "doc_id_1"), ("document2", "url_2", "doc_id_2")]
+    assert data_with_citations == expected_value_with_citations
 
-    assert data == expected_value
     app_with_settings.db.reset()

tests/vectordb/test_elasticsearch_db.py

@@ -60,12 +60,16 @@ class TestEsDB(unittest.TestCase):
 
         # Query the database for the documents that are most similar to the query "This is a document".
         query = ["This is a document"]
-        results = self.db.query(query, n_results=2, where={}, skip_embedding=False)
+        results_without_citations = self.db.query(query, n_results=2, where={}, skip_embedding=False)
+        expected_results_without_citations = ["This is a document.", "This is another document."]
+        self.assertEqual(results_without_citations, expected_results_without_citations)
 
-        # Assert that the results are correct.
+        results_with_citations = self.db.query(query, n_results=2, where={}, skip_embedding=False, citations=True)
-        self.assertEqual(
+        expected_results_with_citations = [
-            results, [("This is a document.", "url_1", "doc_id_1"), ("This is another document.", "url_2", "doc_id_2")]
+            ("This is a document.", "url_1", "doc_id_1"),
-        )
+            ("This is another document.", "url_2", "doc_id_2"),
+        ]
+        self.assertEqual(results_with_citations, expected_results_with_citations)
 
     @patch("embedchain.vectordb.elasticsearch.Elasticsearch")
     def test_query_with_skip_embedding(self, mock_client):
@@ -111,9 +115,7 @@ class TestEsDB(unittest.TestCase):
         results = self.db.query(query, n_results=2, where={}, skip_embedding=True)
 
         # Assert that the results are correct.
-        self.assertEqual(
+        self.assertEqual(results, ["This is a document.", "This is another document."])
-            results, [("This is a document.", "url_1", "doc_id_1"), ("This is another document.", "url_2", "doc_id_2")]
-        )
 
     def test_init_without_url(self):
         # Make sure it's not loaded from env

tests/vectordb/test_weaviate.py

@@ -75,10 +75,6 @@ class TestWeaviateDb(unittest.TestCase):
                             "name": "app_id",
                             "dataType": ["text"],
                         },
-                        {
-                            "name": "text",
-                            "dataType": ["text"],
-                        },
                     ],
                 },
             ]

tests/vectordb/test_zilliz_db.py

@@ -129,7 +129,7 @@ class TestZillizDBCollection:
             query_result = zilliz_db.query(input_query=["query_text"], n_results=1, where={}, skip_embedding=True)
 
             # Assert that MilvusClient.search was called with the correct parameters
-            mock_search.assert_called_once_with(
+            mock_search.assert_called_with(
                 collection_name=mock_config.collection_name,
                 data=["query_text"],
                 limit=1,
@@ -137,7 +137,20 @@ class TestZillizDBCollection:
             )
 
             # Assert that the query result matches the expected result
-            assert query_result == [("result_doc", "url_1", "doc_id_1")]
+            assert query_result == ["result_doc"]
+
+            query_result_with_citations = zilliz_db.query(
+                input_query=["query_text"], n_results=1, where={}, skip_embedding=True, citations=True
+            )
+
+            mock_search.assert_called_with(
+                collection_name=mock_config.collection_name,
+                data=["query_text"],
+                limit=1,
+                output_fields=["text", "url", "doc_id"],
+            )
+
+            assert query_result_with_citations == [("result_doc", "url_1", "doc_id_1")]
 
     @patch("embedchain.vectordb.zilliz.MilvusClient", autospec=True)
     @patch("embedchain.vectordb.zilliz.connections", autospec=True)
@@ -168,7 +181,7 @@ class TestZillizDBCollection:
             query_result = zilliz_db.query(input_query=["query_text"], n_results=1, where={}, skip_embedding=False)
 
             # Assert that MilvusClient.search was called with the correct parameters
-            mock_search.assert_called_once_with(
+            mock_search.assert_called_with(
                 collection_name=mock_config.collection_name,
                 data=["query_vector"],
                 limit=1,
@@ -176,4 +189,17 @@ class TestZillizDBCollection:
             )
 
             # Assert that the query result matches the expected result
-            assert query_result == [("result_doc", "url_1", "doc_id_1")]
+            assert query_result == ["result_doc"]
+
+            query_result_with_citations = zilliz_db.query(
+                input_query=["query_text"], n_results=1, where={}, skip_embedding=False, citations=True
+            )
+
+            mock_search.assert_called_with(
+                collection_name=mock_config.collection_name,
+                data=["query_vector"],
+                limit=1,
+                output_fields=["text", "url", "doc_id"],
+            )
+
+            assert query_result_with_citations == [("result_doc", "url_1", "doc_id_1")]