feat: filter sitemap (#304)

2023-07-19 05:36:39 +05:30
parent c12362486f
commit 3da5724853
3 changed files with 32 additions and 2 deletions
@@ -30,6 +30,14 @@ To add any web page, use the data_type as `web_page`. Eg:
 app.add('web_page', 'a_valid_web_page_url')
 ```

+### Sitemap
+
+Add all web pages from an xml-sitemap. Filters non-text files. Use the data_type as `sitemap`. Eg:
+
+```python
+app.add('sitemap', 'https://example.com/sitemap.xml')
+```
+
 ### Doc file

 To add any doc/docx file, use the data_type as `docx`. Eg: