Knowledge
Website Knowledge Sources
Website sources let you crawl one page or an entire domain and turn that content into searchable knowledge for your agents.
Crawl Modes
KriyaOS supports two crawl modes:
single: index only the provided page URL.domain: follow links within the same domain (bounded by limits).
Crawl Limits
To keep ingestion predictable, website sources have limits such as:
- max depth (how many link-hops)
- max pages (how many pages to index)
Rendered Crawling
Some pages require JavaScript rendering to show real content.
If you have a modern marketing site or a SPA, enable render to improve extraction.
Recommendation
Start with single + a small page limit to verify content quality, then expand to a domain crawl if results look good.
Include and Exclude Patterns
Use include/exclude patterns to control what gets indexed. This helps you avoid low-value pages like tag archives, privacy pages, or parameterized URLs.