This demo web crawler illustrates how token-based identification can be used to access and crawl protected websites. There are two sample URLs provided:
Unprotected Website.
Accessible to the crawler without any token.
Protected Website.
Requires a KYA token to allow crawler access.
No pages crawled yet. Enter a URL and click Crawl to get started.