This demo web crawler illustrates how token-based identification can be used to access and crawl protected websites. There are two sample URLs provided:
Unprotected Website: Accessible to the crawler without any token.
Protected Website: Requires a kya token to allow crawler access.