AsyntaiBot

AsyntaiBot is the web crawler of Asyntai, an AI chatbot for websites. It reads a website so the chatbot on that same website can answer visitors' questions from its pages.

What it does

• It crawls a website when an Asyntai customer adds that website to their account. This is normally the site owner, who wants a chatbot that knows their own content.
• It reads public pages only. It does not log in, submit forms or buy anything.
• The text it reads is used to answer questions on that customer's chatbot. Asyntai does not use it to train AI models.
• It crawls when the customer starts a crawl, or when a periodic refresh keeps that customer's chatbot up to date. It does not crawl the web at large.

How to recognise it

User agent	Mozilla/5.0 (compatible; AsyntaiBot/1.0; +https://asyntai.com/bot)
robots.txt token	AsyntaiBot
Request signatures	Requests are signed with Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519).
Signature-Agent	"https://asyntai.com"
Public key directory	https://asyntai.com/.well-known/http-message-signatures-directory

A signed request can be checked against the key above.

robots.txt

When AsyntaiBot crawls pages, it follows robots.txt (RFC 9309). It uses the AsyntaiBot group when there is one, and the * group otherwise. It supports Allow, Disallow, the * and $ wildcards, and Crawl-delay.

To block AsyntaiBot from the whole site:

User-agent: AsyntaiBot
Disallow: /

To keep it out of one section only:

User-agent: AsyntaiBot
Disallow: /account/
Disallow: /checkout/

To slow it down, in seconds between requests:

User-agent: AsyntaiBot
Crawl-delay: 5

Product feeds and sitemaps that a customer sets up in their Asyntai account are fetched on that customer's request.

Allowing it on Cloudflare

If your site uses Cloudflare and you want your Asyntai chatbot to read it, and your Cloudflare settings block AI crawlers, add AsyntaiBot to your allowed bots, or create a WAF custom rule with the action Skip that matches the AsyntaiBot user agent.

Contact

Questions about AsyntaiBot, or a request to stop crawling a site: [email protected].