AI Access Checker Guide
This checker reads robots.txt for 12 AI crawlers, fetches the homepage once as a control and again as each crawler, then looks at llms.txt and four agent paths on that same site. It does not scan other hosts and it does not call agent tools.
Overview
Use the free check for one URL. AI Access Watch is $9 per origin per month or $90 per year and repeats the check daily. The first watch day is quiet. Later mail is only for a new mismatch, a new private URL, or a new open agent path.
How to read a row
- Policy: Blocked, allowed, or unspecified, from robots.txt. Allow wins when it ties a disallow of the same length.
- Live: Served, blocked, unknown, or not comparable. Not comparable means the control request never loaded the page.
- Mismatch: The file says blocked and the page still came back, or the file says allowed and the server blocked that crawler.
What else is checked
| Check | Counts as a finding | Does not count |
|---|---|---|
| llms.txt | A real text file with an H1 that names a private URL which answers | An HTML page, a redirect home, or a soft 404 |
| Agent path | JSON that lists tools and asks for no login | HTML, a lock, or a tool call |
| Copy box | Training crawlers that were served the homepage | A second form you have to fill in |
Control blocked
Not a policy failure
The control request was blocked or challenged, so crawler rows are not comparable and are not treated as policy failures.Recommended Remediation Flow
- Run the free check Open the AI Access Checker and enter the public homepage.
- Read not comparable as not comparable Do not tell the owner the crawler policy failed when the control request never loaded.
- Start the watch only if you want a daily rerun AI Access Watch is $9 per month or $90 per year for one origin. The first day records a baseline and stays quiet.
Troubleshooting Common Issues
Every row says not comparable
The control fetch was blocked.
- A firewall may be challenging every automated request.
- That is not a robots.txt mismatch.
- Fix the control path, then run the check again.
Validation Checklist
Post-fix validation
- A blocked control fetch does not create mismatch mail.
- An HTML llms.txt is not treated as a map.
- An HTML /mcp response is not an open agent.
FAQ
Does this scan the whole internet for agent servers?
No.
- It only requests the site you submit.
- It requests four known paths on that origin.
- It never calls a tool.