About the AgentsEyeResearch crawler

What this is

This page describes an academic research project that measures whether public websites serve different content to AI agents (crawlers, AI assistants that fetch pages on a user's behalf, and browser agents) than they serve to ordinary web browsers. The goal is to inform web standards, AI-agent developers and site operators about practices such as content negotiation, cloaking, and content that is only shown to automated clients.

What our crawler does

• It fetches a small number of public pages per site (at most about one request per second per site and a handful of pages per site), using several request profiles: a normal browser user agent, our declared research user agent, and user-agent strings published by AI vendors. It compares the responses.
• Our declared crawler identifies itself as AgentsEyeResearch/1.0 with a link to this page. Some requests are cryptographically signed with Web Bot Auth (HTTP Message Signatures, RFC 9421); the signing keys are published at /.well-known/http-message-signatures-directory and our Signature-Agent is "https://agentseye.org".
• It does not log in, submit forms, solve CAPTCHAs, attempt to bypass access controls, or download non-public data.

How to opt out

Our declared research crawler honours robots.txt (RFC 9309). To exclude your site, add:

User-agent: AgentsEyeResearch
Disallow: /

You can also email contact@agentseye.org with the domain(s) to exclude; we will remove them from all future collection and, on request, delete any data already collected from them.

Data handling

• We store the public page content returned to our requests and the response headers, for comparison only.
• We do not collect personal data from the sites we measure. Results are reported in aggregate; site-specific findings that may indicate a security or advertising-disclosure problem are reported privately to the site operator before publication.
• This domain itself logs requests it receives (time, path, request headers, IP address and network) to study how automated agents fetch pages. These logs are used only for this research, are not shared, and are deleted when the study ends.

Test pages on this domain

Pages under /c/ and /hp/ are research test pages. Their content deliberately varies with request headers to validate our measurement tools. They contain no harmful content.

Contact

contact@agentseye.org