Free tool
Are you blocking AI without knowing it?
Many websites block AI crawlers without the owner knowing, for example through a default setting in Cloudflare or an old robots.txt. We test 15 crawlers from OpenAI, Anthropic, Google, Microsoft, Perplexity, Apple and more.
Two places you can block
robots.txt is a file that tells crawlers what they're allowed to do. But a firewall or a CDN like Cloudflare can also turn them away, even if robots.txt allows them. The test sends a request as each crawler and checks whether the server responds normally.
Search or training?
Some crawlers fetch content for AI answers and search, others for training models. Blocking training is perfectly legitimate. What matters is that you don't accidentally block the ones that show you in the answers.
More tools
Good collaboration starts with a conversation
What should we build together?
Tell me about your idea, and we’ll find the right way forward.