Test a path against user-agent groups, wildcard rules and allow/disallow precedence.
Maximum input: 2 MiB. Robots.txt: 500 KiB. Public HTTP/HTTPS targets only; 20 requests per 10 minutes.
Pasted or uploaded data is sent to Revbu for processing. Reports are stored privately for 30 minutes.
Test robots.txt rules with real paths
Separate crawling from indexing
