robots.txt tester

Check if a path is crawlable. We fetch the site's robots.txtand apply Google's longest-match rules.