I'm with you on this, not for huge sites but if even 1% of sites used this behav... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

cmdrfred on May 18, 2015 | parent | context | favorite | on: What one may find in robots.txt

I'm with you on this, not for huge sites but if even 1% of sites used this behavior malicious bots would be forced to honor robots.txt

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact