Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

How often does it crawl? Just this week I tried to look up a reference on a page and found it dead, probably for quite some time (I forget the page I was on).


Well it turns out I may have been mistaken; the bot that I thought did that is defunct, and I cannot easily find another that does that. After thinking about it, I can see some reasons why you might not want a bot automatically marking links as dead. There are these, which I think are a bit more manual in nature: http://en.wikipedia.org/wiki/Wikipedia:Link_rot#Bots


Please share those reasons. The only one I can think of is false positives of temporarily down servers, which could be mitigated by timed retries.


That is the main reason. Another would be some sort of "Content not here anymore" message that doesn't conform to HTTP status code standards, giving a false negative.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: