Pluggable parsers, automatically good error handling and spidering functionality (finding and queueing new links to scrape), great logging, progress stats, exports, pause/resume functionality, and a million other goodies that are seemingly "trivial" but really you don't want to rewrite them every time you write a scraper.
edit: Especially if your scraping jobs take a LONG time - days and weeks, this stuff is extra handy. Might I add a great debugging environment (scrapy shell), error handling, rate limiting, respecting robots.txt, so much more.
How much benefit does the spidering/progress/pause/resume functionality give if you're not just spidering every link on the site, but have complex logic to determine exactly which links to crawl and in what order? Does Scrapy provide convenient extension hooks to change the crawl algorithm?
I haven't had the need to use pause/resume but I do incorporate logic (not necessarily all that complex) in determining which links to crawl. It is very easy to do within each spider especially with how the framework uses generators. It is also easy to extend the pipelines for pre and post processing.
As others have said, managing a project with Scrapy is super easy and highly configurable with sane out of the box settings.
edit: Especially if your scraping jobs take a LONG time - days and weeks, this stuff is extra handy. Might I add a great debugging environment (scrapy shell), error handling, rate limiting, respecting robots.txt, so much more.