I haven't had the need to use pause/resume but I do incorporate logic (not necessarily all that complex) in determining which links to crawl. It is very easy to do within each spider especially with how the framework uses generators. It is also easy to extend the pipelines for pre and post processing.

As others have said, managing a project with Scrapy is super easy and highly configurable with sane out of the box settings.

I haven't done this, but take a peek at these. Perhaps you can have your own linkextractor that rejects certain items, or perhaps a custom scheduler:

https://doc.scrapy.org/en/latest/topics/link-extractors.html https://doc.scrapy.org/en/0.14/topics/architecture.html?high... https://doc.scrapy.org/en/0.9/experimental/scheduler-middlew... https://doc.scrapy.org/en/latest/topics/settings.html#schedu... https://scrapy.readthedocs.io/en/0.14/faq.html#does-scrapy-c...