Installing scrapy-poet

scrapy-poet is a Scrapy extension that runs on Python 3.8 and above.

If you’re already familiar with installation of Python packages, you can install scrapy-poet and its dependencies from PyPI with:

pip install scrapy-poet

Scrapy 2.6.0 or above is required and it has to be installed separately.

Configuring the project

To use scrapy-poet, enable its middlewares in the file of your Scrapy project:

    "scrapy_poet.InjectionMiddleware": 543,
    "scrapy.downloadermiddlewares.stats.DownloaderStats": None,
    "scrapy_poet.DownloaderStatsMiddleware": 850,
    "scrapy_poet.RetryMiddleware": 275,
REQUEST_FINGERPRINTER_CLASS = "scrapy_poet.ScrapyPoetRequestFingerprinter"

Things that are good to know

scrapy-poet is written in pure Python and depends on a few key Python packages (among others):

  • web-poet, core library used for Page Object pattern

  • andi, provides annotation-based dependency injection

  • parsel, responsible for css and xpath selectors