Installation

Installing scrapy-poet

scrapy-poet is a Scrapy extension that runs on Python 3.8 and above.

If you’re already familiar with installation of Python packages, you can install scrapy-poet and its dependencies from PyPI with:

pip install scrapy-poet

Scrapy 2.6.0 or above is required and it has to be installed separately.

Configuring the project

To use scrapy-poet, enable its middlewares in the settings.py file of your Scrapy project:

DOWNLOADER_MIDDLEWARES = {
    "scrapy_poet.InjectionMiddleware": 543,
    "scrapy.downloadermiddlewares.stats.DownloaderStats": None,
    "scrapy_poet.DownloaderStatsMiddleware": 850,
}
SPIDER_MIDDLEWARES = {
    "scrapy_poet.RetryMiddleware": 275,
}
REQUEST_FINGERPRINTER_CLASS = "scrapy_poet.ScrapyPoetRequestFingerprinter"

Things that are good to know

scrapy-poet is written in pure Python and depends on a few key Python packages (among others):

web-poet, core library used for Page Object pattern
andi, provides annotation-based dependency injection
parsel, responsible for css and xpath selectors