Setup

Install

pip install scrapy-poet

Enable

Configure the scrapy-poet add-on to enable it:

settings.py

ADDONS = {
    "scrapy_poet.Addon": 300,
}

This is what the add-on changes:

• In DOWNLOADER_MIDDLEWARES:

  • Sets InjectionMiddleware with value 543.

  • Sets HtmlMetadataMiddleware with value 585.

  • Replaces scrapy.downloadermiddlewares.stats.DownloaderStats with scrapy_poet.DownloaderStatsMiddleware.

• Sets REQUEST_FINGERPRINTER_CLASS to ScrapyPoetRequestFingerprinter.

• In SPIDER_MIDDLEWARES, sets RetryMiddleware with value 275.

Configure

Declare the SCRAPY_POET_DISCOVER setting with a list of modules that define page objects, so that they can be loaded at run-time.

A best practice is to create a pages/ folder in your Scrapy project, a sibling of your spiders/ folder, add an empty __init__.py file to it to make it a Python module, and declare its import path in the setting:

settings.py

SCRAPY_POET_DISCOVER = ["myproject.pages"]