Tests for Page Objects

web-poet provides tools for testing page objects. scrapy-poet projects can use a Scrapy command to easily generate tests:

scrapy savefixture my_project.pages.MyItemPage 'https://quotes.toscrape.com/page/1/'

This will request the provided page, create an instance of the provided page object for this page, request its to_item() method and save both the page object dependencies and the resulting item as a test fixture. These fixtures can then be used with the pytest plugin provided by web-poet.

Configuring the test location

The SCRAPY_POET_TESTS_DIR setting specifies where to create the tests. It can be set in the project settings or with the -s command argument.

Handling time fields

The tests generated by savefixture set the frozen_time metadata value to the time of the test creation.

Using spiders

By default savefixture creates a simple spider that uses the project settings and makes one request to the URL provided to the command. It may be needed instead to use a real spider from the project, for example because of its custom_settings. In this case you can pass the spider name as the third argument:

scrapy savefixture my_project.pages.MyItemPage 'https://quotes.toscrape.com/page/1/' toscrape_listing

The command will try to run the spider overriding its start(), so it should run just one request but it can break on spiders with complicated logic, e.g. ones that use spider_idle to schedule requests or modify items returned from to_item(). You may need to adapt your spiders to this, for example checking if the special _SCRAPY_POET_SAVEFIXTURE setting is set to True and using more simple logic in this case.

Configuring the item adapter

As documented in Item adapters, fixtures can use custom item adapters. The savefixture command uses the adapter specified in the SCRAPY_POET_TESTS_ADAPTER setting to save the fixture.