Configuring the settings denoted below would follow the usual methods used by Scrapy.
dict wherein the keys would be the providers available for your Scrapy
project while the values denotes the priority of the provider.
More info on this at this section: Providers.
List[ApplyRule] which sets the rules to use.
SCRAPY_POET_RULES already has values set from the return value of
make sure to also set the
SCRAPY_POET_DISCOVER setting below.
A list of packages/modules (i.e.
List[str]) which scrapy-poet will look for
page objects annotated with the
web_poet.handle_urls() decorator. Each
package/module is passed into
web_poet.consume_modules where each
module from a package is recursively loaded.
This ensures that when using the default value of
SCRAPY_POET_RULES set to
it should contain all the intended rules.
Note that it’s also possible for
SCRAPY_POET_RULES to have rules not specified
SCRAPY_POET_DISCOVER (e.g. when the annotated page objects are inside your
Scrapy project). However, it’s recommended to still use
to ensure all the intended rules are properly loaded.
The caching mechanism in the providers can be enabled by either setting this
True which configures the file path of the cache into a
in your local Scrapy project.
On the other hand, you can also set this as a
str pointing to any path relative
to your local Scrapy project.
When this is set to
True, any error that arises when retrieving dependencies from
providers would be cached. This could be useful in cases during local development
wherein you outright know that retrieving the dependency would fail and would
choose to ignore it. Caching such errors would reduce the waiting time when
developing Page Objects.
It’s recommended to set this off into
False by default since you might miss
out on sporadic errors.
Sets the location where the
savefixture command creates tests.
More info at Tests for Page Objects.
Sets the class, or its import path, that will be used as an adapter in the generated test fixtures.
More info at Configuring the item adapter.
The default value is the default value of the
setting for the version of Scrapy currently installed (e.g.
You can assign a request fingerprinter class to this setting to configure a custom request fingerprinter class to use for requests.
This class is used to generate a base fingerprint for a request. If that request uses dependency injection, that fingerprint is then modified to account for requested dependencies. Otherwise, the fingerprint is used as is.