Python USES the scrapy method of putting back large downloaded pages during data collection

This article illustrates an example of how Python USES scrapy to retrieve a large page during data collection. Share with you for your reference. Specific analysis is as follows:

Add the following code to settings.py, myproject for your project name

DOWNLOADER_HTTPCLIENTFACTORY = 'myproject.downloader.LimitSizeHTTPClientFactory'

Custom modules that restrict downloading of large pages

MAX_RESPONSE_SIZE = 1048576 # 1Mb

from scrapy.core.downloader.webclient import ScrapyHTTPClientFactory, ScrapyHTTPPageGetter

class LimitSizePageGetter(ScrapyHTTPPageGetter):

    def handleHeader(self, key, value):

        ScrapyHTTPPageGetter.handleHeader(self, key, value)

        if key.lower() == 'content-length' and int(value) > MAX_RESPONSE_SIZE:

            self.connectionLost('oversized')

class LimitSizeHTTPClientFactory(ScrapyHTTPClientFactory):

     protocol = LimitSizePageGetter

I hope this article has been helpful to your Python programming.