How to use performance options


from nyawc.Options import Options
from nyawc.Crawler import Crawler
from nyawc.http.Request import Request

options = Options()

options.performance.max_threads = 20
options.performance.request_timeout = 15

crawler = Crawler(options)

Available performance options

Maximum threads

The maximum amount of simultaneous threads to use for crawling. Default is 40.

options.performance.max_threads = 40

Request timeout

The request timeout in seconds (throws an exception if exceeded). Default is 30.

options.performance.request_timeout = 30