xiaoyu9527
V2EX  ›  问与答

scrapy 下载卡住 如何设置超时?

  •  
  •   xiaoyu9527 · Aug 8, 2017 · 8535 views
    This topic created in 3226 days ago, the information mentioned may be changed or developed.

    正在下载 xxxx 正在保存 xxxx

    就卡住了

    这里写的是一个循环语句。

    1 replies    2017-08-11 14:08:00 +08:00
    860670496
        1
    860670496  
       Aug 11, 2017
    settings 里自带参数,比如:

    DOWNLOAD_TIMEOUT = 300

    如果觉得不能满足要求就写个继承 DownloadTimeoutMiddleware 的类就好

    https://doc.scrapy.org/en/1.4/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.downloadtimeout

    顺带蛋疼的话还可以利用下面这个类获取状态写进度条
    class scrapy.downloadermiddlewares.stats.DownloaderStats
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Solana   ·   952 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 27ms · UTC 21:09 · PVG 05:09 · LAX 14:09 · JFK 17:09
    ♥ Do have faith in what you're doing.