Home Sign Up Sign In
monsabre1

新开源的 python scrapy 爬虫集群 有人用过吗？

  •   
  •   monsabre1 · May 10, 2015 · 5021 views
    This topic created in 4199 days ago, the information mentioned may be changed or developed.

    https://github.com/istresearch/scrapy-cluster

  • 爬虫
  • Scrapy
  • 新开源
    5 replies  •  2015-05-11 10:35:55 +08:00
    zts1993
    zts1993
    May 10, 2015
    1
    用过 Redis 和Scrapy组的集群
    cxl008
    cxl008
    May 10, 2015
    2
    scrapy + redis + mongodb 。。。。。。自己写的。。。。。。。
    zhwei
    zhwei
    May 10, 2015 via iPad
    3
    这货一直2.7也是醉了
    yakczh
    yakczh
    May 10, 2015
    4
    scrapy如果已经抓过的网站又有更新了, 一般是怎么处理 ?是在命令行重新启动运行一遍吗?
    heamon7
    heamon7
    May 11, 2015 via iPhone
    5
    @yakczh 定时任务么？
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Privacy   ·   Solana   ·   1589 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 39ms · UTC 02:06 · PVG 10:06 · LAX 18:06 · JFK 21:06
    ♥ Do have faith in what you're doing.