home • source • nvt

Web Crawling

• BruceDone/awesome-crawler: A collection of awesome web crawler,spider in different languages

Tools

Colly

• gocolly/colly: Fast and Elegant Scraping Framework for Golang

Scrapy

• scrapy/scrapy: Scrapy, a fast high-level web crawling & scraping framework for Python.
• scrapinghub/portia: Visual scraping for Scrapy
• xiyouMc/WebHubBot: Python + Scrapy + MongoDB

Others

• hardikvasa/google-images-download: Python Script to download hundreds of images from ‘Google Images’. It is a ready-to-run code!

Resources

• 为何大量网站不能抓取?爬虫突破封禁的6种常见方法