knowledge/categories/web-scraping.md

A good scraper

• content downloading, download images, contents and so on.
• links retrieve, find new links
• URL management, avoid endless loop
• content analysis and management, export as csv or chart.

good frameworks

• colly
  • https://github.com/asciimoo/colly
• pyspider
  • http://docs.pyspider.org/en/latest/
• scrapy
  • https://github.com/scrapy/scrapy

anti scraper blocking

• multiple ips
• different headers
• proper interval
• dynamic pages -> prentend to be a real man. /phantom.js/selenium/chrome headless
• set proper cookie
• verification code -> ocr
• fetch data from mobile webpage

anti scraper

• block ip
• http headers, such as user-agent
• cookie
• return fake data