# A good scraper

- content downloading, download images, contents and so on.
- links retrieve, find new links
- URL management, avoid endless loop
- content analysis and management, export as csv or chart.

# good frameworks

- colly
	- https://github.com/asciimoo/colly
- pyspider
	- [http://docs.pyspider.org/en/latest/](http://docs.pyspider.org/en/latest/)
- scrapy
	- [https://github.com/scrapy/scrapy](https://github.com/scrapy/scrapy)

# anti scraper blocking

- multiple ips
- different headers
- proper interval
- dynamic pages -> prentend to be a real man. /phantom.js/selenium/chrome headless
- set proper cookie
- verification code -> ocr
- fetch data from mobile webpage

# anti scraper

- block ip
- http headers, such as user-agent
- cookie
- return fake data
