Update scraper

categories/web-scraping.md

@@ -1,11 +1,13 @@
-# A good scraper
+# Web Scraping
+
+## A good scraper
 
 - content downloading, download images, contents and so on.
 - links retrieve, find new links
 - URL management, avoid endless loop
 - content analysis and management, export as csv or chart.
 
-# good frameworks
+## good frameworks
 
 - colly
 	- https://github.com/asciimoo/colly
@@ -14,19 +16,25 @@
 - scrapy
 	- [https://github.com/scrapy/scrapy](https://github.com/scrapy/scrapy)
 
-# anti scraper blocking
+## anti scraper blocking
 
+
+- Ddon't use default user-agent, instead use real browser user agent
+- Proper interval, normally means reduce your interval
 - multiple ips
 - different headers
-- proper interval
 - dynamic pages -> prentend to be a real man. /phantom.js/selenium/chrome headless
 - set proper cookie
 - verification code -> ocr
 - fetch data from mobile webpage
 
-# anti scraper
+## anti scraper
 
 - block ip
 - http headers, such as user-agent
 - cookie
 - return fake data
+
+## How does web scraper work
+
+TODO