From f83c9db2f38c9acaebbe8302d279ed99a1452bd3 Mon Sep 17 00:00:00 2001
From: Junwei Zhao <wahyd4@gmail.com>
Date: Thu, 7 Apr 2022 14:58:37 +1000
Subject: [PATCH] Update scraper

---
 categories/web-scraping.md | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/categories/web-scraping.md b/categories/web-scraping.md
index 8d04d97..d0dfe45 100644
--- a/categories/web-scraping.md
+++ b/categories/web-scraping.md
@@ -1,11 +1,13 @@
-# A good scraper
+# Web Scraping
+
+## A good scraper
 
 - content downloading, download images, contents and so on.
 - links retrieve, find new links
 - URL management, avoid endless loop
 - content analysis and management, export as csv or chart.
 
-# good frameworks
+## good frameworks
 
 - colly
 	- https://github.com/asciimoo/colly
@@ -14,19 +16,25 @@
 - scrapy
 	- [https://github.com/scrapy/scrapy](https://github.com/scrapy/scrapy)
 
-# anti scraper blocking
+## anti scraper blocking
 
+
+- Ddon't use default user-agent, instead use real browser user agent
+- Proper interval, normally means reduce your interval
 - multiple ips
 - different headers
-- proper interval
 - dynamic pages -> prentend to be a real man. /phantom.js/selenium/chrome headless
 - set proper cookie
 - verification code -> ocr
 - fetch data from mobile webpage
 
-# anti scraper
+## anti scraper
 
 - block ip
 - http headers, such as user-agent
 - cookie
 - return fake data
+
+## How does web scraper work
+
+TODO