one-knowledge/github-knowledge/web-scraping.md

title, created, updated, type, tags, external

title	created	updated	type	tags	external
Web Scraping	2022-04-07	2022-04-07	summary	tech reference	https://github.com/wahyd4/knowledge/blob/master/categories/web-scraping.md

Web Scraping

A good scraper

• content downloading, download images, contents and so on.
• links retrieve, find new links
• URL management, avoid endless loop
• content analysis and management, export as csv or chart.

good frameworks

• colly
  • https://github.com/asciimoo/colly
• pyspider
  • http://docs.pyspider.org/en/latest/
• scrapy
  • https://github.com/scrapy/scrapy

anti scraper blocking

• Ddon't use default user-agent, instead use real browser user agent
• Proper interval, normally means reduce your interval
• multiple ips
• different headers
• dynamic pages -> prentend to be a real man. /phantom.js/selenium/chrome headless
• set proper cookie
• verification code -> ocr
• fetch data from mobile webpage

anti scraper

• block ip
• http headers, such as user-agent
• cookie
• return fake data

How does web scraper work

TODO