---
title: Web Scraping
created: 2022-04-07
updated: 2022-04-07
type: summary
tags: [tech, reference]
external: https://github.com/wahyd4/knowledge/blob/master/categories/web-scraping.md
---

# Web Scraping

## A good scraper

- content downloading, download images, contents and so on.
- links retrieve, find new links
- URL management, avoid endless loop
- content analysis and management, export as csv or chart.

## good frameworks

- colly
	- https://github.com/asciimoo/colly
- pyspider
	- [http://docs.pyspider.org/en/latest/](http://docs.pyspider.org/en/latest/)
- scrapy
	- [https://github.com/scrapy/scrapy](https://github.com/scrapy/scrapy)

## anti scraper blocking


- Ddon't use default user-agent, instead use real browser user agent
- Proper interval, normally means reduce your interval
- multiple ips
- different headers
- dynamic pages -> prentend to be a real man. /phantom.js/selenium/chrome headless
- set proper cookie
- verification code -> ocr
- fetch data from mobile webpage

## anti scraper

- block ip
- http headers, such as user-agent
- cookie
- return fake data

## How does web scraper work

TODO