mirror of
https://github.com/wahyd4/one-knowledge.git
synced 2026-08-08 20:59:08 +10:00
50 lines
1.1 KiB
Markdown
50 lines
1.1 KiB
Markdown
---
|
|
title: Web Scraping
|
|
created: 2022-04-07
|
|
updated: 2022-04-07
|
|
type: summary
|
|
tags: [tech, reference]
|
|
external: https://github.com/wahyd4/knowledge/blob/master/categories/web-scraping.md
|
|
---
|
|
|
|
# Web Scraping
|
|
|
|
## A good scraper
|
|
|
|
- content downloading, download images, contents and so on.
|
|
- links retrieve, find new links
|
|
- URL management, avoid endless loop
|
|
- content analysis and management, export as csv or chart.
|
|
|
|
## good frameworks
|
|
|
|
- colly
|
|
- https://github.com/asciimoo/colly
|
|
- pyspider
|
|
- [http://docs.pyspider.org/en/latest/](http://docs.pyspider.org/en/latest/)
|
|
- scrapy
|
|
- [https://github.com/scrapy/scrapy](https://github.com/scrapy/scrapy)
|
|
|
|
## anti scraper blocking
|
|
|
|
|
|
- Ddon't use default user-agent, instead use real browser user agent
|
|
- Proper interval, normally means reduce your interval
|
|
- multiple ips
|
|
- different headers
|
|
- dynamic pages -> prentend to be a real man. /phantom.js/selenium/chrome headless
|
|
- set proper cookie
|
|
- verification code -> ocr
|
|
- fetch data from mobile webpage
|
|
|
|
## anti scraper
|
|
|
|
- block ip
|
|
- http headers, such as user-agent
|
|
- cookie
|
|
- return fake data
|
|
|
|
## How does web scraper work
|
|
|
|
TODO
|