mirror of
https://github.com/wahyd4/kb.git
synced 2026-08-09 05:16:11 +10:00
hermes
1f0201a80f
- 3-layer architecture: raw (monthly buckets) → wiki (domain subdirs) → SCHEMA - Privacy: repo set to private (public-safe content only) - Sync: daily Links App sync via raw/links/YYYY-MM/ - Design: converged via Grok 4.5 2-round review
1.1 KiB
1.1 KiB
title, created, updated, type, tags, external
| title | created | updated | type | tags | external | ||
|---|---|---|---|---|---|---|---|
| Web Scraping | 2022-04-07 | 2022-04-07 | summary |
|
https://github.com/wahyd4/knowledge/blob/master/categories/web-scraping.md |
Web Scraping
A good scraper
- content downloading, download images, contents and so on.
- links retrieve, find new links
- URL management, avoid endless loop
- content analysis and management, export as csv or chart.
good frameworks
- colly
- pyspider
- scrapy
anti scraper blocking
- Ddon't use default user-agent, instead use real browser user agent
- Proper interval, normally means reduce your interval
- multiple ips
- different headers
- dynamic pages -> prentend to be a real man. /phantom.js/selenium/chrome headless
- set proper cookie
- verification code -> ocr
- fetch data from mobile webpage
anti scraper
- block ip
- http headers, such as user-agent
- cookie
- return fake data
How does web scraper work
TODO