UncleCode 16b2318242 feat(api): implement crawler pool manager for improved resource handling ...

Adds a new CrawlerManager class to handle browser instance pooling and failover:
- Implements auto-scaling based on system resources
- Adds primary/backup crawler management
- Integrates memory monitoring and throttling
- Adds streaming support with memory tracking
- Updates API endpoints to use pooled crawlers

BREAKING CHANGE: API endpoints now require CrawlerManager initialization

2025-04-18 22:26:24 +08:00

..

async

feat(browser): add standalone CDP browser launch and lxml extraction strategy

2025-03-07 20:55:56 +08:00

browser

feat(browser): implement browser pooling and page pre-warming

2025-03-31 21:55:07 +08:00

cli

feat(cli): add command line interface with comprehensive features

2025-02-10 16:58:52 +08:00

docker

docs(tests): clarify server URL comments in deep crawl tests

2025-04-15 22:32:27 +08:00

general

feat(markdown): add content source selection for markdown generation

2025-04-17 20:13:53 +08:00

hub

refactor(crawler): improve HTML handling and cleanup codebase

2025-02-07 21:56:27 +08:00

loggers

Release prep (#749)

2025-02-28 19:53:35 +08:00

memory

feat(api): implement crawler pool manager for improved resource handling

2025-04-18 22:26:24 +08:00

__init__.py

- Test all methods

2024-05-14 21:27:41 +08:00

docker_example.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

test_cli_docs.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

test_docker.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

test_llmtxt.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

test_main.py

Apply Ruff Corrections

2025-01-13 19:19:58 +08:00

test_scraping_strategy.py

Release prep (#749)

2025-02-28 19:53:35 +08:00

test_web_crawler.py

feat(browser): add standalone CDP browser launch and lxml extraction strategy

2025-03-07 20:55:56 +08:00