home/crawl4ai
2
0
Fork 0
mirror of https://github.com/wahyd4/crawl4ai.git synced 2026-08-24 04:45:50 +10:00
Code Issues Packages Projects Releases Wiki Activity
296 Commits 1 Branch 0 Tags
870296fa7ee43b221cdede34dbe22a8a2ea4ea4c
Commit Graph
296 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
bizrockmanandGitHub 870296fa7e Rename episode_11_1_Extraction_Strategies:_JSON_CSS.md to episode_11_1_Extraction_Strategies_JSON_CSS.md 
Name that will work in Windows
 2024-11-04 20:18:58 +01:00
bizrockmanandGitHub a28046c233 Rename episode_08_Media_Handling:_Images,_Videos,_and_Audio.md to episode_08_Media_Handling_Images_Videos_and_Audio.md 
Name that will work in Windows
 2024-11-04 20:18:26 +01:00
bizrockmanandGitHub 0bba0e074f Preventing NoneType has no attribute get Errors 
Sometimes the list contains Tag elements that do not have attrs set, resulting in this Error.
 2024-11-04 20:12:24 +01:00
UncleCodeandGitHub de6b43f334 Merge pull request #215 from mjvankampen/build/flexible-requirements 
build: make requirements more flexible
 2024-11-03 08:30:06 +01:00
UncleCodeandGitHub 07f508bd0c Merge pull request #218 from timoa/main 
chore(docs): fix documentation links + markdown lint fix
 2024-11-03 06:59:30 +01:00
UncleCode 62a86dbe8d Refactor mission section in README and add mission diagram 2024-10-31 16:38:56 +08:00
UncleCode 492ada0ed4 Add mission diagram to MISSION.md 2024-10-31 15:26:43 +08:00
UncleCode d8eef02867 Add link to mission statement in README 2024-10-31 15:23:58 +08:00
UncleCode 6c7235d6a7 Add mission.md file 2024-10-31 15:22:00 +08:00
Damien Laureaux 0a09d78fa5 chore(docs): fix documentation links + markdown lint 2024-10-31 05:50:22 +01:00
UncleCode 19c3f3efb2 Refactor tutorial markdown files: Update numbering and formatting 2024-10-30 20:58:07 +08:00
UncleCode e97e8df6ba Update README: Fix typo in project name 2024-10-30 20:45:20 +08:00
UncleCode cb6f5323ae Update README 2024-10-30 20:44:57 +08:00
UncleCode 47464cedec Update README 2024-10-30 20:42:27 +08:00
UncleCode 982d203d91 Merge branch '0.3.73' 2024-10-30 20:40:09 +08:00
UncleCode 9307c19f35 Update documents, upload new version of quickstart. 2024-10-30 20:39:35 +08:00
Mark Jan van Kampen 605a82793b fix dev requirements and lock playwright due to failing tests 2024-10-30 10:41:37 +01:00
Mark Jan van Kampen df9ee44d42 build: make requirements more flexible 
According to #102 the requirements specified are minimum version. Currently they are defined as fixed versions in requirements.txt and setup.py leading to projects consuming this package are limited to using exactly these requirements instead of a more flexible range. This PR addresses this.
 2024-10-30 10:03:22 +01:00
UncleCode e9f7d5e73a Merge branch '0.3.73' 2024-10-30 00:16:49 +08:00
UncleCode 3529c2e732 Update new tutorial documents and added to the docs folder. 2024-10-30 00:16:18 +08:00
UncleCode d9e0b7abab Fix README badge 2024-10-28 15:14:16 +08:00
UncleCode b2800fefc6 Add badges to README 2024-10-28 15:10:12 +08:00
UncleCode d913e20edc Update Readme 2024-10-28 15:09:37 +08:00
UncleCode c2a71a5abe Update Docs folder, prepare branch for new version 0.3.73 2024-10-27 19:35:13 +08:00
UncleCode d61615e0b0 Merge branch '0.3.72' 2024-10-27 19:33:05 +08:00
UncleCode ac9d83c72f Update gitignore 2024-10-27 19:29:04 +08:00
UncleCode ff9149b5c9 Merge branch 'main' of https://github.com/unclecode/crawl4ai 2024-10-27 19:28:05 +08:00
UncleCode 4239654722 Update Documentation 2024-10-27 19:24:46 +08:00
UncleCode 38474bd66a Update version 2024-10-24 20:24:21 +08:00
UncleCode bcfe83f702 feat: enhance crawler with overlay removal and improved screenshot capabilities 
• Add smart overlay removal system for handling popups and modals
• Improve screenshot functionality with configurable timing controls
• Implement URL normalization and enhanced link processing
• Add custom base directory support for cache storage
• Refine external content filtering and social media domain handling

This commit significantly improves the crawler's ability to handle modern
websites by automatically removing intrusive overlays and providing better
screenshot capabilities. URL handling is now more robust with proper
normalization and duplicate detection. The cache system is more flexible
with customizable base directory support.

Breaking changes: None
Issue numbers: None
 2024-10-24 20:22:47 +08:00
UncleCodeandGitHub 32f57c49d6 Merge pull request #194 from IdrisHanafi/feat/customize-crawl-base-directory 
Support for custom crawl base directory
 2024-10-24 13:09:27 +02:00
UncleCode 60ba131ac8 [v0.3.72] Enhance content extraction and proxy support 
- Add ContentCleaningStrategy for improved content extraction
- Implement advanced proxy configuration with authentication
- Enhance image source detection and handling
- Add fit_markdown and fit_html for refined content output
- Improve external link and image handling flexibility
 2024-10-22 20:19:22 +08:00
Idris Hanafi a5f627ba1a feat: customize crawl base directory 2024-10-21 17:58:39 -04:00
UncleCode 04d16e6d2b Fix Base64 image parsing in WebScrappingStrategy (issue 182) 
- Add support for extracting Base64 encoded images
- Improve image format detection to include Base64 images
- Enhance compatibility with locally saved HTML files using Base64 image encoding
 2024-10-20 19:25:25 +08:00
UncleCode 1dd36f9035 Refactor content scrapping strategy and improve error handling 2024-10-20 19:11:18 +08:00
UncleCode 6ec4cb33ca Enhance Markdown generation and external content control 
- Integrate customized html2text library for flexible Markdown output
- Add options to exclude external links and images
- Improve content scraping efficiency and error handling
- Update AsyncPlaywrightCrawlerStrategy for faster closing
- Enhance CosineStrategy with generic embedding model loading
 2024-10-20 18:56:58 +08:00
UncleCode e7cd8a1c2d Update Changelog 2024-10-19 18:37:12 +08:00
UncleCode 4e2852d5ff [v0.3.71] Enhance chunking strategies and improve overall performance 
- Add OverlappingWindowChunking and improve SlidingWindowChunking
- Update CHUNK_TOKEN_THRESHOLD to 2048 tokens
- Optimize AsyncPlaywrightCrawlerStrategy close method
- Enhance flexibility in CosineStrategy with generic embedding model loading
- Improve JSON-based extraction strategies
- Add knowledge graph generation example
 2024-10-19 18:36:59 +08:00
UncleCode b309bc34e1 Fix the model nam ein quick start example 2024-10-18 15:32:25 +08:00
UncleCode b8147b64e0 chore: Bump version to 0.3.71 and improve error handling 
- Update version number to 0.3.71
- Add sleep_on_close option to AsyncPlaywrightCrawlerStrategy
- Enhance context creation with additional options
- Improve error message formatting and visibility
- Update quickstart documentation
 2024-10-18 13:31:12 +08:00
UncleCode aab6ea022e Update requirements and switch to 0.3.8 2024-10-18 12:51:23 +08:00
UncleCode dd17ed0e63 Rename some flags name, introducing magic flag. 2024-10-18 12:35:09 +08:00
UncleCode dbb587d681 Update gitignore 2024-10-17 21:38:48 +08:00
UncleCode 768aa06ceb feat(crawler): Enhance stealth and flexibility, improve error handling 
- Implement playwright_stealth for better bot detection avoidance
- Add user simulation and navigator override options
- Improve iframe processing and browser selection
- Enhance error reporting and debugging capabilities
- Optimize image processing and parallel crawling
- Add new example for user simulation feature
- Added support for including links in Markdown content, by definin g a new flag `include_links_on_markdown` in `crawl` method.
 2024-10-17 21:37:48 +08:00
unclecode 9ffa34b697 Update README 2024-10-14 22:58:27 +08:00
unclecode 740802c491 Merge branch '0.3.6' 2024-10-14 22:55:24 +08:00
unclecode b9ac96c332 Merge branch 'main' of https://github.com/unclecode/crawl4ai 2024-10-14 22:54:23 +08:00
unclecode d06535388a Update gitignore 2024-10-14 22:53:56 +08:00
unclecode 2b73bdf6b0 Update changelog 2024-10-14 21:04:02 +08:00
unclecode 6aa803d712 Update gitignore 2024-10-14 21:03:40 +08:00