home/crawl4ai

Fork 0

mirror of https://github.com/wahyd4/crawl4ai.git synced 2026-08-09 05:15:51 +10:00

Files

T

UncleCode c0fd36982d Update all documentation to import extraction strategies directly from crawl4ai.

2025-06-10 18:08:27 +08:00

34 KiB

Raw Blame History

Crawl4AI 🕷️🤖

Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐

GitHub Repository: https://github.com/unclecode/crawl4ai
Twitter: @unclecode
Website: https://crawl4ai.com

🌟 Meet the Crawl4AI Assistant: Your Copilot for Crawling

Use the Crawl4AI GPT Assistant as your AI-powered copilot! With this assistant, you can:

🧑‍💻 Generate code for complex crawling and extraction tasks
💡 Get tailored support and examples
📘 Learn Crawl4AI faster with step-by-step guidance

Quickstart with Crawl4AI

1. Installation

Install Crawl4AI and necessary dependencies:

In [ ]:

# %%capture
!pip install crawl4ai
!pip install nest_asyncio
!playwright install

In [3]:

import asyncio
import nest_asyncio
nest_asyncio.apply()

2. Basic Setup and Simple Crawl

In [ ]:

import asyncio
from crawl4ai import AsyncWebCrawler

async def simple_crawl():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
            bypass_cache=True # By default this is False, meaning the cache will be used
        )
        print(result.markdown.raw_markdown[:500])  # Print the first 500 characters
        
asyncio.run(simple_crawl())

[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 1.49 seconds
[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.10 seconds
[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler
[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.10 seconds.
IE 11 is not supported. For an optimal experience visit our site on another browser.

[Morning Rundown: Trump and Harris' vastly different closing pitches, why Kim Jong Un is helping Russia, and an ancient city is discovered by accident](https://www.nbcnews.com/news/harris-speech-ellipse-ancient-mayan-city-morning-rundown-rcna177973)[](https://www.nbcnews.com/news/harris-speech-ellipse-ancient-mayan-city-morning-rundown-rcna177973)

Skip to Content

[NBC News Logo](https://www.nbcnews.com)

Spon

3. Dynamic Content Handling

In [ ]:

async def crawl_dynamic_content():
    # You can use wait_for to wait for a condition to be met before returning the result
    # wait_for = """() => {
    #     return Array.from(document.querySelectorAll('article.tease-card')).length > 10;
    # }"""

    # wait_for can be also just a css selector
    # wait_for = "article.tease-card:nth-child(10)"

    async with AsyncWebCrawler(verbose=True) as crawler:
        js_code = [
            "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();"
        ]
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
            js_code=js_code,
            # wait_for=wait_for,
            bypass_cache=True,
        )
        print(result.markdown.raw_markdown[:500])  # Print first 500 characters

asyncio.run(crawl_dynamic_content())

[LOG] 🌤️  Warming up the AsyncWebCrawler
[LOG] 🌞 AsyncWebCrawler is ready to crawl
[LOG] 🕸️ Crawling https://www.nbcnews.com/business using AsyncPlaywrightCrawlerStrategy...
[LOG] ✅ Crawled https://www.nbcnews.com/business successfully!
[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 4.52 seconds
[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.15 seconds
[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler
[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.15 seconds.
IE 11 is not supported. For an optimal experience visit our site on another browser.

[Morning Rundown: Trump and Harris' vastly different closing pitches, why Kim Jong Un is helping Russia, and an ancient city is discovered by accident](https://www.nbcnews.com/news/harris-speech-ellipse-ancient-mayan-city-morning-rundown-rcna177973)[](https://www.nbcnews.com/news/harris-speech-ellipse-ancient-mayan-city-morning-rundown-rcna177973)

Skip to Content

[NBC News Logo](https://www.nbcnews.com)

Spon

4. Content Cleaning and Fit Markdown

In [ ]:

async def clean_content():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://janineintheworld.com/places-to-visit-in-central-mexico",
            excluded_tags=['nav', 'footer', 'aside'],
            remove_overlay_elements=True,
            word_count_threshold=10,
            bypass_cache=True
        )
        full_markdown_length = len(result.markdown.raw_markdown)
        fit_markdown_length = len(result.markdown.fit_markdown)
        print(f"Full Markdown Length: {full_markdown_length}")
        print(f"Fit Markdown Length: {fit_markdown_length}")
        print(result.markdown.fit_markdown[:1000])
        

asyncio.run(clean_content())

5. Link Analysis and Smart Filtering

In [23]:


async def link_analysis():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
            bypass_cache=True,
            exclude_external_links=True,
            exclude_social_media_links=True,
            # exclude_domains=["facebook.com", "twitter.com"]
        )
        print(f"Found {len(result.links['internal'])} internal links")
        print(f"Found {len(result.links['external'])} external links")

        for link in result.links['internal'][:5]:
            print(f"Href: {link['href']}\nText: {link['text']}\n")
                

asyncio.run(link_analysis())

[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 0.93 seconds
[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.11 seconds
[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler
[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.11 seconds.
Found 107 internal links
Found 58 external links
Href: https://www.nbcnews.com/news/harris-speech-ellipse-ancient-mayan-city-morning-rundown-rcna177973
Text: Morning Rundown: Trump and Harris' vastly different closing pitches, why Kim Jong Un is helping Russia, and an ancient city is discovered by accident

Href: https://www.nbcnews.com
Text: NBC News Logo

Href: https://www.nbcnews.com/politics/2024-election/live-blog/kamala-harris-donald-trump-rally-election-live-updates-rcna177529
Text: 2024 Election

Href: https://www.nbcnews.com/politics
Text: Politics

Href: https://www.nbcnews.com/us-news
Text: U.S. News

6. Media Handling

In [25]:

async def media_handling():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business", 
            bypass_cache=True,
            exclude_external_images=False,
            screenshot=True
        )
        for img in result.media['images'][:5]:
            print(f"Image URL: {img['src']}, Alt: {img['alt']}, Score: {img['score']}")
        
asyncio.run(media_handling())

[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 1.42 seconds
[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.11 seconds
[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler
[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.12 seconds.
Image URL: https://media-cldnry.s-nbcnews.com/image/upload/t_focal-762x508,f_auto,q_auto:best/rockcms/2024-10/241023-NM-Chilccare-jg-27b982.jpg, Alt: , Score: 4
Image URL: https://media-cldnry.s-nbcnews.com/image/upload/t_focal-80x80,f_auto,q_auto:best/rockcms/2024-10/241030-china-ev-electric-mb-0746-cae05c.jpg, Alt: Volkswagen Workshop in Hefei, Score: 5
Image URL: https://media-cldnry.s-nbcnews.com/image/upload/t_focal-80x80,f_auto,q_auto:best/rockcms/2024-10/241029-nyc-subway-sandwich-2021-ac-922p-a92374.jpg, Alt: A sub is prepared at a Subway restaurant in Manhattan, New York City, Score: 5
Image URL: https://media-cldnry.s-nbcnews.com/image/upload/t_focal-80x80,f_auto,q_auto:best/rockcms/2024-10/241029-suv-gravity-ch-1618-752415.jpg, Alt: The Lucid Gravity car., Score: 5
Image URL: https://media-cldnry.s-nbcnews.com/image/upload/t_focal-80x80,f_auto,q_auto:best/rockcms/2024-10/241029-dearborn-michigan-f-150-ford-ranger-trucks-assembly-line-ac-426p-614f0b.jpg, Alt: Ford Introduces new F-150 And Ranger Trucks At Their Dearborn Plant, Score: 5

7. Using Hooks for Custom Workflow

Hooks in Crawl4AI allow you to run custom logic at specific stages of the crawling process. This can be invaluable for scenarios like setting custom headers, logging activities, or processing content before it is returned. Below is an example of a basic workflow using a hook, followed by a complete list of available hooks and explanations on their usage.

In [ ]:

async def custom_hook_workflow():
    async with AsyncWebCrawler() as crawler:
        # Set a 'before_goto' hook to run custom code just before navigation
        crawler.crawler_strategy.set_hook("before_goto", lambda page: print("[Hook] Preparing to navigate..."))
        
        # Perform the crawl operation
        result = await crawler.arun(
            url="https://crawl4ai.com",
            bypass_cache=True
        )
        print(result.markdown.raw_markdown[:500])  # Display the first 500 characters

asyncio.run(custom_hook_workflow())

[Hook] Preparing to navigate...
[LOG] 🚀 Crawling done for https://crawl4ai.com, success: True, time taken: 3.49 seconds
[LOG] 🚀 Content extracted for https://crawl4ai.com, success: True, time taken: 0.03 seconds
[LOG] 🔥 Extracting semantic blocks for https://crawl4ai.com, Strategy: AsyncWebCrawler
[LOG] 🚀 Extraction done for https://crawl4ai.com, time taken: 0.03 seconds.
[Crawl4AI Documentation](https://docs.crawl4ai.com/)

  * [ Home ](.)
  * [ Installation ](basic/installation/)
  * [ Quick Start ](basic/quickstart/)
  * [ Search ](#)



  * Home
  * [Installation](basic/installation/)
  * [Quick Start](basic/quickstart/)
  * Basic
    * [Simple Crawling](basic/simple-crawling/)
    * [Output Formats](basic/output-formats/)
    * [Browser Configuration](basic/browser-config/)
    * [Page Interaction](basic/page-interaction/)
    * [Content Selection](basic/con

List of available hooks and examples for each stage of the crawling process:

on_browser_created

async def on_browser_created_hook(browser):
    print("[Hook] Browser created")

before_goto

async def before_goto_hook(page):
    await page.set_extra_http_headers({"X-Test-Header": "test"})

after_goto

async def after_goto_hook(page):
    print(f"[Hook] Navigated to {page.url}")

on_execution_started

async def on_execution_started_hook(page):
    print("[Hook] JavaScript execution started")

before_return_html

async def before_return_html_hook(page, html):
    print(f"[Hook] HTML length: {len(html)}")

8. Session-Based Crawling

When to Use Session-Based Crawling: Session-based crawling is especially beneficial when navigating through multi-page content where each page load needs to maintain the same session context. For instance, in cases where a “Next Page” button must be clicked to load subsequent data, the new data often replaces the previous content. Here, session-based crawling keeps the browser state intact across each interaction, allowing for sequential actions within the same session.

Example: Multi-Page Navigation Using JavaScript In this example, we’ll navigate through multiple pages by clicking a "Next Page" button. After each page load, we extract the new content and repeat the process.

In [ ]:

async def multi_page_session_crawl():
    async with AsyncWebCrawler() as crawler:
        session_id = "page_navigation_session"
        url = "https://example.com/paged-content"

        for page_number in range(1, 4):
            result = await crawler.arun(
                url=url,
                session_id=session_id,
                js_code="document.querySelector('.next-page-button').click();" if page_number > 1 else None,
                css_selector=".content-section",
                bypass_cache=True
            )
            print(f"Page {page_number} Content:")
            print(result.markdown.raw_markdown[:500])  # Print first 500 characters

# asyncio.run(multi_page_session_crawl())

9. Using Extraction Strategies

LLM Extraction

This example demonstrates how to use language model-based extraction to retrieve structured data from a pricing page on OpenAI’s site.

In [31]:

from crawl4ai import LLMExtractionStrategy
from pydantic import BaseModel, Field
import os, json

class OpenAIModelFee(BaseModel):
    model_name: str = Field(..., description="Name of the OpenAI model.")
    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
    output_fee: str = Field(
        ..., description="Fee for output token for the OpenAI model."
    )

async def extract_structured_data_using_llm(provider: str, api_token: str = None, extra_headers: dict = None):
    print(f"\n--- Extracting Structured Data with {provider} ---")
    
    # Skip if API token is missing (for providers that require it)
    if api_token is None and provider != "ollama":
        print(f"API token is required for {provider}. Skipping this example.")
        return

    extra_args = {"extra_headers": extra_headers} if extra_headers else {}

    async with AsyncWebCrawler(verbose=True) as crawler:
        result = await crawler.arun(
            url="https://openai.com/api/pricing/",
            word_count_threshold=1,
            extraction_strategy=LLMExtractionStrategy(
                provider=provider,
                api_token=api_token,
                schema=OpenAIModelFee.schema(),
                extraction_type="schema",
                instruction="""Extract all model names along with fees for input and output tokens."
                "{model_name: 'GPT-4', input_fee: 'US$10.00 / 1M tokens', output_fee: 'US$30.00 / 1M tokens'}.""",
                **extra_args
            ),
            bypass_cache=True,
        )
        print(json.loads(result.extracted_content)[:5])

# Usage:
await extract_structured_data_using_llm("openai/gpt-4o-mini", os.getenv("OPENAI_API_KEY"))

--- Extracting Structured Data with openai/gpt-4o-mini ---
[LOG] 🌤️  Warming up the AsyncWebCrawler
[LOG] 🌞 AsyncWebCrawler is ready to crawl
[LOG] 🕸️ Crawling https://openai.com/api/pricing/ using AsyncPlaywrightCrawlerStrategy...
[LOG] ✅ Crawled https://openai.com/api/pricing/ successfully!
[LOG] 🚀 Crawling done for https://openai.com/api/pricing/, success: True, time taken: 1.29 seconds
[LOG] 🚀 Content extracted for https://openai.com/api/pricing/, success: True, time taken: 0.13 seconds
[LOG] 🔥 Extracting semantic blocks for https://openai.com/api/pricing/, Strategy: AsyncWebCrawler
[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 0
[LOG] Extracted 26 blocks from URL: https://openai.com/api/pricing/ block index: 0
[LOG] 🚀 Extraction done for https://openai.com/api/pricing/, time taken: 15.12 seconds.
[{'model_name': 'gpt-4o', 'input_fee': '$2.50 / 1M input tokens', 'output_fee': '$10.00 / 1M output tokens', 'error': False}, {'model_name': 'gpt-4o-2024-08-06', 'input_fee': '$2.50 / 1M input tokens', 'output_fee': '$10.00 / 1M output tokens', 'error': False}, {'model_name': 'gpt-4o-audio-preview', 'input_fee': '$2.50 / 1M input tokens', 'output_fee': '$10.00 / 1M output tokens', 'error': False}, {'model_name': 'gpt-4o-audio-preview-2024-10-01', 'input_fee': '$2.50 / 1M input tokens', 'output_fee': '$10.00 / 1M output tokens', 'error': False}, {'model_name': 'gpt-4o-2024-05-13', 'input_fee': '$5.00 / 1M input tokens', 'output_fee': '$15.00 / 1M output tokens', 'error': False}]

/Users/unclecode/devs/crawl4ai/venv/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings:
  Expected `PromptTokensDetails` but got `dict` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(

Cosine Similarity Strategy

This strategy uses semantic clustering to extract relevant content based on contextual similarity, which is helpful when extracting related sections from a single topic.

In [32]:

from crawl4ai import CosineStrategy

async def cosine_similarity_extraction():
    async with AsyncWebCrawler() as crawler:
        strategy = CosineStrategy(
            word_count_threshold=10,
            max_dist=0.2, # Maximum distance between two words
            linkage_method="ward", # Linkage method for hierarchical clustering (ward, complete, average, single)
            top_k=3, # Number of top keywords to extract
            sim_threshold=0.3, # Similarity threshold for clustering
            semantic_filter="McDonald's economic impact, American consumer trends", # Keywords to filter the content semantically using embeddings
            verbose=True
        )
        
        result = await crawler.arun(
            url="https://www.nbcnews.com/business/consumer/how-mcdonalds-e-coli-crisis-inflation-politics-reflect-american-story-rcna177156",
            extraction_strategy=strategy
        )
        print(json.loads(result.extracted_content)[:5])

asyncio.run(cosine_similarity_extraction())

[LOG] Loading Extraction Model for mps device.
[LOG] Loading Multilabel Classifier for mps device.
[LOG] Model loaded sentence-transformers/all-MiniLM-L6-v2, models/reuters, took 5.193778038024902 seconds
[LOG] 🚀 Crawling done for https://www.nbcnews.com/business/consumer/how-mcdonalds-e-coli-crisis-inflation-politics-reflect-american-story-rcna177156, success: True, time taken: 1.37 seconds
[LOG] 🚀 Content extracted for https://www.nbcnews.com/business/consumer/how-mcdonalds-e-coli-crisis-inflation-politics-reflect-american-story-rcna177156, success: True, time taken: 0.07 seconds
[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business/consumer/how-mcdonalds-e-coli-crisis-inflation-politics-reflect-american-story-rcna177156, Strategy: AsyncWebCrawler
[LOG] 🚀 Assign tags using mps
[LOG] 🚀 Categorization done in 0.55 seconds
[LOG] 🚀 Extraction done for https://www.nbcnews.com/business/consumer/how-mcdonalds-e-coli-crisis-inflation-politics-reflect-american-story-rcna177156, time taken: 6.63 seconds.
[{'index': 1, 'tags': ['news_&_social_concern'], 'content': "McDonald's 2024 combo: Inflation, a health crisis and a side of politics # McDonald's 2024 combo: Inflation, a health crisis and a side of politics"}, {'index': 2, 'tags': ['business_&_entrepreneurs', 'news_&_social_concern'], 'content': 'Like many major brands, McDonald’s raked in big profits as the economy reopened from the pandemic. In October 2022, [executives were boasting](https://www.cnbc.com/2022/10/27/mcdonalds-mcd-earnings-q3-2022.html) that they’d been raising prices without crimping traffic, even as competitors began to warn that some customers were closing their wallets after inflation peaked above 9% that summer. Still, the U.S. had repeatedly dodged a much-forecast recession, and [Americans kept spending on nonessentials](https://www.nbcnews.com/business/economy/year-peak-inflation-travel-leisure-mostly-cost-less-rcna92760) like travel and dining out — despite regularly relaying to pollsters their dismal views of an otherwise solid economy. Even so, 64% of consumers said they noticed price increases at quick-service restaurants in September, more than at any other type of venue, according to a survey by Datassential, a food and beverage market researcher. Politicians are still drawing attention to fast-food costs, too, as the election season barrels toward a tumultuous finish. A group of Democratic senators this month [denounced McDonald’s for menu prices](https://www.nbcnews.com/news/us-news/democratic-senators-slam-mcdonalds-menu-price-hikes-rcna176380) that they said outstripped inflation, accusing the company of looking to profit “at the expense of people’s ability to put food on the table.” The financial results come toward the end of a humbling year for the nearly $213 billion restaurant chain, whose shares remained steady on the heels of its latest earnings. Kempczinski [sought to reassure investors](https://www.cnbc.com/2024/10/29/mcdonalds-e-coli-outbreak-ceo-comments.html) that [the E. coli outbreak](https://www.nbcnews.com/health/health-news/illnesses-linked-mcdonalds-e-coli-outbreak-rise-75-cdc-says-rcna177260), linked to Quarter Pounder burgers, was under control after the health crisis temporarily dented the company’s stock and caused U.S. foot traffic to drop nearly 10% in the days afterward, according to estimates by Gordon Haskett financial researchers. The fast-food giant [reported Tuesday](https://www.cnbc.com/2024/10/29/mcdonalds-mcd-earnings-q3-2024.html) that it had reversed its recent U.S. sales drop, posting a 0.3% uptick in the third quarter. Foot traffic was still down slightly, but the company said its summer of discounts was paying off. But by early this year, [photos of eye-watering menu prices](https://x.com/sam_learner/status/1681367351143301129) at some McDonald’s locations — including an $18 Big Mac combo at a Connecticut rest stop from July 2023 — went viral, bringing diners’ long-simmering frustrations to a boiling point that the company couldn’t ignore. On an earnings call in April, Kempczinski acknowledged that foot traffic had fallen. “We will stay laser-focused on providing an unparalleled experience with simple, everyday value and affordability that our consumers can count on as they continue to be mindful about their spending,” CEO Chris Kempczinski [said in a statement](https://www.prnewswire.com/news-releases/mcdonalds-reports-third-quarter-2024-results-302289216.html?Fds-Load-Behavior=force-external) alongside the earnings report.'}, {'index': 3, 'tags': ['food_&_dining', 'news_&_social_concern'], 'content': '![mcdonalds drive-thru economy fast food](https://media-cldnry.s-nbcnews.com/image/upload/t_fit-760w,f_auto,q_auto:best/rockcms/2024-10/241024-los-angeles-mcdonalds-drive-thru-ac-1059p-cfc311.jpg)McDonald’s has had some success leaning into discounts this year. Eric Thayer / Bloomberg via Getty Images file'}, {'index': 4, 'tags': ['business_&_entrepreneurs', 'food_&_dining', 'news_&_social_concern'], 'content': 'McDonald’s has faced a customer revolt over pricey Big Macs, an unsolicited cameo in election-season crossfire, and now an E. coli outbreak — just as the company had been luring customers back with more affordable burgers. Despite a difficult quarter, McDonald’s looks resilient in the face of various pressures, analysts say — something the company shares with U.S. consumers overall. “Consumers continue to be even more discriminating with every dollar that they spend,” he said at the time. Going forward, McDonald’s would be “laser-focused” on affordability.  “McDonald’s has also done a good job of embedding the brand in popular culture to enhance its relevance and meaning around fun and family. But it also needed to modify the product line to meet the expectations of a consumer who is on a tight budget,” he said. “The thing that McDonald’s had struggled with, and why I think we’re seeing kind of an inflection point, is a value proposition,” Senatore said. “McDonald’s menu price increases had run ahead of a lot of its restaurant peers. … Consumers are savvy enough to know that.” For many consumers, the fast-food giant’s menus serve as an informal gauge of the economy overall, said Sara Senatore, a Bank of America analyst covering restaurants. “The spotlight is always on McDonald’s because it’s so big” and something of a “bellwether,” she said. McDonald’s didn’t respond to requests for comment.'}, {'index': 5, 'tags': ['business_&_entrepreneurs', 'food_&_dining'], 'content': 'Mickey D’s’ $5 meal deal, which it launched in late June to jumpstart slumping sales, has given the company an appealing price point to advertise nationwide, Senatore said, speculating that it could open the door to a new permanent value offering. But before that promotion rolled out, the company’s reputation as a low-cost option had taken a bruising hit.'}]

10. Conclusion and Next Steps

You’ve explored core features of Crawl4AI, including dynamic content handling, link analysis, and advanced extraction strategies. Visit our documentation for further details on using Crawl4AI’s extensive features.

GitHub Repository: https://github.com/unclecode/crawl4ai
Twitter: @unclecode
Website: https://crawl4ai.com

Happy Crawling with Crawl4AI! 🕷️🤖

34 KiB Raw Blame History Unescape Escape

Crawl4AI 🕷️🤖

🌟 Meet the Crawl4AI Assistant: Your Copilot for Crawling

Quickstart with Crawl4AI

1. Installation

2. Basic Setup and Simple Crawl

3. Dynamic Content Handling

4. Content Cleaning and Fit Markdown

5. Link Analysis and Smart Filtering

6. Media Handling

7. Using Hooks for Custom Workflow

8. Session-Based Crawling

9. Using Extraction Strategies

10. Conclusion and Next Steps

34 KiB

Raw Blame History