> For the complete documentation index, see [llms.txt](https://docs.glbnxt.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.glbnxt.com/tutorials/guides/langflow/crawl4ai-in-langflow.md).

# Using Crawl4AI in LangFlow

This guide walks you through integrating **Crawl4AI**—the open-source, LLM-friendly web crawler—directly into your **Langflow** workflows. By creating a Custom Component in Langflow, you can dynamically bypass heavy HTML, parse raw web data, and stream clean Markdown straight to your LLMs or Vector Stores.

***

### Prerequisites

* An active **Langflow** instance.
* **Crawl4AI** running.
* Your Crawl4AI API Authorization Token (to be found in [GLBNXT Applications](https://platform.glbnxt.com/applications) > Services > Crawl4AI > Actions).

***

### Steps

#### Step 1: Create a Custom Component in Langflow

To interact with Crawl4AI inside Langflow, the cleanest method is to build a **Custom Component** that handles the asynchronous HTTP requests.

1. Open your Langflow dashboard and create a new flow.
2. In the canvas toolbar, click on the options to create a **Custom Component**.
3. Click **View Code** on the new component and replace the default template with the following Python code:

```python
import httpx
from langflow.custom import CustomComponent
from langflow.inputs import StrInput, SecretStrInput, BoolInput
from langflow.io import Output
from langflow.schema import Data

class Crawl4AIComponent(CustomComponent):
    display_name = "Crawl4AI Web Scraper"
    description = "Crawl web pages and convert them to clean Markdown using Crawl4AI."
    icon = "globe"

    inputs = [
        StrInput(
            name="url", 
            display_name="Target URL", 
            info="The web address you want to crawl and scrape.",
            value="https://example.com"
        ),
        StrInput(
            name="base_url", 
            display_name="Crawl4AI Base URL", 
            info="The base URL of your self-hosted Crawl4AI API.",
            value="http://localhost:11235"
        ),
        SecretStrInput(
            name="api_token", 
            display_name="API Bearer Token", 
            info="Authentication token for your Crawl4AI instance if required.",
            required=False
        ),
        BoolInput(
            name="only_text",
            display_name="Only Text",
            info="Strip images, layouts, and non-text elements.",
            value=True
        ),
        BoolInput(
            name="fit_markdown",
            display_name="Fit Markdown",
            info="Extract only the most semantically relevant content (heavily pruned).",
            value=False
        )
    ]

    outputs = [
        Output(name="markdown", display_name="Markdown Output", method="crawl_page")
    ]

    def crawl_page(self) -> Data:
        endpoint = f"{self.base_url.rstrip('/')}/crawl"
        
        headers = {}
        if self.api_token:
            headers["Authorization"] = f"Bearer {self.api_token}"
            
        payload = {
            "urls": [self.url],
            "browser_config": {
                "text_mode": True,
                "light_mode": True
            },
            "crawler_config": {
                "only_text": self.only_text,
                "word_count_threshold": 15,
                "remove_overlay_elements": True,
                "excluded_tags": [
                    "nav", "footer", "header", "aside", "form", "button", 
                    "iframe", "noscript", "style", "script", "svg"
                ]
            }
        }
        
        try:
            with httpx.Client(timeout=60.0) as client:
                response = client.post(endpoint, json=payload, headers=headers)
                response.raise_for_status()
                result_data = response.json()
                
                # Crawl4AI returns results in a nested list
                first_result = result_data.get("results", [{}])[0]
                
                # Check whether to return standard markdown or high-density "fit" markdown
                raw_markdown = first_result.get("markdown", "")
                if self.fit_markdown and isinstance(raw_markdown, dict):
                    output_text = raw_markdown.get("fit_markdown", "")
                elif isinstance(raw_markdown, dict):
                    output_text = raw_markdown.get("raw_markdown", "")
                else:
                    output_text = raw_markdown
                
                return Data(value=output_text)
                
        except Exception as e:
            return Data(value=f"Error crawling page: {str(e)}")
```

#### Step 2: Configure and Test the Component

Once the code is compiled, the component will visually transform on your canvas with all of its inputs:

* **Target URL:** Connect this port to your **Chat Input** (or any database text extractor) to pull URLs dynamically.
* **Crawl4AI Base URL:**

```url
http://crawl4ai.platform
```

* **Only Text:** Toggle this to strip image resources and media tags, speeding up browser load times.
* **Fit Markdown:** Turn this on if you want Crawl4AI to use its BM25 pruning algorithm to return only high-density, contextually relevant text to your LLM.

#### Step 3: Handle the Crawler Output in your Flow

Now, link the output port (**Markdown Output**) directly to upstream LLM Prompt templates or a **Text Splitter** block.

* **To feed an LLM:** Drag a connection from **Markdown Output** directly into an **Agent** or **Prompt** component.
* **To feed Vector Stores (RAG):** Connect it to a **Recursive Character Text Splitter** so the output can be cleanly chunked and indexed.

***

### Tips for Production Flows

> 💡 **Avoid Rate Limits:** When building agent loops, make sure to enable `light_mode` and use a `semaphore_count` in your Crawl4AI instance config to prevent memory exhaustion on concurrent crawler calls.
>
> 💡 **Bypassing Cache:** By default, Crawl4AI uses smart caching. If your AI Agent requires real-time, live data (like news or stock prices), you can append `"cache_mode": "BYPASS"` to your python crawler config parameters inside the custom component.

***

### Need More Information?

For deep dives into CSS/XPath extraction schemas, LLM-based parsing, dynamic browser interaction scripts, or session management, consult the [Crawl4AI Documentation](https://docs.crawl4ai.com/).

***

*Need help? Contact the GLBNXT support team or ask a GLBNXT agent to walk you through the setup.*


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.glbnxt.com/tutorials/guides/langflow/crawl4ai-in-langflow.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.