Using Crawl4AI in LangFlow

This guide walks you through integrating Crawl4AI—the open-source, LLM-friendly web crawler—directly into your Langflow workflows. By creating a Custom Component in Langflow, you can dynamically bypass heavy HTML, parse raw web data, and stream clean Markdown straight to your LLMs or Vector Stores.

---

Prerequisites

• An active Langflow instance.
• Crawl4AI running.
• Your Crawl4AI API Authorization Token (to be found in GLBNXT Applications > Services > Crawl4AI > Actions).

---

Steps

Step 1: Create a Custom Component in Langflow

To interact with Crawl4AI inside Langflow, the cleanest method is to build a Custom Component that handles the asynchronous HTTP requests.

1. Open your Langflow dashboard and create a new flow.
2. In the canvas toolbar, click on the options to create a Custom Component.
3. Click View Code on the new component and replace the default template with the following Python code:

import httpx
from langflow.custom import CustomComponent
from langflow.inputs import StrInput, SecretStrInput, BoolInput
from langflow.io import Output
from langflow.schema import Data

class Crawl4AIComponent(CustomComponent):
    display_name = "Crawl4AI Web Scraper"
    description = "Crawl web pages and convert them to clean Markdown using Crawl4AI."
    icon = "globe"

    inputs = [
        StrInput(
            name="url", 
            display_name="Target URL", 
            info="The web address you want to crawl and scrape.",
            value="https://example.com"
        ),
        StrInput(
            name="base_url", 
            display_name="Crawl4AI Base URL", 
            info="The base URL of your self-hosted Crawl4AI API.",
            value="http://localhost:11235"
        ),
        SecretStrInput(
            name="api_token", 
            display_name="API Bearer Token", 
            info="Authentication token for your Crawl4AI instance if required.",
            required=False
        ),
        BoolInput(
            name="only_text",
            display_name="Only Text",
            info="Strip images, layouts, and non-text elements.",
            value=True
        ),
        BoolInput(
            name="fit_markdown",
            display_name="Fit Markdown",
            info="Extract only the most semantically relevant content (heavily pruned).",
            value=False
        )
    ]

    outputs = [
        Output(name="markdown", display_name="Markdown Output", method="crawl_page")
    ]

    def crawl_page(self) -> Data:
        endpoint = f"{self.base_url.rstrip('/')}/crawl"
        
        headers = {}
        if self.api_token:
            headers["Authorization"] = f"Bearer {self.api_token}"
            
        payload = {
            "urls": [self.url],
            "browser_config": {
                "text_mode": True,
                "light_mode": True
            },
            "crawler_config": {
                "only_text": self.only_text,
                "word_count_threshold": 15,
                "remove_overlay_elements": True,
                "excluded_tags": [
                    "nav", "footer", "header", "aside", "form", "button", 
                    "iframe", "noscript", "style", "script", "svg"
                ]
            }
        }
        
        try:
            with httpx.Client(timeout=60.0) as client:
                response = client.post(endpoint, json=payload, headers=headers)
                response.raise_for_status()
                result_data = response.json()
                
                # Crawl4AI returns results in a nested list
                first_result = result_data.get("results", [{}])[0]
                
                # Check whether to return standard markdown or high-density "fit" markdown
                raw_markdown = first_result.get("markdown", "")
                if self.fit_markdown and isinstance(raw_markdown, dict):
                    output_text = raw_markdown.get("fit_markdown", "")
                elif isinstance(raw_markdown, dict):
                    output_text = raw_markdown.get("raw_markdown", "")
                else:
                    output_text = raw_markdown
                
                return Data(value=output_text)
                
        except Exception as e:
            return Data(value=f"Error crawling page: {str(e)}")

Step 2: Configure and Test the Component

Once the code is compiled, the component will visually transform on your canvas with all of its inputs:

• Target URL: Connect this port to your Chat Input (or any database text extractor) to pull URLs dynamically.
• Crawl4AI Base URL:

http://crawl4ai.platform

• Only Text: Toggle this to strip image resources and media tags, speeding up browser load times.
• Fit Markdown: Turn this on if you want Crawl4AI to use its BM25 pruning algorithm to return only high-density, contextually relevant text to your LLM.

Step 3: Handle the Crawler Output in your Flow

Now, link the output port (Markdown Output) directly to upstream LLM Prompt templates or a Text Splitter block.

• To feed an LLM: Drag a connection from Markdown Output directly into an Agent or Prompt component.
• To feed Vector Stores (RAG): Connect it to a Recursive Character Text Splitter so the output can be cleanly chunked and indexed.

---

Tips for Production Flows

💡 Avoid Rate Limits: When building agent loops, make sure to enable light_mode and use a semaphore_count in your Crawl4AI instance config to prevent memory exhaustion on concurrent crawler calls.

💡 Bypassing Cache: By default, Crawl4AI uses smart caching. If your AI Agent requires real-time, live data (like news or stock prices), you can append "cache_mode": "BYPASS" to your python crawler config parameters inside the custom component.

---

Need More Information?

For deep dives into CSS/XPath extraction schemas, LLM-based parsing, dynamic browser interaction scripts, or session management, consult the Crawl4AI Documentation.

---

Need help? Contact the GLBNXT support team or ask a GLBNXT agent to walk you through the setup.