Using Crawl4AI in n8n

This guide walks you through integrating Crawl4AI—the open-source, LLM-friendly web crawler—directly into your n8n workflows. By executing API calls to Crawl4AI, you can bypass heavy HTML and feed clean, structured Markdown directly to your LLMs or Vector Stores.

---

Prerequisites

• An active n8n instance.
• Crawl4AI running.
• Your Crawl4AI API Authorization Token (to be found in GLBNXT Applications > Services > Crawl4AI > Actions).

---

Steps

Step 1: Add the HTTP Request Node in n8n

To interact with Crawl4AI, add an HTTP Request node to your canvas. This node acts as the bridge that sends URLs to Crawl4AI and receives clean Markdown back.

Configure the node with the following parameters:

• Method: POST
• URL: http://crawl4ai.platform
• Authentication: Header Auth (or add a Header parameter manually)
• Headers:

  • Authorization: Bearer YOUR_CRAWL4AI_API_KEY (to be found in GLBNXT Applications Services)
• Send Body: Checked / True
• Body Content Type: JSON

Step 2: Configure the JSON Payload

Crawl4AI accepts fine-grained instructions directly within the JSON request body. Paste the following structure into your JSON Body section in n8n, utilizing dynamic expressions (such as {{ $json.url }}) to pass input URLs dynamically:

JSON

{
  "urls": ["{{ $json.url }}"],
  "browser_config": {
    "text_mode": true,
    "light_mode": true
  },
  "crawler_config": {
    "only_text": true,
    "word_count_threshold": 15,
    "remove_overlay_elements": true,
    "excluded_tags": [
      "nav", "footer", "header", "aside", "form", "button", 
      "iframe", "noscript", "style", "script", "svg"
    ]
  }
}

Key Parameter Breakdown

Configuration Block	Parameter	Description
browser_config	text_mode	Blocks images and heavy media from loading, speeding up the crawl.
	light_mode	Launches a lightweight browser context to optimize memory usage.
crawler_config	only_text	Instructs the crawler to focus strictly on text elements.
	word_count_threshold	Ignores layout blocks with fewer than 15 words to filter out UI clutter.
	excluded_tags	Completely strips non-content DOM elements like headers, footers, and sidebars before processing.

Step 3: Handle the Crawler Output in n8n

Crawl4AI returns raw crawl results embedded inside a structured JSON payload. When configuring subsequent LLM or Edit Fields nodes in your n8n workflow, you can reference the generated markdown outputs directly.

---

Extracting Raw Markdown

To retrieve the raw, converted Markdown string for your LLM or prompt:

JavaScript

{{ $json.results[0].markdown }}

---

Extracting Fit Markdown

Crawl4AI automatically applies content-pruning algorithms to generate a highly concise version of the page (Fit Markdown). To extract this optimized block:

JavaScript

{{ $json.results[0].markdown.fit_markdown }}

---

Tips for Production Workflows

💡 Inputs: Crawl4AI needs URL(s) to be able to scrape pages. It is therefore necessary that the node before the Crawl4AI node outputs URL(s). We highly recommend that you use an AI Agent with as tool SearXNG. This tool lets the AI Agent search for URLs on the web in real-time.

💡 Memory Management: If you are running high-concurrency crawls (processing many URLs at once), leverage light_mode: true and set a low semaphore_count in Crawl4AI's parameters to prevent your hosting environment from running out of memory.

💡 Caching: By default, Crawl4AI uses smart caching. If you need fresh data every execution, pass "cache_mode": "BYPASS" inside your crawler_config payload.

---

Need More Information?

For deep dives into advanced extraction strategies (like CSS/XPath schemas or LLM-based structured data extractions), concurrency limits, and dynamic browser interaction scripts, visit the official Crawl4AI Documentation.

---

Need help? Contact the GLBNXT support team or ask a GLBNXT agent to walk you through the setup.