Crawl a website

Crawl a website

Follows links from a starting URL and returns the content of every page it reaches, within the limits you set.

Note: The API starts a job and returns its id: read the pages with GET /crawl/{id}; the MCP tool waits for the crawl to finish. Set a limit and include or exclude paths to keep the crawl bounded.

Set up

Use the first option your agent supports.

MCP (official, remote)

Add this server to your agent's MCP settings, then sign in when asked.

{ "mcpServers": { "firecrawl": { "url": "https://mcp.firecrawl.dev/v2/mcp-oauth" } } }

Call the MCP tool firecrawl_crawl.

Server URL: https://mcp.firecrawl.dev/v2/mcp-oauth

CLI (official)

Install the command, then confirm it runs.

npm install -g firecrawl-cli
firecrawl --version

Run firecrawl crawl.

Set $FIRECRAWL_API_KEY in your environment first (get a key: https://www.firecrawl.dev/app/api-keys).

API (official)

• Base URL: https://api.firecrawl.dev/v2
• Endpoint: POST /crawl
• Auth: send the header Authorization: Bearer $FIRECRAWL_API_KEY
• Get a key: https://www.firecrawl.dev/app/api-keys
• Docs: https://docs.firecrawl.dev/api-reference/introduction

Before anything else, confirm access with the cheapest read-only call, like a list or a search. Never send or change anything to test access.

Rules

• Ask the user before anything that sends messages, costs money, or changes data, and say how many records it touches. One approval covers a batch the user has seen.
• Never print API keys.