mirror of
https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl
synced 2025-07-11 03:51:49 +08:00
scrapeURL
New URL scraper for Firecrawl
Signal flow
flowchart TD;
scrapeURL-.->buildFallbackList;
buildFallbackList-.->scrapeURLWithEngine;
scrapeURLWithEngine-.->parseMarkdown;
parseMarkdown-.->wasScrapeSuccessful{{Was scrape successful?}};
wasScrapeSuccessful-."No".->areEnginesLeft{{Are there engines left to try?}};
areEnginesLeft-."Yes, try next engine".->scrapeURLWithEngine;
areEnginesLeft-."No".->NoEnginesLeftError[/NoEnginesLeftError/]
wasScrapeSuccessful-."Yes".->asd;
Differences from WebScraperDataProvider
- The job of
WebScraperDataProvider.validateInitialUrl
has been delegated to the zod layer abovescrapeUrl
. WebScraperDataProvider.mode
has no equivalent, onlyscrape_url
is supported.- You may no longer specify multiple URLs.
- Built on
v1
definitons, instead ofv0
. - PDFs are now converted straight to markdown using LlamaParse, instead of converting to just plaintext.
- DOCXs are now converted straight to HTML (and then later to markdown) using mammoth, instead of converting to just plaintext.
- Using new JSON Schema OpenAI API -- schema fails with LLM Extract will be basically non-existant.