scrapeURL

New URL scraper for Firecrawl

Signal flow

flowchart TD;
    scrapeURL-.->buildFallbackList;
    buildFallbackList-.->scrapeURLWithEngine;
    scrapeURLWithEngine-.->parseMarkdown;
    parseMarkdown-.->wasScrapeSuccessful{{Was scrape successful?}};
    wasScrapeSuccessful-."No".->areEnginesLeft{{Are there engines left to try?}};
    areEnginesLeft-."Yes, try next engine".->scrapeURLWithEngine;
    areEnginesLeft-."No".->NoEnginesLeftError[/NoEnginesLeftError/]
    wasScrapeSuccessful-."Yes".->asd;

Differences from WebScraperDataProvider

  • The job of WebScraperDataProvider.validateInitialUrl has been delegated to the zod layer above scrapeUrl.
  • WebScraperDataProvider.mode has no equivalent, only scrape_url is supported.
  • You may no longer specify multiple URLs.
  • Built on v1 definitons, instead of v0.
  • PDFs are now converted straight to markdown using LlamaParse, instead of converting to just plaintext.
  • DOCXs are now converted straight to HTML (and then later to markdown) using mammoth, instead of converting to just plaintext.
  • Using new JSON Schema OpenAI API -- schema fails with LLM Extract will be basically non-existant.