AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-07-10 09:11:46 +08:00
Code Issues Packages Projects Releases Wiki Activity
firecrawl/apps/api/src/scraper/scrapeURL
History
Gergő Móricz 5c62bb1195
feat: new snips test framework (FIR-414) (#1033) 
* feat: new snips test framework

* Update mock.ts

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-01-13 20:50:47 +01:00
..
engines
feat: new snips test framework (FIR-414) (#1033)
2025-01-13 20:50:47 +01:00
lib
feat: new snips test framework (FIR-414) (#1033)
2025-01-13 20:50:47 +01:00
transformers
Nick: formatting fixes
2025-01-10 18:35:10 -03:00
.gitignore
feat: new snips test framework (FIR-414) (#1033)
2025-01-13 20:50:47 +01:00
error.ts
feat(scrapeURL/fire-engine/chrome-cdp): handle file downloads
2024-12-26 20:29:09 +01:00
index.ts
feat: new snips test framework (FIR-414) (#1033)
2025-01-13 20:50:47 +01:00
README.md
WebScraper refactor into scrapeURL (#714)
2024-11-07 20:57:33 +01:00
scrapeURL.test.ts
Nick: revert trailing comma
2024-12-11 19:51:08 -03:00

README.md

scrapeURL

New URL scraper for Firecrawl

Signal flow

flowchart TD;
    scrapeURL-.->buildFallbackList;
    buildFallbackList-.->scrapeURLWithEngine;
    scrapeURLWithEngine-.->parseMarkdown;
    parseMarkdown-.->wasScrapeSuccessful{{Was scrape successful?}};
    wasScrapeSuccessful-."No".->areEnginesLeft{{Are there engines left to try?}};
    areEnginesLeft-."Yes, try next engine".->scrapeURLWithEngine;
    areEnginesLeft-."No".->NoEnginesLeftError[/NoEnginesLeftError/]
    wasScrapeSuccessful-."Yes".->asd;

Differences from WebScraperDataProvider

  • The job of WebScraperDataProvider.validateInitialUrl has been delegated to the zod layer above scrapeUrl.
  • WebScraperDataProvider.mode has no equivalent, only scrape_url is supported.
  • You may no longer specify multiple URLs.
  • Built on v1 definitons, instead of v0.
  • PDFs are now converted straight to markdown using LlamaParse, instead of converting to just plaintext.
  • DOCXs are now converted straight to HTML (and then later to markdown) using mammoth, instead of converting to just plaintext.
  • Using new JSON Schema OpenAI API -- schema fails with LLM Extract will be basically non-existant.