AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-06-21 03:24:20 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,475 Commits 496 Branches 17 Tags
Commit Graph

3475 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nicolas
783fad90dd Nick: more read replicas 2025-03-06 17:45:51 -03:00
Nicolas
949fb68bdd Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-03-06 17:36:57 -03:00
Nicolas
4c4d51e60b Revert "Update runWebScraper.ts" 
This reverts commit 40eacfacd4c5a782f722fd85e3935f4a1741b27a.
 2025-03-06 17:36:08 -03:00
Gergő Móricz
57b313608c fix(auth): always use replica for acuc 2025-03-06 21:35:30 +01:00
Gergő Móricz
7b05512054 fix(credit_billing): teams check 2025-03-06 21:30:57 +01:00
Nicolas
ae010a76c1 Update blocklist.ts 2025-03-06 17:11:00 -03:00
Gergő Móricz
e6c3f20944 fix(preview): temporarily disable 2025-03-06 21:08:12 +01:00
Nicolas
72d894c2a3 Update rate-limiter.ts 2025-03-06 17:06:42 -03:00
Nicolas
1de5a2c589 Update batch_billing.ts 2025-03-06 16:57:23 -03:00
Nicolas
eba28a4210 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-03-06 16:41:22 -03:00
Nicolas
40eacfacd4 Update runWebScraper.ts 2025-03-06 16:41:21 -03:00
Gergő Móricz
e1cfe1da48
feat(crawl): includes/excludes fixes (FIR-1300) (#1303) 
* feat(crawl): includes/excludes fixes pt. 1

* fix(snips): billing tests

* drop tha logs

* fix(ci): add replica url

* feat(crawl): drop initial scrape if it's not included

* feat(ci): more verbose logging

* fix crawl path in test

* fix(ci): wait for api

* fix(snips/scrape/ad): test for more pixels

* feat(js-sdk/crawl): add regexOnFullURL
 2025-03-06 17:05:15 +01:00
Gergő Móricz
f8df18ed6a feat(feng-check): run on chrome-cdp 2025-03-05 18:53:10 +01:00
Gergő Móricz
c22c87ab0a fix(crawler): check for more strings 2025-03-05 18:48:05 +01:00
Nicolas
4902d0ac64 Nick: fixes 2025-03-04 23:22:58 -03:00
Nicolas
8cfc946cae Nick: fix system prompt missing from extract params 2025-03-04 23:04:20 -03:00
Eric Ciarla
5a1886936c
Truncate llmstxt cache based on maxurls limit & improve maxurls handling (#1285) 
* init

* Update generate-llmstxt-service.ts
 2025-03-03 18:37:33 -03:00
Gergő Móricz
1beadf39f0 fix 2025-03-03 21:06:01 +01:00
Gergő Móricz
6dce8e9ddf asd 2025-03-03 21:05:45 +01:00
Gergő Móricz
8620bf3d5e fix(auth): split load evenly between two instances 2025-03-03 20:53:51 +01:00
Gergő Móricz
67ee2662de feat(auth): force acuc on read replica 2025-03-03 20:36:54 +01:00
Nicolas
64af3ba728 Revert "Update auth.ts" 
This reverts commit fea249c568c4a8d63bcc2e560fd3655e73350be6.
 2025-03-03 16:26:42 -03:00
Eric Ciarla
72eb360eba
Merge pull request #1291 from aparupganguly/feature/claude3.7-extractor 
examples/Add Claude 3.7 web extractor
 2025-03-03 13:41:47 -05:00
Aparup Ganguly
bced299e36 examples/Add Claude 3.7 web extractor 2025-03-03 17:48:52 +05:30
Nicolas
9eb25786dc Update deep-research-service.ts 2025-03-02 18:16:20 -03:00
Nicolas
aa54fd1658 Nick: only new activities/sources in the callback 2025-03-02 17:46:12 -03:00
Nicolas
33c59ee474 Nick: fixes 2025-03-02 17:41:19 -03:00
Nicolas
30c1e92a2e Update deep-research-service.ts 2025-03-02 17:37:15 -03:00
Nicolas
22d4f0de38
(feat/deep-research) Alpha prep + Improvements (#1284) 
* Update index.ts

* Nick:

* Nick: topic -> query

* Update deep-research.ts

* Nick: bump

* Nick: add onSource callback
 2025-03-02 17:33:21 -03:00
Gergő Móricz
9ad947884d
feat(tests/snips): add billing tests + misc billing fixes (FIR-1280) (#1283) 
* feat(tests/snips): add billing tests + misc billing fixes

* add testing key

* asd
 2025-03-02 16:51:42 -03:00
Gergő Móricz
4f25f12a12
fix(ai): handle if AI returns a JSON code block (#1280) 2025-03-02 15:25:24 -03:00
Eric Ciarla
522f2d2e3b
Merge pull request #1267 from ceewaigit/main 
Add groq_web_crawler example and dependencies
 2025-03-02 11:32:04 -05:00
Jean-Marie Porchet
42e92216b6
docs: remove undefined "required" field (#1282) 2025-03-02 17:27:52 +01:00
Gergő Móricz
e8c698d613
feat(crawler): handle cross-origin redirects differently than same-origin redirects (#1279) 2025-03-02 13:32:46 +01:00
Nicolas
fea249c568 Update auth.ts 2025-03-02 02:40:34 -03:00
Nicolas
99e61c967c Update index.ts 2025-03-02 02:39:52 -03:00
Eric Ciarla
ab8dcab6e2
Merge pull request #1276 from aparupganguly/feature/gpt4.5-crawler 2025-02-28 14:57:30 +01:00
Aparup Ganguly
06cdd988a4 examples/Add gpt 4.5 web crawler 2025-02-28 18:17:32 +05:30
Gergő Móricz
05c29e8299 js-sdk: bump 2025-02-28 12:40:39 +01:00
Gergő Móricz
856ec37d4c fix(ci/js-sdk): properly build SDK before publishing 2025-02-28 12:34:53 +01:00
Gergő Móricz
904e69bfbc
feat(supabase): add read replica routing (#1274) 2025-02-28 09:52:26 +01:00
Nicolas
39b61132bf Nick: fixed js sdk 2025-02-27 20:39:53 -03:00
Gergő Móricz
44bf59229a fix(acuc): cache for 1 hour 2025-02-27 21:36:33 +01:00
Nicolas
b72e21a697
Nick: batch billing (#1264) 2025-02-27 20:18:03 +01:00
Nicolas
289e351c14
(feat/deep-research-alpha) Added Max Urls, Sources and Fixes (#1271) 
* Nick: fixes

* Nick:

* Update deep-research-status.ts
 2025-02-27 13:24:40 -03:00
Gergő Móricz
1d3757b391 bump map to 30k 2025-02-27 12:44:23 +01:00
Benjy
78334e4ef4
feat(self-host/ai): pass in the ollama envs into docker compose (#1269) 2025-02-27 11:56:31 +01:00
Grass Huang
7bf04d409a
fix(scraper): improve charset detection regex to accurately parse meta tags (#1265) 2025-02-26 17:31:06 +01:00
ceewaigithub
75ac980fe4 Add groq_web_crawler example and dependencies 2025-02-26 22:24:38 +08:00
Nicolas
bf1a79588e Merge branch 'feat/fire-index' 2025-02-25 21:01:30 -03:00