AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-08-02 05:30:38 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,044 Commits 496 Branches 17 Tags
Commit Graph

3044 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Gergő Móricz
7cf2e52fe6
feat(crawl): add maxDiscoveryDepth (#1329) 2025-03-12 18:46:57 +01:00
Gergő Móricz
d855f5a567 fix(test/snips/scrape): allow more time for stealth proxy 2025-03-12 18:22:40 +01:00
Gergő Móricz
97dee5de21 temp(ci/self-host): remove no-proxy option 2025-03-12 18:19:24 +01:00
Gergő Móricz
337bc261f7 temp: remove flakey ad-block tests 2025-03-12 18:17:52 +01:00
Abdul Rahman ArM
e9804d25f9
fix: resolve circular JSON structure error in search function (#1330) 
Modified error handlers to avoid logging entire response objects that contain circular references
 2025-03-12 18:13:53 +01:00
Gergő Móricz
0154e40685 fix(js-sdk/extract): fix zod type check with zod version discrepancy 2025-03-12 16:02:44 +01:00
Nicolas
4cc33e1e8d Nick: added team_id_o 2025-03-10 22:30:26 -04:00
Gergő Móricz
0c457e6f2f adjust acuc balance 2025-03-07 20:34:43 +01:00
rafaelmmiller
3786ad7f5d restart 2025-03-07 16:19:59 -03:00
Nicolas
7ec1437d14 Merge branch 'fix/preview_token' v1.6.0 2025-03-06 19:17:09 -03:00
Nicolas
1ced546bd0 Update rate-limiter.ts 2025-03-06 19:17:02 -03:00
Nicolas
60346ecfa3
Fix/p token (#1305) 
* preview-token

* Nick:

---------

Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
 2025-03-06 19:03:33 -03:00
Nicolas
f45b3c0155 Merge branch 'main' into fix/preview_token 2025-03-06 19:02:29 -03:00
Nicolas
b66e0ccb7d Nick: 2025-03-06 18:36:01 -03:00
rafaelmmiller
55a4a3d768 preview-token 2025-03-06 18:16:07 -03:00
Gergő Móricz
39b1390289 fix: crawl 2025-03-06 21:57:42 +01:00
Gergő Móricz
a1e6c13b67 move crawl to read replica 2025-03-06 21:57:42 +01:00
Nicolas
982b3da27d Update auth.ts 2025-03-06 17:53:41 -03:00
Nicolas
5a149a1c30 Revert "fix(auth): always use replica for acuc" 
This reverts commit 57b313608cb6e9f3c123e4754c653fef8dfacbdf.
 2025-03-06 17:53:28 -03:00
Nicolas
783fad90dd Nick: more read replicas 2025-03-06 17:45:51 -03:00
Nicolas
949fb68bdd Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-03-06 17:36:57 -03:00
Nicolas
4c4d51e60b Revert "Update runWebScraper.ts" 
This reverts commit 40eacfacd4c5a782f722fd85e3935f4a1741b27a.
 2025-03-06 17:36:08 -03:00
Gergő Móricz
57b313608c fix(auth): always use replica for acuc 2025-03-06 21:35:30 +01:00
Gergő Móricz
7b05512054 fix(credit_billing): teams check 2025-03-06 21:30:57 +01:00
Nicolas
ae010a76c1 Update blocklist.ts 2025-03-06 17:11:00 -03:00
Gergő Móricz
e6c3f20944 fix(preview): temporarily disable 2025-03-06 21:08:12 +01:00
Nicolas
72d894c2a3 Update rate-limiter.ts 2025-03-06 17:06:42 -03:00
Nicolas
1de5a2c589 Update batch_billing.ts 2025-03-06 16:57:23 -03:00
Nicolas
eba28a4210 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-03-06 16:41:22 -03:00
Nicolas
40eacfacd4 Update runWebScraper.ts 2025-03-06 16:41:21 -03:00
Gergő Móricz
e1cfe1da48
feat(crawl): includes/excludes fixes (FIR-1300) (#1303) 
* feat(crawl): includes/excludes fixes pt. 1

* fix(snips): billing tests

* drop tha logs

* fix(ci): add replica url

* feat(crawl): drop initial scrape if it's not included

* feat(ci): more verbose logging

* fix crawl path in test

* fix(ci): wait for api

* fix(snips/scrape/ad): test for more pixels

* feat(js-sdk/crawl): add regexOnFullURL
 2025-03-06 17:05:15 +01:00
Gergő Móricz
f8df18ed6a feat(feng-check): run on chrome-cdp 2025-03-05 18:53:10 +01:00
Gergő Móricz
c22c87ab0a fix(crawler): check for more strings 2025-03-05 18:48:05 +01:00
Nicolas
4902d0ac64 Nick: fixes 2025-03-04 23:22:58 -03:00
Nicolas
8cfc946cae Nick: fix system prompt missing from extract params 2025-03-04 23:04:20 -03:00
Eric Ciarla
5a1886936c
Truncate llmstxt cache based on maxurls limit & improve maxurls handling (#1285) 
* init

* Update generate-llmstxt-service.ts
 2025-03-03 18:37:33 -03:00
Gergő Móricz
1beadf39f0 fix 2025-03-03 21:06:01 +01:00
Gergő Móricz
6dce8e9ddf asd 2025-03-03 21:05:45 +01:00
Gergő Móricz
8620bf3d5e fix(auth): split load evenly between two instances 2025-03-03 20:53:51 +01:00
Gergő Móricz
67ee2662de feat(auth): force acuc on read replica 2025-03-03 20:36:54 +01:00
Nicolas
64af3ba728 Revert "Update auth.ts" 
This reverts commit fea249c568c4a8d63bcc2e560fd3655e73350be6.
 2025-03-03 16:26:42 -03:00
Eric Ciarla
72eb360eba
Merge pull request #1291 from aparupganguly/feature/claude3.7-extractor 
examples/Add Claude 3.7 web extractor
 2025-03-03 13:41:47 -05:00
Aparup Ganguly
bced299e36 examples/Add Claude 3.7 web extractor 2025-03-03 17:48:52 +05:30
Nicolas
9eb25786dc Update deep-research-service.ts 2025-03-02 18:16:20 -03:00
Nicolas
aa54fd1658 Nick: only new activities/sources in the callback 2025-03-02 17:46:12 -03:00
Nicolas
33c59ee474 Nick: fixes 2025-03-02 17:41:19 -03:00
Nicolas
30c1e92a2e Update deep-research-service.ts 2025-03-02 17:37:15 -03:00
Nicolas
22d4f0de38
(feat/deep-research) Alpha prep + Improvements (#1284) 
* Update index.ts

* Nick:

* Nick: topic -> query

* Update deep-research.ts

* Nick: bump

* Nick: add onSource callback
 2025-03-02 17:33:21 -03:00
Gergő Móricz
9ad947884d
feat(tests/snips): add billing tests + misc billing fixes (FIR-1280) (#1283) 
* feat(tests/snips): add billing tests + misc billing fixes

* add testing key

* asd
 2025-03-02 16:51:42 -03:00
Gergő Móricz
4f25f12a12
fix(ai): handle if AI returns a JSON code block (#1280) 2025-03-02 15:25:24 -03:00