AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-06-04 11:24:40 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,460 Commits 496 Branches 17 Tags
Commit Graph

3460 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Gergő Móricz
387dd3aa38 fix(tests/snips/billing): don't wait 40s for nothing when self hosted 2025-03-13 20:07:52 +01:00
Gergő Móricz
c7ae50d2d0
fix(crawler): sitemaps poisoning crawls with unrelated links (#1334) 2025-03-13 19:31:05 +01:00
Dominik Farr
da6b7505f3 Meaningful log message for high resource usage errors 2025-03-13 11:08:04 -07:00
Nicolas
c6cad942ab Nick: errors -> warn 2025-03-13 13:07:51 -04:00
Nicolas
7ec278a908 Nick: fixes 2025-03-13 13:05:12 -04:00
Nicolas
134de67a3b
(fix/map) Map failed to filter by path if indexed (#1333) 
* Nick:

* Update map.ts

* Update map.ts
 2025-03-13 12:48:56 -04:00
Gergő Móricz
f87e11712c fix: don't log bull secret 2025-03-13 17:30:37 +01:00
Loris
c53df8650a
Update searxng.ts (#1319) 
remove default categories. otherwise we cannot use engines
 2025-03-13 17:21:44 +01:00
rafaelmmiller
6a5a4e5b6f improv/types-and-comments-descs 2025-03-13 11:21:35 -03:00
Nicolas
d54af97754 Update rate-limiter.ts 2025-03-12 21:57:50 -04:00
Nicolas
b87b0d9b7f Merge branch 'nsc/concurrent-browsers' 2025-03-12 20:21:26 -04:00
Nicolas
e886189394 Update search.ts 2025-03-12 20:21:19 -04:00
Nicolas
1e6b484add
(feat/pricing) Concurrent Browsers - Improve rate limits (#1331) 
* Update rate-limiter.ts

* Update rate-limiter.ts
 2025-03-12 15:23:09 -04:00
Nicolas
d438b235f6 Update rate-limiter.ts 2025-03-12 15:22:28 -04:00
Gergő Móricz
71b6b83ec2
tally rework api switchover (#1328) 
* tally rework api switchover

* fix and send logs

* temp: force main instance while RPCs propagate

* Revert "temp: force main instance while RPCs propagate"

This reverts commit 4c93379cfa64efd60eb4767dd8eced1bdd302531.
 2025-03-12 20:10:33 +01:00
Nicolas
9edbdc987f Update rate-limiter.ts 2025-03-12 14:17:22 -04:00
Gergő Móricz
7cf2e52fe6
feat(crawl): add maxDiscoveryDepth (#1329) 2025-03-12 18:46:57 +01:00
Gergő Móricz
d855f5a567 fix(test/snips/scrape): allow more time for stealth proxy 2025-03-12 18:22:40 +01:00
Gergő Móricz
97dee5de21 temp(ci/self-host): remove no-proxy option 2025-03-12 18:19:24 +01:00
Gergő Móricz
337bc261f7 temp: remove flakey ad-block tests 2025-03-12 18:17:52 +01:00
Abdul Rahman ArM
e9804d25f9
fix: resolve circular JSON structure error in search function (#1330) 
Modified error handlers to avoid logging entire response objects that contain circular references
 2025-03-12 18:13:53 +01:00
Gergő Móricz
0154e40685 fix(js-sdk/extract): fix zod type check with zod version discrepancy 2025-03-12 16:02:44 +01:00
Nicolas
4cc33e1e8d Nick: added team_id_o 2025-03-10 22:30:26 -04:00
Gergő Móricz
0c457e6f2f adjust acuc balance 2025-03-07 20:34:43 +01:00
rafaelmmiller
3786ad7f5d restart 2025-03-07 16:19:59 -03:00
Nicolas
7ec1437d14 Merge branch 'fix/preview_token' v1.6.0 2025-03-06 19:17:09 -03:00
Nicolas
1ced546bd0 Update rate-limiter.ts 2025-03-06 19:17:02 -03:00
Nicolas
60346ecfa3
Fix/p token (#1305) 
* preview-token

* Nick:

---------

Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
 2025-03-06 19:03:33 -03:00
Nicolas
f45b3c0155 Merge branch 'main' into fix/preview_token 2025-03-06 19:02:29 -03:00
Nicolas
b66e0ccb7d Nick: 2025-03-06 18:36:01 -03:00
rafaelmmiller
55a4a3d768 preview-token 2025-03-06 18:16:07 -03:00
Gergő Móricz
39b1390289 fix: crawl 2025-03-06 21:57:42 +01:00
Gergő Móricz
a1e6c13b67 move crawl to read replica 2025-03-06 21:57:42 +01:00
Nicolas
982b3da27d Update auth.ts 2025-03-06 17:53:41 -03:00
Nicolas
5a149a1c30 Revert "fix(auth): always use replica for acuc" 
This reverts commit 57b313608cb6e9f3c123e4754c653fef8dfacbdf.
 2025-03-06 17:53:28 -03:00
Nicolas
783fad90dd Nick: more read replicas 2025-03-06 17:45:51 -03:00
Nicolas
949fb68bdd Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-03-06 17:36:57 -03:00
Nicolas
4c4d51e60b Revert "Update runWebScraper.ts" 
This reverts commit 40eacfacd4c5a782f722fd85e3935f4a1741b27a.
 2025-03-06 17:36:08 -03:00
Gergő Móricz
57b313608c fix(auth): always use replica for acuc 2025-03-06 21:35:30 +01:00
Gergő Móricz
7b05512054 fix(credit_billing): teams check 2025-03-06 21:30:57 +01:00
Nicolas
ae010a76c1 Update blocklist.ts 2025-03-06 17:11:00 -03:00
Gergő Móricz
e6c3f20944 fix(preview): temporarily disable 2025-03-06 21:08:12 +01:00
Nicolas
72d894c2a3 Update rate-limiter.ts 2025-03-06 17:06:42 -03:00
Nicolas
1de5a2c589 Update batch_billing.ts 2025-03-06 16:57:23 -03:00
Nicolas
eba28a4210 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-03-06 16:41:22 -03:00
Nicolas
40eacfacd4 Update runWebScraper.ts 2025-03-06 16:41:21 -03:00
Gergő Móricz
e1cfe1da48
feat(crawl): includes/excludes fixes (FIR-1300) (#1303) 
* feat(crawl): includes/excludes fixes pt. 1

* fix(snips): billing tests

* drop tha logs

* fix(ci): add replica url

* feat(crawl): drop initial scrape if it's not included

* feat(ci): more verbose logging

* fix crawl path in test

* fix(ci): wait for api

* fix(snips/scrape/ad): test for more pixels

* feat(js-sdk/crawl): add regexOnFullURL
 2025-03-06 17:05:15 +01:00
Gergő Móricz
f8df18ed6a feat(feng-check): run on chrome-cdp 2025-03-05 18:53:10 +01:00
Gergő Móricz
c22c87ab0a fix(crawler): check for more strings 2025-03-05 18:48:05 +01:00
Nicolas
4902d0ac64 Nick: fixes 2025-03-04 23:22:58 -03:00