AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-07-31 11:02:01 +08:00
Code Issues Packages Projects Releases Wiki Activity
1,548 Commits 496 Branches 17 Tags
Commit Graph

1548 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
rafaelsideguide
7727302ef1 Merge remote-tracking branch 'origin/v1-webscraper' into v1/node-sdk 2024-08-20 09:22:55 -03:00
rafaelsideguide
e160d5529a fixed test 2024-08-20 09:22:38 -03:00
rafaelsideguide
e1c9cbf709 bug fixed. crawl should not stop if sitemap url is invalid 2024-08-20 09:11:58 -03:00
Thomas Kosmas
0dce57832d
Merge pull request #555 from mendableai/feat/beta-customers 
added variables to beta customers
 2024-08-20 02:20:23 +03:00
rafaelsideguide
ecd472356b added variables to beta customers 2024-08-19 16:41:54 -03:00
Nicolas
3dc298be54 Nick: 2x rate limits for standard and growth for /scrape 2024-08-19 13:52:54 -03:00
rafaelsideguide
32aba44167 fixing merge issues 2024-08-19 13:37:20 -03:00
rafaelsideguide
72461ce9a6 Update index.test.ts 2024-08-19 13:29:52 -03:00
rafaelsideguide
fd7fdc1d52 added blocklist middleware 2024-08-19 13:28:54 -03:00
Nicolas
5a44191344
Merge pull request #554 from mendableai/nsc/check-team-credits-limit 
Check team credits based on the crawl limit
 2024-08-19 11:54:47 -03:00
rafaelsideguide
e516e4993b Merge remote-tracking branch 'origin/v1-webscraper' into v1/node-sdk 2024-08-19 11:24:57 -03:00
Nicolas
8e4ca86463 Update crawl.ts 2024-08-19 11:02:24 -03:00
Nicolas
36b35dbc67 Update crawl.ts 2024-08-19 11:01:26 -03:00
rafaelsideguide
4ffc60596a Update queue-worker.ts 2024-08-19 09:29:23 -03:00
rafaelsideguide
b8170aaa47 Update blocklist.ts 2024-08-19 08:51:48 -03:00
Nicolas
3fe82b4f12 Update queue-worker.ts 2024-08-17 03:09:31 -04:00
Nicolas
f797380112 Nick: 2024-08-16 22:17:38 -04:00
Nicolas
47123be783 Nick: weird activity block 2024-08-16 22:01:56 -04:00
Nicolas
ff84f1fe5e Update map.ts 2024-08-16 20:42:36 -04:00
Nicolas
4314313477 Update map.ts 2024-08-16 19:56:18 -04:00
Nicolas
af9a0a6f0b Update map.ts 2024-08-16 19:56:03 -04:00
Nicolas
ba5279eafc Nick: all tests passing 2024-08-16 19:55:44 -04:00
Nicolas
5205c5f005 Update map.ts 2024-08-16 19:37:00 -04:00
Nicolas
0c05d096a9 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 19:33:58 -04:00
Nicolas
ab48353226 Nick: /map almost good 2024-08-16 19:33:57 -04:00
Gergő Móricz
eb84673b06 feat: crawl status websocket WIP 2024-08-17 01:04:14 +02:00
Gergő Móricz
e2a6ef26d3 mount v1Router under v1 path 2024-08-16 23:48:50 +02:00
Gergő Móricz
4c1b74dab3 fix(map): remove robots.txt 2024-08-16 23:46:10 +02:00
Gergő Móricz
c281fe62c0 fix(crawl): propagate db fix to preview endpoint 2024-08-16 23:43:54 +02:00
Gergő Móricz
803577eeba feat(crawl): webhook 2024-08-16 23:42:48 +02:00
Gergő Móricz
e6738abf96 fix(crawl-status): retrieve from DB in bulk 2024-08-16 23:39:39 +02:00
rafaelsideguide
086ba6280b fixed markdown format 2024-08-16 18:39:13 -03:00
Gergő Móricz
aabfaf0ac5 clean up crawl-status, fix db ddos 2024-08-16 23:29:39 +02:00
rafaelsideguide
e5b807ccc4 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 17:57:31 -03:00
rafaelsideguide
b311fe1898 Create temp-37564.rdb 2024-08-16 17:57:14 -03:00
rafaelsideguide
7a61325500 map + search + scrape markdown bug 2024-08-16 17:57:11 -03:00
Gergő Móricz
5896153d19 fix: crawl status and redis fixes 2024-08-16 22:52:48 +02:00
Gergő Móricz
3fcb21930e remove log 2024-08-16 22:48:23 +02:00
Gergő Móricz
f20328bdbb crawl status and document stuff 2024-08-16 22:48:05 +02:00
Nicolas
0c057bb649 Update index.test.ts 2024-08-16 16:45:10 -04:00
Nicolas
b32464558a Update index.test.ts 2024-08-16 16:41:09 -04:00
Nicolas
5bac7988a6 Update index.test.ts 2024-08-16 16:08:38 -04:00
Nicolas
290c7ee936 Update index.test.ts 2024-08-16 16:06:46 -04:00
Nicolas
23a033fe61 Nick: fixes and more e2e tests 2024-08-16 16:03:35 -04:00
Nicolas
78ca94251c
Merge pull request #480 from mendableai/nsc/hyper-v81 
Reduce metrics ingestion w/ HyperDX v0.8.1
 2024-08-16 14:34:14 -04:00
Nicolas
37ae9a9043 Update index.test.ts 2024-08-16 14:17:43 -04:00
Nicolas
200ce8e2ce Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 14:16:35 -04:00
Nicolas
21d3798e49 Nick: initial e2e v1 tests for /scrape 2024-08-16 14:16:30 -04:00
rafaelsideguide
3f998b688d scrape ready 2024-08-16 15:14:37 -03:00
Nicolas
b0d211ecc1 Merge branch 'main' into v1-webscraper 2024-08-16 13:43:28 -04:00