AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-08-12 00:09:01 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,127 Commits 496 Branches 17 Tags
Commit Graph

2446 Commits

Author SHA1 Message Date
Gergő Móricz
8620bf3d5e fix(auth): split load evenly between two instances 2025-03-03 20:53:51 +01:00
Gergő Móricz
67ee2662de feat(auth): force acuc on read replica 2025-03-03 20:36:54 +01:00
Nicolas
64af3ba728 Revert "Update auth.ts" 
This reverts commit fea249c568c4a8d63bcc2e560fd3655e73350be6.
 2025-03-03 16:26:42 -03:00
Nicolas
9eb25786dc Update deep-research-service.ts 2025-03-02 18:16:20 -03:00
Nicolas
aa54fd1658 Nick: only new activities/sources in the callback 2025-03-02 17:46:12 -03:00
Nicolas
33c59ee474 Nick: fixes 2025-03-02 17:41:19 -03:00
Nicolas
30c1e92a2e Update deep-research-service.ts 2025-03-02 17:37:15 -03:00
Nicolas
22d4f0de38
(feat/deep-research) Alpha prep + Improvements (#1284) 
* Update index.ts

* Nick:

* Nick: topic -> query

* Update deep-research.ts

* Nick: bump

* Nick: add onSource callback
 2025-03-02 17:33:21 -03:00
Gergő Móricz
9ad947884d
feat(tests/snips): add billing tests + misc billing fixes (FIR-1280) (#1283) 
* feat(tests/snips): add billing tests + misc billing fixes

* add testing key

* asd
 2025-03-02 16:51:42 -03:00
Gergő Móricz
4f25f12a12
fix(ai): handle if AI returns a JSON code block (#1280) 2025-03-02 15:25:24 -03:00
Gergő Móricz
e8c698d613
feat(crawler): handle cross-origin redirects differently than same-origin redirects (#1279) 2025-03-02 13:32:46 +01:00
Nicolas
fea249c568 Update auth.ts 2025-03-02 02:40:34 -03:00
Nicolas
99e61c967c Update index.ts 2025-03-02 02:39:52 -03:00
Gergő Móricz
05c29e8299 js-sdk: bump 2025-02-28 12:40:39 +01:00
Gergő Móricz
904e69bfbc
feat(supabase): add read replica routing (#1274) 2025-02-28 09:52:26 +01:00
Nicolas
39b61132bf Nick: fixed js sdk 2025-02-27 20:39:53 -03:00
Gergő Móricz
44bf59229a fix(acuc): cache for 1 hour 2025-02-27 21:36:33 +01:00
Nicolas
b72e21a697
Nick: batch billing (#1264) 2025-02-27 20:18:03 +01:00
Nicolas
289e351c14
(feat/deep-research-alpha) Added Max Urls, Sources and Fixes (#1271) 
* Nick: fixes

* Nick:

* Update deep-research-status.ts
 2025-02-27 13:24:40 -03:00
Gergő Móricz
1d3757b391 bump map to 30k 2025-02-27 12:44:23 +01:00
Grass Huang
7bf04d409a
fix(scraper): improve charset detection regex to accurately parse meta tags (#1265) 2025-02-26 17:31:06 +01:00
Nicolas
31df234127 Update log_job.ts 2025-02-25 21:01:05 -03:00
Nicolas
ec90aaffd6 Update log_job.ts 2025-02-25 21:01:00 -03:00
Nicolas
59d09f5c45 Update log_job.ts 2025-02-25 19:32:16 -03:00
Gergő Móricz
115b6b61c4 add initial codeowners 2025-02-25 14:28:09 +01:00
Gergő Móricz
8c42b08b7e
feat(v1/crawl-status-ws): update behavior to ignore errors like regular crawl-status (#1234) 2025-02-24 21:44:29 +01:00
Gergő Móricz
15489be542
feat(self-host/ai): use any OpenAI-compatible API (#1245) 2025-02-23 09:07:32 +01:00
Nicolas
b24ac0f6b5
Nick: done (#1237) 2025-02-22 20:18:46 -03:00
Nicolas
5ab86b8b43
(fix/token-slicer) Fixes extract token limit issues (#1236) 
* Nick: fixes extract token limit errors

* Update llmExtract.ts

* Update llmExtract.ts
 2025-02-21 20:44:42 +01:00
Gergő Móricz
76e1f29ae8
Update Dockerfile (#1231) (#1232) 
* Update Dockerfile (#1231)

* Dockerfile: re-add prod-deps stage and fix copies

---------

Co-authored-by: Loris <loris.rion@gmail.com>
 2025-02-21 17:44:01 +01:00
Nicolas
6c51ef401e Update rate-limiter.ts 2025-02-20 22:31:43 -03:00
Nicolas
25d9bdb1f6
(feat/ai-sdk) Migrate to AI-SDK (#1220) 
* Nick: init

* Update llmExtract.ts

* Update llmExtract.ts

* Nick rename

* fix(v1/types): extract json schema validation

* Update url-processor.ts

* feat(ai-sdk): ollama support

* feat(ai-sdk): further ollama support

* Nick: it is broken btw

* feat(ai-sdk): abstract model adapter

* Update pnpm-lock.yaml

* Update analyzeSchemaAndPrompt.ts

* Nick:

* feat(ai-sdk): ollama support

* doc(SELF_HOST): update with embedding param

* Nick:

* Update ranker.ts

* Nick:

* feat(ai-sdk): fixes

* Update llmExtract.ts

* feat: remove zod-to-json-schema

* fix

* Update llmExtract.ts

* use openai

* fixes

---------

Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-20 22:48:58 +01:00
Gergő Móricz
16c305775e
fix(crawl-redis): ignore empty includes/excludes (#1223) 
* fix(crawl-redis): ignore empty includes/excludes

* fix(snips/scrape): bump timeouts
 2025-02-20 19:06:02 +01:00
Gergő Móricz
283a3bfef3
fix(scrapeURL/engines/fetch): discover charset and re-decode (#1221) 
* fix(scrapeURL/engines/fetch): discover charset and re-decode

* fix(snips/scrape): allow more time for stealth proxy
 2025-02-20 18:56:15 +01:00
Gergő Móricz
e417f83c28
feat(self-host): ollama support (#1219) 2025-02-20 16:59:19 +01:00
Gergő Móricz
e84c7325d9 chore: remove dead code 
Fixes #1149
Fixes #1150
 2025-02-20 15:44:34 +01:00
Gergő Móricz
62d11f21fc self-host: remove old playwright-service 2025-02-20 15:39:19 +01:00
Nicolas
2151ca846c Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-02-20 10:50:32 -03:00
Nicolas
7db2d25efa Nick: 2025-02-20 10:50:22 -03:00
Gergő Móricz
c38dcd0432
feat(self-host): proxy support (FIR-1111) (#1212) 
* feat(self-host): proxy support

* fix(playwright-service-ts): return untreated text/plain
 2025-02-20 14:20:03 +01:00
Gergő Móricz
c75522f535
feat(sdk): enforce timeout on client-side if set (#1204) 2025-02-20 13:35:31 +01:00
Loris
100168ddf3
Add searxng for search endpoint (#1193) 
* add searxng.ts

* update to add searxng endpoint

* Apply suggestions from code review

* feat(ci/self-host): add tests with searxng

* feat(ci/self-host): bootstrap searxng for testing

* feat(ci): improvements in syntax

---------

Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-20 12:36:53 +01:00
Gergő Móricz
da1670b78c
feat(map): mock support (FIR-1109) (#1213) 
* feat(map,fetch): mock support

* feat(snips/map): mock out long-running test

* fix(snips/scrape): use more reliable site for adblock testing
 2025-02-20 10:41:43 +01:00
Gergő Móricz
11ed679274 feat(scrapeURL/pdf): support PDF prefetch when parsePDF is off 2025-02-20 09:28:13 +01:00
Gergő Móricz
5eb0235ccb feat(apps/api): remove Sentry builds 2025-02-20 08:19:06 +01:00
Gergő Móricz
10d9b65f96 fix(self-host): update docs and dockerignore 2025-02-20 07:57:57 +01:00
niazarak
e4504b3236
Use correct list typing for py 3.8 support (#931) 
list[str] is not supported in Python 3.8
 2025-02-20 07:39:46 +01:00
Gergő Móricz
55d047b6b3
feat(scrapeURL): handle PDFs behind anti-bot (#1198) 2025-02-20 04:11:30 +01:00
Gergő Móricz
c39cc27866
feat(ci/self-host): add playwright microservice tests (#1210) 
* feat(ci/self-host): add playwright microservice tests

* fix ci

* fix ci 2

* fix ci 3

* fix(playwright-service): get raw JSON if response is JSON
 2025-02-20 02:06:13 +01:00
Shaun Offenbacher
037f64916a
Playwright page error schema (#1172) 
* updated

* udpated

* updated

* updated

* added env

* updated

* updated

* updated

* updated

* updated

* updated docker file

* updated docker compose

* fixed validation error

* reverted local changes to get ready for pr
 2025-02-20 00:43:46 +01:00