AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-06-21 08:47:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,475 Commits 496 Branches 17 Tags
Commit Graph

3475 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nicolas
31df234127 Update log_job.ts 2025-02-25 21:01:05 -03:00
Nicolas
ec90aaffd6 Update log_job.ts 2025-02-25 21:01:00 -03:00
Nicolas
0f0520391a
Update log_job.ts (#1263) 2025-02-25 19:35:57 -03:00
Nicolas
59d09f5c45 Update log_job.ts 2025-02-25 19:32:16 -03:00
Gergő Móricz
115b6b61c4 add initial codeowners 2025-02-25 14:28:09 +01:00
Gergő Móricz
51bc7757ef
fix(self-host/compose): pass SearXNG params 2025-02-25 11:19:27 +01:00
Eric Ciarla
2da6d7bdb4
Merge pull request #1257 from aparupganguly/feature/claude3.7-crawler 
Implemented claude 3.7 web crawler
 2025-02-24 15:46:52 -05:00
Gergő Móricz
8c42b08b7e
feat(v1/crawl-status-ws): update behavior to ignore errors like regular crawl-status (#1234) 2025-02-24 21:44:29 +01:00
Aparup Ganguly
6508afc624 Implemented claude 3.7 2025-02-25 01:54:25 +05:30
Gergő Móricz
15489be542
feat(self-host/ai): use any OpenAI-compatible API (#1245) 2025-02-23 09:07:32 +01:00
Gergő Móricz
b88b573909 docker: force host to 0.0.0.0 to fix env precedence issues 
Fixes #1225
 2025-02-23 08:49:31 +01:00
Nicolas
b24ac0f6b5
Nick: done (#1237) 2025-02-22 20:18:46 -03:00
Nicolas
5ab86b8b43
(fix/token-slicer) Fixes extract token limit issues (#1236) 
* Nick: fixes extract token limit errors

* Update llmExtract.ts

* Update llmExtract.ts
 2025-02-21 20:44:42 +01:00
Gergő Móricz
76e1f29ae8
Update Dockerfile (#1231) (#1232) 
* Update Dockerfile (#1231)

* Dockerfile: re-add prod-deps stage and fix copies

---------

Co-authored-by: Loris <loris.rion@gmail.com>
 2025-02-21 17:44:01 +01:00
Gergő Móricz
bfe6a0ab32 feat(self-host/docker-compose): add option to use Valkey 
Fixes #1228
 2025-02-21 16:19:37 +01:00
Gergő Móricz
82adf81b15 feat(self-host/docker-compose): allow configuring the internal port of the api 2025-02-21 16:19:37 +01:00
Eric Ciarla
9671e68512
Merge pull request #1229 from aparupganguly/feature/gemini-github-analyzer 
Implemented github analyzer
 2025-02-21 09:52:42 -05:00
Aparup Ganguly
448b44cdd9 Implemented github analyzer 2025-02-21 20:08:37 +05:30
Nicolas
6c51ef401e Update rate-limiter.ts 2025-02-20 22:31:43 -03:00
Nicolas
25d9bdb1f6
(feat/ai-sdk) Migrate to AI-SDK (#1220) 
* Nick: init

* Update llmExtract.ts

* Update llmExtract.ts

* Nick rename

* fix(v1/types): extract json schema validation

* Update url-processor.ts

* feat(ai-sdk): ollama support

* feat(ai-sdk): further ollama support

* Nick: it is broken btw

* feat(ai-sdk): abstract model adapter

* Update pnpm-lock.yaml

* Update analyzeSchemaAndPrompt.ts

* Nick:

* feat(ai-sdk): ollama support

* doc(SELF_HOST): update with embedding param

* Nick:

* Update ranker.ts

* Nick:

* feat(ai-sdk): fixes

* Update llmExtract.ts

* feat: remove zod-to-json-schema

* fix

* Update llmExtract.ts

* use openai

* fixes

---------

Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-20 22:48:58 +01:00
rafaelmmiller
943eb775c6 Update eval_run.py 2025-02-20 17:11:42 -03:00
rafaelmmiller
570dc28da2 bugfix eval run 2025-02-20 16:10:36 -03:00
rafaelmmiller
d211240f58 Update eval-prod.yml 2025-02-20 16:02:45 -03:00
rafaelmmiller
04127cca64 Update eval-prod.yml 2025-02-20 15:59:30 -03:00
Rafael Miller
a13863638a
Merge pull request #1224 from mendableai/feat/eval-run-workflow 
Feat/added eval run after deploy workflow
 2025-02-20 15:23:57 -03:00
Gergő Móricz
16c305775e
fix(crawl-redis): ignore empty includes/excludes (#1223) 
* fix(crawl-redis): ignore empty includes/excludes

* fix(snips/scrape): bump timeouts
 2025-02-20 19:06:02 +01:00
rafaelmmiller
d7db58e477 Feat/added eval run after deploy workflow 2025-02-20 15:01:26 -03:00
Gergő Móricz
283a3bfef3
fix(scrapeURL/engines/fetch): discover charset and re-decode (#1221) 
* fix(scrapeURL/engines/fetch): discover charset and re-decode

* fix(snips/scrape): allow more time for stealth proxy
 2025-02-20 18:56:15 +01:00
Gergő Móricz
e417f83c28
feat(self-host): ollama support (#1219) v1.5.0 2025-02-20 16:59:19 +01:00
Eric Ciarla
192f0031cf
Merge pull request #1206 from aparupganguly/feature/gemini-extractor 
gemini extractor Implementation
 2025-02-20 09:48:10 -05:00
Gergő Móricz
e84c7325d9 chore: remove dead code 
Fixes #1149
Fixes #1150
 2025-02-20 15:44:34 +01:00
Gergő Móricz
62d11f21fc self-host: remove old playwright-service 2025-02-20 15:39:19 +01:00
Gergő Móricz
d533718312 feat(self-host): rework docker-compose.yaml 2025-02-20 15:37:19 +01:00
Gergő Móricz
305fbb37e7
temp (#1218) 2025-02-20 15:08:32 +01:00
Gergő Móricz
387cc60668 fix(ci/test-server): clean up old envs 2025-02-20 15:06:37 +01:00
Gergő Móricz
fca349f345
fix(ci/js-sdk): bad sed command 2025-02-20 15:02:56 +01:00
Nicolas
2151ca846c Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-02-20 10:50:32 -03:00
Nicolas
7db2d25efa Nick: 2025-02-20 10:50:22 -03:00
Gergő Móricz
c38dcd0432
feat(self-host): proxy support (FIR-1111) (#1212) 
* feat(self-host): proxy support

* fix(playwright-service-ts): return untreated text/plain
 2025-02-20 14:20:03 +01:00
Gergő Móricz
c75522f535
feat(sdk): enforce timeout on client-side if set (#1204) 2025-02-20 13:35:31 +01:00
Gergő Móricz
dc1501527a
doc(self-host): update with searxng details 2025-02-20 12:41:44 +01:00
Loris
100168ddf3
Add searxng for search endpoint (#1193) 
* add searxng.ts

* update to add searxng endpoint

* Apply suggestions from code review

* feat(ci/self-host): add tests with searxng

* feat(ci/self-host): bootstrap searxng for testing

* feat(ci): improvements in syntax

---------

Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-20 12:36:53 +01:00
Gergő Móricz
04218de2b0 Revert "feat(ci): use pull_request_target (+ manual approval)" 
This reverts commit 9142030881e0d153396279520e127b74af8417c9.
 2025-02-20 10:58:08 +01:00
Gergő Móricz
9142030881 feat(ci): use pull_request_target (+ manual approval) 2025-02-20 10:52:29 +01:00
Gergő Móricz
da1670b78c
feat(map): mock support (FIR-1109) (#1213) 
* feat(map,fetch): mock support

* feat(snips/map): mock out long-running test

* fix(snips/scrape): use more reliable site for adblock testing
 2025-02-20 10:41:43 +01:00
Gergő Móricz
bc5a16d048 feat(ci/test-server): build go markdown parser 2025-02-20 10:05:39 +01:00
Gergő Móricz
7770929dcd
fix(README): Adjust crawling example limit 
Fixes #871
 2025-02-20 09:51:38 +01:00
Gergő Móricz
11ed679274 feat(scrapeURL/pdf): support PDF prefetch when parsePDF is off 2025-02-20 09:28:13 +01:00
Gergő Móricz
5eb0235ccb feat(apps/api): remove Sentry builds 2025-02-20 08:19:06 +01:00
Gergő Móricz
da46736016 fix(self-host): further changes 2025-02-20 08:01:31 +01:00