AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-06-21 13:31:45 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,475 Commits 496 Branches 17 Tags
Commit Graph

3475 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Gergő Móricz
10d9b65f96 fix(self-host): update docs and dockerignore 2025-02-20 07:57:57 +01:00
niazarak
e4504b3236
Use correct list typing for py 3.8 support (#931) 
list[str] is not supported in Python 3.8
 2025-02-20 07:39:46 +01:00
Gergő Móricz
55d047b6b3
feat(scrapeURL): handle PDFs behind anti-bot (#1198) 2025-02-20 04:11:30 +01:00
Gergő Móricz
bec52bef6c
feat(self-host/docs): playwright-service-ts is now default 2025-02-20 03:37:42 +01:00
Gergő Móricz
7f5ff80203
feat(self-host): use playwright-service-ts by default 2025-02-20 03:36:51 +01:00
Gergő Móricz
c39cc27866
feat(ci/self-host): add playwright microservice tests (#1210) 
* feat(ci/self-host): add playwright microservice tests

* fix ci

* fix ci 2

* fix ci 3

* fix(playwright-service): get raw JSON if response is JSON
 2025-02-20 02:06:13 +01:00
Shaun Offenbacher
037f64916a
Playwright page error schema (#1172) 
* updated

* udpated

* updated

* updated

* added env

* updated

* updated

* updated

* updated

* updated

* updated docker file

* updated docker compose

* fixed validation error

* reverted local changes to get ready for pr
 2025-02-20 00:43:46 +01:00
Gergő Móricz
46b187bc64
feat(v1/map): stop mapping if timed out via AbortController (#1205) 2025-02-20 00:42:13 +01:00
Gergő Móricz
2200f084f3
SELFHOST FIXES (#1207) 
* fix(extract): construct OpenAI on demand

Fixes hard-crash if api key not specified in a self-hosting environment.

* fix(ci): try sleeping

* fix(ci): override host

* fix(ci): wait for server to start

* Support /extract and /crawl for self-hosted (FIR-1097) (#1137)

* Support /extract for self-hosted

This returns the job response from redis rather than supabase when db auth is disabled (self hosted mode)

* Use getJob for extract and use correct types

* fix(v1/crawl-status): only poll DB for total count if DB is enabled

* feat(snips): TEST_SUITE_SELF_HOSTED

* fix(ci/test-server-self-host): use pr trigger

* fix(scrapeURL): f-e mocking in selfhosted env

* fix(snips): do not try to eval json format on selfhost

* fix(scrapeURL): further f-e mocking

* fix(snips): don't timeout on hard fail polling

* fix(v1/extract-status): fix-up the db-agnostic impl

unfortunately had to separate the functions since the schema
was too divergent :(

* fix(snips): boost screenshot delay

* feat(ci): test with openai

* feat(ci): extract, search testing

* fix(ci): matrix

* fix(ci): bleh

* Update: fix default google search (#1174)

* fix log title

* search should always work

* asd

* fix ci

---------

Co-authored-by: Nick Roth <nlr06886@gmail.com>
Co-authored-by: William <sdustusun@gmail.com>
 2025-02-20 00:41:22 +01:00
Gergő Móricz
c1ca64fc07 fix(ci): retain fail status 2025-02-19 22:31:16 +01:00
Gergő Móricz
f4f75fe184 fix(ci): path to lock 2025-02-19 22:15:41 +01:00
Gergő Móricz
e9cb8ac956 feat(ci): caching improvements 2025-02-19 22:11:32 +01:00
Gergő Móricz
0eff99007a feat(ci): upload logs 2025-02-19 22:02:32 +01:00
Gergő Móricz
9976560567 feat(ci): self-hosted server test suite 2025-02-19 21:55:48 +01:00
Aparup Ganguly
8caeab2691 minor changes 2025-02-20 02:24:49 +05:30
Aparup Ganguly
21b22d9f5c Output re-structured 2025-02-20 02:21:43 +05:30
Gergő Móricz
dab0016669 feat(ci): publish all JS SDK packages 2025-02-19 21:46:02 +01:00
Aparup Ganguly
3a4ef05a70 Output imporvements 2025-02-20 01:30:59 +05:30
Gergő Móricz
1a9f6b985a feat(github/ci): improvements 2025-02-19 20:51:38 +01:00
Gergő Móricz
055f7d2da0
fix(concurrency-limit): move to renewing a lock on each job instead of estimating time to complete (#1197) 2025-02-19 20:13:22 +01:00
Nicolas
acf1e60608 Nick: llmstxt improvements 2025-02-19 16:09:46 -03:00
Nicolas
d4cf2269ed Update generate-llmstxt-service.ts 2025-02-19 15:50:59 -03:00
Aparup Ganguly
5e0036f8b4 Implemented gemini 2.0 web extractor 2025-02-20 00:05:12 +05:30
Nicolas
f5de803a9d Nick: fixes 2025-02-19 15:21:52 -03:00
Nicolas
a60f3ff645 Nick: fixes 2025-02-19 15:01:47 -03:00
Eric Ciarla
d984b50400
Add llmstxt generator endpoint (#1201) 
* Nick:

* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"

This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.

* Update deep-research-service.ts

* Nick:

* init

* part 2

* Update generate-llmstxt-service.ts

* Fix queue

* Update queue-worker.ts

* Almost there

* Final touches

* Update requests.http

* final touches

* Update requests.http

* Improve logging

* Change endpoint to /llmstxt

* Update queue-worker.ts

* Update generate-llmstxt-service.ts

* Nick: cache

* Update index.ts

* Update firecrawl.py

* Update package.json

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-19 14:42:33 -03:00
Gergő Móricz
e373fab5c1 fix(sdk/js): don't require CrawlScrapeOptions.formats 2025-02-19 17:58:19 +01:00
Gergő Móricz
8e1e5986f7 feat(sdk/js): bump 2025-02-19 17:37:36 +01:00
Gergő Móricz
4fd26a3baa feat(js-sdk): support scrapeOptions in ExtractParams 2025-02-19 17:08:43 +01:00
Nicolas
5c47e97db2
(feat/deep-research) Alpha implementation of deep research (#1202) 
* Nick:

* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"

This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.

* Update deep-research-service.ts

* Nick:

* Nick:

* Nick:

* Nick:

* Nick:

* Nick:

* Update deep-research-service.ts

* Nick:

* Update deep-research-service.ts

* Apply suggestions from code review

---------

Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-19 12:44:21 -03:00
Gergő Móricz
fc64f436ed
fix(v1/types): fix extract -> json rename, ROUND II (FIR-1072) (#1199) 
* Revert "Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)""

This reverts commit e28a44463ae49ffc195507204492cc7c15c438c4.

* fix(v1/types): fix bad transform

* feat(v1): proxy option / stealthProxy flag (FIR-1050) (#1196)

* feat(v1): proxy option / stealthProxy flag

* feat(js-sdk): add proxy option

* fix

* fix extract tests
 2025-02-19 12:07:55 -03:00
Gergő Móricz
42050d3d6e fix 2025-02-18 18:06:39 +01:00
Gergő Móricz
b136e42b53
feat(v1): proxy option / stealthProxy flag (FIR-1050) (#1196) 
* feat(v1): proxy option / stealthProxy flag

* feat(js-sdk): add proxy option
 2025-02-18 18:03:10 +01:00
Nicolas
e28a44463a Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)" 
This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.
 2025-02-18 11:31:23 -03:00
Gergő Móricz
586a10f40d
fix(v1/types): fix extract -> json rename (FIR-1072) (#1195) 
* fix(v1/types): fix extract -> json rename

* fix(types/v1): bad transform
 2025-02-18 10:32:19 -03:00
Tetsuro Yokoyama
5ac6eb7440
Update self-hosted Kubernetes deployments examples for compatibility and consistency (#1177) 
* fix: Quote variables in `docker-entrypoint.sh`

- This commit adds double quotes around variables in the docker-entrypoint.sh script to prevent word splitting and globbing issues, ensuring the script behaves correctly in all cases.

* fix: Ensure worker/api deployment starts with `OPENAI_API_KEY`

* fix: Add missing `FLY_PROCESS_GROUP` env var to deployments

* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`

* fix: Update Docker build options for Apple Silicon compatibility

* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`
 2025-02-18 13:33:20 +01:00
Gergő Móricz
aacbea1d9e fix(tests/snips/map): remove flaky useless test 2025-02-18 13:20:11 +01:00
Gergő Móricz
58564bceb8 Revert "temp: debugging" 
wrong branch
This reverts commit 97e6c28cf96fe837d3934b126e3b5e86fa10df2b.
 2025-02-18 11:28:35 +01:00
Gergő Móricz
97e6c28cf9 temp: debugging 2025-02-18 11:28:12 +01:00
Gergő Móricz
5c4021e8cf fix(scrapeURL/fire-engine): screenshot broken hotfix 2025-02-17 18:59:16 +01:00
Nicolas
b0534d0767 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-02-17 12:42:59 -03:00
Nicolas
03da3c36a6 Update auth.ts 2025-02-17 12:42:57 -03:00
Gergő Móricz
445b906af1
fix(scrapeURL/fire-engine): perform format screenshot after specified actions (#1192) 2025-02-17 12:35:55 -03:00
Gergő Móricz
7ecbff3b20
fix(map): do not remove query parameters from results (FIR-1015) (#1191) 
* fix(map): do not remove query parameters from results

* feat(map): add tests for query parameter handling
 2025-02-17 12:34:58 -03:00
Eric Ciarla
5cb2918368 update name of example 2025-02-16 18:46:10 -05:00
Gergő Móricz
1491b5b141
fix(scrapeURL/sb): enforce timeout (FIR-980) (#1183) 
* fix(scrapeURL/scrapingbee): enforce timeout

* fix(scrapeURL/sb): types

* fix the test

* fixup: remove nix files
 2025-02-16 15:55:03 +01:00
Gergő Móricz
fd8b38902a
fix(crawl-status): consider concurrency limited jobs as prioritized (#1184) 2025-02-16 15:52:17 +01:00
Nicolas
7ac2b99210 Update auth.ts v1.4.4 2025-02-14 11:12:02 -03:00
Nicolas
ce9d3d32f6 Nick: adjust for old compatibility s2 2025-02-14 11:09:09 -03:00
Gergő Móricz
a4cf8f439c Revert "(feat/extract) Multi-entity prompt improvements (#1181)" 
This reverts commit 584221a106216dfcc25de14c16720224cad237e1.
 2025-02-14 11:19:16 +01:00