AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-06-22 04:22:07 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,412 Commits 496 Branches 17 Tags
Commit Graph

2697 Commits

Author SHA1 Message Date
Shaun Offenbacher
037f64916a
Playwright page error schema (#1172) 
* updated

* udpated

* updated

* updated

* added env

* updated

* updated

* updated

* updated

* updated

* updated docker file

* updated docker compose

* fixed validation error

* reverted local changes to get ready for pr
 2025-02-20 00:43:46 +01:00
Gergő Móricz
46b187bc64
feat(v1/map): stop mapping if timed out via AbortController (#1205) 2025-02-20 00:42:13 +01:00
Gergő Móricz
2200f084f3
SELFHOST FIXES (#1207) 
* fix(extract): construct OpenAI on demand

Fixes hard-crash if api key not specified in a self-hosting environment.

* fix(ci): try sleeping

* fix(ci): override host

* fix(ci): wait for server to start

* Support /extract and /crawl for self-hosted (FIR-1097) (#1137)

* Support /extract for self-hosted

This returns the job response from redis rather than supabase when db auth is disabled (self hosted mode)

* Use getJob for extract and use correct types

* fix(v1/crawl-status): only poll DB for total count if DB is enabled

* feat(snips): TEST_SUITE_SELF_HOSTED

* fix(ci/test-server-self-host): use pr trigger

* fix(scrapeURL): f-e mocking in selfhosted env

* fix(snips): do not try to eval json format on selfhost

* fix(scrapeURL): further f-e mocking

* fix(snips): don't timeout on hard fail polling

* fix(v1/extract-status): fix-up the db-agnostic impl

unfortunately had to separate the functions since the schema
was too divergent :(

* fix(snips): boost screenshot delay

* feat(ci): test with openai

* feat(ci): extract, search testing

* fix(ci): matrix

* fix(ci): bleh

* Update: fix default google search (#1174)

* fix log title

* search should always work

* asd

* fix ci

---------

Co-authored-by: Nick Roth <nlr06886@gmail.com>
Co-authored-by: William <sdustusun@gmail.com>
 2025-02-20 00:41:22 +01:00
Gergő Móricz
055f7d2da0
fix(concurrency-limit): move to renewing a lock on each job instead of estimating time to complete (#1197) 2025-02-19 20:13:22 +01:00
Nicolas
acf1e60608 Nick: llmstxt improvements 2025-02-19 16:09:46 -03:00
Nicolas
d4cf2269ed Update generate-llmstxt-service.ts 2025-02-19 15:50:59 -03:00
Nicolas
f5de803a9d Nick: fixes 2025-02-19 15:21:52 -03:00
Nicolas
a60f3ff645 Nick: fixes 2025-02-19 15:01:47 -03:00
Eric Ciarla
d984b50400
Add llmstxt generator endpoint (#1201) 
* Nick:

* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"

This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.

* Update deep-research-service.ts

* Nick:

* init

* part 2

* Update generate-llmstxt-service.ts

* Fix queue

* Update queue-worker.ts

* Almost there

* Final touches

* Update requests.http

* final touches

* Update requests.http

* Improve logging

* Change endpoint to /llmstxt

* Update queue-worker.ts

* Update generate-llmstxt-service.ts

* Nick: cache

* Update index.ts

* Update firecrawl.py

* Update package.json

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-19 14:42:33 -03:00
Gergő Móricz
e373fab5c1 fix(sdk/js): don't require CrawlScrapeOptions.formats 2025-02-19 17:58:19 +01:00
Gergő Móricz
8e1e5986f7 feat(sdk/js): bump 2025-02-19 17:37:36 +01:00
Gergő Móricz
4fd26a3baa feat(js-sdk): support scrapeOptions in ExtractParams 2025-02-19 17:08:43 +01:00
Nicolas
5c47e97db2
(feat/deep-research) Alpha implementation of deep research (#1202) 
* Nick:

* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"

This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.

* Update deep-research-service.ts

* Nick:

* Nick:

* Nick:

* Nick:

* Nick:

* Nick:

* Update deep-research-service.ts

* Nick:

* Update deep-research-service.ts

* Apply suggestions from code review

---------

Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-19 12:44:21 -03:00
Gergő Móricz
fc64f436ed
fix(v1/types): fix extract -> json rename, ROUND II (FIR-1072) (#1199) 
* Revert "Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)""

This reverts commit e28a44463ae49ffc195507204492cc7c15c438c4.

* fix(v1/types): fix bad transform

* feat(v1): proxy option / stealthProxy flag (FIR-1050) (#1196)

* feat(v1): proxy option / stealthProxy flag

* feat(js-sdk): add proxy option

* fix

* fix extract tests
 2025-02-19 12:07:55 -03:00
Gergő Móricz
42050d3d6e fix 2025-02-18 18:06:39 +01:00
Gergő Móricz
b136e42b53
feat(v1): proxy option / stealthProxy flag (FIR-1050) (#1196) 
* feat(v1): proxy option / stealthProxy flag

* feat(js-sdk): add proxy option
 2025-02-18 18:03:10 +01:00
Nicolas
e28a44463a Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)" 
This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.
 2025-02-18 11:31:23 -03:00
Gergő Móricz
586a10f40d
fix(v1/types): fix extract -> json rename (FIR-1072) (#1195) 
* fix(v1/types): fix extract -> json rename

* fix(types/v1): bad transform
 2025-02-18 10:32:19 -03:00
Tetsuro Yokoyama
5ac6eb7440
Update self-hosted Kubernetes deployments examples for compatibility and consistency (#1177) 
* fix: Quote variables in `docker-entrypoint.sh`

- This commit adds double quotes around variables in the docker-entrypoint.sh script to prevent word splitting and globbing issues, ensuring the script behaves correctly in all cases.

* fix: Ensure worker/api deployment starts with `OPENAI_API_KEY`

* fix: Add missing `FLY_PROCESS_GROUP` env var to deployments

* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`

* fix: Update Docker build options for Apple Silicon compatibility

* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`
 2025-02-18 13:33:20 +01:00
Gergő Móricz
aacbea1d9e fix(tests/snips/map): remove flaky useless test 2025-02-18 13:20:11 +01:00
Gergő Móricz
58564bceb8 Revert "temp: debugging" 
wrong branch
This reverts commit 97e6c28cf96fe837d3934b126e3b5e86fa10df2b.
 2025-02-18 11:28:35 +01:00
Gergő Móricz
97e6c28cf9 temp: debugging 2025-02-18 11:28:12 +01:00
Gergő Móricz
5c4021e8cf fix(scrapeURL/fire-engine): screenshot broken hotfix 2025-02-17 18:59:16 +01:00
Nicolas
b0534d0767 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-02-17 12:42:59 -03:00
Nicolas
03da3c36a6 Update auth.ts 2025-02-17 12:42:57 -03:00
Gergő Móricz
445b906af1
fix(scrapeURL/fire-engine): perform format screenshot after specified actions (#1192) 2025-02-17 12:35:55 -03:00
Gergő Móricz
7ecbff3b20
fix(map): do not remove query parameters from results (FIR-1015) (#1191) 
* fix(map): do not remove query parameters from results

* feat(map): add tests for query parameter handling
 2025-02-17 12:34:58 -03:00
Gergő Móricz
1491b5b141
fix(scrapeURL/sb): enforce timeout (FIR-980) (#1183) 
* fix(scrapeURL/scrapingbee): enforce timeout

* fix(scrapeURL/sb): types

* fix the test

* fixup: remove nix files
 2025-02-16 15:55:03 +01:00
Gergő Móricz
fd8b38902a
fix(crawl-status): consider concurrency limited jobs as prioritized (#1184) 2025-02-16 15:52:17 +01:00
Nicolas
7ac2b99210 Update auth.ts 2025-02-14 11:12:02 -03:00
Nicolas
ce9d3d32f6 Nick: adjust for old compatibility s2 2025-02-14 11:09:09 -03:00
Gergő Móricz
a4cf8f439c Revert "(feat/extract) Multi-entity prompt improvements (#1181)" 
This reverts commit 584221a106216dfcc25de14c16720224cad237e1.
 2025-02-14 11:19:16 +01:00
Gergő Móricz
d9c99e5892
fix(v1/batch/scrape): use scrape rate limit (#1182) 2025-02-13 17:32:58 -03:00
Gergő Móricz
73e7884df4
fix(queue-worker/crawl): only report successful page count in num_docs (#1179) 2025-02-13 13:14:24 -03:00
Nicolas
584221a106
(feat/extract) Multi-entity prompt improvements (#1181) 
* Update build-prompts.ts

* Update build-prompts.ts

* Update build-prompts.ts

* Update build-prompts.ts
 2025-02-13 13:11:35 -03:00
Nicolas
73ecaf74c2
feat(v1/extract) Show sources out of __experimental (#1180) 
* Nick:

* Nick:

* Nick:
 2025-02-13 12:41:45 -03:00
Gergő Móricz
1d9a0b9653
feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens (#1178) 
* feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens

* feat(v1/checkCredits): better instructions for upgrade
 2025-02-13 12:14:13 -03:00
Carter LaSalle
78094e2fe4
fix(html-transformer): Update free_string function parameter type (#1163) 
Correct the parameter type to use libc::c_char for better type consistency
 2025-02-11 15:11:29 +01:00
Nicolas
6b313ad0b0
Update types.ts 2025-02-10 17:42:12 -03:00
Gergő Móricz
bc787de62a fix(blocklist): behavioural fixes 2025-02-10 19:10:20 +01:00
Gergő Móricz
290dd033d3
Update apps/api/src/controllers/v1/types.ts 2025-02-10 09:31:23 +01:00
Ademílson F. Tonato
843cec971d
[FIR-796] feat(api/types): Add action and wait time validation for scrape requests 
- Implement max actions limit (15 actions)
- Add total wait time calculation for actions
- Increase max waitFor time to 60 seconds
- Refactor scrape options schema to include validation
 2025-02-07 21:16:38 +00:00
Rafael Miller
ac5c88bffb
added scrapeOptions to extract (#1133) 2025-02-07 13:38:08 -03:00
Gergő Móricz
892f3a41f3
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) (#1138) 
* fix(scrape): allow getting valid JSON via rawHtml

* fix(scrape/test):
 2025-02-06 18:35:28 +01:00
Nicolas
401c18761a Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-02-06 12:31:31 -03:00
Nicolas
394a31064d Update types.ts 2025-02-06 12:31:19 -03:00
Ademílson Tonato
996352c24c
Merge pull request #1117 from mendableai/feat/blocked-domains 
feat(blocklist): Improve URL blocking with tldts parsing
 2025-02-06 15:23:58 +00:00
Nicolas
fe276060cf Update pyproject.toml 2025-02-06 12:19:32 -03:00
Gergő Móricz
844ba39f74 fix(sdks): MIT license 2025-02-06 16:07:50 +01:00
Gergő Móricz
34fe360dc1
fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827) (#1134) 
* fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends

* fix(crawl-redis/tests): adjust tests
 2025-02-05 15:44:56 +01:00