AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-08-05 11:00:38 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,150 Commits 496 Branches 17 Tags
Commit Graph

2007 Commits

Author SHA1 Message Date
Nicolas
a60f3ff645 Nick: fixes 2025-02-19 15:01:47 -03:00
Eric Ciarla
d984b50400
Add llmstxt generator endpoint (#1201) 
* Nick:

* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"

This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.

* Update deep-research-service.ts

* Nick:

* init

* part 2

* Update generate-llmstxt-service.ts

* Fix queue

* Update queue-worker.ts

* Almost there

* Final touches

* Update requests.http

* final touches

* Update requests.http

* Improve logging

* Change endpoint to /llmstxt

* Update queue-worker.ts

* Update generate-llmstxt-service.ts

* Nick: cache

* Update index.ts

* Update firecrawl.py

* Update package.json

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-19 14:42:33 -03:00
Nicolas
5c47e97db2
(feat/deep-research) Alpha implementation of deep research (#1202) 
* Nick:

* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"

This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.

* Update deep-research-service.ts

* Nick:

* Nick:

* Nick:

* Nick:

* Nick:

* Nick:

* Update deep-research-service.ts

* Nick:

* Update deep-research-service.ts

* Apply suggestions from code review

---------

Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-19 12:44:21 -03:00
Gergő Móricz
fc64f436ed
fix(v1/types): fix extract -> json rename, ROUND II (FIR-1072) (#1199) 
* Revert "Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)""

This reverts commit e28a44463ae49ffc195507204492cc7c15c438c4.

* fix(v1/types): fix bad transform

* feat(v1): proxy option / stealthProxy flag (FIR-1050) (#1196)

* feat(v1): proxy option / stealthProxy flag

* feat(js-sdk): add proxy option

* fix

* fix extract tests
 2025-02-19 12:07:55 -03:00
Gergő Móricz
42050d3d6e fix 2025-02-18 18:06:39 +01:00
Gergő Móricz
b136e42b53
feat(v1): proxy option / stealthProxy flag (FIR-1050) (#1196) 
* feat(v1): proxy option / stealthProxy flag

* feat(js-sdk): add proxy option
 2025-02-18 18:03:10 +01:00
Nicolas
e28a44463a Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)" 
This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.
 2025-02-18 11:31:23 -03:00
Gergő Móricz
586a10f40d
fix(v1/types): fix extract -> json rename (FIR-1072) (#1195) 
* fix(v1/types): fix extract -> json rename

* fix(types/v1): bad transform
 2025-02-18 10:32:19 -03:00
Tetsuro Yokoyama
5ac6eb7440
Update self-hosted Kubernetes deployments examples for compatibility and consistency (#1177) 
* fix: Quote variables in `docker-entrypoint.sh`

- This commit adds double quotes around variables in the docker-entrypoint.sh script to prevent word splitting and globbing issues, ensuring the script behaves correctly in all cases.

* fix: Ensure worker/api deployment starts with `OPENAI_API_KEY`

* fix: Add missing `FLY_PROCESS_GROUP` env var to deployments

* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`

* fix: Update Docker build options for Apple Silicon compatibility

* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`
 2025-02-18 13:33:20 +01:00
Gergő Móricz
aacbea1d9e fix(tests/snips/map): remove flaky useless test 2025-02-18 13:20:11 +01:00
Gergő Móricz
58564bceb8 Revert "temp: debugging" 
wrong branch
This reverts commit 97e6c28cf96fe837d3934b126e3b5e86fa10df2b.
 2025-02-18 11:28:35 +01:00
Gergő Móricz
97e6c28cf9 temp: debugging 2025-02-18 11:28:12 +01:00
Gergő Móricz
5c4021e8cf fix(scrapeURL/fire-engine): screenshot broken hotfix 2025-02-17 18:59:16 +01:00
Nicolas
b0534d0767 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-02-17 12:42:59 -03:00
Nicolas
03da3c36a6 Update auth.ts 2025-02-17 12:42:57 -03:00
Gergő Móricz
445b906af1
fix(scrapeURL/fire-engine): perform format screenshot after specified actions (#1192) 2025-02-17 12:35:55 -03:00
Gergő Móricz
7ecbff3b20
fix(map): do not remove query parameters from results (FIR-1015) (#1191) 
* fix(map): do not remove query parameters from results

* feat(map): add tests for query parameter handling
 2025-02-17 12:34:58 -03:00
Gergő Móricz
1491b5b141
fix(scrapeURL/sb): enforce timeout (FIR-980) (#1183) 
* fix(scrapeURL/scrapingbee): enforce timeout

* fix(scrapeURL/sb): types

* fix the test

* fixup: remove nix files
 2025-02-16 15:55:03 +01:00
Gergő Móricz
fd8b38902a
fix(crawl-status): consider concurrency limited jobs as prioritized (#1184) 2025-02-16 15:52:17 +01:00
Nicolas
7ac2b99210 Update auth.ts 2025-02-14 11:12:02 -03:00
Nicolas
ce9d3d32f6 Nick: adjust for old compatibility s2 2025-02-14 11:09:09 -03:00
Gergő Móricz
a4cf8f439c Revert "(feat/extract) Multi-entity prompt improvements (#1181)" 
This reverts commit 584221a106216dfcc25de14c16720224cad237e1.
 2025-02-14 11:19:16 +01:00
Gergő Móricz
d9c99e5892
fix(v1/batch/scrape): use scrape rate limit (#1182) 2025-02-13 17:32:58 -03:00
Gergő Móricz
73e7884df4
fix(queue-worker/crawl): only report successful page count in num_docs (#1179) 2025-02-13 13:14:24 -03:00
Nicolas
584221a106
(feat/extract) Multi-entity prompt improvements (#1181) 
* Update build-prompts.ts

* Update build-prompts.ts

* Update build-prompts.ts

* Update build-prompts.ts
 2025-02-13 13:11:35 -03:00
Nicolas
73ecaf74c2
feat(v1/extract) Show sources out of __experimental (#1180) 
* Nick:

* Nick:

* Nick:
 2025-02-13 12:41:45 -03:00
Gergő Móricz
1d9a0b9653
feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens (#1178) 
* feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens

* feat(v1/checkCredits): better instructions for upgrade
 2025-02-13 12:14:13 -03:00
Carter LaSalle
78094e2fe4
fix(html-transformer): Update free_string function parameter type (#1163) 
Correct the parameter type to use libc::c_char for better type consistency
 2025-02-11 15:11:29 +01:00
Nicolas
6b313ad0b0
Update types.ts 2025-02-10 17:42:12 -03:00
Gergő Móricz
bc787de62a fix(blocklist): behavioural fixes 2025-02-10 19:10:20 +01:00
Gergő Móricz
290dd033d3
Update apps/api/src/controllers/v1/types.ts 2025-02-10 09:31:23 +01:00
Ademílson F. Tonato
843cec971d
[FIR-796] feat(api/types): Add action and wait time validation for scrape requests 
- Implement max actions limit (15 actions)
- Add total wait time calculation for actions
- Increase max waitFor time to 60 seconds
- Refactor scrape options schema to include validation
 2025-02-07 21:16:38 +00:00
Rafael Miller
ac5c88bffb
added scrapeOptions to extract (#1133) 2025-02-07 13:38:08 -03:00
Gergő Móricz
892f3a41f3
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) (#1138) 
* fix(scrape): allow getting valid JSON via rawHtml

* fix(scrape/test):
 2025-02-06 18:35:28 +01:00
Nicolas
401c18761a Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-02-06 12:31:31 -03:00
Nicolas
394a31064d Update types.ts 2025-02-06 12:31:19 -03:00
Ademílson Tonato
996352c24c
Merge pull request #1117 from mendableai/feat/blocked-domains 
feat(blocklist): Improve URL blocking with tldts parsing
 2025-02-06 15:23:58 +00:00
Gergő Móricz
34fe360dc1
fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827) (#1134) 
* fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends

* fix(crawl-redis/tests): adjust tests
 2025-02-05 15:44:56 +01:00
Nick Roth
aa1e820d52
Fix corepack and self hosting setup (#1131) 
* Update Dockerfile

Config corepack and pnpm

* Update Dockerfile

Try using this approach: https://github.com/pnpm/pnpm/issues/9029#issuecomment-2631400936

* Add ulimits command to docker compose

* Remove ulimit from entrypoint

* Add ulimit back to docker-entrypoint.sh

This implements the proposed approach to not fail when the ulimit command can't run due to the environment

* Resolve signature mismatch

Resolves issue encountered when using docker compose up on a clean environment on an intel mac. This just adds the clean command to avoid leveraging any cached data before updating apt-get and installing ca-certificates

---------

Co-authored-by: Nick Roth <nroth@dealnews.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-05 10:19:11 +01:00
Thomas Kosmas
73d19aa01b fix(dockerfile): Temporarily resolve corepack issue 2025-02-05 02:42:16 +02:00
Gergő Móricz
1e7d42e8af fix(concurrency-limit): allow for big crawls 2025-02-04 19:26:30 +01:00
Ademílson F. Tonato
bcd74498e3
test(blocklist): Enhance URL blocking test coverage with decrypted domain validation 2025-01-31 18:23:38 +00:00
Rafael Miller
8d7e8c4f50
added cached scrapes to extract (#1107) 
* added cached scrapes to extract

* dont save if exists

* no duplicates

* experimental tag

* Update requests.http

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
 2025-01-31 13:58:52 -03:00
Ademílson F. Tonato
24e8aaf6b5
feat(blocklist): Improve URL blocking with tldts parsing 2025-01-31 13:14:48 +00:00
Móricz Gergő
8a8d7d645f fix(concurrency): proper job timeouting 2025-01-31 11:22:10 +01:00
Nicolas
e026917698 Nick: disable auto rech for extract requests 2025-01-31 01:10:59 -03:00
Nicolas
6bfd24d903 Nick: waitFor fixes 2025-01-30 23:23:03 -03:00
Nicolas
5819f60688 Update types.ts 2025-01-30 23:21:47 -03:00
Nicolas
26c61b96f6 Update queue-jobs.ts 2025-01-30 22:58:08 -03:00
Rafael Miller
948f7866df
Added instructions for empty string to extract prompts (#1114) 
* added instructions to extract prompts

* getting back the null for not strings
 2025-01-30 16:11:55 -03:00