Gergő Móricz
b136e42b53
feat(v1): proxy option / stealthProxy flag (FIR-1050) ( #1196 )
...
* feat(v1): proxy option / stealthProxy flag
* feat(js-sdk): add proxy option
2025-02-18 18:03:10 +01:00
Nicolas
e28a44463a
Revert "fix(v1/types): fix extract -> json rename (FIR-1072) ( #1195 )"
...
This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.
2025-02-18 11:31:23 -03:00
Gergő Móricz
586a10f40d
fix(v1/types): fix extract -> json rename (FIR-1072) ( #1195 )
...
* fix(v1/types): fix extract -> json rename
* fix(types/v1): bad transform
2025-02-18 10:32:19 -03:00
Tetsuro Yokoyama
5ac6eb7440
Update self-hosted Kubernetes deployments examples for compatibility and consistency ( #1177 )
...
* fix: Quote variables in `docker-entrypoint.sh`
- This commit adds double quotes around variables in the docker-entrypoint.sh script to prevent word splitting and globbing issues, ensuring the script behaves correctly in all cases.
* fix: Ensure worker/api deployment starts with `OPENAI_API_KEY`
* fix: Add missing `FLY_PROCESS_GROUP` env var to deployments
* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`
* fix: Update Docker build options for Apple Silicon compatibility
* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`
2025-02-18 13:33:20 +01:00
Gergő Móricz
aacbea1d9e
fix(tests/snips/map): remove flaky useless test
2025-02-18 13:20:11 +01:00
Gergő Móricz
58564bceb8
Revert "temp: debugging"
...
wrong branch
This reverts commit 97e6c28cf96fe837d3934b126e3b5e86fa10df2b.
2025-02-18 11:28:35 +01:00
Gergő Móricz
97e6c28cf9
temp: debugging
2025-02-18 11:28:12 +01:00
Gergő Móricz
5c4021e8cf
fix(scrapeURL/fire-engine): screenshot broken hotfix
2025-02-17 18:59:16 +01:00
Nicolas
b0534d0767
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-02-17 12:42:59 -03:00
Nicolas
03da3c36a6
Update auth.ts
2025-02-17 12:42:57 -03:00
Gergő Móricz
445b906af1
fix(scrapeURL/fire-engine): perform format screenshot after specified actions ( #1192 )
2025-02-17 12:35:55 -03:00
Gergő Móricz
7ecbff3b20
fix(map): do not remove query parameters from results (FIR-1015) ( #1191 )
...
* fix(map): do not remove query parameters from results
* feat(map): add tests for query parameter handling
2025-02-17 12:34:58 -03:00
Eric Ciarla
5cb2918368
update name of example
2025-02-16 18:46:10 -05:00
Gergő Móricz
1491b5b141
fix(scrapeURL/sb): enforce timeout (FIR-980) ( #1183 )
...
* fix(scrapeURL/scrapingbee): enforce timeout
* fix(scrapeURL/sb): types
* fix the test
* fixup: remove nix files
2025-02-16 15:55:03 +01:00
Gergő Móricz
fd8b38902a
fix(crawl-status): consider concurrency limited jobs as prioritized ( #1184 )
2025-02-16 15:52:17 +01:00
Nicolas
7ac2b99210
Update auth.ts
2025-02-14 11:12:02 -03:00
Nicolas
ce9d3d32f6
Nick: adjust for old compatibility s2
2025-02-14 11:09:09 -03:00
Gergő Móricz
a4cf8f439c
Revert "(feat/extract) Multi-entity prompt improvements ( #1181 )"
...
This reverts commit 584221a106216dfcc25de14c16720224cad237e1.
2025-02-14 11:19:16 +01:00
Gergő Móricz
d9c99e5892
fix(v1/batch/scrape): use scrape rate limit ( #1182 )
2025-02-13 17:32:58 -03:00
dolonfly
254580a284
fix: relative url 2 full url use error base url ( #584 )
...
according to ： https://developer.mozilla.org/en-US/docs/Web/API/URL/URL
the second parameter: A string representing the base URL to use in cases where url is a relative reference
so need use the pageUrl , not url origin
eg:
when pageUrl = http://example.com/a/b/c/d.html and tag a href is '../e.html'
then
correct: new URL('../e.html','http://example.com/a/b/c/d.html ')
incorrect: new URL('../e.html','http://example.com ')
2025-02-13 20:38:47 +01:00
Gergő Móricz
73e7884df4
fix(queue-worker/crawl): only report successful page count in num_docs ( #1179 )
2025-02-13 13:14:24 -03:00
Nicolas
584221a106
(feat/extract) Multi-entity prompt improvements ( #1181 )
...
* Update build-prompts.ts
* Update build-prompts.ts
* Update build-prompts.ts
* Update build-prompts.ts
2025-02-13 13:11:35 -03:00
Nicolas
73ecaf74c2
feat(v1/extract) Show sources out of __experimental ( #1180 )
...
* Nick:
* Nick:
* Nick:
2025-02-13 12:41:45 -03:00
Gergő Móricz
1d9a0b9653
feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens ( #1178 )
...
* feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens
* feat(v1/checkCredits): better instructions for upgrade
2025-02-13 12:14:13 -03:00
Eric Ciarla
582bbf8da4
Merge pull request #1175 from mayooear/feat/gemini-sub-links
...
fix: update gemini library. extract pdf links from scraped content
2025-02-12 12:00:00 -05:00
mayo
32f9897670
fix: update gemini library. extract pdf links from markdowncontent
2025-02-12 16:43:59 +00:00
Eric Ciarla
a1b7d6e696
Merge pull request #1173 from mayooear/feat/gemini-sub-links
2025-02-11 22:11:49 -05:00
mayo
cb3bc5e445
Add detection of PDF/image sub-links and extract text via Gemini
2025-02-12 02:52:20 +00:00
Eric Ciarla
c67b052f4a
Update gemini-2.0-crawler.py
2025-02-11 13:10:48 -05:00
Carter LaSalle
78094e2fe4
fix(html-transformer): Update free_string function parameter type ( #1163 )
...
Correct the parameter type to use libc::c_char for better type consistency
2025-02-11 15:11:29 +01:00
Luke Martin
52c0f78d4c
chore: add Serper and Search API envs to docker-compose ( #1147 )
2025-02-11 11:11:16 -03:00
Eric Ciarla
ef9aad8b8a
Merge pull request #1161 from aparupganguly/feature/gemini-2.0-crawler
...
Implemented Gemini 2.0 crawler
2025-02-11 08:13:06 -05:00
Nicolas
6b313ad0b0
Update types.ts
2025-02-10 17:42:12 -03:00
Gergő Móricz
bc787de62a
fix(blocklist): behavioural fixes
2025-02-10 19:10:20 +01:00
Ademílson Tonato
3608256569
Merge pull request #1146 from mendableai/feat/wait-validation
...
[FIR-796] feat(api/types): Add action and wait time validation for scrape requests
2025-02-10 14:45:47 +00:00
Gergő Móricz
290dd033d3
Update apps/api/src/controllers/v1/types.ts
2025-02-10 09:31:23 +01:00
Ademílson F. Tonato
843cec971d
[FIR-796] feat(api/types): Add action and wait time validation for scrape requests
...
- Implement max actions limit (15 actions)
- Add total wait time calculation for actions
- Increase max waitFor time to 60 seconds
- Refactor scrape options schema to include validation
2025-02-07 21:16:38 +00:00
Aparup Ganguly
f33fa5bfc6
Initial commit
2025-02-08 01:24:21 +05:30
Eric Ciarla
2b7b7400f6
Merge pull request #1144 from aparupganguly/feature/o3-mini-job-resource
...
Feature/o3 mini job resource extractor
2025-02-07 11:39:29 -05:00
Rafael Miller
ac5c88bffb
added scrapeOptions to extract ( #1133 )
2025-02-07 13:38:08 -03:00
Eric Ciarla
42f4f7ef09
Merge pull request #1139 from aparupganguly/feature/o3-mini-review-summarizer
...
Implemented prodcut reviews summarizer using o3 mini
2025-02-07 11:37:57 -05:00
Aparup Ganguly
46f05a750d
Job Resources Analyzer
2025-02-07 20:32:52 +05:30
Gergő Móricz
892f3a41f3
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) ( #1138 )
...
* fix(scrape): allow getting valid JSON via rawHtml
* fix(scrape/test):
2025-02-06 18:35:28 +01:00
Nicolas
401c18761a
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-02-06 12:31:31 -03:00
Nicolas
394a31064d
Update types.ts
2025-02-06 12:31:19 -03:00
Ademílson Tonato
996352c24c
Merge pull request #1117 from mendableai/feat/blocked-domains
...
feat(blocklist): Improve URL blocking with tldts parsing
2025-02-06 15:23:58 +00:00
Nicolas
fe276060cf
Update pyproject.toml
2025-02-06 12:19:32 -03:00
Gergő Móricz
844ba39f74
fix(sdks): MIT license
2025-02-06 16:07:50 +01:00
Aparup Ganguly
310ef66506
Implemented prodcut reviews summarizer using o3 mini
2025-02-06 20:13:35 +05:30
Gergő Móricz
34fe360dc1
fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827) ( #1134 )
...
* fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends
* fix(crawl-redis/tests): adjust tests
2025-02-05 15:44:56 +01:00
