AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-06-04 11:24:40 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,460 Commits 496 Branches 17 Tags
Commit Graph

3460 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Aparup Ganguly
21b22d9f5c Output re-structured 2025-02-20 02:21:43 +05:30
Gergő Móricz
dab0016669 feat(ci): publish all JS SDK packages 2025-02-19 21:46:02 +01:00
Aparup Ganguly
3a4ef05a70 Output imporvements 2025-02-20 01:30:59 +05:30
Gergő Móricz
1a9f6b985a feat(github/ci): improvements 2025-02-19 20:51:38 +01:00
Gergő Móricz
055f7d2da0
fix(concurrency-limit): move to renewing a lock on each job instead of estimating time to complete (#1197) 2025-02-19 20:13:22 +01:00
Nicolas
acf1e60608 Nick: llmstxt improvements 2025-02-19 16:09:46 -03:00
Nicolas
d4cf2269ed Update generate-llmstxt-service.ts 2025-02-19 15:50:59 -03:00
Aparup Ganguly
5e0036f8b4 Implemented gemini 2.0 web extractor 2025-02-20 00:05:12 +05:30
Nicolas
f5de803a9d Nick: fixes 2025-02-19 15:21:52 -03:00
Nicolas
a60f3ff645 Nick: fixes 2025-02-19 15:01:47 -03:00
Eric Ciarla
d984b50400
Add llmstxt generator endpoint (#1201) 
* Nick:

* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"

This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.

* Update deep-research-service.ts

* Nick:

* init

* part 2

* Update generate-llmstxt-service.ts

* Fix queue

* Update queue-worker.ts

* Almost there

* Final touches

* Update requests.http

* final touches

* Update requests.http

* Improve logging

* Change endpoint to /llmstxt

* Update queue-worker.ts

* Update generate-llmstxt-service.ts

* Nick: cache

* Update index.ts

* Update firecrawl.py

* Update package.json

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-19 14:42:33 -03:00
Gergő Móricz
e373fab5c1 fix(sdk/js): don't require CrawlScrapeOptions.formats 2025-02-19 17:58:19 +01:00
Gergő Móricz
8e1e5986f7 feat(sdk/js): bump 2025-02-19 17:37:36 +01:00
Gergő Móricz
4fd26a3baa feat(js-sdk): support scrapeOptions in ExtractParams 2025-02-19 17:08:43 +01:00
Nicolas
5c47e97db2
(feat/deep-research) Alpha implementation of deep research (#1202) 
* Nick:

* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"

This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.

* Update deep-research-service.ts

* Nick:

* Nick:

* Nick:

* Nick:

* Nick:

* Nick:

* Update deep-research-service.ts

* Nick:

* Update deep-research-service.ts

* Apply suggestions from code review

---------

Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-19 12:44:21 -03:00
Gergő Móricz
fc64f436ed
fix(v1/types): fix extract -> json rename, ROUND II (FIR-1072) (#1199) 
* Revert "Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)""

This reverts commit e28a44463ae49ffc195507204492cc7c15c438c4.

* fix(v1/types): fix bad transform

* feat(v1): proxy option / stealthProxy flag (FIR-1050) (#1196)

* feat(v1): proxy option / stealthProxy flag

* feat(js-sdk): add proxy option

* fix

* fix extract tests
 2025-02-19 12:07:55 -03:00
Gergő Móricz
42050d3d6e fix 2025-02-18 18:06:39 +01:00
Gergő Móricz
b136e42b53
feat(v1): proxy option / stealthProxy flag (FIR-1050) (#1196) 
* feat(v1): proxy option / stealthProxy flag

* feat(js-sdk): add proxy option
 2025-02-18 18:03:10 +01:00
Nicolas
e28a44463a Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)" 
This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.
 2025-02-18 11:31:23 -03:00
Gergő Móricz
586a10f40d
fix(v1/types): fix extract -> json rename (FIR-1072) (#1195) 
* fix(v1/types): fix extract -> json rename

* fix(types/v1): bad transform
 2025-02-18 10:32:19 -03:00
Tetsuro Yokoyama
5ac6eb7440
Update self-hosted Kubernetes deployments examples for compatibility and consistency (#1177) 
* fix: Quote variables in `docker-entrypoint.sh`

- This commit adds double quotes around variables in the docker-entrypoint.sh script to prevent word splitting and globbing issues, ensuring the script behaves correctly in all cases.

* fix: Ensure worker/api deployment starts with `OPENAI_API_KEY`

* fix: Add missing `FLY_PROCESS_GROUP` env var to deployments

* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`

* fix: Update Docker build options for Apple Silicon compatibility

* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`
 2025-02-18 13:33:20 +01:00
Gergő Móricz
aacbea1d9e fix(tests/snips/map): remove flaky useless test 2025-02-18 13:20:11 +01:00
Gergő Móricz
58564bceb8 Revert "temp: debugging" 
wrong branch
This reverts commit 97e6c28cf96fe837d3934b126e3b5e86fa10df2b.
 2025-02-18 11:28:35 +01:00
Gergő Móricz
97e6c28cf9 temp: debugging 2025-02-18 11:28:12 +01:00
Gergő Móricz
5c4021e8cf fix(scrapeURL/fire-engine): screenshot broken hotfix 2025-02-17 18:59:16 +01:00
Nicolas
b0534d0767 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-02-17 12:42:59 -03:00
Nicolas
03da3c36a6 Update auth.ts 2025-02-17 12:42:57 -03:00
Gergő Móricz
445b906af1
fix(scrapeURL/fire-engine): perform format screenshot after specified actions (#1192) 2025-02-17 12:35:55 -03:00
Gergő Móricz
7ecbff3b20
fix(map): do not remove query parameters from results (FIR-1015) (#1191) 
* fix(map): do not remove query parameters from results

* feat(map): add tests for query parameter handling
 2025-02-17 12:34:58 -03:00
Eric Ciarla
5cb2918368 update name of example 2025-02-16 18:46:10 -05:00
Gergő Móricz
1491b5b141
fix(scrapeURL/sb): enforce timeout (FIR-980) (#1183) 
* fix(scrapeURL/scrapingbee): enforce timeout

* fix(scrapeURL/sb): types

* fix the test

* fixup: remove nix files
 2025-02-16 15:55:03 +01:00
Gergő Móricz
fd8b38902a
fix(crawl-status): consider concurrency limited jobs as prioritized (#1184) 2025-02-16 15:52:17 +01:00
Nicolas
7ac2b99210 Update auth.ts v1.4.4 2025-02-14 11:12:02 -03:00
Nicolas
ce9d3d32f6 Nick: adjust for old compatibility s2 2025-02-14 11:09:09 -03:00
Gergő Móricz
a4cf8f439c Revert "(feat/extract) Multi-entity prompt improvements (#1181)" 
This reverts commit 584221a106216dfcc25de14c16720224cad237e1.
 2025-02-14 11:19:16 +01:00
Gergő Móricz
d9c99e5892
fix(v1/batch/scrape): use scrape rate limit (#1182) 2025-02-13 17:32:58 -03:00
dolonfly
254580a284
fix: relative url 2 full url use error base url (#584) 
according to ： https://developer.mozilla.org/en-US/docs/Web/API/URL/URL 

the second  parameter:  A string representing the base URL to use in cases where url is a relative reference

so need use the pageUrl , not url origin

eg: 
when pageUrl = http://example.com/a/b/c/d.html and tag a  href is '../e.html'
then 
correct: new URL('../e.html','http://example.com/a/b/c/d.html')
incorrect: new URL('../e.html','http://example.com')
 2025-02-13 20:38:47 +01:00
Gergő Móricz
73e7884df4
fix(queue-worker/crawl): only report successful page count in num_docs (#1179) 2025-02-13 13:14:24 -03:00
Nicolas
584221a106
(feat/extract) Multi-entity prompt improvements (#1181) 
* Update build-prompts.ts

* Update build-prompts.ts

* Update build-prompts.ts

* Update build-prompts.ts
 2025-02-13 13:11:35 -03:00
Nicolas
73ecaf74c2
feat(v1/extract) Show sources out of __experimental (#1180) 
* Nick:

* Nick:

* Nick:
 2025-02-13 12:41:45 -03:00
Gergő Móricz
1d9a0b9653
feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens (#1178) 
* feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens

* feat(v1/checkCredits): better instructions for upgrade
 2025-02-13 12:14:13 -03:00
Eric Ciarla
582bbf8da4
Merge pull request #1175 from mayooear/feat/gemini-sub-links 
fix: update gemini library. extract pdf links from scraped content
 2025-02-12 12:00:00 -05:00
mayo
32f9897670 fix: update gemini library. extract pdf links from markdowncontent 2025-02-12 16:43:59 +00:00
Eric Ciarla
a1b7d6e696
Merge pull request #1173 from mayooear/feat/gemini-sub-links 2025-02-11 22:11:49 -05:00
mayo
cb3bc5e445 Add detection of PDF/image sub-links and extract text via Gemini 2025-02-12 02:52:20 +00:00
Eric Ciarla
c67b052f4a Update gemini-2.0-crawler.py 2025-02-11 13:10:48 -05:00
Carter LaSalle
78094e2fe4
fix(html-transformer): Update free_string function parameter type (#1163) 
Correct the parameter type to use libc::c_char for better type consistency
 2025-02-11 15:11:29 +01:00
Luke Martin
52c0f78d4c
chore: add Serper and Search API envs to docker-compose (#1147) 2025-02-11 11:11:16 -03:00
Eric Ciarla
ef9aad8b8a
Merge pull request #1161 from aparupganguly/feature/gemini-2.0-crawler 
Implemented Gemini 2.0 crawler
 2025-02-11 08:13:06 -05:00
Nicolas
6b313ad0b0
Update types.ts 2025-02-10 17:42:12 -03:00