Nicolas
584221a106
(feat/extract) Multi-entity prompt improvements ( #1181 )
...
* Update build-prompts.ts
* Update build-prompts.ts
* Update build-prompts.ts
* Update build-prompts.ts
2025-02-13 13:11:35 -03:00
Nicolas
73ecaf74c2
feat(v1/extract) Show sources out of __experimental ( #1180 )
...
* Nick:
* Nick:
* Nick:
2025-02-13 12:41:45 -03:00
Gergő Móricz
1d9a0b9653
feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens ( #1178 )
...
* feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens
* feat(v1/checkCredits): better instructions for upgrade
2025-02-13 12:14:13 -03:00
Eric Ciarla
582bbf8da4
Merge pull request #1175 from mayooear/feat/gemini-sub-links
...
fix: update gemini library. extract pdf links from scraped content
2025-02-12 12:00:00 -05:00
mayo
32f9897670
fix: update gemini library. extract pdf links from markdowncontent
2025-02-12 16:43:59 +00:00
Eric Ciarla
a1b7d6e696
Merge pull request #1173 from mayooear/feat/gemini-sub-links
2025-02-11 22:11:49 -05:00
mayo
cb3bc5e445
Add detection of PDF/image sub-links and extract text via Gemini
2025-02-12 02:52:20 +00:00
Eric Ciarla
c67b052f4a
Update gemini-2.0-crawler.py
2025-02-11 13:10:48 -05:00
Carter LaSalle
78094e2fe4
fix(html-transformer): Update free_string function parameter type ( #1163 )
...
Correct the parameter type to use libc::c_char for better type consistency
2025-02-11 15:11:29 +01:00
Luke Martin
52c0f78d4c
chore: add Serper and Search API envs to docker-compose ( #1147 )
2025-02-11 11:11:16 -03:00
Eric Ciarla
ef9aad8b8a
Merge pull request #1161 from aparupganguly/feature/gemini-2.0-crawler
...
Implemented Gemini 2.0 crawler
2025-02-11 08:13:06 -05:00
Nicolas
6b313ad0b0
Update types.ts
2025-02-10 17:42:12 -03:00
Gergő Móricz
bc787de62a
fix(blocklist): behavioural fixes
2025-02-10 19:10:20 +01:00
Ademílson Tonato
3608256569
Merge pull request #1146 from mendableai/feat/wait-validation
...
[FIR-796] feat(api/types): Add action and wait time validation for scrape requests
2025-02-10 14:45:47 +00:00
Gergő Móricz
290dd033d3
Update apps/api/src/controllers/v1/types.ts
2025-02-10 09:31:23 +01:00
Ademílson F. Tonato
843cec971d
[FIR-796] feat(api/types): Add action and wait time validation for scrape requests
...
- Implement max actions limit (15 actions)
- Add total wait time calculation for actions
- Increase max waitFor time to 60 seconds
- Refactor scrape options schema to include validation
2025-02-07 21:16:38 +00:00
Aparup Ganguly
f33fa5bfc6
Initial commit
2025-02-08 01:24:21 +05:30
Eric Ciarla
2b7b7400f6
Merge pull request #1144 from aparupganguly/feature/o3-mini-job-resource
...
Feature/o3 mini job resource extractor
2025-02-07 11:39:29 -05:00
Rafael Miller
ac5c88bffb
added scrapeOptions to extract ( #1133 )
2025-02-07 13:38:08 -03:00
Eric Ciarla
42f4f7ef09
Merge pull request #1139 from aparupganguly/feature/o3-mini-review-summarizer
...
Implemented prodcut reviews summarizer using o3 mini
2025-02-07 11:37:57 -05:00
Aparup Ganguly
46f05a750d
Job Resources Analyzer
2025-02-07 20:32:52 +05:30
Gergő Móricz
892f3a41f3
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) ( #1138 )
...
* fix(scrape): allow getting valid JSON via rawHtml
* fix(scrape/test):
2025-02-06 18:35:28 +01:00
Nicolas
401c18761a
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-02-06 12:31:31 -03:00
Nicolas
394a31064d
Update types.ts
2025-02-06 12:31:19 -03:00
Ademílson Tonato
996352c24c
Merge pull request #1117 from mendableai/feat/blocked-domains
...
feat(blocklist): Improve URL blocking with tldts parsing
2025-02-06 15:23:58 +00:00
Nicolas
fe276060cf
Update pyproject.toml
2025-02-06 12:19:32 -03:00
Gergő Móricz
844ba39f74
fix(sdks): MIT license
2025-02-06 16:07:50 +01:00
Aparup Ganguly
310ef66506
Implemented prodcut reviews summarizer using o3 mini
2025-02-06 20:13:35 +05:30
Gergő Móricz
34fe360dc1
fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827) ( #1134 )
...
* fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends
* fix(crawl-redis/tests): adjust tests
2025-02-05 15:44:56 +01:00
Nick Roth
aa1e820d52
Fix corepack and self hosting setup ( #1131 )
...
* Update Dockerfile
Config corepack and pnpm
* Update Dockerfile
Try using this approach: https://github.com/pnpm/pnpm/issues/9029#issuecomment-2631400936
* Add ulimits command to docker compose
* Remove ulimit from entrypoint
* Add ulimit back to docker-entrypoint.sh
This implements the proposed approach to not fail when the ulimit command can't run due to the environment
* Resolve signature mismatch
Resolves issue encountered when using docker compose up on a clean environment on an intel mac. This just adds the clean command to avoid leveraging any cached data before updating apt-get and installing ca-certificates
---------
Co-authored-by: Nick Roth <nroth@dealnews.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-05 10:19:11 +01:00
Thomas Kosmas
73d19aa01b
fix(dockerfile): Temporarily resolve corepack issue
2025-02-05 02:42:16 +02:00
Eric Ciarla
b0f9e73376
Update o3-mini_web_crawler.py
2025-02-04 16:35:30 -05:00
Gergő Móricz
1e7d42e8af
fix(concurrency-limit): allow for big crawls
2025-02-04 19:26:30 +01:00
Eric Ciarla
1951b60881
Merge pull request #1130 from aparupganguly/feature/o3-company-researcher
...
Updated Model Parameters (o3-mini_company_researcher)
2025-02-04 09:46:16 -05:00
Aparup Ganguly
7aa2db25a3
Updated Model Parameters
2025-02-04 17:54:44 +05:30
Eric Ciarla
5894076fda
Merge pull request #1120 from aparupganguly/feature-o3-crawler
...
Feature o3-mini web crawler
2025-02-03 10:48:02 -05:00
Aparup Ganguly
01d656feb7
Minor Changes
2025-02-03 00:54:45 +05:30
Aparup Ganguly
d5d3df9d10
Handled JSON format structure
2025-02-03 00:39:00 +05:30
Aparup Ganguly
61b989cc64
Initial Commit
2025-02-02 17:36:26 +05:30
Eric Ciarla
e0c292f847
Merge pull request #1115 from aparupganguly/feature-r1-web-extractor
...
Added R1 web extractor feature
2025-01-31 13:35:58 -05:00
Eric Ciarla
cdc747ab7b
small edits
2025-01-31 13:35:46 -05:00
Ademílson F. Tonato
bcd74498e3
test(blocklist): Enhance URL blocking test coverage with decrypted domain validation
2025-01-31 18:23:38 +00:00
Rafael Miller
8d7e8c4f50
added cached scrapes to extract ( #1107 )
...
* added cached scrapes to extract
* dont save if exists
* no duplicates
* experimental tag
* Update requests.http
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-01-31 13:58:52 -03:00
Ademílson Tonato
4341b6e12e
Merge pull request #1118 from sami0596/patch-1
...
docs: fix typo on CONTRIBUTING.md
2025-01-31 16:41:04 +00:00
Nicolas
492b81d7c3
Nick: origin api-sdk in python sdk
2025-01-31 12:20:41 -03:00
sami0596
0a44d11b83
Update CONTRIBUTING.md
...
A letter seemed to be forgotten
2025-01-31 14:53:29 +01:00
Ademílson F. Tonato
24e8aaf6b5
feat(blocklist): Improve URL blocking with tldts parsing
2025-01-31 13:14:48 +00:00
Móricz Gergő
8a8d7d645f
fix(concurrency): proper job timeouting
2025-01-31 11:22:10 +01:00
Nicolas
e026917698
Nick: disable auto rech for extract requests
2025-01-31 01:10:59 -03:00
Nicolas
6bfd24d903
Nick: waitFor fixes
2025-01-30 23:23:03 -03:00
