2761 Commits

Author SHA1 Message Date
Gergő Móricz
d8e3c36ccc feat(api): install git to docker to have proper diffs 2025-04-13 11:38:26 -07:00
Nicolas
870fe56e0a Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-04-13 11:26:31 -07:00
Nicolas
723e7b7c60 Update diff.ts 2025-04-13 11:26:30 -07:00
Devin AI
9ab2a26636 Add waitFor of 5000ms for changeTracking format
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
2025-04-13 18:25:12 +00:00
Gergő Móricz
5658232ed6 feat(acuc): bump 30 2025-04-13 10:32:03 -07:00
Gergő Móricz
d260f367e4 feat(acuc): bump 2025-04-13 10:12:35 -07:00
Gergő Móricz
63a283bfe7 js-sdk: once again again 2025-04-12 17:44:19 -07:00
Gergő Móricz
58ad7f40b6 js-sdk: once more 2025-04-12 17:42:30 -07:00
Gergő Móricz
62c842e64a js-sdk: fix tsup config 2025-04-12 17:40:16 -07:00
Gergő Móricz
32798e2200 revert lock 2025-04-12 17:27:47 -07:00
Gergő Móricz
557df1ab95 js-sdk: bump 2025-04-12 17:26:35 -07:00
Gergő Móricz
4414fbca43 fix lock 2025-04-12 17:23:15 -07:00
Nicolas
950a9512bd Nick: 2025-04-12 17:21:36 -07:00
Gergő Móricz
c5079074bc js-sdk: bump 2025-04-12 17:16:28 -07:00
Nicolas
a03b26a45f Update package.json 2025-04-12 17:11:56 -07:00
Gergő Móricz
f09458ffe3 fix(api/tests/scrape): schema change 2025-04-12 16:49:36 -07:00
devin-ai-integration[bot]
ef341399f0
Add change tracking support to Python and JS SDKs (#1448)
* Add change tracking support to Python and JS SDKs

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Replace test API keys with TEST_API_KEY placeholder

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Replace API keys with dummy values for testing

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Use environment variables for API keys in tests

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Move JS SDK test to correct location and add dependencies

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Remove old test file location

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Update test file to use TEST_API_KEY environment variable

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Update Python SDK test to use TEST_API_KEY environment variable

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Update package.json

* Update __init__.py

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-12 16:47:24 -07:00
devin-ai-integration[bot]
138a9757ae
(feat/change-tracking) Change Tracking Modes (#1445)
* Add git-diff support to change tracking format

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Fix type issues with parse-diff library

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Fix parse-diff type definitions to match actual library structure

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Add structured output/prompt support to change tracking

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* (feat/change-tracking) Change Tracking Modes (#1447)

* Refactor change tracking to use modes array instead of separate formats

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Implement schema-based change tracking with old/new value comparison

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Nick:

* Nick: .json

* Update diff.ts

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>

* Update index.ts

* Update types.ts

* Update diff.ts

* Update scrape.ts

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-12 16:38:56 -07:00
Gergő Móricz
f18a6b20ff extract concurrency hotfix 2025-04-11 20:38:54 +02:00
Gergő Móricz
6e9396dc57 feat(search): add further logging 2025-04-11 18:57:14 +02:00
Ademílson Tonato
b1fdc0e850
Merge pull request #1419 from mendableai/feat/email-notifications
feat: incorporate user preferences and notification categories
2025-04-11 16:51:24 +01:00
Nicolas
f52d6aab91
(feat/deep-research) Improvements to final analysis (#1443)
* Nick: fixes

* Update research-manager.ts

* Update research-manager.ts
2025-04-10 22:59:28 -07:00
Gergő Móricz
0bed648b0d Revert "Revert "Revert "temp: get acuc from main db"""
This reverts commit d3b821e8279414b2b998705aa93b644621a0d4cc.
2025-04-11 07:13:25 +02:00
Gergő Móricz
d3b821e827 Revert "Revert "temp: get acuc from main db""
This reverts commit 8566bff35c56ac042b54afd661a0cba7c4556e53.
2025-04-11 00:12:57 +02:00
Gergő Móricz
8566bff35c Revert "temp: get acuc from main db"
This reverts commit f16f034463d4216de4a9ae12fcad18193d66bd6d.
2025-04-10 20:28:43 +02:00
Gergő Móricz
f16f034463 temp: get acuc from main db 2025-04-10 18:58:59 +02:00
Gergő Móricz
6a10f0689d
ACUC: Dynamic Limits (FIR-1641) (#1434)
* extend acuc definition

* kill plan

* stuff

* stupid tests

* feat: better acuc

* feat(acuc): mock ACUC when not using db auth
2025-04-10 18:49:23 +02:00
Gergő Móricz
f2865f6699 temp: disable acuc caching 2025-04-10 16:08:20 +02:00
Gergő Móricz
a461f72d17 temporarily disable some flaky tests 2025-04-10 13:07:09 +02:00
Gergő Móricz
415603acb0 fixes 2025-04-10 12:50:03 +02:00
Gergő Móricz
4294face78
feat(scrapeURL): reintroduce default timeout for simple queries (#1440)
* feat(scrapeURL): reintroduce default timeout

* fix

* adjust timeouts
2025-04-10 12:29:54 +02:00
Gergő Móricz
7bb5643028 feat(log_job): is_migrated: true 2025-04-10 08:48:37 +02:00
Gergő Móricz
d925bf2f68
feat(log_job): stop putting docs in the db (#1438)
* feat(log_job): stop putting jobs in the db

* fix parens
2025-04-09 21:29:56 +02:00
Ademílson Tonato
45efe3fd46
Merge pull request #1436 from mendableai/feat/increase-search-limit
feat(api/search): add search endpoint and update request limits
2025-04-09 19:04:31 +01:00
Gergő Móricz
78a920af61 fix(api/tests/scrape): bump timeout 2025-04-09 19:47:38 +02:00
Gergő Móricz
d3da790dc4 feat(extraction-service): teamId logging 2025-04-09 18:48:00 +02:00
Ademílson F. Tonato
da2f17c757
feat(api/search): add search endpoint and update request limits
- Introduced a new POST endpoint for searching with a query and limit
- Updated the maximum limit for search results from 20 to 50 in the request schema
- Adjusted the default number of results in the Google search function from 7 to 5
2025-04-09 17:24:29 +01:00
Gergő Móricz
9fd735f3a1 feat(api/test/snips): disable flaky tests 2025-04-09 15:45:07 +02:00
Gergő Móricz
dc1a17d571 remove bad log 2025-04-09 13:04:00 +02:00
Gergő Móricz
3a8de846e3
read from GCS (again) (#1433)
* feat(crawl-status): retrieve job data from GCS

* feat(gcs-jobs/save): retrying saving metadata (might conflict)

* feat(gcs-jobs/save): retry save operation

* fix(gcs-jobs/save): respect metadata rules

* feat(crawl-status): log if gcs job is not found

* feat(ci/test/server): add gcs
2025-04-09 12:47:51 +02:00
Gergő Móricz
670e4a6bf1 Revert "feat(crawl-status): retrieve job data from GCS (#1427)"
This reverts commit 673bf6a2dea0f2340f964ba9c25cab8e92d929e4.
2025-04-09 12:28:46 +02:00
Gergő Móricz
673bf6a2de
feat(crawl-status): retrieve job data from GCS (#1427)
* feat(crawl-status): retrieve job data from GCS

* feat(gcs-jobs/save): retrying saving metadata (might conflict)

* feat(gcs-jobs/save): retry save operation

* fix(gcs-jobs/save): respect metadata rules

* feat(crawl-status): log if gcs job is not found
2025-04-09 12:27:23 +02:00
Gergő Móricz
ab6fb48e6e bump ver 2025-04-08 21:11:42 +02:00
devin-ai-integration[bot]
8c801ed956
Rename 'compare' format and property to 'changeTracking' (#1423) 2025-04-08 21:09:31 +02:00
Gergő Móricz
62265c63c8 feat(log_job): use atob 2025-04-08 20:26:43 +02:00
Gergő Móricz
c69d156179 fix(log_job): use service account credentials 2025-04-08 20:18:48 +02:00
Gergő Móricz
37b13ba146 feat(log_job): allow use of api key if specified 2025-04-08 20:04:59 +02:00
Gergő Móricz
bd1c1b0012
feat(log_job): start saving jobs to GCS (#1424) 2025-04-08 19:28:21 +02:00
Ademílson F. Tonato
80bf732f50
feat: incorporate user preferences and notification categories 2025-04-07 15:59:29 +01:00
Nicolas
66e65d9422 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-04-05 12:42:25 -04:00