2697 Commits

Author SHA1 Message Date
Nicolas
a03b26a45f Update package.json 2025-04-12 17:11:56 -07:00
Gergő Móricz
f09458ffe3 fix(api/tests/scrape): schema change 2025-04-12 16:49:36 -07:00
devin-ai-integration[bot]
ef341399f0
Add change tracking support to Python and JS SDKs (#1448)
* Add change tracking support to Python and JS SDKs

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Replace test API keys with TEST_API_KEY placeholder

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Replace API keys with dummy values for testing

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Use environment variables for API keys in tests

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Move JS SDK test to correct location and add dependencies

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Remove old test file location

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Update test file to use TEST_API_KEY environment variable

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Update Python SDK test to use TEST_API_KEY environment variable

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Update package.json

* Update __init__.py

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-12 16:47:24 -07:00
devin-ai-integration[bot]
138a9757ae
(feat/change-tracking) Change Tracking Modes (#1445)
* Add git-diff support to change tracking format

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Fix type issues with parse-diff library

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Fix parse-diff type definitions to match actual library structure

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Add structured output/prompt support to change tracking

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* (feat/change-tracking) Change Tracking Modes (#1447)

* Refactor change tracking to use modes array instead of separate formats

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Implement schema-based change tracking with old/new value comparison

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Nick:

* Nick: .json

* Update diff.ts

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>

* Update index.ts

* Update types.ts

* Update diff.ts

* Update scrape.ts

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-12 16:38:56 -07:00
Gergő Móricz
f18a6b20ff extract concurrency hotfix 2025-04-11 20:38:54 +02:00
Gergő Móricz
6e9396dc57 feat(search): add further logging 2025-04-11 18:57:14 +02:00
Ademílson Tonato
b1fdc0e850
Merge pull request #1419 from mendableai/feat/email-notifications
feat: incorporate user preferences and notification categories
2025-04-11 16:51:24 +01:00
Nicolas
f52d6aab91
(feat/deep-research) Improvements to final analysis (#1443)
* Nick: fixes

* Update research-manager.ts

* Update research-manager.ts
2025-04-10 22:59:28 -07:00
Gergő Móricz
0bed648b0d Revert "Revert "Revert "temp: get acuc from main db"""
This reverts commit d3b821e8279414b2b998705aa93b644621a0d4cc.
2025-04-11 07:13:25 +02:00
Gergő Móricz
d3b821e827 Revert "Revert "temp: get acuc from main db""
This reverts commit 8566bff35c56ac042b54afd661a0cba7c4556e53.
2025-04-11 00:12:57 +02:00
Gergő Móricz
8566bff35c Revert "temp: get acuc from main db"
This reverts commit f16f034463d4216de4a9ae12fcad18193d66bd6d.
2025-04-10 20:28:43 +02:00
Gergő Móricz
f16f034463 temp: get acuc from main db 2025-04-10 18:58:59 +02:00
Gergő Móricz
6a10f0689d
ACUC: Dynamic Limits (FIR-1641) (#1434)
* extend acuc definition

* kill plan

* stuff

* stupid tests

* feat: better acuc

* feat(acuc): mock ACUC when not using db auth
2025-04-10 18:49:23 +02:00
Gergő Móricz
f2865f6699 temp: disable acuc caching 2025-04-10 16:08:20 +02:00
Gergő Móricz
a461f72d17 temporarily disable some flaky tests 2025-04-10 13:07:09 +02:00
Gergő Móricz
415603acb0 fixes 2025-04-10 12:50:03 +02:00
Gergő Móricz
4294face78
feat(scrapeURL): reintroduce default timeout for simple queries (#1440)
* feat(scrapeURL): reintroduce default timeout

* fix

* adjust timeouts
2025-04-10 12:29:54 +02:00
Gergő Móricz
7bb5643028 feat(log_job): is_migrated: true 2025-04-10 08:48:37 +02:00
Gergő Móricz
d925bf2f68
feat(log_job): stop putting docs in the db (#1438)
* feat(log_job): stop putting jobs in the db

* fix parens
2025-04-09 21:29:56 +02:00
Ademílson Tonato
45efe3fd46
Merge pull request #1436 from mendableai/feat/increase-search-limit
feat(api/search): add search endpoint and update request limits
2025-04-09 19:04:31 +01:00
Gergő Móricz
78a920af61 fix(api/tests/scrape): bump timeout 2025-04-09 19:47:38 +02:00
Gergő Móricz
d3da790dc4 feat(extraction-service): teamId logging 2025-04-09 18:48:00 +02:00
Ademílson F. Tonato
da2f17c757
feat(api/search): add search endpoint and update request limits
- Introduced a new POST endpoint for searching with a query and limit
- Updated the maximum limit for search results from 20 to 50 in the request schema
- Adjusted the default number of results in the Google search function from 7 to 5
2025-04-09 17:24:29 +01:00
Gergő Móricz
9fd735f3a1 feat(api/test/snips): disable flaky tests 2025-04-09 15:45:07 +02:00
Gergő Móricz
dc1a17d571 remove bad log 2025-04-09 13:04:00 +02:00
Gergő Móricz
3a8de846e3
read from GCS (again) (#1433)
* feat(crawl-status): retrieve job data from GCS

* feat(gcs-jobs/save): retrying saving metadata (might conflict)

* feat(gcs-jobs/save): retry save operation

* fix(gcs-jobs/save): respect metadata rules

* feat(crawl-status): log if gcs job is not found

* feat(ci/test/server): add gcs
2025-04-09 12:47:51 +02:00
Gergő Móricz
670e4a6bf1 Revert "feat(crawl-status): retrieve job data from GCS (#1427)"
This reverts commit 673bf6a2dea0f2340f964ba9c25cab8e92d929e4.
2025-04-09 12:28:46 +02:00
Gergő Móricz
673bf6a2de
feat(crawl-status): retrieve job data from GCS (#1427)
* feat(crawl-status): retrieve job data from GCS

* feat(gcs-jobs/save): retrying saving metadata (might conflict)

* feat(gcs-jobs/save): retry save operation

* fix(gcs-jobs/save): respect metadata rules

* feat(crawl-status): log if gcs job is not found
2025-04-09 12:27:23 +02:00
Gergő Móricz
ab6fb48e6e bump ver 2025-04-08 21:11:42 +02:00
devin-ai-integration[bot]
8c801ed956
Rename 'compare' format and property to 'changeTracking' (#1423) 2025-04-08 21:09:31 +02:00
Gergő Móricz
62265c63c8 feat(log_job): use atob 2025-04-08 20:26:43 +02:00
Gergő Móricz
c69d156179 fix(log_job): use service account credentials 2025-04-08 20:18:48 +02:00
Gergő Móricz
37b13ba146 feat(log_job): allow use of api key if specified 2025-04-08 20:04:59 +02:00
Gergő Móricz
bd1c1b0012
feat(log_job): start saving jobs to GCS (#1424) 2025-04-08 19:28:21 +02:00
Ademílson F. Tonato
80bf732f50
feat: incorporate user preferences and notification categories 2025-04-07 15:59:29 +01:00
Nicolas
66e65d9422 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-04-05 12:42:25 -04:00
Nicolas
f45fa12075 Update rate-limiter.ts 2025-04-05 12:42:24 -04:00
Gergő Móricz
f5e5bdb710 fix(llmExtract): arbitrary objects caused error to be thrown 2025-04-05 15:48:47 +02:00
Gergő Móricz
570809aa59 fix(unvisitedUrls): filter with crawler
Fixes #1410
2025-04-04 22:13:02 +02:00
Jingyu
6bed5eca50
fix(rust-sdk): remove rustfmt (#1392)
rustfmt is deprecated, it depends on a outdated extprim crate which cause test failed to run
2025-04-04 22:05:51 +02:00
Nicolas
41e094032f Update email_notification.ts 2025-04-04 14:36:41 -04:00
Nicolas
e1e39f8836 Nick: send notifications for crawl+batch scrape 2025-04-04 14:34:48 -04:00
Gergő Móricz
7128f83a7a
fix(js-sdk): isows import issues (FIR-1586) (FIR-1536) (#1411)
* attempt

* improvements

* kill isows -- there's been native websocket support in node since 21

* clean up the diff
2025-04-04 17:54:37 +02:00
Ademílson F. Tonato
426151c9c9
feat(queue-jobs): add function to determine job type and update notification logic for concurrency limits 2025-04-03 17:02:51 +01:00
Gergő Móricz
8c1579df51 bump cc 2025-04-03 11:56:24 +02:00
Gergő Móricz
2e2c3d52ce feat: add swoogo classes to force include main tags 2025-04-03 09:57:19 +02:00
Gergő Móricz
24f5199359
compare format (FIR-1560) (#1405) 2025-04-02 19:52:43 +02:00
Gergő Móricz
b3b63486f1 cc manual 2025-04-02 19:27:13 +02:00
Ademílson F. Tonato
b900f34b5a
feat(notification): add notification message for concurrency limit reached 2025-04-02 17:36:11 +01:00
rafaelmmiller
7216799ca0 revert mog changes 2025-04-02 10:45:11 -03:00