Gergő Móricz
13bd50ad2b
feat(fetch): don't time out (for smart scrape)
2025-04-15 21:57:05 -07:00
Gergő Móricz
0abe60085b
fix
2025-04-15 20:29:01 -07:00
Gergő Móricz
eea1267b50
feat(batchExtract): thingymajig
2025-04-15 20:28:53 -07:00
Gergő Móricz
d119552eee
bump rate limits
2025-04-15 20:20:29 -07:00
Gergő Móricz
5515ca7a52
fix(llm-cost): update
2025-04-15 19:49:14 -07:00
Nicolas
524b9770cd
Update queue-worker.ts
2025-04-15 17:52:20 -07:00
Gergő Móricz
c8a8e96acc
un-gate scrape status, add test
2025-04-15 17:48:54 -07:00
Gergő Móricz
3ccef5fb66
fix(v1): scrape-status with GCS
2025-04-15 17:42:38 -07:00
rafaelmmiller
252a9ccc89
Refactor robustFetch logging to exclude sensitive parameters and improve error handling.
2025-04-15 17:15:01 -07:00
devin-ai-integration[bot]
a840db9ef3
Set default timeout to 120s when proxy is stealth ( #1464 )
...
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mogery@sideguide.dev <mogery@sideguide.dev>
2025-04-15 23:09:11 +02:00
Gergő Móricz
31e24e907c
FIX DAT
2025-04-15 02:13:31 -07:00
Gergő Móricz
0ee9603945
FIX MORE
2025-04-15 02:09:52 -07:00
Gergő Móricz
9400b1423a
fix typing
2025-04-15 02:07:40 -07:00
Gergő Móricz
edb40d75c1
log session ID
2025-04-15 01:46:31 -07:00
Nicolas
6634d236bf
(feat/fire-1) FIRE-1 ( #1462 )
...
* wip
* integrating smart-scrape
* integrate smartscrape into llmExtract
* wip
* smart scrape multiple links
* fixes
* fix
* wip
* it worked!
* wip. there's a bug on the batchExtract TypeError: Converting circular structure to JSON
* wip
* retry model
* retry models
* feat/scrape+json+extract interfaces ready
* vertex -> googleapi
* fix/transformArrayToObject. required params on schema is still a bug
* change model
* o3-mini -> gemini
* Update extractSmartScrape.ts
* sessionId
* sessionId
* Nick: f-0 start
* Update extraction-service-f0.ts
* Update types.ts
* Nick:
* Update queue-worker.ts
* Nick: new interface
* rename analyzeSchemaAndPrompt -> F0
* refactor: rename agent ID to model in types and extract logic
* agent
* id->model
* id->model
* refactor: standardize agent model handling and validation across extraction logic
* livecast agent
* (feat/f1) sdks (#1459 )
* feat: add FIRE-1 agent support to Python and JavaScript SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add FIRE-1 agent support to scrape methods in both SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add prompt and sessionId to AgentOptions interface
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Update index.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* feat(v1): rate limits
* Update types.ts
* Update llmExtract.ts
* add cost tracking
* remove
* Update requests.http
* fix smart scrape cost calc
* log sm cost
* fix counts
* fix
* expose cost tracking
* models fix
* temp: skipLibcheck
* get rid of it
* fix ts
* dont skip lib check
* Update extractSmartScrape.ts
* Update queue-worker.ts
* Update smartScrape.ts
* Update requests.http
* fix(rate-limiter):
* types: fire-1 refine
* bill 150
* fix credits used on crawl
* ban from crawl
* route cost limit warning
* Update generic-ai.ts
* genres
* Update llmExtract.ts
* test server diff
* cletu
---------
Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
Co-authored-by: Ademílson F. Tonato <ademilsonft@outlook.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-04-15 00:19:45 -07:00
Gergő Móricz
e2c4b0e72f
remove double v0 log
2025-04-14 18:12:36 -07:00
Nicolas
0446443bbe
Nick: acuc cache on now
2025-04-14 16:06:38 -07:00
Gergő Móricz
b415e625a0
feat(scrape): get job result from GCS, avoid Redis ( #1461 )
...
* feat(scrape): get job result from GCS, avoid Redis
* call logjob on scrapes
* Fix inverse bool
* fix more
* migrate gracefully
* refactor
* feat(tests/search): test with scrape
2025-04-15 00:07:44 +02:00
Gergő Móricz
0b50349fed
feat(v0): fix jobs
2025-04-14 11:39:13 -07:00
Gergő Móricz
ebdf182b00
feat(auth): more ip
2025-04-13 23:26:57 -07:00
Gergő Móricz
07cdde7409
feat(auth): preview acuc team more
2025-04-13 23:21:34 -07:00
Gergő Móricz
713d58675e
fix(auth): preview acuc team
2025-04-13 23:16:55 -07:00
Gergő Móricz
2857496356
feat(diff): better rpc (revert this if broken)
2025-04-13 22:45:58 -07:00
Nicolas
cf84324534
Merge branch 'devin/1744568481-add-wait-for-change-tracking'
2025-04-13 11:54:05 -07:00
Nicolas
94d0b4be43
Update types.ts
2025-04-13 11:53:35 -07:00
devin-ai-integration[bot]
6bdae3cf4f
Add waitFor of 5000ms for changeTracking format ( #1450 )
...
* Add waitFor of 5000ms for changeTracking format
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Move waitFor check for changeTracking to types.ts refine layer
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Fix waitFor check to handle undefined values
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Update apps/api/src/__tests__/snips/scrape.test.ts
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-04-13 11:52:01 -07:00
Nicolas
0c5bd85540
Update apps/api/src/__tests__/snips/scrape.test.ts
...
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-04-13 11:50:52 -07:00
Devin AI
4026866c15
Fix waitFor check to handle undefined values
...
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
2025-04-13 18:50:22 +00:00
Devin AI
8bc2f16712
Move waitFor check for changeTracking to types.ts refine layer
...
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
2025-04-13 18:45:17 +00:00
Gergő Móricz
d8e3c36ccc
feat(api): install git to docker to have proper diffs
2025-04-13 11:38:26 -07:00
Nicolas
870fe56e0a
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-04-13 11:26:31 -07:00
Nicolas
723e7b7c60
Update diff.ts
2025-04-13 11:26:30 -07:00
Devin AI
9ab2a26636
Add waitFor of 5000ms for changeTracking format
...
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
2025-04-13 18:25:12 +00:00
Gergő Móricz
5658232ed6
feat(acuc): bump 30
2025-04-13 10:32:03 -07:00
Gergő Móricz
d260f367e4
feat(acuc): bump
2025-04-13 10:12:35 -07:00
Gergő Móricz
f09458ffe3
fix(api/tests/scrape): schema change
2025-04-12 16:49:36 -07:00
devin-ai-integration[bot]
138a9757ae
(feat/change-tracking) Change Tracking Modes ( #1445 )
...
* Add git-diff support to change tracking format
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Fix type issues with parse-diff library
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Fix parse-diff type definitions to match actual library structure
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Add structured output/prompt support to change tracking
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* (feat/change-tracking) Change Tracking Modes (#1447 )
* Refactor change tracking to use modes array instead of separate formats
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Implement schema-based change tracking with old/new value comparison
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Nick:
* Nick: .json
* Update diff.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* Update index.ts
* Update types.ts
* Update diff.ts
* Update scrape.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-12 16:38:56 -07:00
Gergő Móricz
f18a6b20ff
extract concurrency hotfix
2025-04-11 20:38:54 +02:00
Gergő Móricz
6e9396dc57
feat(search): add further logging
2025-04-11 18:57:14 +02:00
Ademílson Tonato
b1fdc0e850
Merge pull request #1419 from mendableai/feat/email-notifications
...
feat: incorporate user preferences and notification categories
2025-04-11 16:51:24 +01:00
Nicolas
f52d6aab91
(feat/deep-research) Improvements to final analysis ( #1443 )
...
* Nick: fixes
* Update research-manager.ts
* Update research-manager.ts
2025-04-10 22:59:28 -07:00
Gergő Móricz
0bed648b0d
Revert "Revert "Revert "temp: get acuc from main db"""
...
This reverts commit d3b821e8279414b2b998705aa93b644621a0d4cc.
2025-04-11 07:13:25 +02:00
Gergő Móricz
d3b821e827
Revert "Revert "temp: get acuc from main db""
...
This reverts commit 8566bff35c56ac042b54afd661a0cba7c4556e53.
2025-04-11 00:12:57 +02:00
Gergő Móricz
8566bff35c
Revert "temp: get acuc from main db"
...
This reverts commit f16f034463d4216de4a9ae12fcad18193d66bd6d.
2025-04-10 20:28:43 +02:00
Gergő Móricz
f16f034463
temp: get acuc from main db
2025-04-10 18:58:59 +02:00
Gergő Móricz
6a10f0689d
ACUC: Dynamic Limits (FIR-1641) ( #1434 )
...
* extend acuc definition
* kill plan
* stuff
* stupid tests
* feat: better acuc
* feat(acuc): mock ACUC when not using db auth
2025-04-10 18:49:23 +02:00
Gergő Móricz
f2865f6699
temp: disable acuc caching
2025-04-10 16:08:20 +02:00
Gergő Móricz
a461f72d17
temporarily disable some flaky tests
2025-04-10 13:07:09 +02:00
Gergő Móricz
415603acb0
fixes
2025-04-10 12:50:03 +02:00
Gergő Móricz
4294face78
feat(scrapeURL): reintroduce default timeout for simple queries ( #1440 )
...
* feat(scrapeURL): reintroduce default timeout
* fix
* adjust timeouts
2025-04-10 12:29:54 +02:00
