Nicolas
524b9770cd
Update queue-worker.ts
2025-04-15 17:52:20 -07:00
Gergő Móricz
c8a8e96acc
un-gate scrape status, add test
2025-04-15 17:48:54 -07:00
Gergő Móricz
3ccef5fb66
fix(v1): scrape-status with GCS
2025-04-15 17:42:38 -07:00
rafaelmmiller
252a9ccc89
Refactor robustFetch logging to exclude sensitive parameters and improve error handling.
2025-04-15 17:15:01 -07:00
devin-ai-integration[bot]
a840db9ef3
Set default timeout to 120s when proxy is stealth ( #1464 )
...
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mogery@sideguide.dev <mogery@sideguide.dev>
2025-04-15 23:09:11 +02:00
Gergő Móricz
31e24e907c
FIX DAT
2025-04-15 02:13:31 -07:00
Gergő Móricz
0ee9603945
FIX MORE
2025-04-15 02:09:52 -07:00
Gergő Móricz
9400b1423a
fix typing
2025-04-15 02:07:40 -07:00
Gergő Móricz
edb40d75c1
log session ID
2025-04-15 01:46:31 -07:00
Nicolas
6634d236bf
(feat/fire-1) FIRE-1 ( #1462 )
...
* wip
* integrating smart-scrape
* integrate smartscrape into llmExtract
* wip
* smart scrape multiple links
* fixes
* fix
* wip
* it worked!
* wip. there's a bug on the batchExtract TypeError: Converting circular structure to JSON
* wip
* retry model
* retry models
* feat/scrape+json+extract interfaces ready
* vertex -> googleapi
* fix/transformArrayToObject. required params on schema is still a bug
* change model
* o3-mini -> gemini
* Update extractSmartScrape.ts
* sessionId
* sessionId
* Nick: f-0 start
* Update extraction-service-f0.ts
* Update types.ts
* Nick:
* Update queue-worker.ts
* Nick: new interface
* rename analyzeSchemaAndPrompt -> F0
* refactor: rename agent ID to model in types and extract logic
* agent
* id->model
* id->model
* refactor: standardize agent model handling and validation across extraction logic
* livecast agent
* (feat/f1) sdks (#1459 )
* feat: add FIRE-1 agent support to Python and JavaScript SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add FIRE-1 agent support to scrape methods in both SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add prompt and sessionId to AgentOptions interface
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Update index.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* feat(v1): rate limits
* Update types.ts
* Update llmExtract.ts
* add cost tracking
* remove
* Update requests.http
* fix smart scrape cost calc
* log sm cost
* fix counts
* fix
* expose cost tracking
* models fix
* temp: skipLibcheck
* get rid of it
* fix ts
* dont skip lib check
* Update extractSmartScrape.ts
* Update queue-worker.ts
* Update smartScrape.ts
* Update requests.http
* fix(rate-limiter):
* types: fire-1 refine
* bill 150
* fix credits used on crawl
* ban from crawl
* route cost limit warning
* Update generic-ai.ts
* genres
* Update llmExtract.ts
* test server diff
* cletu
---------
Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
Co-authored-by: Ademílson F. Tonato <ademilsonft@outlook.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-04-15 00:19:45 -07:00
Gergő Móricz
e2c4b0e72f
remove double v0 log
2025-04-14 18:12:36 -07:00
Nicolas
0446443bbe
Nick: acuc cache on now
2025-04-14 16:06:38 -07:00
Gergő Móricz
b415e625a0
feat(scrape): get job result from GCS, avoid Redis ( #1461 )
...
* feat(scrape): get job result from GCS, avoid Redis
* call logjob on scrapes
* Fix inverse bool
* fix more
* migrate gracefully
* refactor
* feat(tests/search): test with scrape
2025-04-15 00:07:44 +02:00
Eric Ciarla
0ac86abead
Merge pull request #1460 from aparupganguly/examples/gpt-4.1-crawler
2025-04-14 12:55:49 -07:00
Aparup Ganguly
5dca350a11
Add examples/gpt-4.1-crawler
2025-04-15 00:48:19 +05:30
Gergő Móricz
0b50349fed
feat(v0): fix jobs
2025-04-14 11:39:13 -07:00
Gergő Móricz
ebdf182b00
feat(auth): more ip
2025-04-13 23:26:57 -07:00
Gergő Móricz
07cdde7409
feat(auth): preview acuc team more
2025-04-13 23:21:34 -07:00
Gergő Móricz
713d58675e
fix(auth): preview acuc team
2025-04-13 23:16:55 -07:00
Gergő Móricz
2857496356
feat(diff): better rpc (revert this if broken)
2025-04-13 22:45:58 -07:00
Nicolas
cf84324534
Merge branch 'devin/1744568481-add-wait-for-change-tracking'
2025-04-13 11:54:05 -07:00
Nicolas
94d0b4be43
Update types.ts
2025-04-13 11:53:35 -07:00
devin-ai-integration[bot]
6bdae3cf4f
Add waitFor of 5000ms for changeTracking format ( #1450 )
...
* Add waitFor of 5000ms for changeTracking format
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Move waitFor check for changeTracking to types.ts refine layer
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Fix waitFor check to handle undefined values
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Update apps/api/src/__tests__/snips/scrape.test.ts
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-04-13 11:52:01 -07:00
Nicolas
0c5bd85540
Update apps/api/src/__tests__/snips/scrape.test.ts
...
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-04-13 11:50:52 -07:00
Devin AI
4026866c15
Fix waitFor check to handle undefined values
...
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
2025-04-13 18:50:22 +00:00
Devin AI
8bc2f16712
Move waitFor check for changeTracking to types.ts refine layer
...
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
2025-04-13 18:45:17 +00:00
Gergő Móricz
d8e3c36ccc
feat(api): install git to docker to have proper diffs
2025-04-13 11:38:26 -07:00
Nicolas
870fe56e0a
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-04-13 11:26:31 -07:00
Nicolas
723e7b7c60
Update diff.ts
2025-04-13 11:26:30 -07:00
Devin AI
9ab2a26636
Add waitFor of 5000ms for changeTracking format
...
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
2025-04-13 18:25:12 +00:00
Gergő Móricz
5658232ed6
feat(acuc): bump 30
2025-04-13 10:32:03 -07:00
Gergő Móricz
d260f367e4
feat(acuc): bump
2025-04-13 10:12:35 -07:00
Gergő Móricz
63a283bfe7
js-sdk: once again again
2025-04-12 17:44:19 -07:00
Gergő Móricz
58ad7f40b6
js-sdk: once more
2025-04-12 17:42:30 -07:00
Gergő Móricz
62c842e64a
js-sdk: fix tsup config
2025-04-12 17:40:16 -07:00
Gergő Móricz
32798e2200
revert lock
2025-04-12 17:27:47 -07:00
Gergő Móricz
557df1ab95
js-sdk: bump
2025-04-12 17:26:35 -07:00
Gergő Móricz
3cf6d88bff
js-sdk: change ci
2025-04-12 17:26:17 -07:00
Gergő Móricz
4414fbca43
fix lock
2025-04-12 17:23:15 -07:00
Nicolas
950a9512bd
Nick:
2025-04-12 17:21:36 -07:00
Gergő Móricz
c5079074bc
js-sdk: bump
2025-04-12 17:16:28 -07:00
Nicolas
a03b26a45f
Update package.json
2025-04-12 17:11:56 -07:00
Gergő Móricz
f09458ffe3
fix(api/tests/scrape): schema change
2025-04-12 16:49:36 -07:00
devin-ai-integration[bot]
ef341399f0
Add change tracking support to Python and JS SDKs ( #1448 )
...
* Add change tracking support to Python and JS SDKs
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Replace test API keys with TEST_API_KEY placeholder
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Replace API keys with dummy values for testing
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Use environment variables for API keys in tests
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Move JS SDK test to correct location and add dependencies
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Remove old test file location
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Update test file to use TEST_API_KEY environment variable
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Update Python SDK test to use TEST_API_KEY environment variable
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Update package.json
* Update __init__.py
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-12 16:47:24 -07:00
devin-ai-integration[bot]
138a9757ae
(feat/change-tracking) Change Tracking Modes ( #1445 )
...
* Add git-diff support to change tracking format
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Fix type issues with parse-diff library
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Fix parse-diff type definitions to match actual library structure
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Add structured output/prompt support to change tracking
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* (feat/change-tracking) Change Tracking Modes (#1447 )
* Refactor change tracking to use modes array instead of separate formats
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Implement schema-based change tracking with old/new value comparison
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Nick:
* Nick: .json
* Update diff.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* Update index.ts
* Update types.ts
* Update diff.ts
* Update scrape.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-12 16:38:56 -07:00
Gergő Móricz
f18a6b20ff
extract concurrency hotfix
2025-04-11 20:38:54 +02:00
Gergő Móricz
6e9396dc57
feat(search): add further logging
2025-04-11 18:57:14 +02:00
Ademílson Tonato
b1fdc0e850
Merge pull request #1419 from mendableai/feat/email-notifications
...
feat: incorporate user preferences and notification categories
2025-04-11 16:51:24 +01:00
Nicolas
f52d6aab91
(feat/deep-research) Improvements to final analysis ( #1443 )
...
* Nick: fixes
* Update research-manager.ts
* Update research-manager.ts
2025-04-10 22:59:28 -07:00
Gergő Móricz
0bed648b0d
Revert "Revert "Revert "temp: get acuc from main db"""
...
This reverts commit d3b821e8279414b2b998705aa93b644621a0d4cc.
2025-04-11 07:13:25 +02:00
