Gergő Móricz
ba4df67de7
force 2.5
2025-04-16 16:53:04 -07:00
Gergő Móricz
6a93293fd0
feat(smart-scrape): use correct models for multi-entity assembly
2025-04-16 16:39:48 -07:00
Gergő Móricz
751c30f139
feat(extractSmartScrape): better pagination handling
2025-04-16 16:23:12 -07:00
Gergő Móricz
509e6e658c
feat(llmExtract): more logging
2025-04-16 15:40:07 -07:00
Gergő Móricz
4c5120e081
feat(llm-extract): do more logging, even more
2025-04-16 15:10:17 -07:00
Gergő Móricz
b45e3bda13
better logs
2025-04-16 15:08:05 -07:00
Gergő Móricz
ad7e3f62d1
feat(extractSmartScrape): resolve refs in provided schema
2025-04-16 14:32:00 -07:00
Gergő Móricz
5ee2434c9d
more logs
2025-04-16 14:24:09 -07:00
Gergő Móricz
772a3ea751
Bump the SDK
2025-04-16 12:36:43 -07:00
Gergő Móricz
4740254b89
feat(rquests.http): add extract
2025-04-16 12:32:02 -07:00
Gergő Móricz
c713251261
feat(extraction-service): send thisSessionId for single entity
2025-04-16 12:28:35 -07:00
Gergő Móricz
7787a58beb
default timeout
2025-04-16 10:48:03 -07:00
Gergő Móricz
39d10dc734
feat: disable cost tracking
2025-04-16 09:55:32 -07:00
Gergő Móricz
8766bc6f6b
temp: don't... do that
2025-04-16 03:28:33 -07:00
Gergő Móricz
defc80af32
stream session IDs for single URLs
2025-04-16 03:23:18 -07:00
Gergő Móricz
d82f44c93e
feat(extract): log failed extracts
2025-04-16 03:13:02 -07:00
Gergő Móricz
aa20246571
minor cost tracking fix
2025-04-16 02:06:39 -07:00
Nicolas
51967c7c3d
Merge branch 'main' into rafa/sessionIdsExtract
2025-04-16 01:48:13 -07:00
Gergő Móricz
dcef6fbc13
feat(extractSmartScrape): mog it to 100 pages max
2025-04-16 01:13:49 -07:00
Gergő Móricz
129b10e478
fix(llmExtract): cost calculation
2025-04-16 01:04:48 -07:00
rafaelmmiller
f92217e3b6
wip
2025-04-16 00:38:35 -07:00
Gergő Móricz
0d813b628b
feat: correlate smart scrape
2025-04-16 00:25:42 -07:00
Gergő Móricz
edd4c30908
FIX IT
2025-04-16 00:11:27 -07:00
Gergő Móricz
a06910115b
asd
2025-04-16 00:03:07 -07:00
Gergő Móricz
2245650bc3
fix
2025-04-15 23:54:56 -07:00
Gergő Móricz
2193bee133
Improve logging
2025-04-15 23:52:20 -07:00
Gergő Móricz
0935ec210e
feat(smartScrape): better loggin
2025-04-15 23:34:00 -07:00
Gergő Móricz
b6abe4f26b
fix(smartScrape): pass extract id
2025-04-15 23:23:18 -07:00
Gergő Móricz
80b507e64e
correlate with eid
2025-04-15 23:06:13 -07:00
Gergő Móricz
512a2b1cd4
feat(extract): run on original links if reranker is weird
2025-04-15 22:57:20 -07:00
Gergő Móricz
13bd50ad2b
feat(fetch): don't time out (for smart scrape)
2025-04-15 21:57:05 -07:00
Gergő Móricz
0abe60085b
fix
2025-04-15 20:29:01 -07:00
Gergő Móricz
eea1267b50
feat(batchExtract): thingymajig
2025-04-15 20:28:53 -07:00
Gergő Móricz
d119552eee
bump rate limits
2025-04-15 20:20:29 -07:00
Gergő Móricz
5515ca7a52
fix(llm-cost): update
2025-04-15 19:49:14 -07:00
Nicolas
524b9770cd
Update queue-worker.ts
2025-04-15 17:52:20 -07:00
Gergő Móricz
c8a8e96acc
un-gate scrape status, add test
2025-04-15 17:48:54 -07:00
Gergő Móricz
3ccef5fb66
fix(v1): scrape-status with GCS
2025-04-15 17:42:38 -07:00
rafaelmmiller
252a9ccc89
Refactor robustFetch logging to exclude sensitive parameters and improve error handling.
2025-04-15 17:15:01 -07:00
devin-ai-integration[bot]
a840db9ef3
Set default timeout to 120s when proxy is stealth ( #1464 )
...
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mogery@sideguide.dev <mogery@sideguide.dev>
2025-04-15 23:09:11 +02:00
Gergő Móricz
31e24e907c
FIX DAT
2025-04-15 02:13:31 -07:00
Gergő Móricz
0ee9603945
FIX MORE
2025-04-15 02:09:52 -07:00
Gergő Móricz
9400b1423a
fix typing
2025-04-15 02:07:40 -07:00
Gergő Móricz
edb40d75c1
log session ID
2025-04-15 01:46:31 -07:00
Nicolas
6634d236bf
(feat/fire-1) FIRE-1 ( #1462 )
...
* wip
* integrating smart-scrape
* integrate smartscrape into llmExtract
* wip
* smart scrape multiple links
* fixes
* fix
* wip
* it worked!
* wip. there's a bug on the batchExtract TypeError: Converting circular structure to JSON
* wip
* retry model
* retry models
* feat/scrape+json+extract interfaces ready
* vertex -> googleapi
* fix/transformArrayToObject. required params on schema is still a bug
* change model
* o3-mini -> gemini
* Update extractSmartScrape.ts
* sessionId
* sessionId
* Nick: f-0 start
* Update extraction-service-f0.ts
* Update types.ts
* Nick:
* Update queue-worker.ts
* Nick: new interface
* rename analyzeSchemaAndPrompt -> F0
* refactor: rename agent ID to model in types and extract logic
* agent
* id->model
* id->model
* refactor: standardize agent model handling and validation across extraction logic
* livecast agent
* (feat/f1) sdks (#1459 )
* feat: add FIRE-1 agent support to Python and JavaScript SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add FIRE-1 agent support to scrape methods in both SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add prompt and sessionId to AgentOptions interface
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Update index.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* feat(v1): rate limits
* Update types.ts
* Update llmExtract.ts
* add cost tracking
* remove
* Update requests.http
* fix smart scrape cost calc
* log sm cost
* fix counts
* fix
* expose cost tracking
* models fix
* temp: skipLibcheck
* get rid of it
* fix ts
* dont skip lib check
* Update extractSmartScrape.ts
* Update queue-worker.ts
* Update smartScrape.ts
* Update requests.http
* fix(rate-limiter):
* types: fire-1 refine
* bill 150
* fix credits used on crawl
* ban from crawl
* route cost limit warning
* Update generic-ai.ts
* genres
* Update llmExtract.ts
* test server diff
* cletu
---------
Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
Co-authored-by: Ademílson F. Tonato <ademilsonft@outlook.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-04-15 00:19:45 -07:00
Gergő Móricz
e2c4b0e72f
remove double v0 log
2025-04-14 18:12:36 -07:00
Nicolas
0446443bbe
Nick: acuc cache on now
2025-04-14 16:06:38 -07:00
Gergő Móricz
b415e625a0
feat(scrape): get job result from GCS, avoid Redis ( #1461 )
...
* feat(scrape): get job result from GCS, avoid Redis
* call logjob on scrapes
* Fix inverse bool
* fix more
* migrate gracefully
* refactor
* feat(tests/search): test with scrape
2025-04-15 00:07:44 +02:00
Eric Ciarla
0ac86abead
Merge pull request #1460 from aparupganguly/examples/gpt-4.1-crawler
2025-04-14 12:55:49 -07:00
Aparup Ganguly
5dca350a11
Add examples/gpt-4.1-crawler
2025-04-15 00:48:19 +05:30
