dependabot[bot]
40d6750e16
apps/test-suite(deps): bump the prod-deps group
...
Bumps the prod-deps group in /apps/test-suite with 8 updates:
| Package | From | To |
| --- | --- | --- |
| [@anthropic-ai/sdk](https://github.com/anthropics/anthropic-sdk-typescript ) | `0.24.3` | `0.52.0` |
| [@dqbd/tiktoken](https://github.com/dqbd/tiktoken ) | `1.0.15` | `1.0.21` |
| [@supabase/supabase-js](https://github.com/supabase/supabase-js ) | `2.44.2` | `2.49.8` |
| [dotenv](https://github.com/motdotla/dotenv ) | `16.4.5` | `16.5.0` |
| [openai](https://github.com/openai/openai-node ) | `4.52.2` | `4.104.0` |
| [playwright](https://github.com/microsoft/playwright ) | `1.45.0` | `1.52.0` |
| [supertest](https://github.com/ladjs/supertest ) | `7.0.0` | `7.1.1` |
| [ts-jest](https://github.com/kulshekhar/ts-jest ) | `29.1.5` | `29.3.4` |
Updates `@anthropic-ai/sdk` from 0.24.3 to 0.52.0
- [Release notes](https://github.com/anthropics/anthropic-sdk-typescript/releases )
- [Changelog](https://github.com/anthropics/anthropic-sdk-typescript/blob/main/CHANGELOG.md )
- [Commits](https://github.com/anthropics/anthropic-sdk-typescript/compare/sdk-v0.24.3...sdk-v0.52.0 )
Updates `@dqbd/tiktoken` from 1.0.15 to 1.0.21
- [Release notes](https://github.com/dqbd/tiktoken/releases )
- [Changelog](https://github.com/dqbd/tiktoken/blob/main/CHANGELOG.md )
- [Commits](https://github.com/dqbd/tiktoken/compare/@dqbd/tiktoken@1.0.15...@dqbd/tiktoken@1.0.21 )
Updates `@supabase/supabase-js` from 2.44.2 to 2.49.8
- [Release notes](https://github.com/supabase/supabase-js/releases )
- [Changelog](https://github.com/supabase/supabase-js/blob/master/RELEASE.md )
- [Commits](https://github.com/supabase/supabase-js/compare/v2.44.2...v2.49.8 )
Updates `dotenv` from 16.4.5 to 16.5.0
- [Changelog](https://github.com/motdotla/dotenv/blob/master/CHANGELOG.md )
- [Commits](https://github.com/motdotla/dotenv/compare/v16.4.5...v16.5.0 )
Updates `openai` from 4.52.2 to 4.104.0
- [Release notes](https://github.com/openai/openai-node/releases )
- [Changelog](https://github.com/openai/openai-node/blob/master/CHANGELOG.md )
- [Commits](https://github.com/openai/openai-node/compare/v4.52.2...v4.104.0 )
Updates `playwright` from 1.45.0 to 1.52.0
- [Release notes](https://github.com/microsoft/playwright/releases )
- [Commits](https://github.com/microsoft/playwright/compare/v1.45.0...v1.52.0 )
Updates `supertest` from 7.0.0 to 7.1.1
- [Release notes](https://github.com/ladjs/supertest/releases )
- [Commits](https://github.com/ladjs/supertest/compare/v7.0.0...v7.1.1 )
Updates `ts-jest` from 29.1.5 to 29.3.4
- [Release notes](https://github.com/kulshekhar/ts-jest/releases )
- [Changelog](https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md )
- [Commits](https://github.com/kulshekhar/ts-jest/compare/v29.1.5...v29.3.4 )
---
updated-dependencies:
- dependency-name: "@anthropic-ai/sdk"
dependency-version: 0.52.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: prod-deps
- dependency-name: "@dqbd/tiktoken"
dependency-version: 1.0.21
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: prod-deps
- dependency-name: "@supabase/supabase-js"
dependency-version: 2.49.8
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: prod-deps
- dependency-name: dotenv
dependency-version: 16.5.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: prod-deps
- dependency-name: openai
dependency-version: 4.104.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: prod-deps
- dependency-name: playwright
dependency-version: 1.52.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: prod-deps
- dependency-name: supertest
dependency-version: 7.1.1
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: prod-deps
- dependency-name: ts-jest
dependency-version: 29.3.4
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: prod-deps
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-06-02 07:00:00 +00:00
Nicolas
9297afd1ff
Nick: search
2025-05-29 17:00:13 -03:00
Gergő Móricz
a8e0482718
feat(search): bill for PDFs properly
2025-05-29 20:59:15 +02:00
Gergő Móricz
a2f41fb650
feat(api/server): wait 60s for GCE load balancer drain timeout
...
To minimize 502s.
2025-05-29 20:08:52 +02:00
Gergő Móricz
3ea221b093
fix(api/queue): tighten expiries on indexQueue jobs
2025-05-29 16:36:55 +02:00
Gergő Móricz
c9dd0e609a
fix(api/queue): tighten expiries on billingQueue jobs
2025-05-29 16:26:52 +02:00
Gergő Móricz
93655b5c0b
feat(scrapeURL/pdf): bill n credits per page (FIR-1934) ( #1553 )
...
* feat(scrapeURL/pdf): bill n credits per page
* Update scrape.ts
* Update queue-worker.ts
* separate billing logi
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-29 16:01:08 +02:00
Gergő Móricz
38c96b524f
feat(scrapeURL): handle contentType JSON better in markdown conversion ( #1604 )
2025-05-29 15:26:07 +02:00
Gergő Móricz
7e73b01599
fix(queue-worker): call webhook after job is in DB
2025-05-29 14:40:47 +02:00
Gergő Móricz
706d378a89
feat(api/v1/scrape-status): log supa lookup errors
2025-05-29 13:02:54 +02:00
Gergő Móricz
3557c90210
feat(js-sdk): auto mode proxy (FIR-2145) ( #1602 )
...
* feat(js-sdk): auto mode proxy
* Nick: py sdk
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-28 14:31:48 -03:00
Gergő Móricz
a5efff07f9
feat(apps/api): add support for a separate, non-eviction Redis ( #1600 )
...
* feat(apps/api): add support for a separate, non-eviction Redis
* fix: misimport
2025-05-28 09:58:04 +02:00
Nicolas
756b452a01
Update batch_billing.ts
2025-05-27 19:05:00 -03:00
Nicolas
299e3e29e0
Update batch_billing.ts
2025-05-27 18:44:24 -03:00
Gergő Móricz
a36c6a4f40
feat(scrapeURL): add unnormalizedSourceURL for url matching DX (FIR-2137) ( #1601 )
...
* feat(scrapeURL): add unnormalizedSourceURL for url matching DX
* fix(tests): fixc
2025-05-27 21:33:44 +02:00
Gergő Móricz
474e5a0543
fix(crawler): always set expiry on sitemap links in redis
2025-05-27 15:39:31 +02:00
Gergő Móricz
c3738063cf
less logs even more
2025-05-25 15:50:20 +02:00
Gergő Móricz
492d97e889
reduce logging
2025-05-24 00:09:13 +02:00
Gergő Móricz
a3145ccacc
fix(extract-status): be able to get extract status even after TTL lapses ( #1599 )
2025-05-23 22:33:09 +02:00
Gergő Móricz
8389a1a78d
fix(html-transformer): bad outName for og:locale:alternate (FIR-2101) ( #1597 )
...
* fix(html-transformer): bad outName for og:locale:alternate
* oops
2025-05-23 17:10:09 +02:00
Gergő Móricz
3ec17e2d1a
fix(v1): avoid overwriting rateLimiterMode with FIRE-1 rate limiter ( #1593 )
2025-05-23 11:50:59 -03:00
Gergő Móricz
3df687e4db
feat(queue-worker/afterJobDone): improved ccq insert logic ( #1595 )
2025-05-23 11:50:14 -03:00
Gergő Móricz
a7894a2714
fix(scrapeURL/pdf): even better timeout detection
2025-05-23 16:29:28 +02:00
Gergő Móricz
8571b5a99d
Revert "feat(queue-worker/afterJobDone): improved ccq insert logic"
...
This reverts commit 97c635676d228ed1342cdd1468cb2a1aef4fcfc9.
2025-05-23 15:42:15 +02:00
Gergő Móricz
97c635676d
feat(queue-worker/afterJobDone): improved ccq insert logic
2025-05-23 15:41:57 +02:00
Gergő Móricz
f41af8241e
fix(scrapeURL/pdf): better timeout error
2025-05-23 13:59:53 +02:00
Gergő Móricz
bfe731309c
fix(scrapeURL/pdf/mu): remove log
2025-05-23 13:47:34 +02:00
Gergő Móricz
b03670a8b7
feat: parse PDFs on fc side and reject if too long for timeout (FIR-2083) ( #1592 )
...
* feat: pdf-parser, implementation in scrapeURL
* use pdf-parser for page count instead of mu
* fix(pdf-parser): bindings
* feat(scrapeURL/pdf): adjust MILLISECONDS_PER_PAGE
* implement post-runsync polling and fix
* fix(Dockerfile): copy in the pdf-parser source code
* fix(scrapeURL/pdf): better error for timeout below 0
2025-05-23 13:45:53 +02:00
Gergő Móricz
321fff1695
ok what
2025-05-23 11:41:34 +02:00
Gergő Móricz
00cc733972
more logs
2025-05-23 11:29:34 +02:00
Gergő Móricz
bb67b9812b
check if enum is being overwritten somehow
2025-05-23 11:27:49 +02:00
Gergő Móricz
d4e7bde03d
add stack
2025-05-23 10:18:30 +02:00
Gergő Móricz
6776292cc2
more log
2025-05-23 09:57:15 +02:00
Gergő Móricz
2e863da334
feat(api/v1/authMiddleware): add log to debug extract agent preview mode
2025-05-23 09:35:29 +02:00
Gergő Móricz
3e736f1e0d
feat(concurrency-log): add cclog endpoint (FIR-2067) ( #1589 )
...
* feat(concurrency-log): add cclog endpoint
* fix(api/routes/admin): misimport
* more misimports
2025-05-22 18:13:35 -03:00
Gergő Móricz
fd74299134
feat(scrapeURL, logJob): log pdf page count to db (FIR-2068) ( #1587 )
...
* feat(scrapeURL, logJob): log pdf page count to db
* devin stop the test littering pls
2025-05-22 17:26:01 -03:00
Gergő Móricz
cc2c968425
fix(robustFetch): selective logging ( #1588 )
2025-05-22 17:23:49 -03:00
Gergő Móricz
749d89a551
feat(api/v1/extract): ignoreInvalidURLs ( #1585 )
2025-05-21 21:46:55 -03:00
Gergő Móricz
6478754f1b
feat(api/extract): show extract as origin for scrapes originating from it ( #1584 )
2025-05-21 21:45:44 -03:00
Gergő Móricz
8522103228
feat(api/v1/extract): log requests
2025-05-21 23:25:32 +02:00
Gergő Móricz
938ef1cfb2
feat(api/v1/map): log requests
2025-05-21 23:00:43 +02:00
Gergő Móricz
a3aee9be13
fix(queue-worker): finish crawl if all addable URLs were already locked ( #1582 )
2025-05-21 17:40:31 -03:00
Gergő Móricz
9bb97388e2
feat(search): ignoreBlockedURLs (FIR-1954) ( #1580 )
...
* feat(search): ignoreBlockedURLs
* change name
2025-05-21 13:02:58 -03:00
Gergő Móricz
3d4692f47c
Update SELF_HOST.md
2025-05-21 17:50:26 +02:00
devin-ai-integration[bot]
c7bb9e77ab
docs: add MAX_RAM and MAX_CPU environment variables documentation ( #1581 )
...
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mogery@sideguide.dev <mogery@sideguide.dev>
2025-05-21 17:49:16 +02:00
Gergő Móricz
b8ea4021c1
fix(services/webhook): greatly improved logging
2025-05-21 16:02:01 +02:00
devin-ai-integration[bot]
9949403b59
FIR-2006: Fix maxUrls and timeLimit parameters in Deep Research API ( #1569 )
...
* FIR-2006: Fix maxUrls and timeLimit enforcement in Deep Research API
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
* FIR-2006: Add tests for maxUrls and timeLimit enforcement
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
* FIR-2006: Replace mocked tests with end-to-end tests for deep research
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
* Delete apps/api/src/__tests__/snips/deep-research-service.test.ts
* Delete apps/api/src/__tests__/snips/lib.ts
* Revert "Delete apps/api/src/__tests__/snips/lib.ts"
This reverts commit a2af9baff89d64adc1930ea5b37b4f07f0735a67.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-20 18:39:56 -03:00
Gergő Móricz
513f469b0f
feat(python-sdk/CrawlWatcher): remove max payload size from WebSocket (FIR-2038) ( #1577 )
...
* feat(python-sdk/CrawlWatcher): remove max payload size from WebSocket
* Update __init__.py
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-20 16:59:08 -03:00
Rafael Miller
6d75161302
Fix sdk/undefined response handle error ( #1578 )
...
* fix(sdk): update version to 1.25.1 and enhance error handling in FirecrawlApp (#1576 )
* package version bump
2025-05-20 16:55:07 -03:00
devin-ai-integration[bot]
a5a915d639
Fix: Concatenate metadata arrays into strings with exceptions ( #1574 )
...
* Fix: Concatenate metadata arrays into strings except for ogLocaleAlternate
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
* Fix: Only concatenate description field, preserve other metadata arrays
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
* Fix: Only concatenate description field, keep other metadata fields in original format
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-20 12:40:53 -03:00
