Gergő Móricz
f8183af88e
sws
2025-06-03 21:02:10 +02:00
Gergő Móricz
7ba8dd25d6
no log
2025-06-03 20:50:29 +02:00
Gergő Móricz
11f27fd7f8
stupid
2025-06-03 20:46:21 +02:00
Gergő Móricz
1a6e1134c4
fix(api/tests/scrape/status): propagation time
2025-06-03 19:11:25 +02:00
Gergő Móricz
359711656a
weird thing
2025-06-03 19:04:51 +02:00
Gergő Móricz
1dc6a443cb
cache issues with billing test
2025-06-03 18:46:55 +02:00
Gergő Móricz
f03676cf7a
more timeout
2025-06-03 17:49:20 +02:00
Gergő Móricz
d3b6b0da34
feat: re-enable billing tests
2025-06-03 15:41:40 +02:00
Gergő Móricz
1307648781
yeet ad blocking tests until further notice
2025-06-03 15:40:34 +02:00
Nicolas
e108ff3525
Update search.ts
2025-06-02 23:46:55 -03:00
Nicolas
9347de6a41
Update scrape.ts
2025-06-02 23:15:59 -03:00
Nicolas
86a9d3525b
Update queue-jobs.ts
2025-06-02 23:09:09 -03:00
Nicolas
cbc47305cc
Update search.ts
2025-06-02 23:09:02 -03:00
Nicolas
ce425d966f
Merge branch 'nsc/bypass-billing-internal'
2025-06-02 22:37:56 -03:00
Nicolas
8c661f5329
Update scrape.ts
2025-06-02 22:37:49 -03:00
Nicolas
dc8cc99b1d
Nick: bypass billing ( #1622 )
2025-06-02 21:57:28 -03:00
Nicolas
8967b31465
Nick: bypass billing
2025-06-02 21:51:46 -03:00
Nicolas
bf919ceb82
Nick: __searchPreviewToken
2025-06-02 21:16:34 -03:00
Nicolas
ef789ce8d7
Nick: __experimental
2025-06-02 19:58:56 -03:00
Gergő Móricz
72be73473f
feat(api/scrape): credits_billed column + handle billing for /scrape calls on worker side with stricter timeout enforcement (FIR-2162) ( #1607 )
...
* feat(api/scrape): stricten timeout and handle billing and logging on queue-worker
* fix: abortsignal pre-check
* fix: proper level
* add comment to clarify is_scrape
* reenable billing tests
* Revert "reenable billing tests"
This reverts commit 98236fdfa03dde8cecdd6b763fcf86810e468a28.
* oof
* fix searxng logging
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-06-02 17:56:27 -03:00
Gergő Móricz
4167ec53eb
fix(scrapeURL): only allow disabling the adblock on playwright (FIR-2200) ( #1616 )
...
* fix(scrapeURL): only allow disabling the adblock on playwright
* feat(api/tests/scrape): re-enable ad blocking tests
2025-06-02 22:48:16 +02:00
Gergő Móricz
7a8be13220
remove indexes that are no longer used
2025-06-02 22:09:55 +02:00
Gergő Móricz
98ceda9bd5
feat(search): ignore concurrency limit for search (FIR-2187) ( #1617 )
...
* feat(search): ignore concurrency limit for search (temp)
* feat(search): only for low tier users for good DX
2025-06-02 17:07:44 -03:00
Gergő Móricz
1396451d31
bump rust version pt.2
2025-06-02 18:10:14 +02:00
Gergő Móricz
07fb651a91
bump rust version
2025-06-02 18:09:12 +02:00
Supasin Liulak
6a76ccfacb
webhook param for crawl ( #1609 )
2025-06-02 18:08:32 +02:00
Nicolas
9297afd1ff
Nick: search
2025-05-29 17:00:13 -03:00
Gergő Móricz
a8e0482718
feat(search): bill for PDFs properly
2025-05-29 20:59:15 +02:00
Gergő Móricz
a2f41fb650
feat(api/server): wait 60s for GCE load balancer drain timeout
...
To minimize 502s.
2025-05-29 20:08:52 +02:00
Gergő Móricz
3ea221b093
fix(api/queue): tighten expiries on indexQueue jobs
2025-05-29 16:36:55 +02:00
Gergő Móricz
c9dd0e609a
fix(api/queue): tighten expiries on billingQueue jobs
2025-05-29 16:26:52 +02:00
Gergő Móricz
93655b5c0b
feat(scrapeURL/pdf): bill n credits per page (FIR-1934) ( #1553 )
...
* feat(scrapeURL/pdf): bill n credits per page
* Update scrape.ts
* Update queue-worker.ts
* separate billing logi
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-29 16:01:08 +02:00
Gergő Móricz
38c96b524f
feat(scrapeURL): handle contentType JSON better in markdown conversion ( #1604 )
2025-05-29 15:26:07 +02:00
Gergő Móricz
7e73b01599
fix(queue-worker): call webhook after job is in DB
2025-05-29 14:40:47 +02:00
Gergő Móricz
706d378a89
feat(api/v1/scrape-status): log supa lookup errors
2025-05-29 13:02:54 +02:00
Gergő Móricz
3557c90210
feat(js-sdk): auto mode proxy (FIR-2145) ( #1602 )
...
* feat(js-sdk): auto mode proxy
* Nick: py sdk
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-28 14:31:48 -03:00
Gergő Móricz
a5efff07f9
feat(apps/api): add support for a separate, non-eviction Redis ( #1600 )
...
* feat(apps/api): add support for a separate, non-eviction Redis
* fix: misimport
2025-05-28 09:58:04 +02:00
Nicolas
756b452a01
Update batch_billing.ts
2025-05-27 19:05:00 -03:00
Nicolas
299e3e29e0
Update batch_billing.ts
2025-05-27 18:44:24 -03:00
Gergő Móricz
a36c6a4f40
feat(scrapeURL): add unnormalizedSourceURL for url matching DX (FIR-2137) ( #1601 )
...
* feat(scrapeURL): add unnormalizedSourceURL for url matching DX
* fix(tests): fixc
2025-05-27 21:33:44 +02:00
Gergő Móricz
474e5a0543
fix(crawler): always set expiry on sitemap links in redis
2025-05-27 15:39:31 +02:00
Gergő Móricz
c3738063cf
less logs even more
2025-05-25 15:50:20 +02:00
Gergő Móricz
492d97e889
reduce logging
2025-05-24 00:09:13 +02:00
Gergő Móricz
a3145ccacc
fix(extract-status): be able to get extract status even after TTL lapses ( #1599 )
2025-05-23 22:33:09 +02:00
Gergő Móricz
8389a1a78d
fix(html-transformer): bad outName for og:locale:alternate (FIR-2101) ( #1597 )
...
* fix(html-transformer): bad outName for og:locale:alternate
* oops
2025-05-23 17:10:09 +02:00
Gergő Móricz
3ec17e2d1a
fix(v1): avoid overwriting rateLimiterMode with FIRE-1 rate limiter ( #1593 )
2025-05-23 11:50:59 -03:00
Gergő Móricz
3df687e4db
feat(queue-worker/afterJobDone): improved ccq insert logic ( #1595 )
2025-05-23 11:50:14 -03:00
Gergő Móricz
a7894a2714
fix(scrapeURL/pdf): even better timeout detection
2025-05-23 16:29:28 +02:00
Gergő Móricz
8571b5a99d
Revert "feat(queue-worker/afterJobDone): improved ccq insert logic"
...
This reverts commit 97c635676d228ed1342cdd1468cb2a1aef4fcfc9.
2025-05-23 15:42:15 +02:00
Gergő Móricz
97c635676d
feat(queue-worker/afterJobDone): improved ccq insert logic
2025-05-23 15:41:57 +02:00
