2740 Commits

Author SHA1 Message Date
Gergő Móricz
1396451d31 bump rust version pt.2 2025-06-02 18:10:14 +02:00
Gergő Móricz
07fb651a91 bump rust version 2025-06-02 18:09:12 +02:00
Supasin Liulak
6a76ccfacb
webhook param for crawl (#1609) 2025-06-02 18:08:32 +02:00
Nicolas
9297afd1ff Nick: search 2025-05-29 17:00:13 -03:00
Gergő Móricz
a8e0482718 feat(search): bill for PDFs properly 2025-05-29 20:59:15 +02:00
Gergő Móricz
a2f41fb650 feat(api/server): wait 60s for GCE load balancer drain timeout
To minimize 502s.
2025-05-29 20:08:52 +02:00
Gergő Móricz
3ea221b093 fix(api/queue): tighten expiries on indexQueue jobs 2025-05-29 16:36:55 +02:00
Gergő Móricz
c9dd0e609a fix(api/queue): tighten expiries on billingQueue jobs 2025-05-29 16:26:52 +02:00
Gergő Móricz
93655b5c0b
feat(scrapeURL/pdf): bill n credits per page (FIR-1934) (#1553)
* feat(scrapeURL/pdf): bill n credits per page

* Update scrape.ts

* Update queue-worker.ts

* separate billing logi

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-29 16:01:08 +02:00
Gergő Móricz
38c96b524f
feat(scrapeURL): handle contentType JSON better in markdown conversion (#1604) 2025-05-29 15:26:07 +02:00
Gergő Móricz
7e73b01599 fix(queue-worker): call webhook after job is in DB 2025-05-29 14:40:47 +02:00
Gergő Móricz
706d378a89 feat(api/v1/scrape-status): log supa lookup errors 2025-05-29 13:02:54 +02:00
Gergő Móricz
3557c90210
feat(js-sdk): auto mode proxy (FIR-2145) (#1602)
* feat(js-sdk): auto mode proxy

* Nick: py sdk

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-28 14:31:48 -03:00
Gergő Móricz
a5efff07f9
feat(apps/api): add support for a separate, non-eviction Redis (#1600)
* feat(apps/api): add support for a separate, non-eviction Redis

* fix: misimport
2025-05-28 09:58:04 +02:00
Nicolas
756b452a01 Update batch_billing.ts 2025-05-27 19:05:00 -03:00
Nicolas
299e3e29e0 Update batch_billing.ts 2025-05-27 18:44:24 -03:00
Gergő Móricz
a36c6a4f40
feat(scrapeURL): add unnormalizedSourceURL for url matching DX (FIR-2137) (#1601)
* feat(scrapeURL): add unnormalizedSourceURL for url matching DX

* fix(tests): fixc
2025-05-27 21:33:44 +02:00
Gergő Móricz
474e5a0543 fix(crawler): always set expiry on sitemap links in redis 2025-05-27 15:39:31 +02:00
Gergő Móricz
c3738063cf less logs even more 2025-05-25 15:50:20 +02:00
Gergő Móricz
492d97e889 reduce logging 2025-05-24 00:09:13 +02:00
Gergő Móricz
a3145ccacc
fix(extract-status): be able to get extract status even after TTL lapses (#1599) 2025-05-23 22:33:09 +02:00
Gergő Móricz
8389a1a78d
fix(html-transformer): bad outName for og:locale:alternate (FIR-2101) (#1597)
* fix(html-transformer): bad outName for og:locale:alternate

* oops
2025-05-23 17:10:09 +02:00
Gergő Móricz
3ec17e2d1a
fix(v1): avoid overwriting rateLimiterMode with FIRE-1 rate limiter (#1593) 2025-05-23 11:50:59 -03:00
Gergő Móricz
3df687e4db
feat(queue-worker/afterJobDone): improved ccq insert logic (#1595) 2025-05-23 11:50:14 -03:00
Gergő Móricz
a7894a2714 fix(scrapeURL/pdf): even better timeout detection 2025-05-23 16:29:28 +02:00
Gergő Móricz
8571b5a99d Revert "feat(queue-worker/afterJobDone): improved ccq insert logic"
This reverts commit 97c635676d228ed1342cdd1468cb2a1aef4fcfc9.
2025-05-23 15:42:15 +02:00
Gergő Móricz
97c635676d feat(queue-worker/afterJobDone): improved ccq insert logic 2025-05-23 15:41:57 +02:00
Gergő Móricz
f41af8241e fix(scrapeURL/pdf): better timeout error 2025-05-23 13:59:53 +02:00
Gergő Móricz
bfe731309c fix(scrapeURL/pdf/mu): remove log 2025-05-23 13:47:34 +02:00
Gergő Móricz
b03670a8b7
feat: parse PDFs on fc side and reject if too long for timeout (FIR-2083) (#1592)
* feat: pdf-parser, implementation in scrapeURL

* use pdf-parser for page count instead of mu

* fix(pdf-parser): bindings

* feat(scrapeURL/pdf): adjust MILLISECONDS_PER_PAGE

* implement post-runsync polling and fix

* fix(Dockerfile): copy in the pdf-parser source code

* fix(scrapeURL/pdf): better error for timeout below 0
2025-05-23 13:45:53 +02:00
Gergő Móricz
321fff1695 ok what 2025-05-23 11:41:34 +02:00
Gergő Móricz
00cc733972 more logs 2025-05-23 11:29:34 +02:00
Gergő Móricz
bb67b9812b check if enum is being overwritten somehow 2025-05-23 11:27:49 +02:00
Gergő Móricz
d4e7bde03d add stack 2025-05-23 10:18:30 +02:00
Gergő Móricz
6776292cc2 more log 2025-05-23 09:57:15 +02:00
Gergő Móricz
2e863da334 feat(api/v1/authMiddleware): add log to debug extract agent preview mode 2025-05-23 09:35:29 +02:00
Gergő Móricz
3e736f1e0d
feat(concurrency-log): add cclog endpoint (FIR-2067) (#1589)
* feat(concurrency-log): add cclog endpoint

* fix(api/routes/admin): misimport

* more misimports
2025-05-22 18:13:35 -03:00
Gergő Móricz
fd74299134
feat(scrapeURL, logJob): log pdf page count to db (FIR-2068) (#1587)
* feat(scrapeURL, logJob): log pdf page count to db

* devin stop the test littering pls
2025-05-22 17:26:01 -03:00
Gergő Móricz
cc2c968425
fix(robustFetch): selective logging (#1588) 2025-05-22 17:23:49 -03:00
Gergő Móricz
749d89a551
feat(api/v1/extract): ignoreInvalidURLs (#1585) 2025-05-21 21:46:55 -03:00
Gergő Móricz
6478754f1b
feat(api/extract): show extract as origin for scrapes originating from it (#1584) 2025-05-21 21:45:44 -03:00
Gergő Móricz
8522103228 feat(api/v1/extract): log requests 2025-05-21 23:25:32 +02:00
Gergő Móricz
938ef1cfb2 feat(api/v1/map): log requests 2025-05-21 23:00:43 +02:00
Gergő Móricz
a3aee9be13
fix(queue-worker): finish crawl if all addable URLs were already locked (#1582) 2025-05-21 17:40:31 -03:00
Gergő Móricz
9bb97388e2
feat(search): ignoreBlockedURLs (FIR-1954) (#1580)
* feat(search): ignoreBlockedURLs

* change name
2025-05-21 13:02:58 -03:00
Gergő Móricz
b8ea4021c1 fix(services/webhook): greatly improved logging 2025-05-21 16:02:01 +02:00
devin-ai-integration[bot]
9949403b59
FIR-2006: Fix maxUrls and timeLimit parameters in Deep Research API (#1569)
* FIR-2006: Fix maxUrls and timeLimit enforcement in Deep Research API

Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>

* FIR-2006: Add tests for maxUrls and timeLimit enforcement

Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>

* FIR-2006: Replace mocked tests with end-to-end tests for deep research

Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>

* Delete apps/api/src/__tests__/snips/deep-research-service.test.ts

* Delete apps/api/src/__tests__/snips/lib.ts

* Revert "Delete apps/api/src/__tests__/snips/lib.ts"

This reverts commit a2af9baff89d64adc1930ea5b37b4f07f0735a67.

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-20 18:39:56 -03:00
Gergő Móricz
513f469b0f
feat(python-sdk/CrawlWatcher): remove max payload size from WebSocket (FIR-2038) (#1577)
* feat(python-sdk/CrawlWatcher): remove max payload size from WebSocket

* Update __init__.py

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-20 16:59:08 -03:00
Rafael Miller
6d75161302
Fix sdk/undefined response handle error (#1578)
* fix(sdk): update version to 1.25.1 and enhance error handling in FirecrawlApp (#1576)

* package version bump
2025-05-20 16:55:07 -03:00
devin-ai-integration[bot]
a5a915d639
Fix: Concatenate metadata arrays into strings with exceptions (#1574)
* Fix: Concatenate metadata arrays into strings except for ogLocaleAlternate

Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>

* Fix: Only concatenate description field, preserve other metadata arrays

Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>

* Fix: Only concatenate description field, keep other metadata fields in original format

Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-20 12:40:53 -03:00