Gergő Móricz
|
3ec17e2d1a
|
fix(v1): avoid overwriting rateLimiterMode with FIRE-1 rate limiter (#1593)
|
2025-05-23 11:50:59 -03:00 |
|
Gergő Móricz
|
321fff1695
|
ok what
|
2025-05-23 11:41:34 +02:00 |
|
Gergő Móricz
|
00cc733972
|
more logs
|
2025-05-23 11:29:34 +02:00 |
|
Gergő Móricz
|
bb67b9812b
|
check if enum is being overwritten somehow
|
2025-05-23 11:27:49 +02:00 |
|
Gergő Móricz
|
d4e7bde03d
|
add stack
|
2025-05-23 10:18:30 +02:00 |
|
Gergő Móricz
|
6776292cc2
|
more log
|
2025-05-23 09:57:15 +02:00 |
|
Gergő Móricz
|
2e863da334
|
feat(api/v1/authMiddleware): add log to debug extract agent preview mode
|
2025-05-23 09:35:29 +02:00 |
|
Gergő Móricz
|
3e736f1e0d
|
feat(concurrency-log): add cclog endpoint (FIR-2067) (#1589)
* feat(concurrency-log): add cclog endpoint
* fix(api/routes/admin): misimport
* more misimports
|
2025-05-22 18:13:35 -03:00 |
|
Gergő Móricz
|
fa581995e6
|
feat(acuc): propagate team flags (FIR-1879) (#1522)
* feat(acuc): propagate team flags
* feat(flags): further functionality
|
2025-05-08 20:23:35 +02:00 |
|
Nicolas
|
8b88c26a47
|
Update v1.ts (#1509)
|
2025-04-30 13:02:07 -03:00 |
|
Nicolas
|
c69acdffd8
|
Update v1.ts
|
2025-04-18 01:57:03 -07:00 |
|
Rafael Miller
|
29b36c5f9a
|
[python-SDK] improvs/async (#1337)
* improv/types-and-comments-descs
* async
* removed v0 in example
* tomkosms review
* refator: dry request and error handling
* fixed websocket params
* added origin to requests
* Update firecrawl.py
* Update firecrawl.py
* added agent options types
* Update firecrawl.py
* generic
* Update firecrawl.py
* scrape params commentary
* Update firecrawl.py
* Update firecrawl.py
* Update firecrawl.py
* Update firecrawl.py
* async scrape
* Update firecrawl.py
* Nick: new examples
* Nick: python sdk 2.0
* async functions
* Nick:
* Nick:
---------
Co-authored-by: Ademílson F. Tonato <ademilsonft@outlook.com>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
|
2025-04-18 01:32:55 -07:00 |
|
Gergő Móricz
|
f844b329f1
|
remove agent preview rate limiter
|
2025-04-17 11:04:45 -07:00 |
|
Nicolas
|
6634d236bf
|
(feat/fire-1) FIRE-1 (#1462)
* wip
* integrating smart-scrape
* integrate smartscrape into llmExtract
* wip
* smart scrape multiple links
* fixes
* fix
* wip
* it worked!
* wip. there's a bug on the batchExtract TypeError: Converting circular structure to JSON
* wip
* retry model
* retry models
* feat/scrape+json+extract interfaces ready
* vertex -> googleapi
* fix/transformArrayToObject. required params on schema is still a bug
* change model
* o3-mini -> gemini
* Update extractSmartScrape.ts
* sessionId
* sessionId
* Nick: f-0 start
* Update extraction-service-f0.ts
* Update types.ts
* Nick:
* Update queue-worker.ts
* Nick: new interface
* rename analyzeSchemaAndPrompt -> F0
* refactor: rename agent ID to model in types and extract logic
* agent
* id->model
* id->model
* refactor: standardize agent model handling and validation across extraction logic
* livecast agent
* (feat/f1) sdks (#1459)
* feat: add FIRE-1 agent support to Python and JavaScript SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add FIRE-1 agent support to scrape methods in both SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add prompt and sessionId to AgentOptions interface
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Update index.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* feat(v1): rate limits
* Update types.ts
* Update llmExtract.ts
* add cost tracking
* remove
* Update requests.http
* fix smart scrape cost calc
* log sm cost
* fix counts
* fix
* expose cost tracking
* models fix
* temp: skipLibcheck
* get rid of it
* fix ts
* dont skip lib check
* Update extractSmartScrape.ts
* Update queue-worker.ts
* Update smartScrape.ts
* Update requests.http
* fix(rate-limiter):
* types: fire-1 refine
* bill 150
* fix credits used on crawl
* ban from crawl
* route cost limit warning
* Update generic-ai.ts
* genres
* Update llmExtract.ts
* test server diff
* cletu
---------
Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
Co-authored-by: Ademílson F. Tonato <ademilsonft@outlook.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
|
2025-04-15 00:19:45 -07:00 |
|
Gergő Móricz
|
6a10f0689d
|
ACUC: Dynamic Limits (FIR-1641) (#1434)
* extend acuc definition
* kill plan
* stuff
* stupid tests
* feat: better acuc
* feat(acuc): mock ACUC when not using db auth
|
2025-04-10 18:49:23 +02:00 |
|
Gergő Móricz
|
670ca84ae9
|
fix(v1/checkCredits): snap crawl limit to remaining credits if over without erroring out (#1350)
|
2025-03-16 19:57:08 +01:00 |
|
Gergő Móricz
|
9ad947884d
|
feat(tests/snips): add billing tests + misc billing fixes (FIR-1280) (#1283)
* feat(tests/snips): add billing tests + misc billing fixes
* add testing key
* asd
|
2025-03-02 16:51:42 -03:00 |
|
Eric Ciarla
|
d984b50400
|
Add llmstxt generator endpoint (#1201)
* Nick:
* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"
This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.
* Update deep-research-service.ts
* Nick:
* init
* part 2
* Update generate-llmstxt-service.ts
* Fix queue
* Update queue-worker.ts
* Almost there
* Final touches
* Update requests.http
* final touches
* Update requests.http
* Improve logging
* Change endpoint to /llmstxt
* Update queue-worker.ts
* Update generate-llmstxt-service.ts
* Nick: cache
* Update index.ts
* Update firecrawl.py
* Update package.json
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
|
2025-02-19 14:42:33 -03:00 |
|
Nicolas
|
5c47e97db2
|
(feat/deep-research) Alpha implementation of deep research (#1202)
* Nick:
* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"
This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.
* Update deep-research-service.ts
* Nick:
* Nick:
* Nick:
* Nick:
* Nick:
* Nick:
* Update deep-research-service.ts
* Nick:
* Update deep-research-service.ts
* Apply suggestions from code review
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
|
2025-02-19 12:44:21 -03:00 |
|
Gergő Móricz
|
d9c99e5892
|
fix(v1/batch/scrape): use scrape rate limit (#1182)
|
2025-02-13 17:32:58 -03:00 |
|
Gergő Móricz
|
1d9a0b9653
|
feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens (#1178)
* feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens
* feat(v1/checkCredits): better instructions for upgrade
|
2025-02-13 12:14:13 -03:00 |
|
Móricz Gergő
|
9c40e0cc8d
|
fix(v1): test override for team
|
2025-01-25 08:27:59 +01:00 |
|
Móricz Gergő
|
afea2eeaac
|
feat(v1): add insufficient credits stuff
|
2025-01-25 08:16:19 +01:00 |
|
Nicolas
|
d786949639
|
Reapply "Merge pull request #1068 from mendableai/nsc/llm-usage-extract"
This reverts commit 8b17af40018688c34f95727ceaec289b02ab2023.
|
2025-01-19 22:04:12 -03:00 |
|
Nicolas
|
8b17af4001
|
Revert "Merge pull request #1068 from mendableai/nsc/llm-usage-extract"
This reverts commit 406f28c04aff2ba3ae65f483627da13f02943cc3, reversing
changes made to 34ad9ec25d73f37deb1e3adec2315a121ec52f0e.
|
2025-01-19 22:00:28 -03:00 |
|
Nicolas
|
382476cb36
|
Nick: auth extract
|
2025-01-18 23:16:25 -03:00 |
|
Gergő Móricz
|
e6531278f6
|
feat(v1): crawl/batch scrape errors route
|
2025-01-17 17:12:04 +01:00 |
|
Nicolas
|
f4d10c5031
|
Nick: formatting fixes
|
2025-01-10 18:35:10 -03:00 |
|
Gergő Móricz
|
29c1f126ab
|
feat(scrape-status): adapt
|
2025-01-09 19:14:00 +01:00 |
|
Nicolas
|
51cb4b1615
|
Nick: temp rl for /extract
|
2025-01-08 15:24:38 -03:00 |
|
Nicolas
|
27457ed5db
|
Nick: init
|
2025-01-03 20:44:27 -03:00 |
|
Nicolas
|
d2742bec4d
|
Nick: v1 search
|
2025-01-02 19:31:03 -03:00 |
|
Nicolas
|
d1f3e26f9e
|
Nick: blocklist string
|
2024-12-20 18:09:49 -03:00 |
|
Nicolas
|
6222152249
|
Nick: credit usage endpoint
|
2024-12-20 15:44:17 -03:00 |
|
Nicolas
|
664ba69f08
|
Nick: f-eng monitoring test
|
2024-12-14 21:40:46 -03:00 |
|
Eric Ciarla
|
a2998d4499
|
Hash Urls
|
2024-12-12 16:10:10 -05:00 |
|
Nicolas
|
8a1c404918
|
Nick: revert trailing comma
|
2024-12-11 19:51:08 -03:00 |
|
Nicolas
|
52f2e733e2
|
Nick: fixes
|
2024-12-11 19:48:22 -03:00 |
|
Nicolas
|
00335e2ba9
|
Nick: fixed prettier
|
2024-12-11 19:46:11 -03:00 |
|
Nicolas
|
d817aa744f
|
Update v1.ts
|
2024-11-24 19:46:31 -08:00 |
|
Nicolas
|
93e106d321
|
Update v0.ts
|
2024-11-20 16:43:02 -08:00 |
|
Nicolas
|
a4f15260a7
|
Nick:
|
2024-11-12 12:23:24 -05:00 |
|
Gergő Móricz
|
8d467c8ca7
|
WebScraper refactor into scrapeURL (#714)
* feat: use strictNullChecking
* feat: switch logger to Winston
* feat(scrapeURL): first batch
* fix(scrapeURL): error swallow
* fix(scrapeURL): add timeout to EngineResultsTracker
* fix(scrapeURL): report unexpected error to sentry
* chore: remove unused modules
* feat(transfomers/coerce): warn when a format's response is missing
* feat(scrapeURL): feature flag priorities, engine quality sorting, PDF and DOCX support
* (add note)
* feat(scrapeURL): wip readme
* feat(scrapeURL): LLM extract
* feat(scrapeURL): better warnings
* fix(scrapeURL/engines/fire-engine;playwright): fix screenshot
* feat(scrapeURL): add forceEngine internal option
* feat(scrapeURL/engines): scrapingbee
* feat(scrapeURL/transformars): uploadScreenshot
* feat(scrapeURL): more intense tests
* bunch of stuff
* get rid of WebScraper (mostly)
* adapt batch scrape
* add staging deploy workflow
* fix yaml
* fix logger issues
* fix v1 test schema
* feat(scrapeURL/fire-engine/chrome-cdp): remove wait inserts on actions
* scrapeURL: v0 backwards compat
* logger fixes
* feat(scrapeurl): v0 returnOnlyUrls support
* fix(scrapeURL/v0): URL leniency
* fix(batch-scrape): ts non-nullable
* fix(scrapeURL/fire-engine/chromecdp): fix wait action
* fix(logger): remove error debug key
* feat(requests.http): use dotenv expression
* fix(scrapeURL/extractMetadata): extract custom metadata
* fix crawl option conversion
* feat(scrapeURL): Add retry logic to robustFetch
* fix(scrapeURL): crawl stuff
* fix(scrapeURL): LLM extract
* fix(scrapeURL/v0): search fix
* fix(tests/v0): grant larger response size to v0 crawl status
* feat(scrapeURL): basic fetch engine
* feat(scrapeURL): playwright engine
* feat(scrapeURL): add url-specific parameters
* Update readme and examples
* added e2e tests for most parameters. Still a few actions, location and iframes to be done.
* fixed type
* Nick:
* Update scrape.ts
* Update index.ts
* added actions and base64 check
* Nick: skipTls feature flag?
* 403
* todo
* todo
* fixes
* yeet headers from url specific params
* add warning when final engine has feature deficit
* expose engine results tracker for ScrapeEvents implementation
* ingest scrape events
* fixed some tests
* comment
* Update index.test.ts
* fixed rawHtml
* Update index.test.ts
* update comments
* move geolocation to global f-e option, fix removeBase64Images
* Nick:
* trim url-specific params
* Update index.ts
---------
Co-authored-by: Eric Ciarla <ericciarla@yahoo.com>
Co-authored-by: rafaelmmiller <8574157+rafaelmmiller@users.noreply.github.com>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
|
2024-11-07 20:57:33 +01:00 |
|
Nicolas
|
d8abd15716
|
Nick: from bulk to batch
|
2024-10-23 15:37:24 -03:00 |
|
Gergő Móricz
|
70c4e7c334
|
feat(bulk/scrape): check credits via url list length
|
2024-10-23 19:42:02 +02:00 |
|
Nicolas
|
66e505317e
|
Merge branch 'main' into mog/bulk-scrape
|
2024-10-23 14:36:26 -03:00 |
|
Gergő Móricz
|
6ed3104eb6
|
feat: clear ACUC cache endpoint based on team ID
|
2024-10-22 20:28:10 +02:00 |
|
Nicolas
|
79e65f31ef
|
Update v1.ts
|
2024-10-17 17:57:44 -03:00 |
|
Gergő Móricz
|
03b37998fd
|
feat: bulk scrape
|
2024-10-17 19:40:18 +02:00 |
|
Nicolas
|
18f9cd09e1
|
Nick: fixed more stuff
|
2024-10-01 16:04:39 -03:00 |
|