17 Commits

Author SHA1 Message Date
Nicolas
1f6abf95e8 Nick: extract billing works 2025-01-17 20:59:44 -03:00
Nicolas
4db023280d Nick: introduce llm-usage cost analysis 2025-01-15 21:01:29 -03:00
Nicolas
957eea4113 Nick: extract without a schema should work as expected 2025-01-14 11:37:00 -03:00
Nicolas
61e6af2b16 Nick: streaming callback experimental 2025-01-14 02:13:42 -03:00
Nicolas
2dc87a2e1c Update extraction-service.ts 2025-01-14 01:59:52 -03:00
Nicolas
033e9bbf29 Nick: __experimental_streamSteps 2025-01-14 01:45:50 -03:00
Nicolas
5e5b5ee0e2
(feat/extract) New re-ranker + multi entity extraction (#1061)
* agent that decides if splits schema or not

* split and merge properties done

* wip

* wip

* changes

* ch

* array merge working!

* comment

* wip

* dereferentiate schema

* dereference schemas

* Nick: new re-ranker

* Create llm-links.txt

* Nick: format

* Update extraction-service.ts

* wip: cooking schema mix and spread functions

* wip

* wip getting there!!!

* nick:

* moved functions to helpers

* nick:

* cant reproduce the error anymore

* error handling all scrapes failed

* fix

* Nick: added the sitemap index

* Update sitemap-index.ts

* Update map.ts

* deduplicate and merge arrays

* added error handler for object transformations

* Update url-processor.ts

* Nick:

* Nick: fixes

* Nick: big improvements to rerank of multi-entity

* Nick: working

* Update reranker.ts

* fixed transformations for nested objs

* fix merge nulls

* Nick: fixed error piping

* Update queue-worker.ts

* Update extraction-service.ts

* Nick: format

* Update queue-worker.ts

* Update pnpm-lock.yaml

* Update queue-worker.ts

---------

Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
2025-01-13 22:30:15 -03:00
Nicolas
f4d10c5031 Nick: formatting fixes 2025-01-10 18:35:10 -03:00
Nicolas
aa31508ccd Nick: links-billed update (temp) 2025-01-08 15:13:33 -03:00
Gergő Móricz
1f2a76fc23
Update apps/api/src/lib/extract/extraction-service.ts 2025-01-07 20:18:10 +01:00
Nicolas
eb254547e5 Nick: 2025-01-07 16:16:01 -03:00
Nicolas
27457ed5db Nick: init 2025-01-03 20:44:27 -03:00
rafaelmmiller
ef0fc8d0d3 broader search if didnt find results 2025-01-02 18:00:18 -03:00
Nicolas
33632d2fe3 Update extraction-service.ts 2024-12-31 15:22:50 -03:00
Nicolas
e6da214aeb Nick: async background index 2024-12-30 21:42:01 -03:00
Nicolas
4332f18a8f Nick: making it optional for the user 2024-12-26 12:43:58 -03:00
Nicolas
233f347f5e Nick: refactor 2024-12-26 12:41:37 -03:00