Gergő Móricz
71b6b83ec2
tally rework api switchover ( #1328 )
...
* tally rework api switchover
* fix and send logs
* temp: force main instance while RPCs propagate
* Revert "temp: force main instance while RPCs propagate"
This reverts commit 4c93379cfa64efd60eb4767dd8eced1bdd302531.
2025-03-12 20:10:33 +01:00
Gergő Móricz
97dee5de21
temp(ci/self-host): remove no-proxy option
2025-03-12 18:19:24 +01:00
Gergő Móricz
e1cfe1da48
feat(crawl): includes/excludes fixes (FIR-1300) ( #1303 )
...
* feat(crawl): includes/excludes fixes pt. 1
* fix(snips): billing tests
* drop tha logs
* fix(ci): add replica url
* feat(crawl): drop initial scrape if it's not included
* feat(ci): more verbose logging
* fix crawl path in test
* fix(ci): wait for api
* fix(snips/scrape/ad): test for more pixels
* feat(js-sdk/crawl): add regexOnFullURL
2025-03-06 17:05:15 +01:00
Gergő Móricz
9ad947884d
feat(tests/snips): add billing tests + misc billing fixes (FIR-1280) ( #1283 )
...
* feat(tests/snips): add billing tests + misc billing fixes
* add testing key
* asd
2025-03-02 16:51:42 -03:00
Gergő Móricz
856ec37d4c
fix(ci/js-sdk): properly build SDK before publishing
2025-02-28 12:34:53 +01:00
Nicolas
39b61132bf
Nick: fixed js sdk
2025-02-27 20:39:53 -03:00
Gergő Móricz
115b6b61c4
add initial codeowners
2025-02-25 14:28:09 +01:00
rafaelmmiller
943eb775c6
Update eval_run.py
2025-02-20 17:11:42 -03:00
rafaelmmiller
570dc28da2
bugfix eval run
2025-02-20 16:10:36 -03:00
rafaelmmiller
d211240f58
Update eval-prod.yml
2025-02-20 16:02:45 -03:00
rafaelmmiller
04127cca64
Update eval-prod.yml
2025-02-20 15:59:30 -03:00
rafaelmmiller
d7db58e477
Feat/added eval run after deploy workflow
2025-02-20 15:01:26 -03:00
Gergő Móricz
305fbb37e7
temp ( #1218 )
2025-02-20 15:08:32 +01:00
Gergő Móricz
387cc60668
fix(ci/test-server): clean up old envs
2025-02-20 15:06:37 +01:00
Gergő Móricz
fca349f345
fix(ci/js-sdk): bad sed command
2025-02-20 15:02:56 +01:00
Gergő Móricz
c38dcd0432
feat(self-host): proxy support (FIR-1111) ( #1212 )
...
* feat(self-host): proxy support
* fix(playwright-service-ts): return untreated text/plain
2025-02-20 14:20:03 +01:00
Loris
100168ddf3
Add searxng for search endpoint ( #1193 )
...
* add searxng.ts
* update to add searxng endpoint
* Apply suggestions from code review
* feat(ci/self-host): add tests with searxng
* feat(ci/self-host): bootstrap searxng for testing
* feat(ci): improvements in syntax
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-20 12:36:53 +01:00
Gergő Móricz
04218de2b0
Revert "feat(ci): use pull_request_target (+ manual approval)"
...
This reverts commit 9142030881e0d153396279520e127b74af8417c9.
2025-02-20 10:58:08 +01:00
Gergő Móricz
9142030881
feat(ci): use pull_request_target (+ manual approval)
2025-02-20 10:52:29 +01:00
Gergő Móricz
bc5a16d048
feat(ci/test-server): build go markdown parser
2025-02-20 10:05:39 +01:00
Gergő Móricz
5eb0235ccb
feat(apps/api): remove Sentry builds
2025-02-20 08:19:06 +01:00
Gergő Móricz
c39cc27866
feat(ci/self-host): add playwright microservice tests ( #1210 )
...
* feat(ci/self-host): add playwright microservice tests
* fix ci
* fix ci 2
* fix ci 3
* fix(playwright-service): get raw JSON if response is JSON
2025-02-20 02:06:13 +01:00
Gergő Móricz
2200f084f3
SELFHOST FIXES ( #1207 )
...
* fix(extract): construct OpenAI on demand
Fixes hard-crash if api key not specified in a self-hosting environment.
* fix(ci): try sleeping
* fix(ci): override host
* fix(ci): wait for server to start
* Support /extract and /crawl for self-hosted (FIR-1097) (#1137 )
* Support /extract for self-hosted
This returns the job response from redis rather than supabase when db auth is disabled (self hosted mode)
* Use getJob for extract and use correct types
* fix(v1/crawl-status): only poll DB for total count if DB is enabled
* feat(snips): TEST_SUITE_SELF_HOSTED
* fix(ci/test-server-self-host): use pr trigger
* fix(scrapeURL): f-e mocking in selfhosted env
* fix(snips): do not try to eval json format on selfhost
* fix(scrapeURL): further f-e mocking
* fix(snips): don't timeout on hard fail polling
* fix(v1/extract-status): fix-up the db-agnostic impl
unfortunately had to separate the functions since the schema
was too divergent :(
* fix(snips): boost screenshot delay
* feat(ci): test with openai
* feat(ci): extract, search testing
* fix(ci): matrix
* fix(ci): bleh
* Update: fix default google search (#1174 )
* fix log title
* search should always work
* asd
* fix ci
---------
Co-authored-by: Nick Roth <nlr06886@gmail.com>
Co-authored-by: William <sdustusun@gmail.com>
2025-02-20 00:41:22 +01:00
Gergő Móricz
c1ca64fc07
fix(ci): retain fail status
2025-02-19 22:31:16 +01:00
Gergő Móricz
f4f75fe184
fix(ci): path to lock
2025-02-19 22:15:41 +01:00
Gergő Móricz
e9cb8ac956
feat(ci): caching improvements
2025-02-19 22:11:32 +01:00
Gergő Móricz
0eff99007a
feat(ci): upload logs
2025-02-19 22:02:32 +01:00
Gergő Móricz
9976560567
feat(ci): self-hosted server test suite
2025-02-19 21:55:48 +01:00
Gergő Móricz
dab0016669
feat(ci): publish all JS SDK packages
2025-02-19 21:46:02 +01:00
Gergő Móricz
1a9f6b985a
feat(github/ci): improvements
2025-02-19 20:51:38 +01:00
Gergő Móricz
fc64f436ed
fix(v1/types): fix extract -> json rename, ROUND II (FIR-1072) ( #1199 )
...
* Revert "Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195 )""
This reverts commit e28a44463ae49ffc195507204492cc7c15c438c4.
* fix(v1/types): fix bad transform
* feat(v1): proxy option / stealthProxy flag (FIR-1050) (#1196 )
* feat(v1): proxy option / stealthProxy flag
* feat(js-sdk): add proxy option
* fix
* fix extract tests
2025-02-19 12:07:55 -03:00
Gergő Móricz
86f05a07ba
feat(github/ci): connect to tailscale (FIR-748) ( #1112 )
...
* feat(github/ci): connect to tailscale
* fix(tests/snips): adjust timeouts
2025-01-29 16:59:43 +01:00
Móricz Gergő
5c1b67511c
feat(github/ci): run snips tests instead of always-failing tests
2025-01-29 08:18:09 +01:00
Móricz Gergő
1c55ce41be
feat(ci): add sentry auth token to builds
2024-11-11 10:32:17 +01:00
Gergő Móricz
8d467c8ca7
WebScraper refactor into scrapeURL (#714 )
...
* feat: use strictNullChecking
* feat: switch logger to Winston
* feat(scrapeURL): first batch
* fix(scrapeURL): error swallow
* fix(scrapeURL): add timeout to EngineResultsTracker
* fix(scrapeURL): report unexpected error to sentry
* chore: remove unused modules
* feat(transfomers/coerce): warn when a format's response is missing
* feat(scrapeURL): feature flag priorities, engine quality sorting, PDF and DOCX support
* (add note)
* feat(scrapeURL): wip readme
* feat(scrapeURL): LLM extract
* feat(scrapeURL): better warnings
* fix(scrapeURL/engines/fire-engine;playwright): fix screenshot
* feat(scrapeURL): add forceEngine internal option
* feat(scrapeURL/engines): scrapingbee
* feat(scrapeURL/transformars): uploadScreenshot
* feat(scrapeURL): more intense tests
* bunch of stuff
* get rid of WebScraper (mostly)
* adapt batch scrape
* add staging deploy workflow
* fix yaml
* fix logger issues
* fix v1 test schema
* feat(scrapeURL/fire-engine/chrome-cdp): remove wait inserts on actions
* scrapeURL: v0 backwards compat
* logger fixes
* feat(scrapeurl): v0 returnOnlyUrls support
* fix(scrapeURL/v0): URL leniency
* fix(batch-scrape): ts non-nullable
* fix(scrapeURL/fire-engine/chromecdp): fix wait action
* fix(logger): remove error debug key
* feat(requests.http): use dotenv expression
* fix(scrapeURL/extractMetadata): extract custom metadata
* fix crawl option conversion
* feat(scrapeURL): Add retry logic to robustFetch
* fix(scrapeURL): crawl stuff
* fix(scrapeURL): LLM extract
* fix(scrapeURL/v0): search fix
* fix(tests/v0): grant larger response size to v0 crawl status
* feat(scrapeURL): basic fetch engine
* feat(scrapeURL): playwright engine
* feat(scrapeURL): add url-specific parameters
* Update readme and examples
* added e2e tests for most parameters. Still a few actions, location and iframes to be done.
* fixed type
* Nick:
* Update scrape.ts
* Update index.ts
* added actions and base64 check
* Nick: skipTls feature flag?
* 403
* todo
* todo
* fixes
* yeet headers from url specific params
* add warning when final engine has feature deficit
* expose engine results tracker for ScrapeEvents implementation
* ingest scrape events
* fixed some tests
* comment
* Update index.test.ts
* fixed rawHtml
* Update index.test.ts
* update comments
* move geolocation to global f-e option, fix removeBase64Images
* Nick:
* trim url-specific params
* Update index.ts
---------
Co-authored-by: Eric Ciarla <ericciarla@yahoo.com>
Co-authored-by: rafaelmmiller <8574157+rafaelmmiller@users.noreply.github.com>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2024-11-07 20:57:33 +01:00
Nicolas
7b1df22698
Delete check-queues.yml
2024-10-15 13:31:29 -03:00
Nicolas
30a375d693
Nick:
2024-10-09 19:26:25 -03:00
Nicolas
a40fb3b062
Update deploy-image.yml
2024-10-03 19:17:52 -03:00
Nicolas
9a056919b9
Create deploy-image.yml
2024-10-03 18:41:41 -03:00
Nicolas
43d8563bb1
Update fly-direct.yml
2024-09-17 14:15:34 -04:00
Nicolas
a0189acbec
Update fly.yml
2024-09-17 12:58:49 -04:00
Nicolas
66577d1d03
Update fly.yml
2024-09-16 21:53:35 -04:00
Nicolas
358f8f9def
Update fly-direct.yml
2024-09-16 21:50:56 -04:00
Nicolas
4884439fd9
Update fly.yml
2024-09-16 21:48:12 -04:00
Nicolas
d21a797ef9
Update fly.yml
2024-09-16 21:46:40 -04:00
Nicolas
eec22a56d3
Nick: self host issue template
2024-09-12 11:43:40 -04:00
rafaelsideguide
7561bfe173
added envs to github action workflows
2024-09-05 12:59:32 -03:00
Nicolas
28c5635502
Update ci.yml
2024-09-04 16:45:56 -03:00
rafaelsideguide
f98a8541c2
fix(cicd): added use_db_auth to deploy workflow
2024-09-04 11:41:50 -03:00
rafaelsideguide
28df353829
fix(cicd): wait and moved rust publish
2024-09-04 09:16:31 -03:00
