Gergő Móricz
82adf81b15
feat(self-host/docker-compose): allow configuring the internal port of the api
2025-02-21 16:19:37 +01:00
Eric Ciarla
9671e68512
Merge pull request #1229 from aparupganguly/feature/gemini-github-analyzer
...
Implemented github analyzer
2025-02-21 09:52:42 -05:00
Aparup Ganguly
448b44cdd9
Implemented github analyzer
2025-02-21 20:08:37 +05:30
Nicolas
6c51ef401e
Update rate-limiter.ts
2025-02-20 22:31:43 -03:00
Nicolas
25d9bdb1f6
(feat/ai-sdk) Migrate to AI-SDK ( #1220 )
...
* Nick: init
* Update llmExtract.ts
* Update llmExtract.ts
* Nick rename
* fix(v1/types): extract json schema validation
* Update url-processor.ts
* feat(ai-sdk): ollama support
* feat(ai-sdk): further ollama support
* Nick: it is broken btw
* feat(ai-sdk): abstract model adapter
* Update pnpm-lock.yaml
* Update analyzeSchemaAndPrompt.ts
* Nick:
* feat(ai-sdk): ollama support
* doc(SELF_HOST): update with embedding param
* Nick:
* Update ranker.ts
* Nick:
* feat(ai-sdk): fixes
* Update llmExtract.ts
* feat: remove zod-to-json-schema
* fix
* Update llmExtract.ts
* use openai
* fixes
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-20 22:48:58 +01:00
rafaelmmiller
943eb775c6
Update eval_run.py
2025-02-20 17:11:42 -03:00
rafaelmmiller
570dc28da2
bugfix eval run
2025-02-20 16:10:36 -03:00
rafaelmmiller
d211240f58
Update eval-prod.yml
2025-02-20 16:02:45 -03:00
rafaelmmiller
04127cca64
Update eval-prod.yml
2025-02-20 15:59:30 -03:00
Rafael Miller
a13863638a
Merge pull request #1224 from mendableai/feat/eval-run-workflow
...
Feat/added eval run after deploy workflow
2025-02-20 15:23:57 -03:00
Gergő Móricz
16c305775e
fix(crawl-redis): ignore empty includes/excludes ( #1223 )
...
* fix(crawl-redis): ignore empty includes/excludes
* fix(snips/scrape): bump timeouts
2025-02-20 19:06:02 +01:00
rafaelmmiller
d7db58e477
Feat/added eval run after deploy workflow
2025-02-20 15:01:26 -03:00
Gergő Móricz
283a3bfef3
fix(scrapeURL/engines/fetch): discover charset and re-decode ( #1221 )
...
* fix(scrapeURL/engines/fetch): discover charset and re-decode
* fix(snips/scrape): allow more time for stealth proxy
2025-02-20 18:56:15 +01:00
Gergő Móricz
e417f83c28
feat(self-host): ollama support ( #1219 )
2025-02-20 16:59:19 +01:00
Eric Ciarla
192f0031cf
Merge pull request #1206 from aparupganguly/feature/gemini-extractor
...
gemini extractor Implementation
2025-02-20 09:48:10 -05:00
Gergő Móricz
e84c7325d9
chore: remove dead code
...
Fixes #1149
Fixes #1150
2025-02-20 15:44:34 +01:00
Gergő Móricz
62d11f21fc
self-host: remove old playwright-service
2025-02-20 15:39:19 +01:00
Gergő Móricz
d533718312
feat(self-host): rework docker-compose.yaml
2025-02-20 15:37:19 +01:00
Gergő Móricz
305fbb37e7
temp ( #1218 )
2025-02-20 15:08:32 +01:00
Gergő Móricz
387cc60668
fix(ci/test-server): clean up old envs
2025-02-20 15:06:37 +01:00
Gergő Móricz
fca349f345
fix(ci/js-sdk): bad sed command
2025-02-20 15:02:56 +01:00
Nicolas
2151ca846c
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-02-20 10:50:32 -03:00
Nicolas
7db2d25efa
Nick:
2025-02-20 10:50:22 -03:00
Gergő Móricz
c38dcd0432
feat(self-host): proxy support (FIR-1111) ( #1212 )
...
* feat(self-host): proxy support
* fix(playwright-service-ts): return untreated text/plain
2025-02-20 14:20:03 +01:00
Gergő Móricz
c75522f535
feat(sdk): enforce timeout on client-side if set ( #1204 )
2025-02-20 13:35:31 +01:00
Gergő Móricz
dc1501527a
doc(self-host): update with searxng details
2025-02-20 12:41:44 +01:00
Loris
100168ddf3
Add searxng for search endpoint ( #1193 )
...
* add searxng.ts
* update to add searxng endpoint
* Apply suggestions from code review
* feat(ci/self-host): add tests with searxng
* feat(ci/self-host): bootstrap searxng for testing
* feat(ci): improvements in syntax
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-20 12:36:53 +01:00
Gergő Móricz
04218de2b0
Revert "feat(ci): use pull_request_target (+ manual approval)"
...
This reverts commit 9142030881e0d153396279520e127b74af8417c9.
2025-02-20 10:58:08 +01:00
Gergő Móricz
9142030881
feat(ci): use pull_request_target (+ manual approval)
2025-02-20 10:52:29 +01:00
Gergő Móricz
da1670b78c
feat(map): mock support (FIR-1109) ( #1213 )
...
* feat(map,fetch): mock support
* feat(snips/map): mock out long-running test
* fix(snips/scrape): use more reliable site for adblock testing
2025-02-20 10:41:43 +01:00
Gergő Móricz
bc5a16d048
feat(ci/test-server): build go markdown parser
2025-02-20 10:05:39 +01:00
Gergő Móricz
7770929dcd
fix(README): Adjust crawling example limit
...
Fixes #871
2025-02-20 09:51:38 +01:00
Gergő Móricz
11ed679274
feat(scrapeURL/pdf): support PDF prefetch when parsePDF is off
2025-02-20 09:28:13 +01:00
Gergő Móricz
5eb0235ccb
feat(apps/api): remove Sentry builds
2025-02-20 08:19:06 +01:00
Gergő Móricz
da46736016
fix(self-host): further changes
2025-02-20 08:01:31 +01:00
Gergő Móricz
10d9b65f96
fix(self-host): update docs and dockerignore
2025-02-20 07:57:57 +01:00
niazarak
e4504b3236
Use correct list typing for py 3.8 support ( #931 )
...
list[str] is not supported in Python 3.8
2025-02-20 07:39:46 +01:00
Gergő Móricz
55d047b6b3
feat(scrapeURL): handle PDFs behind anti-bot ( #1198 )
2025-02-20 04:11:30 +01:00
Gergő Móricz
bec52bef6c
feat(self-host/docs): playwright-service-ts is now default
2025-02-20 03:37:42 +01:00
Gergő Móricz
7f5ff80203
feat(self-host): use playwright-service-ts by default
2025-02-20 03:36:51 +01:00
Gergő Móricz
c39cc27866
feat(ci/self-host): add playwright microservice tests ( #1210 )
...
* feat(ci/self-host): add playwright microservice tests
* fix ci
* fix ci 2
* fix ci 3
* fix(playwright-service): get raw JSON if response is JSON
2025-02-20 02:06:13 +01:00
Shaun Offenbacher
037f64916a
Playwright page error schema ( #1172 )
...
* updated
* udpated
* updated
* updated
* added env
* updated
* updated
* updated
* updated
* updated
* updated docker file
* updated docker compose
* fixed validation error
* reverted local changes to get ready for pr
2025-02-20 00:43:46 +01:00
Gergő Móricz
46b187bc64
feat(v1/map): stop mapping if timed out via AbortController ( #1205 )
2025-02-20 00:42:13 +01:00
Gergő Móricz
2200f084f3
SELFHOST FIXES ( #1207 )
...
* fix(extract): construct OpenAI on demand
Fixes hard-crash if api key not specified in a self-hosting environment.
* fix(ci): try sleeping
* fix(ci): override host
* fix(ci): wait for server to start
* Support /extract and /crawl for self-hosted (FIR-1097) (#1137 )
* Support /extract for self-hosted
This returns the job response from redis rather than supabase when db auth is disabled (self hosted mode)
* Use getJob for extract and use correct types
* fix(v1/crawl-status): only poll DB for total count if DB is enabled
* feat(snips): TEST_SUITE_SELF_HOSTED
* fix(ci/test-server-self-host): use pr trigger
* fix(scrapeURL): f-e mocking in selfhosted env
* fix(snips): do not try to eval json format on selfhost
* fix(scrapeURL): further f-e mocking
* fix(snips): don't timeout on hard fail polling
* fix(v1/extract-status): fix-up the db-agnostic impl
unfortunately had to separate the functions since the schema
was too divergent :(
* fix(snips): boost screenshot delay
* feat(ci): test with openai
* feat(ci): extract, search testing
* fix(ci): matrix
* fix(ci): bleh
* Update: fix default google search (#1174 )
* fix log title
* search should always work
* asd
* fix ci
---------
Co-authored-by: Nick Roth <nlr06886@gmail.com>
Co-authored-by: William <sdustusun@gmail.com>
2025-02-20 00:41:22 +01:00
Gergő Móricz
c1ca64fc07
fix(ci): retain fail status
2025-02-19 22:31:16 +01:00
Gergő Móricz
f4f75fe184
fix(ci): path to lock
2025-02-19 22:15:41 +01:00
Gergő Móricz
e9cb8ac956
feat(ci): caching improvements
2025-02-19 22:11:32 +01:00
Gergő Móricz
0eff99007a
feat(ci): upload logs
2025-02-19 22:02:32 +01:00
Gergő Móricz
9976560567
feat(ci): self-hosted server test suite
2025-02-19 21:55:48 +01:00
Aparup Ganguly
8caeab2691
minor changes
2025-02-20 02:24:49 +05:30
