Yanlong Wang
8456fcecbd
fix: somehow side-loading chromewebstore would 100% crash the browser
2025-03-08 20:44:02 +08:00
Yanlong Wang
440ff4d729
fix: expect failure while loading pdf
2025-03-08 20:25:18 +08:00
Yanlong Wang
4bc6394692
fix: potential invalid pdf issue
2025-03-08 20:19:50 +08:00
Yanlong Wang
4ab28fe971
deps: bump
2025-03-08 20:04:24 +08:00
Yanlong Wang
66db31788e
cleanup: use local project code as much as possible
2025-03-08 19:32:30 +08:00
Yanlong Wang
512f225692
fix: sideload redirections
2025-03-08 18:59:36 +08:00
Yanlong Wang
c19ca2147c
fix: bug in pptr injections
2025-03-08 18:43:39 +08:00
Yanlong Wang
e551695d17
fix: fail early on special cookie redirects
2025-03-08 18:43:10 +08:00
Yanlong Wang
26f6202f79
fix: curl cookie mimicking
2025-03-08 18:14:43 +08:00
Yanlong Wang
89e5dbbe9c
fix: curl cookie behavior
2025-03-08 18:01:38 +08:00
Yanlong Wang
3b1978fd1d
fix: implement DNT in alt-gen and pdf-extract
2025-03-08 17:52:49 +08:00
Yanlong Wang
1a2754c674
fix: sideLoad header detection
2025-03-08 17:46:09 +08:00
Yanlong Wang
63a2e15f4d
fix: curl redirection location
2025-03-08 17:20:38 +08:00
Yanlong Wang
fb43578fdd
fix: curl implicit redirect
2025-03-08 17:18:53 +08:00
Yanlong Wang
8597daa96b
fix: side load context bridging
2025-03-08 16:49:14 +08:00
Yanlong Wang
e92ff33ad0
fix
2025-03-08 15:49:22 +08:00
Yanlong Wang
b674d26f76
fix: clean HTML timer
2025-03-08 13:33:10 +08:00
Yanlong Wang
4e299bf8e2
fix: remove tailwind classes instead of the opposite
2025-03-08 13:30:06 +08:00
Yanlong Wang
434c6707fd
deps: bump
2025-03-08 13:06:01 +08:00
Yanlong Wang
4d345f40f9
fix: curl error wrapping
2025-03-08 12:59:22 +08:00
Yanlong Wang
4744b8645f
deps: bump
2025-03-08 11:30:05 +08:00
Yanlong Wang
c326088327
fix: deps
2025-03-08 10:54:08 +08:00
Yanlong Wang
23a3b807c9
restructure: nolonger a firebase application ( #1160 )
...
* fix: fine allow redefining Function.prototype.toString
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* fix: contentType encoding
* wip
* fix: error throwing
* wip
* fix
* wip
* fix
* fix
* fix: jsdom
* wip
* wip
* fix: links summary uniqueness
* wip
* wip
* robots-txt catch no robots.txt
* deps: remove puppeteer-extra-plugin-stealth
* fix: dont change waring type
* fix: curl
* fix: replace firebase-roundtrip-check with blackhole-detector
* fix: black hole detection
* sercher: black hole detecting
* fix: no h2c for searcher
* fix: bhd
* fix: search and crawl conflict
* fix: bhd
* fix
* fix: server script
* canvas: fixed avif issue
* logging: move some to debug
* fix
* fix: pptr declare ready only when page can be created without issues
* fix: bhd
* cd: cloud run deploy-health-check cannot complete pptr newPage
* cd: fix
* fix: curl body can be null
* fix
* fix
* fix: major fix regarding TC pdfs
* fix
* fix
* deps: fix civkit trie router issue
* fix
* boom: total restructure
* cd: fix docker ctx
* fix
* fix: switch to h2c
* cd: ensure http2
2025-03-08 00:46:52 +08:00
Aaron Ji
ed80c9a4a2
chore: expose 'num' param for search API ( #1157 )
...
* chore: expose 'num' param for search API
* fix: wrong count value
2025-03-05 11:20:55 +08:00
Aaron Ji
a7ffb58741
chore: support 'gl', 'hl', 'location' and 'page' when search ( #1156 )
2025-03-04 17:34:49 +08:00
Aaron Ji
21ae52a55a
feat: support fetching favicon ( #1155 )
...
* feat: support fetching favicon
* Update backend/functions/src/dto/scrapping-options.ts
Co-authored-by: Yanlong Wang <yanlong.wang@naiver.org>
* Update backend/functions/src/dto/scrapping-options.ts
Co-authored-by: Yanlong Wang <yanlong.wang@naiver.org>
* chore: remove 'withFavicon' from CrawlerOptions
---------
Co-authored-by: Yanlong Wang <yanlong.wang@naiver.org>
2025-03-04 11:32:42 +08:00
yanlong.wang
cd6a4eedf9
cd: add dev branch
2025-03-03 18:52:07 +08:00
Aaron Ji
3d29b3069a
fix: use 'no-content' ( #1154 )
...
* fix: use 'no-content' in x-respond-with
* fix: use 'no-content' in x-respond-with
2025-03-03 17:00:18 +08:00
Aaron Ji
51109e01d2
chore: improve type definitions and type safety ( #1153 )
...
* chore: improve type definitions and type safety
* chore: format search result
* chore: use 'x-details'
* chore: remove 'domain' from search result
* chore: use 'x-respond-with'
* chore: optimize logic to return content
* chore: cleanup
* chore: optimize logic
2025-02-28 18:44:38 +08:00
Yanlong Wang
4873578c62
Revert "serving: switching to h2c"
...
This reverts commit 690984d44a960a27556fc46873366f785d60666f.
2025-02-27 11:37:06 +08:00
Sha Zhou
efc405b1a0
remove version params
2025-02-27 11:33:29 +08:00
Yanlong Wang
f8a0dbd0ac
deps: ditch firebase logger
2025-02-27 10:36:02 +08:00
Yanlong Wang
690984d44a
serving: switching to h2c
2025-02-27 10:11:04 +08:00
Yanlong Wang
20a7d4d538
fix: search return amount
2025-02-27 09:55:10 +08:00
Sha Zhou
1ca29b4b0e
fix: update charge amount for search v2
2025-02-27 09:43:38 +08:00
Aaron Ji
fcb59591c2
fix: always return 10 results ( #1152 )
...
* fix: always return 10 results
* chore: get 2 extra results
2025-02-25 18:11:08 +08:00
Sha Zhou
4d48797f17
switch to search serper
2025-02-25 16:36:28 +08:00
Sha Zhou
abd37e5266
feat: search v2 ( #1151 )
2025-02-25 16:33:23 +08:00
Yanlong Wang
29774ac637
fix: scrapMany and searcher
2025-02-24 23:44:29 +08:00
yanlong.wang
380bbffc0c
fix: formatting in some edge cases
2025-02-24 13:29:56 +08:00
yanlong.wang
ecb1af07f9
cleanup
2025-02-24 11:38:47 +08:00
yanlong.wang
6335c526ec
fine: stop worrying about the global environment
2025-02-24 11:31:38 +08:00
yanlong.wang
a2f810cce6
fix: allow redefine Function.prototype.toString
2025-02-24 11:05:36 +08:00
yanlong.wang
6a1f391339
ci: fix mmdb download
2025-02-20 18:46:23 +08:00
yanlong.wang
7340028c64
fix(stand-alone-server): set ip to local context
2025-02-20 18:36:04 +08:00
yanlong.wang
9dbecbbc9f
fix
2025-02-20 17:50:12 +08:00
yanlong.wang
ef2f87cd7d
fix: serper search caching
2025-02-20 17:49:24 +08:00
yanlong.wang
7c029bf429
search: new branch using serper as provider
2025-02-20 17:32:45 +08:00
yanlong.wang
b44247c008
deps: block avif images for now
...
Because it causes segfaults.
See https://github.com/Brooooooklyn/canvas/issues/996
2025-02-19 17:52:51 +08:00
yanlong.wang
dc1dd7e30d
deps: update @napi-rs/canvas
...
Getting segfault in some cases, try updating.
2025-02-19 15:38:17 +08:00
