270 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
yanlong.wang	a54816d12d	fix	2024-10-14 17:33:24 +08:00
yanlong.wang	6a97f0bfa6	fix: uri encoding	2024-10-14 17:27:29 +08:00
Zhaofeng Miao	f82504540b	fix(adaptive-crawler): fix cache problem	2024-10-10 16:37:12 +08:00
Zhaofeng Miao	db432645c3	feat: change deployment machine type to improve cpu utilization	2024-10-10 11:21:42 +08:00
Zhaofeng Miao	b9124a2ec1	chore	2024-10-10 11:20:31 +08:00
Zhaofeng Miao	b3ca557f6e	chore: security	2024-10-10 11:18:38 +08:00
Zhaofeng Miao	86d69eebd1	chore: fix security dependencies	2024-10-10 11:17:18 +08:00
Zhaofeng Miao	14322140ba	docs: readme changelog	2024-10-10 10:34:25 +08:00
yanlong.wang	e9258af742	fix: pdf mode and google web cache	2024-10-09 17:47:53 +08:00
yanlong.wang	f6bbddcb48	fix: pageshot missing in cache	2024-10-09 15:07:30 +08:00
Zhaofeng Miao	a44d9a2d2a	feat(adaptive-crawler): optimize relevance detection	2024-10-08 15:19:03 +08:00
Zhaofeng Miao	af282eec43	fix(adaptive-crawler): useSitemap should be rewritten in certain condition	2024-10-08 14:18:13 +08:00
yanlong.wang	339af19192	fix: request to unknown domain	2024-10-08 12:02:27 +08:00
Zhaofeng Miao	5a4b35e4b9	fix(adaptive-crawler): if no sitemap, use recursive instead	2024-10-08 11:50:50 +08:00
Yanlong Wang	ee29be58f1	fix: gfm strikethrough	2024-10-01 18:57:12 +08:00
Yanlong Wang	f0c3a9b70e	fix	2024-10-01 12:55:06 +08:00
Yanlong Wang	a66791d85f	fix	2024-09-27 13:30:29 +08:00
yanlong.wang	f531056bbd	fix: pageshot not removed from page snapshot	2024-09-26 15:55:16 +08:00
Zhaofeng Miao	8008e53d57	feat(adaptive-crawl): disable invalid link	2024-09-25 14:18:28 +08:00
Zhaofeng Miao	3f88f8d2f7	fix(adaptive): url hash	2024-09-23 16:21:46 +08:00
Yanlong Wang	39e49cac63	fix: 3xx not considered errors	2024-09-18 02:35:36 +08:00
Yanlong Wang	96ce7f5aac	fix: iframe should not actively report snapshot	2024-09-18 02:33:11 +08:00
Yanlong Wang	87a6578970	chore: deployment tweak	2024-09-18 00:12:29 +08:00
Yanlong Wang	c36aa730b4	fix: target selector	2024-09-17 17:47:01 +08:00
Zhaofeng Miao	e27bcaca77	feat: add adaptive crawler (#112)	2024-09-13 14:08:07 +08:00
Yanlong Wang	f8bc4877ef	fix	2024-09-12 19:50:46 +08:00
Yanlong Wang	1bd3ed7125	fix: description from jsdom	2024-09-12 19:09:19 +08:00
Yanlong Wang	6e05ea2243	feat: warn on non 200 response	2024-09-12 19:05:06 +08:00
Zhaofeng Miao	6147a28609	feat: return description	2024-09-11 15:21:39 +08:00
yanlong.wang	aad9096119	fix: bump deps	2024-09-10 17:13:13 +08:00
yanlong.wang	5b85fe450f	bump: deps	2024-09-10 16:41:54 +08:00
yanlong.wang	0da04847b5	bump: deps	2024-09-10 16:31:08 +08:00
yanlong.wang	145a9f8f88	bump: deps	2024-09-10 15:38:00 +08:00
yanlong.wang	e2aed6dd97	fix: attachment pdf	2024-09-10 12:02:22 +08:00
Yanlong Wang	c5abdf8570	tweak: try alleviate oom killed issue	2024-09-08 16:29:45 +08:00
Yanlong Wang	e324c4667f	feat: support explicit q passing in search	2024-09-08 10:38:35 +08:00
Yanlong Wang	607407f740	fix: pdf detection	2024-09-08 10:14:34 +08:00
Yanlong Wang	94170db060	fix: performance issue of jsdom	2024-09-08 00:50:15 +08:00
yanlong.wang	5171e5f94b	fix: req cap issues	2024-09-02 14:30:23 +08:00
yanlong.wang	405fe6372e	fix: bump deps	2024-09-02 13:59:40 +08:00
Yanlong Wang	42700d1a85	fix: cache with locale	2024-08-30 18:37:23 +08:00
Zhaofeng Miao	6900e0241c	feat: allow passing pdf without url param (#111)	2024-08-30 11:04:32 +08:00
Yanlong Wang	9a514cd473	fix: cap browser request freq to avoid block from google	2024-08-29 09:28:17 +08:00
Zhaofeng Miao	7e6c2fcf48	feat: add referer param	2024-08-22 16:48:47 +08:00
Zhaofeng Miao	080056e889	feat: allow passing base64 encoded pdf	2024-08-22 14:56:09 +08:00
Zhaofeng Miao	de50c93825	feat: expose X-Locale parameter	2024-08-20 16:14:48 +08:00
Yanlong Wang	fb5bd58ee4	feat: return usage tokens in json	2024-08-16 20:32:38 +08:00
Yanlong Wang	c7860e615c	fix: set-cookie	2024-08-16 19:37:13 +08:00
Yanlong Wang	df58fcb3fa	fix: alleviate search performance issue	2024-08-09 15:03:24 +08:00
Yanlong Wang	eb74e9c6f8	fix: remove select element from markdown to walk around turndown performance issue	2024-08-09 10:55:36 +08:00