648 Commits

Author	SHA1	Message	Date
Gergo Moricz	8dbac0268c	feat: offload crawl results to the DB	2024-08-13 21:40:59 +02:00
Gergo Moricz	4bbc9db1df	fix: prioritize scrape jobs over crawl jobs	2024-08-13 21:31:34 +02:00
Gergo Moricz	5f2af37880	fix(scrape): remove scrape job from queue after the job is done	2024-08-13 21:26:41 +02:00
Gergo Moricz	2413e33359	fix(queue-worker): remove console.log	2024-08-13 21:07:36 +02:00
Gergo Moricz	d7549d4dc5	feat: remove webScraperQueue	2024-08-13 21:03:24 +02:00
Gergő Móricz	4a2c37dcf5	Merge branch 'main' into feat/queue-scrapes	2024-08-13 20:53:49 +02:00
Gergo Moricz	86e136beca	feat: crawl to scrape conversion	2024-08-13 20:51:43 +02:00
rafaelsideguide	a4be95ac27	fixed tests	2024-08-13 13:42:26 -03:00
Nicolas	6810338271	Update search.ts	2024-08-12 16:51:43 -04:00
Thomas Kosmas	98be29c963	Update parameters for platform.openai.com	2024-08-12 22:49:28 +03:00
Nicolas	74a5125185	Nick: removed redlock	2024-08-12 15:07:30 -04:00
Nicolas	0bd1a820ee	Update auth.ts	2024-08-12 13:42:09 -04:00
Nicolas	25a899eae3	Nick: redlock cache in auth	2024-08-12 13:37:47 -04:00
Nicolas	e28c415cf4	Nick:	2024-08-09 14:07:46 -04:00
rafaelsideguide	0591000b64	bugfix includes excludes	2024-08-09 14:30:41 -03:00
Nicolas	f1f5605010	Update website_params.ts	2024-08-08 12:31:58 -04:00
Nicolas	b0abad07da	Merge pull request #496 from tak-s/improve-logging-level ... Improve logs	2024-08-07 22:01:12 -04:00
Gergo Moricz	920b7f2f44	fix(runWebScraper): don't filter empty docs	2024-08-07 21:00:22 +02:00
Gergo Moricz	55ec96c23f	fix(queue-worker): bad job lock extension time	2024-08-07 20:24:16 +02:00
Gergo Moricz	ab7a35c581	fix(queue-worker): log lock extensions	2024-08-07 19:49:48 +02:00
Gergo Moricz	a1c2ee5aa9	fix: always complete job, no try	2024-08-07 19:39:09 +02:00
Gergo Moricz	191dfbd9ca	fix: move to completed in one place	2024-08-07 18:49:58 +02:00
Nicolas	457c082ba1	Nick: fixed tests	2024-08-07 11:08:53 -04:00
Gergő Móricz	5fc7fcb77c	Merge branch 'main' into feat/queue-scrapes	2024-08-07 16:35:44 +02:00
Gergo Moricz	fe9fdb578b	revert bad hotfixes	2024-08-07 16:34:25 +02:00
Gergo Moricz	b7c01dcb9b	fix(webScraperQueue): reduce retries to 2	2024-08-07 16:31:50 +02:00
Gergo Moricz	cdf7bad5b4	fix(runWebScraper): don't move to completed	2024-08-07 15:20:56 +02:00
Gergo Moricz	9df8719efa	fix(queue-worker): raise queue log level to info	2024-08-07 14:56:04 +02:00
Gergo Moricz	7bb922071c	fix(queue-worker): manually renew lock (testing)	2024-08-07 14:35:20 +02:00
Gergo Moricz	8216266d16	fix(scrape_log): display error properly	2024-08-07 14:19:20 +02:00
Gergo Moricz	2e2e80d679	fix(scrape-events): updateScrapeResult fix	2024-08-07 14:17:50 +02:00
Gergo Moricz	b5ec47fd96	fix(runWebScraper): don't fetch next job	2024-08-07 13:53:04 +02:00
Nicolas	3321ca9398	Merge pull request #504 from mendableai/feat/fullpage-screenshot ... [Feat] Added fullpagescreenshot capabilities	2024-08-06 13:52:29 -04:00
Gergo Moricz	b60ee30dba	fix(single_url): accept 500	2024-08-06 18:00:56 +02:00
Gergo Moricz	06751a8e21	fix(crawl-status): missing partial data after cancel	2024-08-06 17:31:20 +02:00
Gergo Moricz	810b98ec38	fix(scrape): fix timeout error code	2024-08-06 17:30:01 +02:00
Gergo Moricz	3ae95a2740	fix(scrape): consider timeout property	2024-08-06 17:25:58 +02:00
Gergo Moricz	8566ece700	fix(scrape): pass extractorOptions	2024-08-06 17:15:19 +02:00
Gergo Moricz	8e0aa69603	fix(crawl-status): partial_data	2024-08-06 17:06:21 +02:00
Gergo Moricz	1ab119c874	fix(scrape): don't double-bill for scrape	2024-08-06 16:57:23 +02:00
Gergo Moricz	7c5cda7b45	fix(queue-worker): concurrency	2024-08-06 16:57:00 +02:00
Gergo Moricz	d7d63790e5	fix(crawl-status): isCancelled should be status failed	2024-08-06 16:35:55 +02:00
Gergo Moricz	03c84a9372	cleanup and fix cancelling	2024-08-06 16:26:46 +02:00
rafaelsideguide	4d24a99d50	fix params	2024-08-06 09:34:43 -03:00
rafaelsideguide	3edc3a3d15	added fullpagescreenshot capabilities, wip on fire-engine side	2024-08-05 18:17:37 -03:00
rafaelsideguide	f32e8de156	fixes the empty excludes.filter undefined bug	2024-08-05 18:13:31 -03:00
tak-s	af9bc5c8bb	Suppressed repetitive logs	2024-08-04 15:09:36 +09:00
Nicolas	1742e4ceae	Nick:	2024-08-02 19:25:15 -04:00
Nicolas	39aecd974b	Update redis-health.ts	2024-08-02 17:43:45 -04:00
Nicolas	b448e3c3ad	Update website_params.ts	2024-08-02 14:26:35 -04:00