1245 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Gergő Móricz	fc08ff450d	search port	2024-08-15 20:10:59 +02:00
Nicolas	86326f34e9	Update single_url.test.ts	2024-08-15 13:48:42 -04:00
Gergő Móricz	129a882bcc	fix(scrape): give scrapes their real job id	2024-08-15 19:29:47 +02:00
Gergő Móricz	965a5817d1	fix(queue-worker): log jobs correctly	2024-08-15 19:27:15 +02:00
Gergő Móricz	dad9d353d9	use thomas's url validation	2024-08-15 19:19:02 +02:00
Gergő Móricz	e3279274f1	fix: make playground crawl work	2024-08-15 19:14:32 +02:00
Gergő Móricz	c5597bc722	fix: robots.txt laoding	2024-08-15 19:11:07 +02:00
Gergő Móricz	29f0d9ec94	propagate priority to fire-engine	2024-08-15 19:04:46 +02:00
Gergő Móricz	b79d3d1754	fix	2024-08-15 19:02:05 +02:00
Gergő Móricz	57730f6a35	priority changes	2024-08-15 18:58:07 +02:00
Gergő Móricz	846610681b	fix: fix posthog, add dummy crawl DB items	2024-08-15 18:55:18 +02:00
Gergő Móricz	8a5cad72f6	fix(queue-worker): variable name collision	2024-08-14 22:02:05 +02:00
Gergő Móricz	b8ec40dd72	fix(crawl): submit sitemapped jobs in bulk	2024-08-14 20:34:19 +02:00
Gergő Móricz	2ca1017fc3	fix(crawl): make request 0 of crawl jobs higher priority	2024-08-14 19:34:18 +02:00
Gergő Móricz	f4466f6bb0	fix(test-suite): add artillery	2024-08-14 19:33:09 +02:00
Gergő Móricz	cfad067a63	fix(fly): change proxy limits	2024-08-14 18:52:40 +02:00
Gergő Móricz	a6c81f9d62	fix: return all data when calling webhook	2024-08-14 17:53:47 +02:00
Gergo Moricz	2e5e480cc2	fix(crawl): call webhooks	2024-08-13 22:10:17 +02:00
Gergo Moricz	a33596de3c	fix(log_job): add crawl_id	2024-08-13 22:03:46 +02:00
Gergo Moricz	9252940b52	fix(crawl-status): sort data	2024-08-13 21:55:13 +02:00
Gergo Moricz	8dbac0268c	feat: offload crawl results to the DB	2024-08-13 21:40:59 +02:00
Gergo Moricz	4bbc9db1df	fix: prioritize scrape jobs over crawl jobs	2024-08-13 21:31:34 +02:00
Gergo Moricz	5f2af37880	fix(scrape): remove scrape job from queue after the job is done	2024-08-13 21:26:41 +02:00
Gergo Moricz	2413e33359	fix(queue-worker): remove console.log	2024-08-13 21:07:36 +02:00
Gergo Moricz	d7549d4dc5	feat: remove webScraperQueue	2024-08-13 21:03:24 +02:00
Gergő Móricz	4a2c37dcf5	Merge branch 'main' into feat/queue-scrapes	2024-08-13 20:53:49 +02:00
Gergo Moricz	86e136beca	feat: crawl to scrape conversion	2024-08-13 20:51:43 +02:00
Nicolas	c82e9f3ebc	Merge pull request #536 from mendableai/fix/e2e-tests ... Fixed e2e tests	2024-08-13 14:44:14 -04:00
rafaelsideguide	a4be95ac27	fixed tests	2024-08-13 13:42:26 -03:00
Eric Ciarla	dd387c53c3	Merge pull request #534 from KentHsu/fix/go-sdk-module-name ... fix: go-sdk module name	2024-08-13 13:53:28 +02:00
KentHsu	fd060c7ef1	fix: go-sdk module name	2024-08-13 10:10:45 +08:00
Nicolas	c6bf78cfb4	Update fly-direct.yml	2024-08-12 21:14:00 -04:00
Nicolas	09ca165d2e	Merge pull request #531 from kevinswiber/fix/respect-docker-env-file-comments ... Self-host fix: Moving comments of .env.example values from end-of-line to above-line.	2024-08-12 16:54:56 -04:00
Nicolas	d06f40810c	Merge pull request #515 from wahpiangle/main ... Update redis urls in example .env	2024-08-12 16:51:59 -04:00
Nicolas	6810338271	Update search.ts	2024-08-12 16:51:43 -04:00
Thomas Kosmas	98be29c963	Update parameters for platform.openai.com	2024-08-12 22:49:28 +03:00
Kevin Swiber	33aa5cf0de	Moving comments of .env.example values from end-of-line to above-line. Self-host docs suggest using .env.example as a base. However, Docker doesn't respect end-of-line comments. It sets the comment as the actual value of the variable. This fix prevents that.	2024-08-12 12:24:46 -07:00
Nicolas	9d187bf3bb	Merge pull request #529 from mendableai/nsc/redlock-cache-auth ... Redlock cache in auth	2024-08-12 15:14:07 -04:00
Nicolas	74a5125185	Nick: removed redlock	2024-08-12 15:07:30 -04:00
Nicolas	0bd1a820ee	Update auth.ts	2024-08-12 13:42:09 -04:00
Nicolas	25a899eae3	Nick: redlock cache in auth	2024-08-12 13:37:47 -04:00
Rafael Miller	36e4b2cf49	Update .env.example	2024-08-12 10:37:00 -03:00
Quan Ming	fe179d0cb1	Update redis troubleshooting in self host guide	2024-08-10 12:39:22 +08:00
Quan Ming	a96ad4b0e2	Update redis url to use comment	2024-08-10 12:33:26 +08:00
Nicolas	e28c415cf4	Nick:	2024-08-09 14:07:46 -04:00
Gergo Moricz	5a778f2c22	fix(js-sdk): add type metadata to exports	2024-08-09 20:05:36 +02:00
Rafael Miller	6a78f6fe78	Merge pull request #497 from KentHsu/feat/add-go-sdk ... [Feat] Add Go SDK implementation	2024-08-09 14:58:20 -03:00
rafaelsideguide	0591000b64	bugfix includes excludes	2024-08-09 14:30:41 -03:00
Kent (Chia-Hao), Hsu	1fda882983	Merge branch 'mendableai:main' into feat/add-go-sdk	2024-08-10 00:46:15 +08:00
Quan Ming	0221872a70	Update redis urls in example .env	2024-08-10 00:40:11 +08:00