1362 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Nicolas	1e08e6d317	Merge branch 'main' into nsc/job-priority	2024-08-27 15:01:58 -03:00
Nicolas	861e2ebdf1	Nick: 2x rate limits	2024-08-27 12:08:50 -03:00
Nicolas	3850b000f0	Nick: removing credit notification for now	2024-08-27 11:14:04 -03:00
Nicolas	fa7dc5b10b	Update rate-limiter.ts	2024-08-26 16:33:34 -03:00
Nicolas	5606fe5870	Nick:	2024-08-26 16:05:11 -03:00
Gergo Moricz	d591e0f51c	block corterix.com for performance issues	2024-08-25 20:06:12 +02:00
rafaelsideguide	6f9a2687ae	fixed turndown bug	2024-08-25 15:04:32 -03:00
Nicolas	4e196a9146	Delete autoscale.yml	2024-08-25 01:48:51 -03:00
Nicolas	1f99bfd3c8	Update queue.ts	2024-08-23 22:47:12 -03:00
Nicolas	b80277d4de	Update queue.ts	2024-08-23 22:46:44 -03:00
Nicolas	d87b62fed9	Nick:	2024-08-23 22:33:17 -03:00
Nicolas	b9e06e27f4	Update queue.ts	2024-08-23 22:17:27 -03:00
Nicolas	8e78511ed4	Update queue.ts	2024-08-23 22:15:47 -03:00
Nicolas	28d7a637c2	Update queue.ts	2024-08-23 22:07:49 -03:00
Nicolas	b23bf2eef9	Update autoscale.yml	2024-08-23 20:14:35 -03:00
Nicolas	0dc592b3e4	Update autoscale.yml	2024-08-23 20:11:30 -03:00
Nicolas	173f4ee1bf	Nick: chrome cdp main | simple autoscaler	2024-08-23 20:09:59 -03:00
Eric Ciarla	732e6af8b9	Add internal link opportunities example	2024-08-23 15:49:41 -04:00
Gergő Móricz	2ab0dd2e15	fix(scrape): add further llm extraction catch	2024-08-23 19:20:17 +02:00
Nicolas	3d53f4e213	Nick: unblocking pin	2024-08-23 13:56:05 -03:00
Gergő Móricz	5ef3926d2a	fix(scrape,search): handle failed jobs	2024-08-23 18:47:56 +02:00
Gergő Móricz	52a05b8c6e	rename "dragonfly" to "redis"	2024-08-23 17:05:59 +02:00
Gergő Móricz	64e9be0cd4	feat(redis): use bitnami image	2024-08-22 23:38:04 +02:00
Gergő Móricz	8d9ff90bcb	feat(fire-engine): propagate sentry trace	2024-08-22 23:38:04 +02:00
Nicolas	1f0abacadf	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-08-22 18:30:54 -03:00
Nicolas	1f779e261a	Update rate-limiter.ts	2024-08-22 18:30:45 -03:00
Gergő Móricz	8e3c2b2855	fix(crawler): verify URL	2024-08-22 23:30:19 +02:00
Gergő Móricz	e690a6fda7	fix: remove QueueEvents	2024-08-22 22:38:39 +02:00
Gergő Móricz	76c8e9f996	fix	2024-08-22 22:24:24 +02:00
Gergő Móricz	ad82175fb8	fix(scrape): poll	2024-08-22 22:12:02 +02:00
Gergő Móricz	dd737f1235	feat(sentry): add queue instrumentation to	2024-08-22 19:17:51 +02:00
Gergő Móricz	7265ab7c67	fix(search): filter docs properly	2024-08-22 18:46:56 +02:00
Gergő Móricz	d036738da0	fix(bullmq): duplicate redis connection for QueueEvents	2024-08-22 18:04:09 +02:00
Gergő Móricz	6d48dbcd38	feat(sentry): add trace continuity for queue	2024-08-22 16:47:38 +02:00
Gergő Móricz	6d92b8524d	feat(scrape): record job result in span	2024-08-22 16:00:13 +02:00
Gergő Móricz	5ca36fe9fc	feat(api): add more captureExceptions	2024-08-22 15:49:16 +02:00
Gergő Móricz	0e8fd6ce70	fix(scrape): ensure extractionSchema is an object if llm-extraction is specified	2024-08-22 14:50:51 +02:00
Gergő Móricz	4bd2ff26d3	fix(llm-extract): pass stacktrace properly	2024-08-22 14:37:09 +02:00
Gergő Móricz	e4adbaa88e	fix(llm-extract): handle llm-extract if scrape failed	2024-08-22 14:12:52 +02:00
Gergő Móricz	670d253a8c	fix(auth): fix error reporting	2024-08-22 14:08:09 +02:00
Gergő Móricz	7d9f5bf8b1	fix(crawl): don't use sitemap if it's empty ... Fixes FIRECRAWL-SCRAPER-JS-11	2024-08-22 13:41:33 +02:00
Gergő Móricz	1f580deefc	fix(crawl): validate includes.excludes regexes	2024-08-22 13:29:11 +02:00
Gergő Móricz	fbbc3878f1	fix(crawler): make sure includes/excludes is an array	2024-08-22 13:18:26 +02:00
Gergő Móricz	508568f943	fix(search): handle scrape timeouts on search ... Fixes FIRECRAWL-SCRAPER-JS-15	2024-08-22 13:10:58 +02:00
Gergő Móricz	14fa75cae6	fix(crawl): send error if url is not a string ... Fixes FIRECRAWL-SCRAPER-JS-1E and FIRECRAWL-SCRAPER-JS-Z	2024-08-22 13:09:08 +02:00
Nicolas	8a778278a9	Merge branch 'main' into nsc/job-priority	2024-08-21 22:57:55 -03:00
Gergo Moricz	0cdf41587e	feat(sentry): add error handles to try-catch blocks	2024-08-22 03:55:40 +02:00
Nicolas	53ca704620	Update index.ts	2024-08-21 22:55:39 -03:00
Nicolas	477c3257dc	Nick:	2024-08-21 22:53:33 -03:00
Nicolas	c7bfe4ffe8	Nick:	2024-08-21 22:20:40 -03:00