2271 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Nicolas	d02a8bcb82	Nick: extract urls to extract	2024-11-19 13:49:23 -08:00
rafaelmmiller	2fb8a3c8dc	fix schema	2024-11-19 10:04:42 -03:00
rafaelmmiller	53134b7c85	Rafa: removed throw error and added map to requests	2024-11-19 09:34:52 -03:00
rafaelmmiller	36cf49c959	Merge remote-tracking branch 'origin/main' into nsc/new-extract	2024-11-19 09:34:08 -03:00
Eric Ciarla	a31336752c	Create README.md	2024-11-18 14:04:29 -05:00
rafaelmmiller	77e152cba8	added team_id to scrape-status endpoint	2024-11-18 15:02:00 -03:00
Gergő Móricz	31a0471bfa	fix(crawl-redis): ordered push to wrong side of list	2024-11-15 21:56:15 +01:00
Gergő Móricz	1a0f13c0eb	fix(webhook): add logging	2024-11-15 21:43:02 +01:00
Gergő Móricz	1b032b05fa	fix(map): make sitemapOnly simpler	2024-11-15 21:14:32 +01:00
Gergő Móricz	a4d3dba865	fix(map): ignore limit when using sitemapOnly	2024-11-15 21:03:20 +01:00
Gergő Móricz	63787bc504	fix(scrapeURL/fire-engine): wait longer if timeout is not specified	2024-11-15 20:25:16 +01:00
Gergő Móricz	4cddcd5206	fix(scrapeURL/fire-engine): timeout-less scrape support (initial)	2024-11-15 20:15:25 +01:00
Gergő Móricz	350d00d27a	fix(crawler): treat XML files as sitemaps (temporarily)	2024-11-15 20:09:20 +01:00
Gergő Móricz	ca2e33db0a	fix(log_job): add force option to retry on supabase failure	2024-11-15 19:55:23 +01:00
Gergő Móricz	7b02c45dd0	fix(v1/types): better timeout primitives	2024-11-15 19:35:54 +01:00
Gergő Móricz	c95a4a26c9	fix(v1/batch/scrape): raise default timeout	2024-11-15 18:58:03 +01:00
Móricz Gergő	3a342bfbf0	fix(scrapeURL/playwright): JSON body fix	2024-11-15 15:18:40 +01:00
Nicolas	3c1b1909f8	Update map.ts	2024-11-14 17:52:15 -05:00
Nicolas	9519897102	Merge branch 'nsc/sitemap-only'	2024-11-14 17:44:39 -05:00
Nicolas	7f084c6c43	Nick:	2024-11-14 17:44:32 -05:00
Nicolas	e8bd089c8a	Merge pull request #901 from mendableai/nsc/sitemap-only ... Allows `/map` to only return links present in the sitemap	2024-11-14 17:32:37 -05:00
Nicolas	3fcdf57d2f	Update fireEngine.ts	2024-11-14 17:31:30 -05:00
Nicolas	d62f12c9d9	Nick: moved away from axios	2024-11-14 17:31:23 -05:00
Nicolas	f155449458	Nick: sitemap only	2024-11-14 17:29:53 -05:00
Móricz Gergő	431e64e752	fix(batch/scrape/webhook): add batch_scrape.started	2024-11-14 22:40:03 +01:00
Nicolas	7bca4486b4	Update package.json	2024-11-14 16:37:53 -05:00
Móricz Gergő	df05124ef5	feat(v1/batch/scrape): webhooks	2024-11-14 22:36:28 +01:00
Nicolas	91f4fd815f	Update extract.ts	2024-11-14 15:41:42 -05:00
Nicolas	22848af5ae	Nick:	2024-11-14 15:34:02 -05:00
Nicolas	ebe9de2ac5	Nick:	2024-11-14 15:26:15 -05:00
Nicolas	5056dcd8e9	Update index.test.ts	2024-11-14 15:06:22 -05:00
rafaelmmiller	be5e6da5ff	tests	2024-11-14 17:03:54 -03:00
rafaelmmiller	45091430ab	Merge branch 'nsc/new-extract' of https://github.com/mendableai/firecrawl into nsc/new-extract	2024-11-14 17:03:15 -03:00
Nicolas	796cd0746d	Update extract.ts	2024-11-14 15:03:06 -05:00
Nicolas	1b5f6a0959	Update extract.ts	2024-11-14 14:59:34 -05:00
Nicolas	d6749c211d	Nick: refactor and /* glob pattern support	2024-11-14 14:57:38 -05:00
rafaelmmiller	41b45a844b	sdk allowexternallinks	2024-11-14 15:56:12 -03:00
rafaelmmiller	3d6d650f0b	Merge branch 'nsc/new-extract' of https://github.com/mendableai/firecrawl into nsc/new-extract	2024-11-14 15:51:32 -03:00
rafaelmmiller	80d6cb16fb	sdks wip	2024-11-14 15:51:27 -03:00
Gergő Móricz	359c30fbda	fix(cache): don't cache on failure error code	2024-11-14 19:49:34 +01:00
Gergő Móricz	49ff37afb4	feat: cache	2024-11-14 19:47:12 +01:00
Gergő Móricz	86a78a03cb	fix(sitemap): scrape with tlsclient	2024-11-14 18:56:32 +01:00
Eric Ciarla	62c8b63b84	Create README.md	2024-11-14 11:55:00 -05:00
Móricz Gergő	5519f077aa	fix(scrapeURL): adjust error message for clarity	2024-11-14 10:13:48 +01:00
Móricz Gergő	0a1c99074f	fix(html-to-markdown): make error reporting less intrusive	2024-11-14 08:58:00 +01:00
Nicolas	bd928b1512	Nick: changed email from hello to help	2024-11-13 20:27:20 -05:00
Nicolas	a1c018fdb0	Merge branch 'main' into nsc/new-extract	2024-11-13 17:14:43 -05:00
rafaelmmiller	904c904971	wip	2024-11-13 18:06:20 -03:00
Gergő Móricz	0310cd2afa	fix(crawl): redirect rebase	2024-11-13 21:38:44 +01:00
Nicolas	0d1c4e4e09	Update package.json	2024-11-13 13:54:22 -05:00