2251 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
ad-angelo	4248c68f5a	Add Mobile Scraping ... https://www.firecrawl.dev/blog/launch-week-ii-day-6-introducing-mobile-scraping	2024-11-20 21:14:49 +01:00
Gergő Móricz	b1eaecfdb0	fix 2	2024-11-20 20:19:16 +01:00
Gergő Móricz	e2ddc6c65c	fix handling of badly formatted URLs	2024-11-20 20:18:40 +01:00
Gergő Móricz	ba6f29cdda	crawl fix, again	2024-11-20 19:55:35 +01:00
Gergő Móricz	b468bb4014	crawl fixes	2024-11-20 19:48:01 +01:00
Nicolas	103c3f28e6	Update rate-limiter.ts	2024-11-19 17:51:31 -08:00
Eric Ciarla	aa01c0b684	Create mastering-the-crawl-endpoint.ipynb	2024-11-19 12:50:31 -05:00
Gergő Móricz	79a75e088a	feat(crawl): allowSubdomain	2024-11-19 18:38:59 +01:00
Nicolas	91caa01c5e	Update CONTRIBUTING.md	2024-11-18 16:13:24 -08:00
Nicolas	1328ae0fa3	Update README.md	2024-11-18 15:49:19 -08:00
Eric Ciarla	a31336752c	Create README.md	2024-11-18 14:04:29 -05:00
rafaelmmiller	77e152cba8	added team_id to scrape-status endpoint	2024-11-18 15:02:00 -03:00
Gergő Móricz	31a0471bfa	fix(crawl-redis): ordered push to wrong side of list	2024-11-15 21:56:15 +01:00
Gergő Móricz	1a0f13c0eb	fix(webhook): add logging	2024-11-15 21:43:02 +01:00
Gergő Móricz	1b032b05fa	fix(map): make sitemapOnly simpler	2024-11-15 21:14:32 +01:00
Gergő Móricz	a4d3dba865	fix(map): ignore limit when using sitemapOnly	2024-11-15 21:03:20 +01:00
Gergő Móricz	63787bc504	fix(scrapeURL/fire-engine): wait longer if timeout is not specified	2024-11-15 20:25:16 +01:00
Gergő Móricz	4cddcd5206	fix(scrapeURL/fire-engine): timeout-less scrape support (initial)	2024-11-15 20:15:25 +01:00
Gergő Móricz	350d00d27a	fix(crawler): treat XML files as sitemaps (temporarily)	2024-11-15 20:09:20 +01:00
Gergő Móricz	ca2e33db0a	fix(log_job): add force option to retry on supabase failure	2024-11-15 19:55:23 +01:00
Gergő Móricz	7b02c45dd0	fix(v1/types): better timeout primitives	2024-11-15 19:35:54 +01:00
Gergő Móricz	c95a4a26c9	fix(v1/batch/scrape): raise default timeout	2024-11-15 18:58:03 +01:00
Móricz Gergő	3a342bfbf0	fix(scrapeURL/playwright): JSON body fix	2024-11-15 15:18:40 +01:00
Nicolas	3c1b1909f8	Update map.ts	2024-11-14 17:52:15 -05:00
Nicolas	9519897102	Merge branch 'nsc/sitemap-only'	2024-11-14 17:44:39 -05:00
Nicolas	7f084c6c43	Nick:	2024-11-14 17:44:32 -05:00
Nicolas	e8bd089c8a	Merge pull request #901 from mendableai/nsc/sitemap-only ... Allows `/map` to only return links present in the sitemap	2024-11-14 17:32:37 -05:00
Nicolas	3fcdf57d2f	Update fireEngine.ts	2024-11-14 17:31:30 -05:00
Nicolas	d62f12c9d9	Nick: moved away from axios	2024-11-14 17:31:23 -05:00
Nicolas	f155449458	Nick: sitemap only	2024-11-14 17:29:53 -05:00
Móricz Gergő	431e64e752	fix(batch/scrape/webhook): add batch_scrape.started	2024-11-14 22:40:03 +01:00
Nicolas	7bca4486b4	Update package.json	2024-11-14 16:37:53 -05:00
Móricz Gergő	df05124ef5	feat(v1/batch/scrape): webhooks	2024-11-14 22:36:28 +01:00
Gergő Móricz	86a78a03cb	fix(sitemap): scrape with tlsclient	2024-11-14 18:56:32 +01:00
Eric Ciarla	62c8b63b84	Create README.md	2024-11-14 11:55:00 -05:00
Móricz Gergő	5519f077aa	fix(scrapeURL): adjust error message for clarity	2024-11-14 10:13:48 +01:00
Móricz Gergő	0a1c99074f	fix(html-to-markdown): make error reporting less intrusive	2024-11-14 08:58:00 +01:00
Nicolas	bd928b1512	Nick: changed email from hello to help	2024-11-13 20:27:20 -05:00
Gergő Móricz	0310cd2afa	fix(crawl): redirect rebase	2024-11-13 21:38:44 +01:00
Nicolas	0d1c4e4e09	Update package.json	2024-11-13 13:54:22 -05:00
Gergő Móricz	32be2cf786	feat(v1/webhook): complex webhook object w/ headers (#899) ... * feat(v1/webhook): complex webhook object w/ headers * feat(js-sdk/crawl): add complex webhook support	2024-11-13 19:36:44 +01:00
Nicolas	ea1302960f	Merge pull request #895 from mendableai/nsc/redlock-email ... Redlock for sending email notifications	2024-11-13 12:45:55 -05:00
Nicolas	1a636b4e59	Update email_notification.ts	2024-11-12 20:09:01 -05:00
Gergő Móricz	5ce4aaf0ec	fix(crawl): initialURL setting is unnecessary	2024-11-12 23:35:07 +01:00
Gergő Móricz	93ac20f930	fix(queue-worker): do not kill crawl on one-page error	2024-11-12 22:53:29 +01:00
Gergő Móricz	16e850288c	fix(scrapeURL/pdf,docx): ignore SSL when downloading PDF	2024-11-12 22:46:58 +01:00
Gergő Móricz	7081beff1f	fix(scrapeURL/pdf): retry	2024-11-12 22:26:36 +01:00
Gergő Móricz	9ace2ad071	fix(scrapeURL/pdf): fix llamaparse upload	2024-11-12 20:55:14 +01:00
Gergő Móricz	687ea69621	fix(requests.http): default to localhost baseUrl	2024-11-12 19:59:09 +01:00
Gergő Móricz	3a5eee6e3f	feat: improve requests.http using format features	2024-11-12 19:58:07 +01:00