1112 Commits

Author	SHA1	Message	Date
Nicolas	a9773a24a3	Nick: increased timeout for chrome-cdp due to smart wait	2024-09-25 19:27:02 -04:00
Nicolas	1da026b26e	Update single_url.ts	2024-09-24 23:29:48 -04:00
Nicolas	b8266cc329	Update website_params.ts	2024-09-24 23:28:58 -04:00
Gergő Móricz	f00c0b82f9	fix(v1/scrape): add total wait specified in request to timeout	2024-09-24 21:56:22 +02:00
Nicolas	3f138e559e	Update website_params.ts	2024-09-24 15:14:26 -04:00
Gergő Móricz	43730b5db6	feat(WebScraper): always report error of last scraper in order	2024-09-24 20:03:49 +02:00
Gergő Móricz	3e661a2087	fix(v1/crawl-cancel): avoid double authing	2024-09-24 20:01:34 +02:00
Gergő Móricz	4194525640	fix(blocklist): unblock TikTok Business page ... This is just a regular business site, not social media.	2024-09-24 16:55:19 +02:00
Gergő Móricz	4a623c084a	fix(fly): don't use Depot builders (doesn't work)	2024-09-24 10:50:30 +02:00
Gergő Móricz	a59b5836d5	Revert error tallying	2024-09-24 10:27:49 +02:00
Gergő Móricz	483f97d21b	fix(v0/search): don't sent scrape fail errors to Sentry	2024-09-23 18:49:27 +02:00
Gergő Móricz	d927cafeea	fix(queue-worker): don't send scraping errors to sentry	2024-09-23 18:48:01 +02:00
Gergő Móricz	677faa27f3	fix(WebScraper): explicitly ignore 404s	2024-09-23 18:47:07 +02:00
Gergő Móricz	83d8287c14	fix(v0, sentry): don't send all scraping methods failed errors to Sentry	2024-09-23 18:40:21 +02:00
Gergő Móricz	d2f7031069	fix(WebScraper): fatal error handler triggering for 404s	2024-09-23 18:33:10 +02:00
Nicolas	dfdbae74c6	Update fireEngine.ts	2024-09-21 21:10:05 -04:00
Nicolas	db161ac55a	Nick: press + write	2024-09-20 19:45:23 -04:00
Nicolas	3fc5ce17d2	Nick: fixed error handling for v0 scrape	2024-09-20 18:35:30 -04:00
Nicolas	0690cfeaad	Merge branch 'main' into feat/actions	2024-09-20 18:24:13 -04:00
Gergő Móricz	e1a34b0a99	Revert "feat(scrape): scroll down/up with actions if fullpagescreenshot" ... This reverts commit 815bfc8f07b3493ed91ff90afda123e7d84f44f9.	2024-09-20 21:43:22 +02:00
Gergő Móricz	815bfc8f07	feat(scrape): scroll down/up with actions if fullpagescreenshot ... revert this if unneeded	2024-09-20 21:42:09 +02:00
Gergő Móricz	d663bbf0ca	feat(actions): add scroll	2024-09-20 21:41:53 +02:00
Gergő Móricz	3dd912ec91	feat(actions): add typeText, pressKey, fix playwright screenshot/waitFor	2024-09-20 21:02:53 +02:00
Gergő Móricz	01f42b980d	feat(scrape): add error tallying instead of empty response	2024-09-19 19:21:13 +02:00
Nicolas	b8b8522b33	Nick: fixed map exception	2024-09-19 12:49:33 -04:00
Nicolas	ef71940429	Merge pull request #674 from mendableai/nsc/map-pagination ... v1/map improvements + higher limits	2024-09-19 11:31:23 -04:00
Nicolas	00bb958b32	Update v1-openapi.json	2024-09-18 15:59:11 -04:00
Nicolas	03eb47d426	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-09-18 15:58:54 -04:00
Nicolas	3a4dd8fc7e	Nick: v1 openapi spec	2024-09-18 15:58:53 -04:00
Gergő Móricz	712ca31615	minor fixes	2024-09-18 21:34:09 +02:00
Gergő Móricz	093c064bff	feat(v1): add public actions api	2024-09-18 20:39:25 +02:00
Gergő Móricz	42d677fe3c	feat(fire-engine): port waitFor and screenshot to use actions	2024-09-18 20:04:54 +02:00
Gergő Móricz	c28e1e2959	fix(v1/zod): formats: add error message if both sc and sc@fP	2024-09-18 19:35:38 +02:00
Nicolas	3c2bfe2da2	Update single_url.ts	2024-09-17 01:58:47 -04:00
Nicolas	18b024c238	Update single_url.ts	2024-09-17 01:41:46 -04:00
Nicolas	a4039bd008	Revert "Update single_url.ts" ... This reverts commit 0f8c0a570dca877d14d590e6002eaffd345a3927.	2024-09-16 23:36:38 -04:00
Nicolas	0f8c0a570d	Update single_url.ts	2024-09-16 21:44:56 -04:00
Nicolas	9da3432596	Update map.ts	2024-09-16 12:03:14 -04:00
Nicolas	af4804e13b	Merge branch 'main' into nsc/map-pagination	2024-09-16 11:04:44 -04:00
Nicolas	030ecab6ee	Update rate-limiter.ts	2024-09-13 18:09:59 -04:00
Gergo Moricz	000a316cc3	fix(fire-engine): poll more frequently	2024-09-13 16:41:27 +02:00
Gergo Moricz	f7c4cee404	fix(queue-worker): don't send LLM extract hallucination error to Sentry	2024-09-13 14:02:49 +02:00
Gergő Móricz	d497284b40	feat(api/queue): auto-remove completed jobs after 25 hours	2024-09-12 19:47:22 +02:00
Nicolas	620b02f9ca	Nick:	2024-09-12 12:51:14 -04:00
Nicolas	e1171ade1b	Merge branch 'main' into mog/fix-status-job-get	2024-09-12 11:35:36 -04:00
Nicolas	4cd1065ae2	Update rate-limiter.ts	2024-09-11 14:03:34 -04:00
Gergő Móricz	97ffabff3a	fix(v1): converting bad docs always gives null	2024-09-10 21:21:20 +02:00
Gergő Móricz	83a165db0f	fix(v0/scrape): ensure url is string	2024-09-10 21:18:53 +02:00
Gergő Móricz	ad1a6fbc74	fix(v1/map): handle invalid URLs gracefully	2024-09-10 19:41:01 +02:00
Gergő Móricz	4ebc35c9dd	fix(crawl-status): add success: true	2024-09-10 18:59:09 +02:00