2623 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Gergő Móricz	d1f3b96388	feat: add scrapeId in document.metadata v1.2.1	2025-01-09 20:52:12 +01:00
Gergő Móricz	29c1f126ab	feat(scrape-status): adapt	2025-01-09 19:14:00 +01:00
Gergő Móricz	2849ce2f13	fix(queue-worker): errored job logging	2025-01-09 18:48:47 +01:00
Gergő Móricz	97bf54214f	fix(scrapeURL/loop): re-add is long enough check with lt 0	2025-01-09 18:43:50 +01:00
Gergő Móricz	0da386914d	fix(queue-worker): graceful shutdown	2025-01-09 16:04:59 +01:00
Móricz Gergő	3c614a2e5c	fix(scrapeURL/engines/pdf,docx): support authorization	2025-01-09 10:03:27 +01:00
Móricz Gergő	49e584f8e1	fix(queue-worker/crawl): use SCARD to generate num_docs field	2025-01-09 09:51:34 +01:00
Móricz Gergő	9e8c629ff4	fix(log_job): don't redact with auth header	2025-01-09 09:51:34 +01:00
Nicolas	14f696805c	Update auth.ts	2025-01-08 17:04:57 -03:00
Nicolas	51cb4b1615	Nick: temp rl for /extract	2025-01-08 15:24:38 -03:00
Nicolas	a199208e21	Update rate-limiter.ts	2025-01-08 15:15:21 -03:00
Nicolas	aa31508ccd	Nick: links-billed update (temp)	2025-01-08 15:13:33 -03:00
Móricz Gergő	363021ea78	feat(crawl): ensure url trimming	2025-01-08 12:35:42 +01:00
Móricz Gergő	977a3e13c5	fix(scrapeURL): remove short content check	2025-01-08 11:23:25 +01:00
Nicolas	0a41fdd35d	Merge branch 'nsc/extract-queue'	2025-01-07 18:21:57 -03:00
Nicolas	7918d0e1c9	Nick: bump 1.12.0	2025-01-07 18:20:56 -03:00
Nicolas	f82a742cd1	Merge pull request #1044 from mendableai/nsc/extract-queue ... (feat/extract) Move extract to a queue system	2025-01-07 18:10:46 -03:00
Nicolas	b98e289f03	Nick:	2025-01-07 17:49:21 -03:00
Nicolas	a185c05a5c	Nick: sdk async and get status	2025-01-07 17:27:40 -03:00
Nicolas	9ec08d7020	Nick: fixed the sdks	2025-01-07 17:20:49 -03:00
Nicolas	dd14744850	Update types.ts	2025-01-07 16:55:55 -03:00
Nicolas	9fdcfb9314	Update index.ts	2025-01-07 16:24:46 -03:00
Nicolas	51636352a6	Merge branch 'nsc/extract-queue' of https://github.com/mendableai/firecrawl into nsc/extract-queue	2025-01-07 16:21:58 -03:00
Nicolas	11af214db1	Nick: update extract in case there is an error	2025-01-07 16:21:51 -03:00
Gergő Móricz	1f2a76fc23	Update apps/api/src/lib/extract/extraction-service.ts	2025-01-07 20:18:10 +01:00
Nicolas	eb254547e5	Nick:	2025-01-07 16:16:01 -03:00
Gergő Móricz	c6a63793bb	crawl incomplete issues	2025-01-07 19:38:17 +01:00
Gergő Móricz	ccfada98ca	various queue fixes	2025-01-07 19:15:23 +01:00
Nicolas	86e34d7c6c	Nick: wip	2025-01-07 12:13:12 -03:00
Móricz Gergő	7a03275575	add comment	2025-01-07 13:57:47 +01:00
Móricz Gergő	7d73ebdbf1	fix(crawl): never invalidate first crawl scrape if redirects	2025-01-07 13:57:23 +01:00
Móricz Gergő	b96b97ed72	fix(crawl): don't push rawhtml to db unless requested	2025-01-07 10:09:15 +01:00
Móricz Gergő	35d1d85978	fix(crawler): also take the hostname of the base url when determining isInternalLink	2025-01-07 09:29:58 +01:00
Nicolas	bb27594443	Merge branch 'main' into nsc/extract-queue	2025-01-06 13:01:15 -03:00
Nicolas	b82cfa8540	Merge pull request #1038 from 1101-1/add_actual_random_useragent ... feat: use new random user agent instead of the old one	2025-01-06 11:51:15 -03:00
Kirill	736c3675b6	use new agent generation instead of expired one	2025-01-05 17:07:14 +04:00
Nicolas	ceb2104960	Merge pull request #1034 from mendableai/sdk/fixed-none-undefined-on-response ... [SDK] fixed none and undefined on response	2025-01-04 16:31:41 -03:00
Gergő Móricz	461842fe8c	fix(v1/crawl-status): handle job's returnvalue being explicitly null (db race)	2025-01-04 17:24:33 +01:00
Gergő Móricz	b92a4eb79b	fix(queue-worker): only do redirect handling logic on crawls, not batch scrape	2025-01-04 16:59:35 +01:00
Nicolas	d48ddb8820	Update canonical-url.test.ts	2025-01-03 23:55:05 -03:00
Nicolas	f2e0bfbfe3	Nick: url normalization	2025-01-03 23:54:03 -03:00
Nicolas	f25c0c6d21	Nick: added canonical tests	2025-01-03 23:16:33 -03:00
Nicolas	aef040b41e	Nick: from cache fixes	2025-01-03 23:07:15 -03:00
Nicolas	e8a9d8ddcd	Merge branch 'main' of https://github.com/mendableai/firecrawl	2025-01-03 22:55:42 -03:00
Nicolas	05e845a971	Update cache.ts	2025-01-03 22:55:38 -03:00
Nicolas	6bfdbcdba9	Merge pull request #1037 from mendableai/nsc/semantic-index-extract ... WIP Indexes, Caching for /extract, Improvements	2025-01-03 22:52:22 -03:00
Nicolas	c655c6859f	Nick: fixed	2025-01-03 22:50:53 -03:00
Nicolas	a4f7c38834	Nick: fixed	2025-01-03 22:15:23 -03:00
Nicolas	8df1c67961	Update queue-worker.ts	2025-01-03 21:48:28 -03:00
Nicolas	499479c85e	Update url-processor.ts	2025-01-03 21:28:52 -03:00