2680 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Nicolas	e8a9d8ddcd	Merge branch 'main' of https://github.com/mendableai/firecrawl	2025-01-03 22:55:42 -03:00
Nicolas	05e845a971	Update cache.ts	2025-01-03 22:55:38 -03:00
Nicolas	6bfdbcdba9	Merge pull request #1037 from mendableai/nsc/semantic-index-extract ... WIP Indexes, Caching for /extract, Improvements	2025-01-03 22:52:22 -03:00
Nicolas	c655c6859f	Nick: fixed	2025-01-03 22:50:53 -03:00
Nicolas	a4f7c38834	Nick: fixed	2025-01-03 22:15:23 -03:00
Nicolas	8df1c67961	Update queue-worker.ts	2025-01-03 21:48:28 -03:00
Nicolas	499479c85e	Update url-processor.ts	2025-01-03 21:28:52 -03:00
Nicolas	432b410678	Update queue-worker.ts	2025-01-03 21:26:05 -03:00
Nicolas	6b2e1cbb28	Nick: cache /extract scrapes	2025-01-03 21:19:40 -03:00
Nicolas	27457ed5db	Nick: init	2025-01-03 20:44:27 -03:00
Nicolas	81cf05885b	Merge branch 'main' into nsc/semantic-index-extract	2025-01-03 19:57:29 -03:00
Rafael Miller	a54a5dbb45	removed warnings	2025-01-03 17:13:34 -03:00
Rafael Miller	12cd9f083c	removed warnings	2025-01-03 17:12:30 -03:00
rafaelmmiller	b223f6ae68	Merge branch 'sdk/fixed-none-undefined-on-response' of https://github.com/mendableai/firecrawl into sdk/fixed-none-undefined-on-response	2025-01-03 13:56:46 -03:00
rafaelmmiller	55dad5ea13	fixed empty data with next causing infinite loop	2025-01-03 13:56:39 -03:00
Rafael Miller	2e53eb98ce	Merge branch 'main' into sdk/fixed-none-undefined-on-response	2025-01-03 13:27:52 -03:00
rafaelmmiller	bafcc008bc	[SDK] fixed none and undefined on response	2025-01-03 13:27:00 -03:00
Nicolas	87757d9b8e	Nick: fixed schemas on extract for node	2025-01-03 13:19:08 -03:00
Nicolas	ad49503f8a	Update search.ts	2025-01-02 21:15:47 -03:00
Nicolas	cbe0716439	Update search.ts	2025-01-02 21:13:24 -03:00
Nicolas	e37ab8431a	Update search.ts	2025-01-02 21:07:14 -03:00
Nicolas	8b64e915b3	Update search.ts	2025-01-02 21:02:55 -03:00
Nicolas	7ce780ac81	Update search.ts	2025-01-02 20:40:38 -03:00
Nicolas	b244afbc82	Update README.md	2025-01-02 20:26:51 -03:00
Nicolas	a4b6dfecd1	Nick: v1.8.0 - added /v1/search support v1.2.0	2025-01-02 20:02:07 -03:00
Nicolas	b61a1ccfd3	Merge pull request #1032 from mendableai/nsc/v1-search ... (feat/v1) Search	2025-01-02 19:58:44 -03:00
Nicolas	21bf89b6cc	Update search.ts	2025-01-02 19:57:51 -03:00
Nicolas	22ae1730bd	Update search.ts	2025-01-02 19:57:41 -03:00
Nicolas	a0dbf20c40	Update types.ts	2025-01-02 19:55:28 -03:00
Nicolas	25da20efd2	Nick: e2e	2025-01-02 19:53:54 -03:00
Nicolas	eae393afb5	Nick: fixed js sdk	2025-01-02 19:52:50 -03:00
Nicolas	07a6ba5d91	Nick:	2025-01-02 19:34:37 -03:00
Nicolas	35d7202894	Update search.ts	2025-01-02 19:33:21 -03:00
Nicolas	d2742bec4d	Nick: v1 search	2025-01-02 19:31:03 -03:00
rafaelmmiller	ef0fc8d0d3	broader search if didnt find results	2025-01-02 18:00:18 -03:00
Nicolas	c9d91af86f	Merge branch 'main' into nsc/semantic-index-extract	2025-01-02 15:26:40 -03:00
Nicolas	c822e34d37	Nick: fixed extract schema	2025-01-02 14:03:23 -03:00
Nicolas	c3fd13a82b	Nick: fixed re-ranker and enabled url cache of 2hrs	2024-12-31 18:06:07 -03:00
Nicolas	07f4b714af	Update removeUnwantedElements.ts	2024-12-31 15:23:02 -03:00
Nicolas	33632d2fe3	Update extraction-service.ts	2024-12-31 15:22:50 -03:00
Nicolas	27cc3dba30	Nick: rm vscode settings	2024-12-31 12:53:07 -03:00
Nicolas	bd81b41d5f	Update queue-worker.ts	2024-12-30 21:43:59 -03:00
Nicolas	e6da214aeb	Nick: async background index	2024-12-30 21:42:01 -03:00
Nicolas	7a31306be5	Nick: url normalization + max metadata size	2024-12-30 20:04:22 -03:00
Nicolas	bf9d41d0b2	Nick: index exploration	2024-12-30 19:37:48 -03:00
Nicolas	0847a6038e	Merge pull request #1014 from mendableai/nsc/extract-url-trace ... /extract URL trace	2024-12-30 19:00:58 -03:00
Gergő Móricz	71a8f7452c	fix(WebScraper/sitemap): await urlsHandler to fix race condition v1.1.1	2024-12-30 16:09:22 +01:00
Nicolas	8ae34a0d31	Nick: rm .xml from isFile	2024-12-30 11:57:01 -03:00
Gergő Móricz	9005757de3	fix(queue-worker): do not follow redirect URLs if they are not allowed by the crawl options	2024-12-30 14:41:31 +01:00
Gergő Móricz	4d1f92f4c8	fix(scrapeURL/fetch): block loopback and link-local IPs	2024-12-29 17:35:14 +01:00