101 Commits

Author	SHA1	Message	Date
Gergő Móricz	5c62bb1195	feat: new snips test framework (FIR-414) (#1033) ... * feat: new snips test framework * Update mock.ts --------- Co-authored-by: Nicolas <nicolascamara29@gmail.com>	2025-01-13 20:50:47 +01:00
Nicolas	f4d10c5031	Nick: formatting fixes	2025-01-10 18:35:10 -03:00
Gergő Móricz	d1f3b96388	feat: add scrapeId in document.metadata	2025-01-09 20:52:12 +01:00
Gergő Móricz	97bf54214f	fix(scrapeURL/loop): re-add is long enough check with lt 0	2025-01-09 18:43:50 +01:00
Móricz Gergő	3c614a2e5c	fix(scrapeURL/engines/pdf,docx): support authorization	2025-01-09 10:03:27 +01:00
Móricz Gergő	363021ea78	feat(crawl): ensure url trimming	2025-01-08 12:35:42 +01:00
Móricz Gergő	977a3e13c5	fix(scrapeURL): remove short content check	2025-01-08 11:23:25 +01:00
Nicolas	aef040b41e	Nick: from cache fixes	2025-01-03 23:07:15 -03:00
Nicolas	e8a9d8ddcd	Merge branch 'main' of https://github.com/mendableai/firecrawl	2025-01-03 22:55:42 -03:00
Nicolas	05e845a971	Update cache.ts	2025-01-03 22:55:38 -03:00
Nicolas	a4f7c38834	Nick: fixed	2025-01-03 22:15:23 -03:00
Nicolas	6b2e1cbb28	Nick: cache /extract scrapes	2025-01-03 21:19:40 -03:00
Nicolas	c9d91af86f	Merge branch 'main' into nsc/semantic-index-extract	2025-01-02 15:26:40 -03:00
Nicolas	c3fd13a82b	Nick: fixed re-ranker and enabled url cache of 2hrs	2024-12-31 18:06:07 -03:00
Nicolas	07f4b714af	Update removeUnwantedElements.ts	2024-12-31 15:23:02 -03:00
Nicolas	e6da214aeb	Nick: async background index	2024-12-30 21:42:01 -03:00
Gergő Móricz	4d1f92f4c8	fix(scrapeURL/fetch): block loopback and link-local IPs	2024-12-29 17:35:14 +01:00
Nicolas	e255301005	Update index.ts	2024-12-27 21:31:29 -03:00
Nicolas	1eca61bffb	Update index.ts	2024-12-27 20:59:18 -03:00
Nicolas	f9d55efba8	Update index.ts	2024-12-27 20:54:26 -03:00
Nicolas	b8d7f9f257	Nick: we are using runpod	2024-12-27 19:59:05 -03:00
Nicolas	5fcf3fa97e	Merge branch 'main' into mog/mineru	2024-12-27 19:53:09 -03:00
Nicolas	05d5f84d87	Merge pull request #1018 from mendableai/feat/add-favicon-metadata ... [FIR-37] feat: extract and return favicon URL during scraping	2024-12-27 17:44:03 -03:00
Nicolas	eba5fda9a1	Merge pull request #955 from mendableai/rafa/fix-default-on-schema-llm-extract ... fixed optional+default bug on llm schema	2024-12-27 16:33:04 -03:00
Ademílson F. Tonato	a4cf814f70	feat: return favicon url when scraping	2024-12-27 19:18:53 +00:00
Gergő Móricz	4772951313	feat(scrapeURL/fire-engine): explicitly delete job after scrape	2024-12-27 16:44:41 +01:00
Gergő Móricz	0b55fb836b	feat(scrapeURL/pdf): switch to MinerU	2024-12-27 16:37:32 +01:00
Gergő Móricz	c543f4f76c	feat(scrapeURL/pdf): update mock Blob implementation to pass TypeScript	2024-12-26 20:31:51 +01:00
Gergő Móricz	f15ef0e758	feat(scrapeURL/fire-engine/chrome-cdp): handle file downloads	2024-12-26 20:29:09 +01:00
Nicolas	ed24853ca6	Merge pull request #996 from mendableai/fix/title-extra-info ... [BUG] fixed title extra info	2024-12-19 16:05:49 -03:00
Gergő Móricz	071b9a01c3	fix(scrapeURL/fire-engine): pass geolocation	2024-12-19 18:23:21 +01:00
rafaelmmiller	cf2ec77131	fixed title extra info	2024-12-19 08:32:10 -03:00
Nicolas	066071cd54	Update llmExtract.ts	2024-12-18 23:45:43 -03:00
Nicolas	e899ecbe44	Update llmExtract.ts	2024-12-18 16:52:05 -03:00
Nicolas	3b6edef9fa	chore: formatting	2024-12-17 16:58:57 -03:00
Eric Ciarla	a20a003c74	revert to pdf parse	2024-12-17 12:12:22 -05:00
Eric Ciarla	1402831a0a	Replace pdf parse with pdf to md	2024-12-17 09:59:52 -05:00
Eric Ciarla	ed7d15d2af	Update index.ts	2024-12-17 09:50:29 -05:00
Gergő Móricz	654d6c6e0b	fix(scrapeURL): increase timeToRun	2024-12-17 13:21:24 +01:00
Gergő Móricz	47b968fede	fix(scrapeURL/fire-engine): timeout calculation issues	2024-12-17 13:17:55 +01:00
Gergő Móricz	7f57c868be	Revert "fix(scrapeURL): better timeToRun distribution" ... This reverts commit 284a6ccedd1baede825571ee933eb7e4f773e2de.	2024-12-16 23:08:20 +01:00
Gergő Móricz	284a6ccedd	fix(scrapeURL): better timeToRun distribution	2024-12-16 23:01:34 +01:00
rafaelmmiller	b6802bc443	merged with main	2024-12-16 11:41:59 -03:00
Rafael Miller	8192d756e9	Merge branch 'main' into rafa/fix-default-on-schema-llm-extract	2024-12-16 09:33:36 -03:00
rafaelmmiller	eab30c474b	added unit tests	2024-12-16 09:30:40 -03:00
Nicolas	1214d219e1	Nick: fix actions errors	2024-12-15 15:43:12 -03:00
Gergő Móricz	0f3a27bf27	fix(scrapeURL/engines): better timeouts	2024-12-15 18:58:29 +01:00
Nicolas	a5256827c0	Update index.ts	2024-12-15 14:36:09 -03:00
Gergő Móricz	b4a5e1a6e9	fix(scrapeURL/fire-engine): timeout handling	2024-12-15 16:04:17 +01:00
Gergő Móricz	afbd01299a	fix(scrapeURL/fire-engine): timeouts	2024-12-15 15:58:27 +01:00