437 Commits

Author	SHA1	Message	Date
Nicolas	1bbfb98d7e	Merge pull request #186 from Keredu/main ... Limit on /search is not deterministic	2024-05-26 18:08:16 -07:00
Nicolas	7e2df7bd5e	Update auth.ts	2024-05-26 18:07:21 -07:00
Simon H	115204e6b6	Feat: Provide more details for 429 error msg ... - Added better error code for when rate limit exceeded including consumed/remaining points, reset date and retry-after seconds	2024-05-25 12:03:20 -04:00
Keredu	2192978f91	Limit on /search is not deterministic	2024-05-25 00:12:26 +02:00
Nicolas	e98434606d	Update blocklist.ts	2024-05-24 15:04:15 -07:00
Nicolas	e5c8719554	Update blocklist.ts	2024-05-24 14:53:04 -07:00
rafaelsideguide	d39860c08b	Merge branch 'main' into feat/idempotency-key	2024-05-24 14:15:37 -03:00
Nicolas	53a214cefb	Merge pull request #168 from mendableai/nsc/allowed-keywords-in-blocklist ... feat: Allow privacy/legal/ other pages in social media websites	2024-05-24 09:43:15 -07:00
Jakob Stadlhuber	9fc5a0ff98	Update comment in .env.example for proxy settings ... This commit modifies the comment in .env.example to specify that proxy settings are for Playwright. This clarification aims to provide users a more clear context about when and why these proxy settings are used.	2024-05-24 17:45:59 +02:00
Jakob Stadlhuber	b001aded46	Add proxy and media blocking configurations ... Updated environment variables and application settings to include proxy configurations and media blocking option. The proxy settings allow users to use a proxy service, while the media blocking is an optional feature that can help save bandwidth. Changes have been made in the .env.example, docker-compose.yaml, and main.py files.	2024-05-24 17:41:34 +02:00
rafaelsideguide	35927a65a5	Merge branch 'main' into feat/idempotency-key	2024-05-23 12:20:06 -03:00
rafaelsideguide	184e4678f1	bugfix on idempotency key check	2024-05-23 11:47:04 -03:00
rafaelsideguide	aa6df4305e	crawl load tests 6 and 7	2024-05-22 18:20:24 -03:00
rafaelsideguide	73f1d09d39	Update website_params.ts	2024-05-22 15:07:12 -03:00
rafaelsideguide	4dfc371241	Update index.test.ts	2024-05-22 14:38:41 -03:00
rafaelsideguide	f4a3469b9e	Merge branch 'main' into bug/crawl-limit	2024-05-22 14:27:28 -03:00
Nicolas	0d187f0425	Merge pull request #77 from tractorjuice/patch-1 ... Add additional file extensions to crawler.ts	2024-05-22 10:16:49 -07:00
rafaelsideguide	04a0bef0fb	Merge branch 'main' into test/load-testing	2024-05-22 11:26:19 -03:00
rafaelsideguide	e4573c08ca	Update website_params.ts	2024-05-22 11:24:48 -03:00
rafaelsideguide	068a240ab4	load tests for scrape route	2024-05-22 09:30:32 -03:00
Nicolas	cb2bd0e71f	Update index.test.ts	2024-05-21 19:03:32 -07:00
Nicolas	253abb849f	Update rate-limiter.ts	2024-05-21 18:53:58 -07:00
Nicolas	229b9908d2	Nick: only enable hyper dx in prod	2024-05-21 18:52:46 -07:00
Nicolas	a8ff295977	Update single_url.ts	2024-05-21 18:50:42 -07:00
Nicolas	a5e718b084	Nick: improvements	2024-05-21 18:34:23 -07:00
Nicolas	6285f12cd1	Merge pull request #167 from mendableai/nsc/hyper-dx-integration ... feat: HyperDX Integration	2024-05-21 13:19:38 -07:00
rafaelsideguide	75f4e34d8e	Merge branch 'main' into test/load-testing	2024-05-21 10:28:02 -03:00
rafaelsideguide	ec46065066	Update rate-limiter.ts	2024-05-21 10:07:27 -03:00
Nicolas	7f64fe884a	Update blocklist.ts	2024-05-20 17:26:01 -07:00
Nicolas	756f54466d	Nick: allowed keywords for now	2024-05-20 17:24:21 -07:00
Nicolas	01783dc336	Update openapi.json	2024-05-20 17:10:55 -07:00
Nicolas	77a79b5a79	Nick: max num tokens for llm extract (for now) + slice the max	2024-05-20 17:07:38 -07:00
Nicolas	2644e1c029	Update .env.example	2024-05-20 13:36:51 -07:00
Nicolas	9e61d431f0	Nick: hyper dx integration init	2024-05-20 13:36:34 -07:00
Nicolas	c74f757b53	Update rate-limiter.ts	2024-05-19 13:05:36 -07:00
Nicolas	98a39b39ab	Nick: increased rate limits	2024-05-19 12:59:29 -07:00
Nicolas	18fa15df25	Update index.test.ts	2024-05-19 12:50:06 -07:00
Nicolas	614c073af0	Nick: improvements	2024-05-19 12:45:46 -07:00
Nicolas	f473793ba3	Merge branch 'main' into feat/rate-limits	2024-05-19 12:23:34 -07:00
Nicolas	4efebf7a4b	Merge branch 'test/load-testing' of https://github.com/mendableai/firecrawl into test/load-testing	2024-05-19 12:22:51 -07:00
Nicolas	5792cd022c	Update fly.staging.toml	2024-05-19 12:22:49 -07:00
rafaelsideguide	d667e1417b	added fly staging load test ... - being rate limited. Need to add the token to the rate-limit functions	2024-05-17 19:09:19 -03:00
Nicolas	7630565c26	Create fly.staging.toml	2024-05-17 14:33:59 -07:00
rafaelsideguide	a480595aa7	Update index.test.ts	2024-05-17 15:41:27 -03:00
rafaelsideguide	54049be539	Added e2e tests	2024-05-17 15:37:47 -03:00
Nicolas	6feb21cc35	Update website_params.ts	2024-05-17 11:21:26 -07:00
Nicolas	5be208f595	Nick: fixed	2024-05-17 10:40:44 -07:00
Nicolas	eb88447e8b	Update index.test.ts	2024-05-17 10:00:05 -07:00
Nicolas	df6c3d1e7d	Merge branch 'main' into detect-pdfs	2024-05-17 09:55:51 -07:00
Nicolas	9d635cb2a3	Nick: docx support	2024-05-16 11:48:02 -07:00