rafaelmmiller
9443a823b2
feat: script that generates all sdk examples for openapi
2025-05-30 17:09:03 -03:00
rafaelmmiller
2faa45162c
sdk(v3): all tests complete
2025-05-30 10:16:44 -03:00
rafaelmmiller
91099e2dba
sdk(v3) map ok
2025-05-29 16:21:42 -03:00
rafaelmmiller
b7f54d874f
sdk(v3): crawl + async crawl
2025-05-29 16:16:03 -03:00
rafaelmmiller
3d7ed80db7
sdk(v3): removed deep research and llmtxt
2025-05-28 08:32:36 -03:00
rafaelmmiller
c5b64bd294
sdk(v3): async batch scrape (async and sync clients)
2025-05-27 15:47:05 -03:00
rafaelmmiller
2e14859de0
sdk(v3): batch scrape sync and ascyn -- this test is not reliable bc of a bug in batch scrape
2025-05-27 11:57:57 -03:00
rafaelmmiller
ab1e244693
sdk(v3): scrape and async scrape ok
2025-05-27 11:23:02 -03:00
Nicolas
94220a772b
Merge branch 'main' into python-sdk/e2e-tests-all-params
2025-05-16 15:54:30 -03:00
devin-ai-integration[bot]
ab30c8e4ac
Fix Supabase client configuration errors when USE_DB_AUTHENTICATION is false ( #1534 )
...
* Fix Supabase client configuration errors when USE_DB_AUTHENTICATION is false
Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
* Add USE_DB_AUTHENTICATION checks to map and search controllers
Add test for USE_DB_AUTHENTICATION=false
Add USE_DB_AUTHENTICATION checks to billing services
Add USE_DB_AUTHENTICATION checks to batch_billing.ts
Add USE_DB_AUTHENTICATION checks to cached-docs.ts
Add USE_DB_AUTHENTICATION checks to supabase-jobs.ts
Add USE_DB_AUTHENTICATION checks to team-id-sync.ts
Add USE_DB_AUTHENTICATION checks to test-suite log.ts
Add USE_DB_AUTHENTICATION checks to idempotency services
Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
* Revert "Add USE_DB_AUTHENTICATION checks to map and search controllers"
This reverts commit 834a5d51a68c74ada67800fa3a0aa45bde22d745.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-05-16 12:56:33 -03:00
devin-ai-integration[bot]
526165e1b9
Add caching for RunPod PDF markdown results in GCS ( #1561 )
...
* Add caching for RunPod PDF markdown results in GCS
Co-Authored-By: thomas@sideguide.dev <thomas@sideguide.dev>
* Update PDF caching to hash base64 directly and add metadata
Co-Authored-By: thomas@sideguide.dev <thomas@sideguide.dev>
* Fix PDF caching to directly hash content and fix test expectations
Co-Authored-By: thomas@sideguide.dev <thomas@sideguide.dev>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: thomas@sideguide.dev <thomas@sideguide.dev>
2025-05-16 12:04:38 -03:00
Gergő Móricz
bd9673e104
Mog/cachable lookup ( #1560 )
...
* feat(scrapeURL): use cacheableLookup
* feat(queue-worker): add cacheablelookup
* fix(cacheable-lookup): make it work with tailscale on local
* add devenv
* try again
* allow querying all
* log
* fixes
* asd
* fix:
* fix(lookup):
* lookup
2025-05-16 15:44:52 +02:00
Gergő Móricz
d46ba95924
Revert "feat: use cacheable lookup everywhere ( #1559 )"
...
This reverts commit b8703b2a720765b92f5c4cab94cc90ea624198a8.
2025-05-16 15:31:06 +02:00
Gergő Móricz
b8703b2a72
feat: use cacheable lookup everywhere ( #1559 )
...
* feat(scrapeURL): use cacheableLookup
* feat(queue-worker): add cacheablelookup
* fix(cacheable-lookup): make it work with tailscale on local
* add devenv
* try again
* allow querying all
* log
* fixes
* asd
* fix:
* fix(lookup):
2025-05-16 15:27:24 +02:00
Gergő Móricz
f936befcdb
feat(queue-worker): liveness check endpoint
2025-05-16 14:15:48 +02:00
Gergő Móricz
b5b612c35b
feat(api/extract/fire-0): error logging ( #1556 )
2025-05-15 11:32:59 -03:00
Will
b0c203e512
Fix/optional chaining operators missing ( #1549 )
...
* fix: missing optional chaining operator in req.acuc.flags
* fix: missing optional chaining operator in req.acuc.flags
2025-05-15 00:04:04 +02:00
Gergő Móricz
cee481a3a9
fix(fire-engine): sslerror passthrough
2025-05-14 23:50:57 +02:00
Gergő Móricz
3db2294b97
feat(scrapeURL): better error for SSL failures ( #1552 )
2025-05-14 23:34:59 +02:00
rafaelmmiller
2da8abe6cb
Merge branch 'python-sdk/e2e-tests-all-params' of https://github.com/mendableai/firecrawl into python-sdk/e2e-tests-all-params
2025-05-14 15:42:04 -03:00
rafaelmmiller
06cf6be9d5
e2e tests for python methods
2025-05-14 15:41:50 -03:00
rafaelmmiller
f0b0afc5d6
Enhance Firecrawl SDK: Add LocationConfig to __init__.py and update ScrapeOptions formats to remove 'content' type. Introduce new E2E tests for async and standard scraping functionalities, including batch scraping and URL mapping.
2025-05-14 15:41:50 -03:00
Devin AI
f17b1cdc86
Update check_deep_research_status return type annotation
...
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-14 15:41:50 -03:00
Devin AI
788733b3fc
Add type hints for deep_research response structure
...
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-14 15:41:50 -03:00
Devin AI
fc366f948c
Fix deep_research method to convert camelCase to snake_case and add dot notation access
...
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-14 15:41:50 -03:00
rafaelmmiller
6d8033402c
e2e tests for python methods
2025-05-14 15:41:39 -03:00
Ademílson Tonato
06189b9646
refactor: increase max limit for search request schema from 50 to 100 ( #1545 )
2025-05-13 17:40:32 -03:00
Yohann Prigent
505924875e
create openAI provider using base url parameter ( #1480 )
...
Co-authored-by: Yohann Prigent <yohann@pandascore.co>
2025-05-12 20:43:07 +02:00
Gergő Móricz
0fd05a67a0
Revert "Revert "fix(queue-worker, scrape): match billing logic and add billing for stealth proxies ( #1521 )""
...
This reverts commit 017a915ae8f550ceaa01ad607b4e6a684385eadf.
2025-05-12 17:46:09 +02:00
rafaelmmiller
476e042496
Enhance Firecrawl SDK: Add LocationConfig to __init__.py and update ScrapeOptions formats to remove 'content' type. Introduce new E2E tests for async and standard scraping functionalities, including batch scraping and URL mapping.
2025-05-09 18:32:05 -03:00
Devin AI
054d99c1af
Update check_deep_research_status return type annotation
...
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-09 15:51:12 +00:00
Devin AI
99ec770f4d
Add type hints for deep_research response structure
...
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-09 15:50:10 +00:00
Devin AI
f25edc4474
Fix deep_research method to convert camelCase to snake_case and add dot notation access
...
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-09 15:45:32 +00:00
Gergő Móricz
fdeb01847d
feat(queue-worker): add more logs around crawl finishing logic
2025-05-09 16:52:38 +02:00
Nicolas
907cf1cf41
Update __init__.py
2025-05-08 20:29:20 -03:00
devin-ai-integration[bot]
21adf047be
[Bug Fix] Make WaitAction milliseconds field optional in firecrawl-py ( #1533 )
...
* This fixes issue #1512 by making the milliseconds field optional in WaitAction and adding a validator to ensure exactly one of milliseconds or selector is provided.
Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
* Update firecrawl.py
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-08 20:27:27 -03:00
Nicolas
7b03ab36a7
Update openapi.json
2025-05-08 20:15:49 -03:00
Gergő Móricz
fa581995e6
feat(acuc): propagate team flags (FIR-1879) ( #1522 )
...
* feat(acuc): propagate team flags
* feat(flags): further functionality
2025-05-08 20:23:35 +02:00
Gergő Móricz
017a915ae8
Revert "fix(queue-worker, scrape): match billing logic and add billing for stealth proxies ( #1521 )"
...
This reverts commit e06c7cc234b9ee0c4bf112cc338e515f28674e11.
2025-05-08 18:34:13 +02:00
Gergő Móricz
e06c7cc234
fix(queue-worker, scrape): match billing logic and add billing for stealth proxies ( #1521 )
2025-05-08 10:51:38 -03:00
Gergő Móricz
0f32500149
fix(queue-jobs): never cc timeout jobs that are crawl-associated (makes no sense)
2025-05-08 12:54:23 +02:00
Gergő Móricz
7ad9a00ea8
fix(concurrency-limit): rework cc queue to work by time not priority ( #1526 )
2025-05-08 12:40:13 +02:00
Ademílson Tonato
5d07cccd65
Merge pull request #1523 from mendableai/refactor/map-limit
...
refactor: maximum links limit for map endpoint from 5000 to 30000
2025-05-06 21:06:19 +01:00
Ademílson Tonato
ae12c326f0
refactor: maximum links limit for map endpoint from 5000 to 30000
2025-05-06 16:00:15 -04:00
Nicolas
17728379df
Revert "Nick: log su usage"
...
This reverts commit 6567ef81f6ca6dfd5a92eafb04f4eed8d37e0a6c.
2025-05-05 18:19:15 -03:00
Nicolas
6567ef81f6
Nick: log su usage
2025-05-05 18:00:34 -03:00
devin-ai-integration[bot]
0512ad6bce
Add delay parameter to crawl options in all SDKs ( #1514 )
...
* Add delay parameter to crawl options in all SDKs
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* Update terminology from 'between crawl requests' to 'between scrapes'
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* Apply suggestions from code review
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mogery@sideguide.dev <mogery@sideguide.dev>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-05-02 18:00:15 +02:00
devin-ai-integration[bot]
411ecdf04b
Add crawl delay functionality with per-crawl concurrency limiting (FIR-249) ( #1413 )
...
* feat: Add crawl delay functionality with per-crawl concurrency limiting (FIR-249)
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* fix: Skip crawl delay in test environment to fix CI tests
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* refactor: Use crawlerOptions.delay instead of separate fields
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* refactor: Rename crawlDelay to delay in type definitions for uniformity
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* refactor: Fix crawl concurrency implementation based on PR feedback
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* refactor: Simplify if/else structure in queue-jobs.ts based on PR feedback
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* human fixes
* test: Add tests for crawl delay functionality
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* test: Move crawl delay tests to existing crawl.test.ts file
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* fix: Ensure sitemapped URLs are added to crawl concurrency queue and update crawl status endpoint
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* dbg
* fix: Ensure jobs with crawl delay are properly added to BullMQ
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* fix: Remove duplicate job addition to BullMQ for jobs with crawl delay
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* fixes
* warning for devin
* test: Simplify crawl delay test as requested in PR feedback
Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>
* bump delay test timeout
* fix operation order
* bump further???
* fix: broken on self-host
* Update apps/api/src/services/queue-jobs.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: import
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mogery@sideguide.dev <mogery@sideguide.dev>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-05-02 17:20:57 +02:00
Eric Ciarla
510171cabe
Delete qwen 3
2025-05-01 16:54:12 -04:00
Eric Ciarla
a0ed76d53d
Merge pull request #1510 from mendableai/devin/1746062769-qwen3-web-crawler-example
...
Add Qwen3 web crawler example using OpenRouter
2025-05-01 16:34:43 -04:00
