222 Commits

Author SHA1 Message Date
Gergő Móricz
3557c90210
feat(js-sdk): auto mode proxy (FIR-2145) (#1602)
* feat(js-sdk): auto mode proxy

* Nick: py sdk

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-28 14:31:48 -03:00
Gergő Móricz
513f469b0f
feat(python-sdk/CrawlWatcher): remove max payload size from WebSocket (FIR-2038) (#1577)
* feat(python-sdk/CrawlWatcher): remove max payload size from WebSocket

* Update __init__.py

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-20 16:59:08 -03:00
devin-ai-integration[bot]
7ccbbec488
Fix LLMs.txt cache bug with subdomains and add bypass option (#1557)
* Fix LLMs.txt cache bug with subdomains and add bypass option (#1519)

Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>

* Nick:

* Update LLMs.txt test file to use helper functions and concurrent tests

Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>

* Remove LLMs.txt test file as requested

Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>

* Change parameter name to 'cache' and keep 7-day expiration

Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>

* Update generate-llmstxt-supabase.ts

* Update JS and Python SDKs to include cache parameter

Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>

* Fix LLMs.txt cache implementation to use normalizeUrl and exact matching

Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>

* Revert "Fix LLMs.txt cache implementation to use normalizeUrl and exact matching"

This reverts commit d05b9964677b7b2384453329d2ac99d841467053.

* Nick:

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-16 16:29:09 -03:00
Nicolas
907cf1cf41 Update __init__.py 2025-05-08 20:29:20 -03:00
devin-ai-integration[bot]
21adf047be
[Bug Fix] Make WaitAction milliseconds field optional in firecrawl-py (#1533)
* This fixes issue #1512 by making the milliseconds field optional in WaitAction and adding a validator to ensure exactly one of milliseconds or selector is provided.

Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>

* Update firecrawl.py

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-05-08 20:27:27 -03:00
devin-ai-integration[bot]
0512ad6bce
Add delay parameter to crawl options in all SDKs (#1514)
* Add delay parameter to crawl options in all SDKs

Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>

* Update terminology from 'between crawl requests' to 'between scrapes'

Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev>

* Apply suggestions from code review

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mogery@sideguide.dev <mogery@sideguide.dev>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-05-02 18:00:15 +02:00
devin-ai-integration[bot]
f0b1507290
Fix: Handle both dict and model instances in actions parameter (#1508)
* Fix: Handle both dict and model instances in actions parameter

Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>

* Update __init__.py

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nicolascamara29@gmail.com>
2025-04-29 13:06:12 -03:00
Nicolas
6dbfd54e2c Update __init__.py 2025-04-29 12:22:30 -03:00
Rafael Miller
317fa43f9e
Fix sdk/schemas (#1507)
* sdk-fix/schema-check

* version bump

* schema validation for extract and jsonOptions parameters

* Update firecrawl.py

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-29 12:19:08 -03:00
Nicolas
a0a1675829
Nick: (#1506) 2025-04-29 11:06:35 -03:00
Nicolas
8053a7cedd Nick: updates on pypi 2025-04-28 15:01:12 -03:00
Arvid Andersson
c164370298
Webhook param for batch scrape (#1505)
The API endpoint supports the webhook param, align the client to support this.
2025-04-28 13:27:10 -03:00
Rafael Miller
e3e730f2c1
Update version to 2.4.0 and enhance ExtractResponse model with additional fields for id, status, and expiresAt. (#1501) 2025-04-25 23:19:08 -04:00
Nicolas
5c3951b42e Update __init__.py 2025-04-24 12:31:02 -04:00
John Bledsoe
ca82015bca
Use async job status monitor for AsyncFirecrawlApp (#1498) 2025-04-24 12:28:29 -04:00
Rafael Miller
e10d4c7b0c
[fix/sdk] kwargs params (#1490)
* fix sdk kwargs params

* version

* Update __init__.py

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-22 15:15:32 -04:00
rafaelmmiller
a4323d8f23 fix:python-sdk 2025-04-19 12:53:37 -07:00
Nicolas
2c72097c3f Nick: 2025-04-18 13:44:16 -07:00
Nicolas
c7df80e2a8 Update __init__.py 2025-04-18 13:42:21 -07:00
rafaelmmiller
91ebd140e5 version bump 2025-04-18 13:37:34 -07:00
rafaelmmiller
0aedef7210 fix 2025-04-18 13:37:09 -07:00
rafaelmmiller
79bc54c11e scrape options fixing types 2025-04-18 13:00:05 -07:00
Nicolas
16439b1c7e Nick: examples 2025-04-18 02:00:00 -07:00
Nicolas
a74b2dc59f Nick: json config instead of extract config 2025-04-18 01:54:15 -07:00
Nicolas
06c54bc41c Update __init__.py 2025-04-18 01:43:18 -07:00
Nicolas
9e67d7ba22 Nick: 2025-04-18 01:30:40 -07:00
Nicolas
37e076e151 Merge branch 'sdk-improv/async' of https://github.com/mendableai/firecrawl into sdk-improv/async 2025-04-18 01:28:39 -07:00
Nicolas
9ba1ae9ae1 Nick: 2025-04-18 01:28:31 -07:00
rafaelmmiller
55c04d615e Merge branch 'sdk-improv/async' of https://github.com/mendableai/firecrawl into sdk-improv/async 2025-04-18 01:21:29 -07:00
rafaelmmiller
0915db515c async functions 2025-04-18 01:20:16 -07:00
Nicolas
a3f3168212 Nick: python sdk 2.0 2025-04-18 01:15:14 -07:00
Nicolas
f3522666db Nick: new examples 2025-04-18 01:13:53 -07:00
Nicolas
0001d6ea25 Merge branch 'sdk-improv/async' of https://github.com/mendableai/firecrawl into sdk-improv/async 2025-04-18 01:12:31 -07:00
Nicolas
0b62be5874 Update firecrawl.py 2025-04-18 01:12:24 -07:00
rafaelmmiller
5db76992de Merge branch 'sdk-improv/async' of https://github.com/mendableai/firecrawl into sdk-improv/async 2025-04-18 01:07:30 -07:00
rafaelmmiller
8cd82b5600 async scrape 2025-04-18 01:06:58 -07:00
Nicolas
1aa0c092e0 Update firecrawl.py 2025-04-18 01:01:01 -07:00
Nicolas
390f3d44a3 Update firecrawl.py 2025-04-18 00:51:06 -07:00
Nicolas
d8792d2301 Update firecrawl.py 2025-04-18 00:48:07 -07:00
Nicolas
5e6e41ab17 Update firecrawl.py 2025-04-18 00:37:34 -07:00
rafaelmmiller
a655d24e7c scrape params commentary 2025-04-18 00:29:20 -07:00
Nicolas
8c5509cbb4 Update firecrawl.py 2025-04-18 00:26:00 -07:00
rafaelmmiller
b67bc707d5 Merge branch 'sdk-improv/async' of https://github.com/mendableai/firecrawl into sdk-improv/async 2025-04-18 00:00:21 -07:00
rafaelmmiller
85247991bc generic 2025-04-18 00:00:18 -07:00
Nicolas
287032017b Merge branch 'sdk-improv/async' of https://github.com/mendableai/firecrawl into sdk-improv/async 2025-04-17 23:50:57 -07:00
Nicolas
8eb4e1a96a Update firecrawl.py 2025-04-17 23:50:56 -07:00
rafaelmmiller
23ef26655c params 2025-04-17 23:37:45 -07:00
rafaelmmiller
22cfdd6ae3 added agent options types 2025-04-17 23:31:28 -07:00
Nicolas
f48937a55d Update firecrawl.py 2025-04-17 23:17:00 -07:00
Nicolas
d9780412f5 Update firecrawl.py 2025-04-17 23:08:33 -07:00