* Fix LLMs.txt cache bug with subdomains and add bypass option (#1519)
Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
* Nick:
* Update LLMs.txt test file to use helper functions and concurrent tests
Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
* Remove LLMs.txt test file as requested
Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
* Change parameter name to 'cache' and keep 7-day expiration
Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
* Update generate-llmstxt-supabase.ts
* Update JS and Python SDKs to include cache parameter
Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
* Fix LLMs.txt cache implementation to use normalizeUrl and exact matching
Co-Authored-By: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
* Revert "Fix LLMs.txt cache implementation to use normalizeUrl and exact matching"
This reverts commit d05b9964677b7b2384453329d2ac99d841467053.
* Nick:
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello+firecrawl@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* Add change tracking support to Python and JS SDKs
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Replace test API keys with TEST_API_KEY placeholder
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Replace API keys with dummy values for testing
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Use environment variables for API keys in tests
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Move JS SDK test to correct location and add dependencies
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Remove old test file location
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Update test file to use TEST_API_KEY environment variable
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Update Python SDK test to use TEST_API_KEY environment variable
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Update package.json
* Update __init__.py
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* Add git-diff support to change tracking format
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Fix type issues with parse-diff library
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Fix parse-diff type definitions to match actual library structure
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Add structured output/prompt support to change tracking
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* (feat/change-tracking) Change Tracking Modes (#1447)
* Refactor change tracking to use modes array instead of separate formats
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Implement schema-based change tracking with old/new value comparison
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Nick:
* Nick: .json
* Update diff.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* Update index.ts
* Update types.ts
* Update diff.ts
* Update scrape.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* feat(crawl): includes/excludes fixes pt. 1
* fix(snips): billing tests
* drop tha logs
* fix(ci): add replica url
* feat(crawl): drop initial scrape if it's not included
* feat(ci): more verbose logging
* fix crawl path in test
* fix(ci): wait for api
* fix(snips/scrape/ad): test for more pixels
* feat(js-sdk/crawl): add regexOnFullURL
