Aparup Ganguly
448b44cdd9
Implemented github analyzer
2025-02-21 20:08:37 +05:30
Aparup Ganguly
8caeab2691
minor changes
2025-02-20 02:24:49 +05:30
Aparup Ganguly
21b22d9f5c
Output re-structured
2025-02-20 02:21:43 +05:30
Aparup Ganguly
3a4ef05a70
Output imporvements
2025-02-20 01:30:59 +05:30
Aparup Ganguly
5e0036f8b4
Implemented gemini 2.0 web extractor
2025-02-20 00:05:12 +05:30
Tetsuro Yokoyama
5ac6eb7440
Update self-hosted Kubernetes deployments examples for compatibility and consistency ( #1177 )
...
* fix: Quote variables in `docker-entrypoint.sh`
- This commit adds double quotes around variables in the docker-entrypoint.sh script to prevent word splitting and globbing issues, ensuring the script behaves correctly in all cases.
* fix: Ensure worker/api deployment starts with `OPENAI_API_KEY`
* fix: Add missing `FLY_PROCESS_GROUP` env var to deployments
* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`
* fix: Update Docker build options for Apple Silicon compatibility
* fix: Correct `PLAYWRIGHT_MICROSERVICE_URL` in `firecrawl-config`
2025-02-18 13:33:20 +01:00
Eric Ciarla
5cb2918368
update name of example
2025-02-16 18:46:10 -05:00
mayo
32f9897670
fix: update gemini library. extract pdf links from markdowncontent
2025-02-12 16:43:59 +00:00
mayo
cb3bc5e445
Add detection of PDF/image sub-links and extract text via Gemini
2025-02-12 02:52:20 +00:00
Eric Ciarla
c67b052f4a
Update gemini-2.0-crawler.py
2025-02-11 13:10:48 -05:00
Eric Ciarla
ef9aad8b8a
Merge pull request #1161 from aparupganguly/feature/gemini-2.0-crawler
...
Implemented Gemini 2.0 crawler
2025-02-11 08:13:06 -05:00
Aparup Ganguly
f33fa5bfc6
Initial commit
2025-02-08 01:24:21 +05:30
Eric Ciarla
2b7b7400f6
Merge pull request #1144 from aparupganguly/feature/o3-mini-job-resource
...
Feature/o3 mini job resource extractor
2025-02-07 11:39:29 -05:00
Eric Ciarla
42f4f7ef09
Merge pull request #1139 from aparupganguly/feature/o3-mini-review-summarizer
...
Implemented prodcut reviews summarizer using o3 mini
2025-02-07 11:37:57 -05:00
Aparup Ganguly
46f05a750d
Job Resources Analyzer
2025-02-07 20:32:52 +05:30
Aparup Ganguly
310ef66506
Implemented prodcut reviews summarizer using o3 mini
2025-02-06 20:13:35 +05:30
Eric Ciarla
b0f9e73376
Update o3-mini_web_crawler.py
2025-02-04 16:35:30 -05:00
Eric Ciarla
1951b60881
Merge pull request #1130 from aparupganguly/feature/o3-company-researcher
...
Updated Model Parameters (o3-mini_company_researcher)
2025-02-04 09:46:16 -05:00
Aparup Ganguly
7aa2db25a3
Updated Model Parameters
2025-02-04 17:54:44 +05:30
Eric Ciarla
5894076fda
Merge pull request #1120 from aparupganguly/feature-o3-crawler
...
Feature o3-mini web crawler
2025-02-03 10:48:02 -05:00
Aparup Ganguly
01d656feb7
Minor Changes
2025-02-03 00:54:45 +05:30
Aparup Ganguly
d5d3df9d10
Handled JSON format structure
2025-02-03 00:39:00 +05:30
Aparup Ganguly
61b989cc64
Initial Commit
2025-02-02 17:36:26 +05:30
Eric Ciarla
cdc747ab7b
small edits
2025-01-31 13:35:46 -05:00
Aparup Ganguly
db740a0c96
Update R1_web_crawler.py
2025-01-31 02:58:56 +05:30
Aparup Ganguly
a3b5666cba
Delete examples/R1_web_extractor/tempCodeRunnerFile.py
2025-01-31 02:57:52 +05:30
Aparup Ganguly
2dafe7fb6b
minor changes
2025-01-31 02:51:20 +05:30
Aparup Ganguly
768e80574c
Added R1 web extractor feature
2025-01-31 01:29:30 +05:30
Eric Ciarla
669c694b32
R1 Crawler
2025-01-22 09:37:10 -03:00
Eric Ciarla
94267ff406
Create o1_web_extractor.py
2024-12-18 17:31:45 -05:00
Eric Ciarla
e06647b4b0
Move full app examples to other repo
2024-12-12 14:41:11 -05:00
Eric Ciarla
e5fe9e1534
Create .env.example
2024-12-11 15:31:08 -05:00
BexTuychiev
a093d55e6d
Add assets for GitHub actions tutorialc
2024-12-09 16:55:04 +05:00
BexTuychiev
57ef400473
Add README to auotmated price tracking project
2024-12-06 23:15:12 +05:00
BexTuychiev
4fc94aba94
Add a link to the assets directory that will work once the PR is merged
2024-12-06 23:11:35 +05:00
BexTuychiev
742c3df67e
Add assets for the Automated Amazon Price Tracking article
2024-12-06 23:05:18 +05:00
BexTuychiev
5b7bd571cc
Add assets for a new 'Deploying web scrapers' article
2024-12-05 14:42:39 +05:00
BexTuychiev
c5d75bef0f
Add assets for a new 'Scheduling web scrapers' article
2024-12-05 14:31:21 +05:00
Eric Ciarla
2988a56ee5
Merge pull request #928 from BexTuychiev/hacker-news-scraper
...
Add a new project to examples that shows how to scrape Hacker News website
2024-11-29 08:49:00 -05:00
BexTuychiev
137726ca4f
feature: Add a log message for every file saved
2024-11-28 15:02:29 +05:00
BexTuychiev
d777633b30
Add a new project to examples that shows how to scrape Hacker News website
2024-11-28 14:30:09 +05:00
BexTuychiev
c0fd256021
Add notebook and markdown files for two articles: mastering /scrape and mastering /map
2024-11-23 21:47:36 +05:00
Eric Ciarla
aa01c0b684
Create mastering-the-crawl-endpoint.ipynb
2024-11-19 12:50:31 -05:00
Eric Ciarla
a31336752c
Create README.md
2024-11-18 14:04:29 -05:00
Eric Ciarla
62c8b63b84
Create README.md
2024-11-14 11:55:00 -05:00
Gergő Móricz
8d467c8ca7
WebScraper refactor into scrapeURL (#714 )
...
* feat: use strictNullChecking
* feat: switch logger to Winston
* feat(scrapeURL): first batch
* fix(scrapeURL): error swallow
* fix(scrapeURL): add timeout to EngineResultsTracker
* fix(scrapeURL): report unexpected error to sentry
* chore: remove unused modules
* feat(transfomers/coerce): warn when a format's response is missing
* feat(scrapeURL): feature flag priorities, engine quality sorting, PDF and DOCX support
* (add note)
* feat(scrapeURL): wip readme
* feat(scrapeURL): LLM extract
* feat(scrapeURL): better warnings
* fix(scrapeURL/engines/fire-engine;playwright): fix screenshot
* feat(scrapeURL): add forceEngine internal option
* feat(scrapeURL/engines): scrapingbee
* feat(scrapeURL/transformars): uploadScreenshot
* feat(scrapeURL): more intense tests
* bunch of stuff
* get rid of WebScraper (mostly)
* adapt batch scrape
* add staging deploy workflow
* fix yaml
* fix logger issues
* fix v1 test schema
* feat(scrapeURL/fire-engine/chrome-cdp): remove wait inserts on actions
* scrapeURL: v0 backwards compat
* logger fixes
* feat(scrapeurl): v0 returnOnlyUrls support
* fix(scrapeURL/v0): URL leniency
* fix(batch-scrape): ts non-nullable
* fix(scrapeURL/fire-engine/chromecdp): fix wait action
* fix(logger): remove error debug key
* feat(requests.http): use dotenv expression
* fix(scrapeURL/extractMetadata): extract custom metadata
* fix crawl option conversion
* feat(scrapeURL): Add retry logic to robustFetch
* fix(scrapeURL): crawl stuff
* fix(scrapeURL): LLM extract
* fix(scrapeURL/v0): search fix
* fix(tests/v0): grant larger response size to v0 crawl status
* feat(scrapeURL): basic fetch engine
* feat(scrapeURL): playwright engine
* feat(scrapeURL): add url-specific parameters
* Update readme and examples
* added e2e tests for most parameters. Still a few actions, location and iframes to be done.
* fixed type
* Nick:
* Update scrape.ts
* Update index.ts
* added actions and base64 check
* Nick: skipTls feature flag?
* 403
* todo
* todo
* fixes
* yeet headers from url specific params
* add warning when final engine has feature deficit
* expose engine results tracker for ScrapeEvents implementation
* ingest scrape events
* fixed some tests
* comment
* Update index.test.ts
* fixed rawHtml
* Update index.test.ts
* update comments
* move geolocation to global f-e option, fix removeBase64Images
* Nick:
* trim url-specific params
* Update index.ts
---------
Co-authored-by: Eric Ciarla <ericciarla@yahoo.com>
Co-authored-by: rafaelmmiller <8574157+rafaelmmiller@users.noreply.github.com>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2024-11-07 20:57:33 +01:00
Eric Ciarla
ed5a0d3cf2
Update readme and examples
2024-11-05 18:06:40 -05:00
Eric Ciarla
71e512b80f
predicted-outputs
2024-11-05 16:16:24 -05:00
Eric Ciarla
a5c9823495
haiku example
2024-11-04 14:58:05 -05:00
Eric Ciarla
983f344fa8
Create claude_stock_analyzer.py
2024-10-29 15:38:57 -04:00
