Gergő Móricz
8389a1a78d
fix(html-transformer): bad outName for og:locale:alternate (FIR-2101) ( #1597 )
...
* fix(html-transformer): bad outName for og:locale:alternate
* oops
2025-05-23 17:10:09 +02:00
Gergő Móricz
b03670a8b7
feat: parse PDFs on fc side and reject if too long for timeout (FIR-2083) ( #1592 )
...
* feat: pdf-parser, implementation in scrapeURL
* use pdf-parser for page count instead of mu
* fix(pdf-parser): bindings
* feat(scrapeURL/pdf): adjust MILLISECONDS_PER_PAGE
* implement post-runsync polling and fix
* fix(Dockerfile): copy in the pdf-parser source code
* fix(scrapeURL/pdf): better error for timeout below 0
2025-05-23 13:45:53 +02:00
devin-ai-integration[bot]
a5a915d639
Fix: Concatenate metadata arrays into strings with exceptions ( #1574 )
...
* Fix: Concatenate metadata arrays into strings except for ogLocaleAlternate
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
* Fix: Only concatenate description field, preserve other metadata arrays
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
* Fix: Only concatenate description field, keep other metadata fields in original format
Co-Authored-By: Nicolas Camara <nicolascamara29@gmail.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nicolascamara29@gmail.com>
2025-05-20 12:40:53 -03:00
Gergő Móricz
2e2c3d52ce
feat: add swoogo classes to force include main tags
2025-04-03 09:57:19 +02:00
Carter LaSalle
78094e2fe4
fix(html-transformer): Update free_string function parameter type ( #1163 )
...
Correct the parameter type to use libc::c_char for better type consistency
2025-02-11 15:11:29 +01:00
Gergő Móricz
892f3a41f3
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) ( #1138 )
...
* fix(scrape): allow getting valid JSON via rawHtml
* fix(scrape/test):
2025-02-06 18:35:28 +01:00
Thomas Kosmas
2a0b408181
chore(go-html-to-md): Update html-to-markdown dependency
2025-01-28 18:36:01 +02:00
Móricz Gergő
173028295b
fix(crawl): relative URL page discovery issues
2025-01-28 09:41:37 +01:00
Gergő Móricz
8af4e4b8dd
fix(html-transformer): preserve title tag
2025-01-27 16:13:24 +01:00
Gergő Móricz
4d8f4109b5
fix(rust): further select fixes
2025-01-25 18:48:40 +01:00
Nicolas
7fdecdc4d3
Nick: fixed include tags bug
2025-01-25 14:12:10 -03:00
Móricz Gergő
dacc5d4f45
fix(rust): improve
2025-01-25 12:59:14 +01:00
Móricz Gergő
4a1ab6f01c
fix(rust): handle bad tok_1
2025-01-25 12:53:03 +01:00
Móricz Gergő
e8a6c1bb65
fix(rust): avoid panic always
2025-01-25 10:15:12 +01:00
Móricz Gergő
ce2c51f6c1
fix(rust): bad comp
2025-01-25 10:11:05 +01:00
Móricz Gergő
a2d94b525f
feat: rewrite html transformer in rust
2025-01-25 09:41:33 +01:00
Gergő Móricz
b005450a34
port most of cheerio stuff to rust ( #1089 )
2025-01-24 22:04:54 +01:00
Gergő Móricz
5cb46dc494
fix(html-to-markdown): build error
2024-11-11 21:09:27 +01:00
Gergő Móricz
56bebc8107
fix(html-to-markdown): reduce logging frequency
2024-11-11 20:53:21 +01:00
Gergő Móricz
8d467c8ca7
WebScraper refactor into scrapeURL (#714 )
...
* feat: use strictNullChecking
* feat: switch logger to Winston
* feat(scrapeURL): first batch
* fix(scrapeURL): error swallow
* fix(scrapeURL): add timeout to EngineResultsTracker
* fix(scrapeURL): report unexpected error to sentry
* chore: remove unused modules
* feat(transfomers/coerce): warn when a format's response is missing
* feat(scrapeURL): feature flag priorities, engine quality sorting, PDF and DOCX support
* (add note)
* feat(scrapeURL): wip readme
* feat(scrapeURL): LLM extract
* feat(scrapeURL): better warnings
* fix(scrapeURL/engines/fire-engine;playwright): fix screenshot
* feat(scrapeURL): add forceEngine internal option
* feat(scrapeURL/engines): scrapingbee
* feat(scrapeURL/transformars): uploadScreenshot
* feat(scrapeURL): more intense tests
* bunch of stuff
* get rid of WebScraper (mostly)
* adapt batch scrape
* add staging deploy workflow
* fix yaml
* fix logger issues
* fix v1 test schema
* feat(scrapeURL/fire-engine/chrome-cdp): remove wait inserts on actions
* scrapeURL: v0 backwards compat
* logger fixes
* feat(scrapeurl): v0 returnOnlyUrls support
* fix(scrapeURL/v0): URL leniency
* fix(batch-scrape): ts non-nullable
* fix(scrapeURL/fire-engine/chromecdp): fix wait action
* fix(logger): remove error debug key
* feat(requests.http): use dotenv expression
* fix(scrapeURL/extractMetadata): extract custom metadata
* fix crawl option conversion
* feat(scrapeURL): Add retry logic to robustFetch
* fix(scrapeURL): crawl stuff
* fix(scrapeURL): LLM extract
* fix(scrapeURL/v0): search fix
* fix(tests/v0): grant larger response size to v0 crawl status
* feat(scrapeURL): basic fetch engine
* feat(scrapeURL): playwright engine
* feat(scrapeURL): add url-specific parameters
* Update readme and examples
* added e2e tests for most parameters. Still a few actions, location and iframes to be done.
* fixed type
* Nick:
* Update scrape.ts
* Update index.ts
* added actions and base64 check
* Nick: skipTls feature flag?
* 403
* todo
* todo
* fixes
* yeet headers from url specific params
* add warning when final engine has feature deficit
* expose engine results tracker for ScrapeEvents implementation
* ingest scrape events
* fixed some tests
* comment
* Update index.test.ts
* fixed rawHtml
* Update index.test.ts
* update comments
* move geolocation to global f-e option, fix removeBase64Images
* Nick:
* trim url-specific params
* Update index.ts
---------
Co-authored-by: Eric Ciarla <ericciarla@yahoo.com>
Co-authored-by: rafaelmmiller <8574157+rafaelmmiller@users.noreply.github.com>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2024-11-07 20:57:33 +01:00
Thomas Kosmas
ae919ad896
Fix go parser
2024-10-31 17:36:32 +02:00
