Nicolas
ec2c0f671c
Added more safety guards to auto-rech
2025-01-30 12:24:37 -03:00
Gergő Móricz
71878cf4d9
fix(cc): hotfix
2025-01-30 16:07:39 +01:00
Móricz Gergő
a7eb2f7c6a
fix(crawler/rust): dedupe
2025-01-30 08:16:51 +01:00
Nicolas
c88176a596
Update blocklist.ts
2025-01-29 22:23:50 -03:00
Gergő Móricz
86f05a07ba
feat(github/ci): connect to tailscale (FIR-748) ( #1112 )
...
* feat(github/ci): connect to tailscale
* fix(tests/snips): adjust timeouts
2025-01-29 16:59:43 +01:00
Gergő Móricz
aaa16ee388
feat(v0): store v0 users (team ID) in Redis for collection ( #1111 )
2025-01-29 12:43:55 -03:00
Nicolas
fa99c62f64
(feat/extract) Improved completions to use model's limits ( #1109 )
...
* Update analyzeSchemaAndPrompt.ts
* Nick: fixes FIR-663
* Update llmExtract.ts
* Update llmExtract.ts
2025-01-29 12:37:14 -03:00
Nicolas
cf8f7d0ce3
Update analyzeSchemaAndPrompt.ts ( #1108 )
2025-01-29 12:36:13 -03:00
Gergő Móricz
d09e0603f8
feat(scrapeUrl/fire-engine): add blockAds flag (FIR-692) ( #1106 )
...
* feat(scrapeUrl/fire-engine): add blockAds flag
* feat(v1/scrape): blockAds test
2025-01-29 15:03:37 +01:00
Gergő Móricz
5733b82e9d
fix(scrapeURL/fire-engine): default to separate US-generic proxy list if no location is specified (FIR-728) ( #1104 )
...
* feat(location/country): default to us-generic
* add tests + fix mock
2025-01-29 08:23:36 +01:00
Móricz Gergő
5c1b67511c
feat(github/ci): run snips tests instead of always-failing tests
2025-01-29 08:18:09 +01:00
Gergő Móricz
74438a4048
Revert "Revert "feat(v1/map): timeout"" ( #1105 )
...
This reverts commit 831c61706d7b1ef9da525b2e1602913bc4f678e3.
2025-01-29 08:12:50 +01:00
Nicolas
70562261bc
Update source-tracker.ts
2025-01-28 15:20:22 -03:00
Nicolas
04c6f511b5
(feat/extract) Add sources to the extraction ( #1101 )
...
* Nick: good state
* Nick: source tracker class
* Nick: show sources under flag
2025-01-28 13:46:21 -03:00
Thomas Kosmas
2a0b408181
chore(go-html-to-md): Update html-to-markdown dependency
2025-01-28 18:36:01 +02:00
Gergő Móricz
831c61706d
Revert "feat(v1/map): timeout"
...
This reverts commit 57e98e83d7ab43249963b0a14c157aaec7fd4ec7.
2025-01-28 16:45:47 +01:00
Gergő Móricz
57e98e83d7
feat(v1/map): timeout
2025-01-28 16:44:44 +01:00
Móricz Gergő
173028295b
fix(crawl): relative URL page discovery issues
2025-01-28 09:41:37 +01:00
Hercules Smith
b8c4e198d1
Fix bad WebSocket URL in CrawlWatcher ( #1053 )
...
* fix: bad websocket url in crawl watcher
Fixed CrawlWatcher creating WebSocket using standard http url from base app.
* Use regex to improve url replacement
2025-01-28 08:40:30 +01:00
Nicolas
6b9e65c4f6
(feat/extract) Refactor and Reranker improvements ( #1100 )
...
* Reapply "Nick: extract api reference"
This reverts commit 61d7ba76f76ce74e0d230f89a93436f29dc8d9df.
* Nick: refactor analyzer
* Nick: formatting
* Nick:
* Update extraction-service.ts
* Nick: fixes
* NIck:
* Nick: wip
* Nick: reverted to the old re-ranker
* Nick:
* Update extract-status.ts
2025-01-27 20:07:01 -03:00
rafaelmmiller
ad06cde422
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-01-27 14:31:18 -03:00
rafaelmmiller
c1a2981d59
default onlyMainContent=false for extract
2025-01-27 14:31:16 -03:00
Gergő Móricz
9d448d18d3
feat(v1): support cyrillic URLs
2025-01-27 16:39:40 +01:00
Gergő Móricz
8af4e4b8dd
fix(html-transformer): preserve title tag
2025-01-27 16:13:24 +01:00
Nicolas
61d7ba76f7
Revert "Nick: extract api reference"
...
This reverts commit 522c5b35da7d5cd997aa5ebe2002a38ede7ace93.
2025-01-26 21:06:37 -03:00
Nicolas
522c5b35da
Nick: extract api reference
2025-01-26 21:00:40 -03:00
Gergő Móricz
ce3c54d7c7
fix(html-transformer.test): add further images
2025-01-25 19:06:32 +01:00
Nicolas
cf17479626
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-01-25 15:03:15 -03:00
Nicolas
d8d159b268
Nick:
2025-01-25 15:03:09 -03:00
Gergő Móricz
eb22848eba
feat(test/html-transformer): add test for absolute URLs
2025-01-25 19:02:52 +01:00
Gergő Móricz
f3982c0894
fix: adapt preview team checks
2025-01-25 19:02:32 +01:00
Gergő Móricz
4d8f4109b5
fix(rust): further select fixes
2025-01-25 18:48:40 +01:00
Nicolas
02caa72f02
Nick: added html-transformer unit tests
2025-01-25 14:28:09 -03:00
Nicolas
7fdecdc4d3
Nick: fixed include tags bug
2025-01-25 14:12:10 -03:00
Móricz Gergő
dacc5d4f45
fix(rust): improve
2025-01-25 12:59:14 +01:00
Móricz Gergő
4a1ab6f01c
fix(rust): handle bad tok_1
2025-01-25 12:53:03 +01:00
Móricz Gergő
e8a6c1bb65
fix(rust): avoid panic always
2025-01-25 10:15:12 +01:00
Móricz Gergő
ce2c51f6c1
fix(rust): bad comp
2025-01-25 10:11:05 +01:00
Móricz Gergő
a2d94b525f
feat: rewrite html transformer in rust
2025-01-25 09:41:33 +01:00
Móricz Gergő
9c40e0cc8d
fix(v1): test override for team
2025-01-25 08:27:59 +01:00
Móricz Gergő
afea2eeaac
feat(v1): add insufficient credits stuff
2025-01-25 08:16:19 +01:00
Nicolas
fa5544add8
Merge pull request #1090 from mendableai/nsc/new-re-rank
...
Re-ranker changes
2025-01-24 19:20:39 -03:00
Nicolas
4747c6f569
Update build-prompts.ts
2025-01-24 19:19:18 -03:00
Gergő Móricz
ca78739a48
fix(koffi): duplicate type name?
2025-01-24 22:56:43 +01:00
Nicolas
10133adcc6
Update reranker.ts
2025-01-24 18:35:36 -03:00
Nicolas
2c391b0105
Nick:
2025-01-24 18:09:25 -03:00
Gergő Móricz
b005450a34
port most of cheerio stuff to rust ( #1089 )
2025-01-24 22:04:54 +01:00
Nicolas
d547192f37
Nick: fixed spread schemas
2025-01-24 17:55:16 -03:00
Gergő Móricz
0d9c9f36b8
feat(queue-worker): add verbosity for lock extension
2025-01-24 19:35:25 +01:00
Gergő Móricz
ce1fe6f06a
update bullmq
2025-01-24 18:56:03 +01:00
