AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-08-12 21:29:00 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix(crawler): always set expiry on sitemap links in redis

Browse Source
This commit is contained in:
Gergő Móricz 2025-05-27 15:39:31 +02:00
parent c3738063cf
commit 474e5a0543
1 changed files with 6 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
apps/api/src/scraper/WebScraper/crawler.ts
View File

@ -334,6 +334,12 @@ export class WebCrawler {
count++;
}
await redisConnection.expire(
"sitemap:" + this.jobId + ":links",
3600,
"NX",
);
return count;
} catch (error) {
if (error.message === "Sitemap fetch timeout") {