AI/firecrawl
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/mendableai/firecrawl synced 2025-06-04 11:24:40 +08:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • 8dd5bf7bd9
         feat(api/tests/scrape): Playwright test improvements (#1626) main Gergő Móricz 2025-06-04 01:24:19 +02:00
  • db75e560a6 fix json mog/better-ip-tests Gergő Móricz 2025-06-04 01:13:11 +02:00
  • 5747af9557 debug Gergő Móricz 2025-06-04 01:11:05 +02:00
  • dba5c28059 fix tests Gergő Móricz 2025-06-04 01:06:46 +02:00
  • 1c86e80a46 feat(playwright): add contentType relaying Gergő Móricz 2025-06-04 01:00:44 +02:00
  • 017a39f143 remove logs Gergő Móricz 2025-06-04 00:57:55 +02:00
  • 3958a33734 debug: logs Gergő Móricz 2025-06-04 00:49:01 +02:00
  • b744f40be9 feat(api/tests/scrape): verify that proxy works on Playwright Gergő Móricz 2025-06-04 00:44:21 +02:00
  • 782702d536 Merge branch 'main' into mog/ongoing-crawls mog/ongoing-crawls Nicolas 2025-06-03 16:40:40 -03:00
  • 95f204aab7
         Index (FIR-2177) (#1605) v1.10.0 Gergő Móricz 2025-06-03 21:30:19 +02:00
  • e1a593d162
         Merge branch 'main' into mog/index mog/index Gergő Móricz 2025-06-03 21:17:29 +02:00
  • 6a1d284fcf Testing improvements (FIR-2209) (#1623) Gergő Móricz 2025-06-03 21:16:36 +02:00
  • 0d3d18be65 feat(selfhost): deploy a playwright image (#1625) Gergő Móricz 2025-06-03 19:19:08 +02:00
  • e89ecc4e4a feat: enhance metadata extraction by including 'itemprop' attribute in HTML (#1624) Ademílson Tonato 2025-06-03 17:16:46 +01:00
  • 406d696667
         Testing improvements (FIR-2209) (#1623) Gergő Móricz 2025-06-03 21:16:36 +02:00
  • 60525220a2 async saving to index Gergő Móricz 2025-06-03 21:16:13 +02:00
  • f8183af88e sws mog/testing-improvements Gergő Móricz 2025-06-03 21:02:10 +02:00
  • 7ba8dd25d6 no log Gergő Móricz 2025-06-03 20:50:29 +02:00
  • 11f27fd7f8 stupid Gergő Móricz 2025-06-03 20:46:21 +02:00
  • e297cf8a0d
         feat(selfhost): deploy a playwright image (#1625) Gergő Móricz 2025-06-03 19:19:08 +02:00
  • c5da155eb7 feat(selfhost): deploy a playwright image mog/publish-playwright Gergő Móricz 2025-06-03 19:18:14 +02:00
  • 1a6e1134c4 fix(api/tests/scrape/status): propagation time Gergő Móricz 2025-06-03 19:11:25 +02:00
  • 359711656a weird thing Gergő Móricz 2025-06-03 19:04:51 +02:00
  • 1dc6a443cb cache issues with billing test Gergő Móricz 2025-06-03 18:46:55 +02:00
  • 41897139da
         feat: enhance metadata extraction by including 'itemprop' attribute in HTML (#1624) Ademílson Tonato 2025-06-03 17:16:46 +01:00
  • 8d22fe9d97
         feat: enhance metadata extraction by including 'itemprop' attribute in HTML feat/itemprop-metadata Ademílson Tonato 2025-06-03 17:09:17 +01:00
  • f03676cf7a more timeout Gergő Móricz 2025-06-03 17:49:20 +02:00
  • d1b5e2ef47 revert Gergő Móricz 2025-06-03 17:02:27 +02:00
  • 1b3f037a26 wth Gergő Móricz 2025-06-03 16:55:39 +02:00
  • ede7aec1f9 ok fixed Gergő Móricz 2025-06-03 16:47:32 +02:00
  • 4e5feca3dd wow i'm an idiot Gergő Móricz 2025-06-03 16:41:28 +02:00
  • 71271cc4b8 try again Gergő Móricz 2025-06-03 16:38:40 +02:00
  • c75fad5e79 improve fns Gergő Móricz 2025-06-03 16:33:14 +02:00
  • 6ba57306c3 asd Gergő Móricz 2025-06-03 16:26:43 +02:00
  • 37d1de09f3 workflow test run Gergő Móricz 2025-06-03 16:25:02 +02:00
  • 2fe35a4e3d remove extraneous log Gergő Móricz 2025-06-03 16:24:02 +02:00
  • 39dd721781 clean up on map Gergő Móricz 2025-06-03 16:22:07 +02:00
  • 7426e54e6c further fixes Gergő Móricz 2025-06-03 16:12:31 +02:00
  • d7fef33224
         Merge branch 'main' into mog/index Gergő Móricz 2025-06-03 16:09:57 +02:00
  • da9a9b0d19 cleanup Gergő Móricz 2025-06-03 16:07:59 +02:00
  • d3b6b0da34 feat: re-enable billing tests Gergő Móricz 2025-06-03 15:41:40 +02:00
  • 1307648781 yeet ad blocking tests until further notice Gergő Móricz 2025-06-03 15:40:34 +02:00
  • e108ff3525 Update search.ts Nicolas 2025-06-02 23:46:55 -03:00
  • 9347de6a41 Update scrape.ts Nicolas 2025-06-02 23:15:59 -03:00
  • 86a9d3525b Update queue-jobs.ts Nicolas 2025-06-02 23:09:09 -03:00
  • cbc47305cc Update search.ts Nicolas 2025-06-02 23:09:02 -03:00
  • ce425d966f Merge branch 'nsc/bypass-billing-internal' Nicolas 2025-06-02 22:37:56 -03:00
  • 8c661f5329 Update scrape.ts nsc/bypass-billing-internal Nicolas 2025-06-02 22:37:49 -03:00
  • dc8cc99b1d
         Nick: bypass billing (#1622) Nicolas 2025-06-02 21:57:28 -03:00
  • 8967b31465 Nick: bypass billing Nicolas 2025-06-02 21:51:46 -03:00
  • bf919ceb82 Nick: __searchPreviewToken Nicolas 2025-06-02 21:16:34 -03:00
  • ef789ce8d7 Nick: __experimental Nicolas 2025-06-02 19:58:56 -03:00
  • 84d0a37d78 feat(api/crawl/ongoing): return more details Gergő Móricz 2025-06-02 23:33:15 +02:00
  • bf9929da3e fix: routers in wrong order Gergő Móricz 2025-06-02 23:22:06 +02:00
  • c26cbb9109 feat(api): GET /crawl/ongoing Gergő Móricz 2025-06-02 23:09:37 +02:00
  • 72be73473f
         feat(api/scrape): credits_billed column + handle billing for /scrape calls on worker side with stricter timeout enforcement (FIR-2162) (#1607) Gergő Móricz 2025-06-02 22:56:27 +02:00
  • 103abc83f2 fix searxng logging mog/stricten-timeout Gergő Móricz 2025-06-02 22:54:56 +02:00
  • 3805a8aa28 oof Gergő Móricz 2025-06-02 22:50:23 +02:00
  • 4167ec53eb
         fix(scrapeURL): only allow disabling the adblock on playwright (FIR-2200) (#1616) Gergő Móricz 2025-06-02 22:48:16 +02:00
  • 0b00833f2e
         apps/api(deps): bump the prod-deps group across 1 directory with 62 updates dependabot/npm_and_yarn/apps/api/prod-deps-a16878348f dependabot[bot] 2025-06-02 20:39:45 +00:00
  • 19d45f5fd0
         Merge branch 'main' into mog/stricten-timeout Gergő Móricz 2025-06-02 22:18:02 +02:00
  • f67baaca2f
         apps/api(deps-dev): bump the dev-deps group across 1 directory with 10 updates dependabot/npm_and_yarn/apps/api/dev-deps-6bbb3bf245 dependabot[bot] 2025-06-02 20:13:50 +00:00
  • 7a8be13220 remove indexes that are no longer used Gergő Móricz 2025-06-02 22:09:50 +02:00
  • 98ceda9bd5
         feat(search): ignore concurrency limit for search (FIR-2187) (#1617) Gergő Móricz 2025-06-02 22:07:44 +02:00
  • cdb37d17df Merge branch 'main' into mog/stricten-timeout Nicolas 2025-06-02 16:53:41 -03:00
  • 96f75f898f feat(search): only for low tier users for good DX mog/search-ignore-concurrency-limit Gergő Móricz 2025-06-02 21:49:45 +02:00
  • 95d1cd2f78 feat(search): ignore concurrency limit for search (temp) Gergő Móricz 2025-06-02 21:39:35 +02:00
  • 7aaabfec2a feat(api/tests/scrape): re-enable ad blocking tests mog/no-adblock-to-playwright Gergő Móricz 2025-06-02 21:23:48 +02:00
  • 9f8c70d9db fix(scrapeURL): only allow disabling the adblock on playwright Gergő Móricz 2025-06-02 21:20:36 +02:00
  • 014a99ef91 map benchmarks rafaelmmiller 2025-06-02 13:38:43 -03:00
  • bb4dfcf094 Revert "reenable billing tests" Gergő Móricz 2025-06-02 18:18:37 +02:00
  • 1396451d31 bump rust version pt.2 Gergő Móricz 2025-06-02 18:10:14 +02:00
  • 07fb651a91 bump rust version Gergő Móricz 2025-06-02 18:09:12 +02:00
  • 6a76ccfacb
         webhook param for crawl (#1609) Supasin Liulak 2025-06-02 23:08:32 +07:00
  • 8b864345e3 feat(api/test): index envs Gergő Móricz 2025-06-02 18:07:38 +02:00
  • 98236fdfa0 reenable billing tests Gergő Móricz 2025-06-02 18:04:42 +02:00
  • b9dc3e738e feat(index): FIRECRAWL_INDEX_WRITE_ONLY Gergő Móricz 2025-06-02 18:00:47 +02:00
  • b3eecdc81b chore(js-sdk): bump Gergő Móricz 2025-06-02 17:57:47 +02:00
  • 297d783585 feat(js-sdk): dontStoreInCache Gergő Móricz 2025-06-02 17:52:46 +02:00
  • b2aeb99dd4 disable cacheable lookup for self hosting tests Gergő Móricz 2025-06-02 17:45:24 +02:00
  • dceca07837 fix(api/tests/scrape): fix index test to work with batching Gergő Móricz 2025-06-02 17:41:45 +02:00
  • 18a7462fea feat(index): batch insert Gergő Móricz 2025-06-02 17:07:21 +02:00
  • 885e08e5d0
         apps/test-suite(deps-dev): bump the dev-deps group across 1 directory with 5 updates dependabot/npm_and_yarn/apps/test-suite/dev-deps-47a9dc5be7 dependabot[bot] 2025-06-02 07:02:58 +00:00
  • 40d6750e16
         apps/test-suite(deps): bump the prod-deps group dependabot/npm_and_yarn/apps/test-suite/prod-deps-10263567f6 dependabot[bot] 2025-06-02 07:00:00 +00:00
  • 369a8f6050 feat(map): ignoreIndex Gergő Móricz 2025-06-01 11:51:36 +02:00
  • 22c7685239 feat/added benchmark for scrapes rafaelmmiller 2025-05-30 18:38:20 -03:00
  • 9443a823b2 feat: script that generates all sdk examples for openapi python-sdk/e2e-tests-all-params rafaelmmiller 2025-05-30 17:09:03 -03:00
  • 1abc6f0f48 add comment to clarify is_scrape Gergő Móricz 2025-05-30 17:53:25 +02:00
  • daa7fc58f6 fix: proper level Gergő Móricz 2025-05-30 17:36:51 +02:00
  • 656c769f73 fix: abortsignal pre-check Gergő Móricz 2025-05-30 17:36:14 +02:00
  • 42c8adf9e5 feat(api/scrape): stricten timeout and handle billing and logging on queue-worker Gergő Móricz 2025-05-30 17:35:32 +02:00
  • 2faa45162c sdk(v3): all tests complete rafaelmmiller 2025-05-30 10:16:44 -03:00
  • 99d3db743d feat(scrapeURL/index): behaviour on non-200 index entries Gergő Móricz 2025-05-30 15:14:16 +02:00
  • 8c250426b3 feat(queue-worker/kickoff): use index links to kickoff crawl Gergő Móricz 2025-05-30 14:16:49 +02:00
  • 96c753f9a9 feat: use url split columns Gergő Móricz 2025-05-30 13:56:28 +02:00
  • 9297afd1ff Nick: search Nicolas 2025-05-29 17:00:13 -03:00
  • 91099e2dba sdk(v3) map ok rafaelmmiller 2025-05-29 16:21:42 -03:00
  • b7f54d874f sdk(v3): crawl + async crawl rafaelmmiller 2025-05-29 16:16:03 -03:00
  • a8e0482718 feat(search): bill for PDFs properly Gergő Móricz 2025-05-29 20:59:15 +02:00
  • a2f41fb650 feat(api/server): wait 60s for GCE load balancer drain timeout Gergő Móricz 2025-05-29 20:08:52 +02:00