{"id":249244,"date":"2025-12-24T13:03:10","date_gmt":"2025-12-24T13:03:10","guid":{"rendered":"https:\/\/www.europesays.com\/ie\/249244\/"},"modified":"2025-12-24T13:03:10","modified_gmt":"2025-12-24T13:03:10","slug":"cactus-v1-cross-platform-llm-inference-on-mobile-with-zero-latency-and-full-privacy","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ie\/249244\/","title":{"rendered":"Cactus v1: Cross-Platform LLM Inference on Mobile with Zero Latency and Full Privacy"},"content":{"rendered":"

Cactus<\/a>, a Y Combinator-backed startup, enables local AI inference to mobile phones, wearables, and other low-power devices through cross-platform, energy-efficient kernels and a native runtime. It delivers sub-50ms time-to-first-token for on-device inference, eliminates network latency, and defaults to complete privacy.<\/p>\n

Version v1<\/a> of the SDK, now in beta, improves performance on lower-end hardware and adds optional cloud fallback to ensure greater reliability.<\/p>\n

Cactus enables developers to deploy models locally in any app using its SDK, which offers native bindings for React Native<\/a>, Flutter<\/a>, and Kotlin Multiplatform<\/a>. Native Swift<\/a> support is still minimal and less mature as support for other languages, but iOS developers can use the Kotlin Multiplatform bindings within their Swift apps.<\/p>\n

On iOS and Android devices, Cactus takes a more general approach to on-device AI inference than the platform-native solutions offered by Apple and Google, Apple Foundation frameworks and Google AI Edge, which are platform-specific and expose only a limited, vendor-controlled set of capabilities. Cactus supports a wide variety of models, including Qwen, Gemma, Llama, DeepSeek, Phi, Mistral and many others. For improved efficiency and performance, it supports a range of quantization levels, from FP32 down to 2-bit.<\/p>\n

<\/p>\n

Cactus provides built-in model versioning and over-the-air updates. You can push new model versions to users without requiring app updates. The SDK handles downloading, caching, and switching between model versions seamlessly in the background<\/p>\n

\n<\/p><\/blockquote>\n

The Cactus SDK also supports fallback to cloud-based models for complex or large-context tasks, ensuring robustness and high availability. According to Cactus, “this addresses one of the most common requests from v0 users who needed guaranteed response times for critical user-facing features”.<\/p>\n

In version v1, Cactus has completely overhauled thitseir inference engine, transitioning from GGUF to a proprietary format and including optimized ARM-CPU kernels for better performance across all supported devices. The SDKs were rebuilt from the ground up to improve API consistency across languages, while maintaining backward compatibility wherever possible. The new version also introduces updated telemetry and monitoring systems.<\/p>\n

<\/p>\n

[This] give[s] developers granular insights into their AI model performance, usage patterns, and potential optimization opportunities. This data-driven approach enables teams to make informed decisions about model selection and deployment strategies.<\/p>\n

\n<\/p><\/blockquote>\n

In addition to LLM inference, Cactus v1 also supports tool calling and voice transcription. The Flutter SDK additionally offers RAG fine-tuning, while the React Native SDK provides image embedding. According to Cactus\u2019 roadmap, these features will soon be extended to other SDK variants, along with voice synthesis.<\/p>\n

To better appreciate Cactus’ performance across distinct devices<\/a>, the company has published benchmarks using the LFM2-VL-450m and Whisper-Small models, measuring metrics such as tokens per second, prefill\/decode, and more. The benchmarks employ INT8 quantization and leverage NPUs for real-time performance and large context handling. Results show that a Mac M4 Pro achieves 173\u202ftok\/s, an iPhone 17\u202fPro reaches 136\u202ftok\/s, a Galaxy S25\u202fUltra 91\u202ftok\/s, and a Raspberry Pi 5 24\u202ftok\/s.<\/p>\n

For reference, the smallest model, gemma-3-270m-it, has a compressed footprint of 172\u202fMB and supports only completion. Qwen3-0.6B occupies 394\u202fMB and supports completion, tool calling, embedding, and speech. On the upper-end, Gemma-3-1b-it requires 642\u202fMB, while Qwen3-1.7B takes 1,161\u202fMB.<\/p>\n

Cactus has also published chat apps for iOS<\/a> and Android<\/a> that can use distinct LLMs, allowing developers to benchmark latency and throughput when accessed through the Cactus SDK.<\/p>\n

Cactus is open-source free for students, educators, non-profits, and small businesses. It can be cloned from GitHub<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"Cactus, a Y Combinator-backed startup, enables local AI inference to mobile phones, wearables, and other low-power devices through…\n","protected":false},"author":2,"featured_media":249245,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[74],"tags":[291,821,311,129812,7266,18,823,19,988,17,14632,14630,279,129813,104484,82,23186],"class_list":{"0":"post-249244","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-technology","8":"tag-ai","9":"tag-android","10":"tag-apple","11":"tag-cactus-on-device-inference","12":"tag-development","13":"tag-eire","14":"tag-google","15":"tag-ie","16":"tag-ios","17":"tag-ireland","18":"tag-large-language-models","19":"tag-ml-data-engineering","20":"tag-mobile","21":"tag-mobile-privacy","22":"tag-raspberrypi","23":"tag-technology","24":"tag-wearable-tech"},"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@ie\/115774685521594697","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts\/249244","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/comments?post=249244"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts\/249244\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/media\/249245"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/media?parent=249244"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/categories?post=249244"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/tags?post=249244"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}