{"id":209344,"date":"2025-06-24T03:29:10","date_gmt":"2025-06-24T03:29:10","guid":{"rendered":"https:\/\/www.europesays.com\/uk\/209344\/"},"modified":"2025-06-24T03:29:10","modified_gmt":"2025-06-24T03:29:10","slug":"introducing-mu-language-model-and-how-it-enabled-the-agent-in-windows-settings","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/uk\/209344\/","title":{"rendered":"Introducing Mu language model and how it enabled the agent in Windows Settings\u202f"},"content":{"rendered":"

We are excited to introduce our newest on-device small language model, Mu. This model addresses scenarios that require inferring complex input-output relationships and has been designed to operate efficiently, delivering high performance while running locally. Specifically, this is the language model that powers the agent in Settings<\/a>,\u202f available to Windows Insiders in the Dev Channel with Copilot+ PCs<\/a>, by mapping natural language input queries to Settings function calls.<\/p>\n

Mu is fully offloaded onto the Neural Processing Unit (NPU) and responds at over 100 tokens per second, meeting the demanding UX requirements of the agent in Settings scenario. This blog will provide further details on Mu\u2019s design and training and how it was fine-tuned to build the agent in Settings.<\/p>\n

Model training Mu<\/strong><\/p>\n

Enabling Phi Silica<\/a> to run on NPUs provided us with valuable insights about tuning models for optimal performance and efficiency. These informed the development of Mu, a micro-sized, task-specific language model designed from the ground up to run efficiently on NPUs and edge devices.<\/p>\n

\"Encoder-Decoder Architecture compared to Decoder-only Architecture<\/p>\n

Mu is an efficient 330M encoder\u2013decoder language model optimized for small-scale deployment, particularly on the NPUs on Copilot+ PCs. It follows a transformer encoder\u2013decoder architecture, meaning an encoder first converts the input into a fixed-length latent representation, and a decoder then generates output tokens based on that representation.<\/p>\n

This design yields significant efficiency benefits. The figure above illustrates how an encoder-decoder reuses the input\u2019s latent representation whereas a decoder-only must consider the full input + output sequence. By separating the input tokens from output tokens, Mu\u2019s one-time encoding greatly reduces computation and memory overhead. In practice, this translates to lower latency and higher throughput on specialized hardware. For example, on a Qualcomm Hexagon NPU (a mobile AI accelerator), Mu\u2019s encoder\u2013decoder approach achieved about 47% lower first-token latency and 4.7\u00d7 higher decoding speed compared to a decoder-only model of similar size. These gains are crucial for on-device and real-time applications.<\/p>\n

Mu\u2019s design was carefully tuned for the constraints and capabilities of NPUs. This involved adjusting model architecture and parameter shapes to better fit the hardware\u2019s parallelism and memory limits. We chose layer dimensions (such as hidden sizes and feed-forward network widths) that align with the NPU\u2019s preferred tensor sizes and vectorization units, ensuring that matrix multiplications and other operations run at peak efficiency. We also optimized the parameter distribution between the encoder and decoder \u2013 empirically favoring a 2\/3\u20131\/3 split (e.g. 32 encoder layers vs 12 decoder layers in one configuration) to maximize performance per parameter.<\/p>\n

Additionally, Mu employs weight sharing in certain components to reduce the total parameter count. For instance, it ties the input token embeddings and output embeddings, so that one set of weights is used for both representing input tokens and generating output logits. This not only saves memory (important on memory-constrained NPUs) but can also improve consistency between encoding and decoding vocabularies.<\/p>\n

Finally, Mu restricts its operations to those NPU-optimized operators supported by the deployment runtime. By avoiding any unsupported or inefficient ops, Mu fully utilizes the NPU\u2019s acceleration capabilities. These hardware-aware optimizations collectively make Mu highly suited for fast, on-device inference.<\/p>\n

Packing performance in a tenth the size<\/strong>
\n\"Graph<\/p>\n

Mu adds three key transformer upgrades that squeeze more performance from a smaller model:<\/p>\n