Meta releases Muse Spark Open as an open-weight model

Meta releases Muse Spark Open, an 816-billion-parameter multimodal model designed for code and AI agents, featuring a one-million-token context window.

816 billion parameters, a context length of one million tokens, and the ability to analyze text, images, video, and audio recordings in a single environment. Muse Spark Open becomes Meta's new open-weights model, primarily intended for software development and autonomous agents. The group is releasing two downloadable versions and accompanying this release with integrations for Hugging Face, inference tools, and the Apple ecosystem. This open release concerns a large-scale model whose local execution nevertheless requires hardware with several hundred gigabytes of memory. 816 billion parameters, but 42 billion active

Muse Spark Open is built on a Mixture-of-Experts (MoE) architecture. Unlike a dense model that mobilizes all of its parameters, it selects a subset of its components for each processed token.

The model has approximately 816 billion total parameters, with 42 billion active per token. Each MoE block contains 256 experts, eight of which are selected during processing.

This design reduces the amount of compute required for generation without eliminating the constraints related to storing the full model in memory.

The experts represent approximately 97% of the parameters. The language component consists of 62 layers, complemented by visual perception and audio processing modules.

Muse Spark Open notably succeeds Muse Glimmer, a more compact model from the same family released in August. Both share a tokenization system and certain multimodal processing features, but Muse Spark Open adopts a significantly larger architecture. One million tokens for extended tasks

The context window reaches 1,048,576 tokens, combining textual information and representations of visual and audio content.

This capacity targets operations where an agent must retain a large amount of information while working: inspecting a vast software repository, analyzing numerous documents, reviewing tool call histories, or exploring a project requiring multiple correction cycles.

Meta specifically trained Muse Spark Open for software development. The model can examine a repository, propose modifications, write code, execute tools, and revise its work based on the obtained results.

It also features multiple reasoning effort levels, from low to xhigh, making it possible to adapt generation resources to the complexity of the requests. However, reasoning cannot be completely disabled.

Maximum context capacity remains dependent on the infrastructure used. For instance, certain configurations published for vLLM limit sessions to 131,072 tokens, even though the model theoretically supports one million tokens. Images, videos, and audio in a single model

Muse Spark Open accepts four input modalities: text, image, video, and audio. Its responses are generated in text form.

A visual encoder of approximately 1.8 billion parameters transforms images into representations usable by the language model. It is also used to process images extracted from videos.

By default, videos are sampled at two frames per second, capped at 96 frames. This setting can be adjusted according to needs and available resources.

Audio processing relies on an encoder capable of producing a representation every 80 milliseconds, or 12.5 audio units per second.

This information can be combined within the same conversation. An agent can thus review a screenshot, listen to a voice instruction, analyze a screen recording, and use these elements to proceed with a computer task.

This multimodality applies to content understanding. Muse Spark Open does not directly generate images, videos, or audio files. A giant model on Apple computers

One of the key aspects of the launch involves optimizations developed with Apple to run Muse Spark Open on machines powered by Apple Silicon.

Meta notably highlights a setup based on two Mac Studio M5 Ultra machines, leveraging adaptations to the MLX framework.

Hugging Face also documents several experiments on Apple machines with large amounts of unified memory.

A 2-bit quantized version was tested using llama.cpp on a Mac M3 Ultra equipped with 512 GB of memory. It uses roughly 304 GB and reaches 18 generated tokens per second under the reported conditions.

Another 3-bit adaptation, designed for oMLX, utilizes about 344 GB and achieves 21 tokens per second on the same category of machine.

These results show that local execution is not exclusively reserved for traditional GPU servers. However, they do not mean that Muse Spark Open can run on an ordinary consumer computer.

Available memory remains the deciding factor, particularly for long context windows, which add substantial memory requirements on top of the model's base needs. Two checkpoints and multiple compatible infrastructures

Meta is releasing two main versions of Muse Spark Open on Hugging Face.

The BF16 version preserves high numerical precision. Its files represent approximately 1.5 TB, and deployment examples rely on distributed setups, such as 32 NVIDIA H200 GPUs.

The NVFP4 version uses a more compact numerical representation tailored for NVIDIA Blackwell GPUs. It aims to reduce the memory required for inference, with a reported minimal trade-off in accuracy.

The documentation outlines execution possibilities with Transformers, llama.cpp, SGLang, vLLM, and MLX.

Supported features vary across engines. The SGLang integration covers text, image, video, and audio, whereas the baseline vLLM setup described at launch primarily focuses on text and image.

The model is also offered on hosted infrastructures, notably through Hugging Face Inference Endpoints, as well as providers like Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter, and Together AI. LoRA fine-tuning on 64 H100 GPUs

The availability of open weights allows technical teams to fine-tune and specialize the model according to their needs.

Hugging Face notably published an additional training method using TRL and LoRA, which consists of adapting specific components of the model without retraining all of its parameters.

In the documented