Provide a metal or jazz track and get the sheet music for each instrument

Kyutai and Mirelo AI released MuScriptor, an open multi-instrument music transcription model trained on 170,000 real tracks to output MIDI files.

The Paris-based laboratory Kyutai, along with Mirelo AI, has released MuScriptor, a multi-instrument music transcription model. Given a recording in any genre—pop, classical, metal, jazz—it outputs the transcription of the separated instruments in MIDI format. The team presents it as the best open model in the field to date.

For years, automatic transcription has stumbled over the same obstacle: the lack of training data, meaning real audio aligned with note annotations. Models were therefore trained mostly on synthetic audio and generalized poorly to real recordings. This is where MuScriptor's central contribution lies: a dataset of 170,000 real tracks (11,000 hours) of various genres, paired with their MIDI transcriptions.

Training is conducted in three stages. Pre-training on 1.5 million MIDI files synthesized in numerous variations, followed by fine-tuning on real data, presented as the primary factor for quality, and finally, reinforcement learning (RL) post-training on 300 tracks with hand-verified transcriptions. The architecture remains intentionally simple: a decoder-only transformer that predicts a stream of tokens from the spectrogram of a five-second excerpt, with longer tracks processed in slices. Sizes range from 100 million to 1.3 billion parameters.

The authors add instrument-presence conditioning, which stabilizes predictions throughout a track and allows the user to adjust their transcription. Weights, code, paper, and an online demo interface are public.