DDSP-VST

DDSP-VST is a real-time neural audio plugin that runs in a digital audio workstation as a synthesizer or an effect. Musicians can turn a voice or other sound into an instrument, or play neural instruments over MIDI, and train their own models for free.

The plugin tracks pitch with a small distilled neural network and measures loudness, then a second network predicts controls for a harmonic synthesizer and a noise synthesizer. This approach, called DDSP, combines classic DSP building blocks with machine learning so it stays light enough for low-latency use.

A companion web trainer builds a custom instrument model from a few minutes of audio at no cost, and models can be shared with others. The plugin is open source, so developers can add features of their own.