AI Companion
Advanced topic: Real-time AI-generated musical accompaniment
Not included in this release
Elm (the AI Companion) is experimental, off by default, and excluded from App Store builds by default in this release. Everything below describes development-only behavior; it does not ship in the current public build.
Overview
The AI Companion listens to your playing and generates complementary musical parts in real-time using machine learning models.
Features
- Multi-track sequence-to-sequence — Generates bass, chords, pads, lead, and drums
- Context-aware — Adapts to your key, tempo, and playing style
- Real-time — Low-latency inference using CoreML
- Optional — Enable only when needed, never affects core routing
How It Works
- AI listens to incoming MIDI stream
- Analyzes musical context (key, chords, rhythm)
- Generates complementary parts
- Outputs to designated destinations
Enabling AI Companion
Settings → Advanced → AI Companion
- Enable — Turn on AI processing
- Model — Select trained model
- Destinations — Route AI output
- Responsiveness — Latency vs quality trade-off
Training (Advanced)
AI models are trained using PyTorch and converted to CoreML. See the project repository for technical details:
- AI Design Documentation (in repository)
- Training Schema Documentation (in repository)
Performance
AI Companion runs on Neural Engine (Apple Silicon) or GPU:
- Latency: 10-50ms depending on model size
- CPU usage: Minimal (dedicated hardware acceleration)
- Memory: ~200MB for models
Next Steps
- NeuroScript routing model — Integrate AI signals without impacting routing
- Performance Optimization — Minimize AI latency