After nearly a month of training, refactoring, fine tuning and losing hair, I hit a wall with my disentangler / decoder models. No matter what I did, no matter how sophisticated I made the decoder, I could not get it to reconstruct the high-frequency parts of the voice outputs. My investigations lead me to take another look at the embeddings being output by the disentangler, and sure enough, the high-frequency elements were almost non-existent.
The disentangler was doing an amazing job at separating voice from noise. I mean I was absolutely amazed at how well it was working, and for the longest time, I was convinced that my decoder was the problem. It too was doing a great job of reconstructing the voice and noise, but everything seemed a bit muffled.
To give you an example of how the disentangler / decoder performed here is some audio that was processed through the disentangler then through the decoder.
The output
To give you a sense of what the disentangler had to work with
And finally, this is what I was hoping it would sound like (the original voice clip before mixing in the noise)
So you can see, while it did a good job of really separating the voice from the noise, the resulting voice was too muffled and really lacked the important vocal elements needed to create crisp, rich voice reconstruction.
Ultimately, I needed to rethink how the disentangler model should be architected and settled on a slightly more modular and flexible approach. Firstly, I moved away from treating both voice and noise with the same set of conditions, this was a major change and maybe one might say should have been a design consideration from the start. But starting with the obvious doesn't help me find the limitations I need to understand to build better more reliable AGI.
The enhanced disentanglers new core philosophy has become:
Primary: Strong self-learning (NTXENT + BYOL - Same as before) for AGI compatibility
Secondary: Supervised guidance using clean voice / noise targets for audio quality
Specialised processing: Voice-specific and noise-specific feature extraction ā This is new
Maintains both batch and streaming capabilities for different use cases
Further, I pushed the model to be a bit more detailed. Previously I was using 25ms frames with 10ms hops from the raw embeddings and 100 Hz for the PCEN. Now it uses a hop length of~5ms frames with a much more feature rich 400Hz for stronger temperal dimensions after embedding (2.5ms per embedding frame). Given the inference speeds I saw with the original model, this extra level of detail was still quite achievable without impacting the models ability to remain real-time.
Before we had multiple embeddings:
Voice/Noise Raw, Voice/Noise PCEN, Voice/Noise Fused, Raw Features and PCEN Features. All 128-dim (except the two features which were 64-dim). In the new model I still have multiple embeddings, but a much more feature focus set per voice / noise path. These are
More importantly, the loss weights were a lot more skewed to preserving high-frequency elements in the voice path to ensure reconstruction could be achieved with much more crispness than I saw in the first prototype.
The training approach, while largely the same as the first model was enhanced with some supervised guidance, but in maintaining the primary requirement that this be AGI ready, I kept that to 20% of the total weights.
Training Approach:
Total Loss Weights:
āāā NTXENT (Contrastive): 50% - Primary disentanglement signal
So the big design changes were all centered around domain-specific feature extraction rather than simply separating voice and noise loosely.
Training was done using 1.9 million 2 second triplets with clean voice, clean noise and mix of both covering 10 languages, with 1 million English samples and 100,000 samples from other languages with strong variations from English (Arabic, Mandarin, Swahili etc).
It took 78 hours of continuous training, and 67,000 steps before the model reached very good metrics and plateaued.
Now I am generating the dataset for the new decoder with the new disentangler (1 million samples). While I have done some tests on the embeddings and the numbers look much better (especially the pain point of HF absence), only after the decoder has been trained and I going to know for sure if the overhaul has paid of.. It's always a tense few days training and waiting to see if things are improving, but that's part of the journey I suppose.
Training the decoder starts tomorrow (The dataset creation has a few hundred thousand more samples to prepare).
I'm excited to see the results in a few days time.
Rebuilding Sound from its DNAIn my audio processing pipeline, I've two components that work together to transform messy, mixed audio into clear voice and noise streams.The D...
You may have read a bit about the exciting new project I am working on - Spwig - an all inclusive e-commerce platform.
Ā
One thing that has been expl...
Another wee side project Iām working on is an e-commerce platform. Iām tired of paying for subscriptions, plugins that break and slow my store dow...
My AI Setup - Power, Efficiency, and Color As an AI consultant and model creator, my tools are my partners in experimentation, discovery, and creation...
After nearly a month of training, refactoring, fine tuning and losing hair, I hit a wall with my disentangler / decoder models. No matter what I did, ...
The other morning I was sleeping in a little while the morning sky was still dark. My wife had woken up earlier than me to tend to our boy, and at som...
