From Voice to Analysis: Designing the Voice Processing Pipeline

A voice pipeline is a structure that, rather than using raw audio directly for judgment, refines it into an increasingly meaningful signal through the stages capture → preprocessing/noise reduction → feature extraction → (optional) STT → modeling. Each stage produces the input the next stage uses, and if an earlier stage is low quality, no matter how sophisticated the later stages are, there are limits.

This article does not explain a specific product’s implementation. It lays out the general design stages and trade-offs you repeatedly encounter when building voice-based systems.

---

01. Capture: The Quality of the Input Sets the Ceiling

The starting point of the pipeline is the stage that records the voice. What is decided here is the sample rate, channels, encoding, and under what conditions the recording is made.

There are two principles for the capture stage. First, secure the quality the later stages require. Second, collect only the minimum needed for the purpose. Retaining excessively long raw audio for a long time may be advantageous later, but privacy risk grows along with it. So the capture stage already decides “what will not be kept.”

02. Preprocessing and Noise Reduction: The Inconspicuous Foundation

Real-world voice mixes in ambient noise, reverberation, microphone characteristics, and volume variance. If you do not clean these up, later stages risk learning differences in the environment rather than a person’s state.

• Volume normalization reduces loudness differences due to device and distance.
• Noise suppression and voice activity detection (VAD) keep only the actual speech segments.
• If needed, resampling and filtering match the input format of later stages.

This stage is not flashy but is the foundation that determines overall quality. That said, excessive noise reduction can erase even the subtle signals related to state, so this too is a matter of balance.

03. Feature Extraction: Turning Manner into Numbers

This is the stage of extracting features from the refined voice that may relate to state. What matters here is not the content of speech but the manner of speaking.

• Prosody: fundamental frequency (F0), speaking rate, pause patterns, energy.
• Voice quality: stability metrics such as jitter, shimmer, and HNR.
• Spectral: timbre- and articulation-related features such as MFCCs and formants.

Why such features can be linked to state is covered in the Voice Biomarker Guide, and the specific connection to stress is covered in Voice Stress Biomarkers. If you separate the feature extraction method into a single module, later changes to the method do not require touching other parts of the pipeline wholesale.

04. STT: An Optional Stage Added Only When Needed

Speech recognition (STT, Speech-to-Text) is not always a required stage. If you only need the manner of speaking, analysis is possible without text transcription. STT is added optionally only when you need to obtain linguistic signals from the content of speech.

Adding STT brings two burdens. One is the transcription accuracy problem (errors increase in noisy, accented, or multilingual environments), and the other is the privacy burden created by the transcribed text. So STT is used only when strictly necessary for the purpose, and only once you have decided how the resulting text will be retained and processed.

05. Modeling: From Signal to Interpretation

The final stage is producing an interpretable signal based on the extracted features. The core principle here is that we look at change relative to a personal baseline, not absolute values.

Since people differ in their usual pitch and speaking habits, applying a single absolute standard to everyone can mistake individual differences for changes in state. So we take a person’s usual pattern as a baseline and observe change using data from many points in time. And we treat voice not as a decisive judgment but as a supporting signal interpreted alongside other signals.

06. The Trade-off That Runs Through It All: On-device versus Server

When designing the pipeline, the decision that spans nearly every stage is where the processing happens.

• On-device: the raw audio does not leave the device, which favors privacy and latency. But it is constrained in compute resources and model updates.
• Server: flexible, powerful, and easy to update. But it requires managing the risk that comes with moving data.
• In many cases a hybrid is the answer. Sensitive preprocessing and feature extraction happen close to the device, while heavy modeling happens on the server.

Latency, battery, and cost are also intertwined here, so there is no single right answer. What matters is loosely coupling each stage so it can be swapped and evaluated independently. That way, when privacy requirements change or a better method appears, you do not have to rebuild the whole pipeline.

How a pipeline like this is assembled within a single product is best understood alongside the flow in the KKEBI Architecture Overview.

---

References

1. Cummins N, Scherer S, Krajewski J, et al. A review of depression and suicide risk assessment using speech analysis. Speech Communication. 2015;71:10–49.
2. Teixeira JP, Oliveira C, Lopes C. Vocal acoustic analysis – Jitter, Shimmer and HNR parameters. Procedia Technology. 2013;9:1112–1122.
3. Low DM, Bentley KH, Ghosh SS. Automated assessment of psychiatric disorders using speech: A systematic review. Laryngoscope Investigative Otolaryngology. 2020;5(1):96–116.