Peter Vary is former Head of the Institute of Communication Systems at RWTH Aachen University, Germany. Professor Vary is a Fellow of IEEE, EURASIP, and ITG, and has been a Distinguished Lecturer of the IEEE Signal Processing Society.

Rainer Martin is Head of the Institute of Communication Acoustics at Ruhr-Universität Bochum, Germany. Professor Martin is a Fellow of the IEEE.
Both authors have been actively involved in speech processing research and teaching over several decades.

Preface xv

1 Introduction 1

2 Models of Speech Production and Hearing 5

2.1 Sound Waves 5

2.2 Organs of Speech Production 7

2.3 Characteristics of Speech Signals 9

2.4 Model of Speech Production 10

2.4.1 Acoustic Tube Model of the Vocal Tract 12

2.4.2 Discrete Time All-Pole Model of the Vocal Tract 19

2.5 Anatomy of Hearing 25

2.6 Psychoacoustic Properties of the Auditory System 27

2.6.1 Hearing and Loudness 27

2.6.2 Spectral Resolution 29

2.6.3 Masking 31

2.6.4 Spatial Hearing 32

2.6.4.1 Head-Related Impulse Responses and Transfer Functions 33

2.6.4.2 Law of The First Wavefront 34

References 35

3 Spectral Transformations 37

3.1 Fourier Transform of Continuous Signals 37

3.2 Fourier Transform of Discrete Signals 38

3.3 Linear Shift Invariant Systems 41

3.3.1 Frequency Response of LSI Systems 42

3.4 The z-transform 42

3.4.1 Relation to Fourier Transform 43

3.4.2 Properties of the ROC 44

3.4.3 Inverse z-Transform 44

3.4.4 z-Transform Analysis of LSI Systems 46

3.5 The Discrete Fourier Transform 47

3.5.1 Linear and Cyclic Convolution 48

3.5.2 The DFT of Windowed Sequences 51

3.5.3 Spectral Resolution and Zero Padding 54

3.5.4 The Spectrogram 55

3.5.5 Fast Computation of the DFT: The FFT 56

3.5.6 Radix-2 Decimation-in-Time FFT 57

3.6 Fast Convolution 60

3.6.1 Fast Convolution of Long Sequences 60

3.6.2 Fast Convolution by Overlap-Add 61

3.6.3 Fast Convolution by Overlap-Save 61

3.7 Analysis-Modification-Synthesis Systems 64

3.8 Cepstral Analysis 66

3.8.1 Complex Cepstrum 67

3.8.2 Real Cepstrum 69

3.8.3 Applications of the Cepstrum 70

3.8.3.1 Construction of Minimum-Phase Sequences 70

3.8.3.2 Deconvolution by Cepstral Mean Subtraction 71

3.8.3.3 Computation of the Spectral Distortion Measure 72

3.8.3.4 Fundamental Frequency Estimation 73

References 75

4 Filter Banks for Spectral Analysis and Synthesis 79

4.1 Spectral Analysis Using Narrowband Filters 79

4.1.1 Short-Term Spectral Analyzer 83

4.1.2 Prototype Filter Design for the Analysis Filter Bank 86

4.1.3 Short-Term Spectral Synthesizer 87

4.1.4 Short-Term Spectral Analysis and Synthesis 88

4.1.5 Prototype Filter Design for the Analysis-Synthesis filter bank 90

4.1.6 Filter Bank Interpretation of the DFT 92

4.2 Polyphase Network Filter Banks 94

4.2.1 PPN Analysis Filter Bank 95

4.2.2 PPN Synthesis Filter Bank 101

4.3 Quadrature Mirror Filter Banks 104

4.3.1 Analysis-Synthesis Filter Bank 104

4.3.2 Compensation of Aliasing and Signal Reconstruction 106

4.3.3 Efficient Implementation 109

4.4 Filter Bank Equalizer 112

4.4.1 The Reference Filter Bank 112

4.4.2 Uniform Frequency Resolution 113

4.4.3 Adaptive Filter Bank Equalizer: Gain Computation 117

4.4.3.1 Conventional Spectral Subtraction 117

4.4.3.2 Filter Bank Equalizer 118

4.4.4 Non-uniform Frequency Resolution 120

4.4.5 Design Aspects & Implementation 122

References 123

5 Stochastic Signals and Estimation 127

5.1 Basic Concepts 127

5.1.1 Random Events and Probability 127

5.1.2 Conditional Probabilities 128

5.1.3 Random Variables 129

5.1.4 Probability Distributions and Probability Density Functions 129

5.1.5 Conditional PDFs 130

5.2 Expectations and Moments 130

5.2.1 Conditional Expectations and Moments 131

5.2.2 Examples 131

5.2.2.1 The Uniform Distribution 132

5.2.2.2 The Gaussian Density 132

5.2.2.3 The Exponential Density 132

5.2.2.4 The Laplace Density 133

5.2.2.5 The Gamma Density 134

5.2.2.6 chi²-Distribution 134

5.2.3 Transformation of a Random Variable 135

5.2.4 Relative Frequencies and Histograms 136

5.3 Bivariate Statistics 137

5.3.1 Marginal Densities 137

5.3.2 Expectations and Moments 137

5.3.3 Uncorrelatedness and Statistical Independence 138

5.3.4 Examples of Bivariate PDFs 139

5.3.4.1 The Bivariate Uniform Density 139

5.3.4.2 The Bivariate Gaussian Density 139

5.3.5 Functions of Two Random Variables 140

5.4 Probability and Information 141

5.4.1 Entropy 141

5.4.2 Kullback-Leibler Divergence 141

5.4.3 Cross-Entropy 142

5.4.4 Mutual Information 142

5.5 Multivariate Statistics 142

5.5.1 Multivariate Gaussian Distribution 143

5.5.2 Gaussian Mixture Models 144

5.6 Stochastic Processes 145

5.6.1 Stationary Processes 145

5.6.2 Auto-Correlation and Auto-Covariance Functions 146

5.6.3 Cross-Correlation and Cross-Covariance Functions 147

5.6.4 Markov Processes 147

5.6.5 Multivariate Stochastic Processes 148

5.7 Estimation of Statistical Quantities by Time Averages 150

5.7.1 Ergodic Processes 150

5.7.2 Short-Time Stationary Processes 150

5.8 Power Spectrum and its Estimation 151

5.8.1 White Noise 152

5.8.2 The Periodogram 152

5.8.3 Smoothed Periodograms 153

5.8.3.1 Non Recursive Smoothing in Time 153

5.8.3.2 Recursive Smoothing in Time 154

5.8.3.3 Log-Mel Filter Bank Features 154

5.8.4 Power Spectra and Linear Shift-Invariant Systems 156

5.9 Statistical Properties of Speech Signals 157

5.10 Statistical Properties of DFT Coefficients 157

5.10.1 Asymptotic Statistical Properties 158

5.10.2 Signal-Plus-Noise Model 159

5.10.3 Statistics of DFT Coefficients for Finite Frame Lengths 160

5.11 Optimal Estimation 162

5.11.1 MMSE Estimation 163

5.11.2 Estimation of Discrete Random Variables 164

5.11.3 Optimal Linear Estimator 164

5.11.4 The Gaussian Case 165

5.11.5 Joint Detection and Estimation 166

5.12 Non-Linear Estimation with Deep Neural Networks 167

5.12.1 Basic Network Components 168

5.12.1.1 The Perceptron 168

5.12.1.2 Convolutional Neural Network 170

5.12.2 Basic DNN Structures 170

5.12.2.1 Fully-Connected Feed-Forward Network 171

5.12.2.2 Autoencoder Networks 171

5.12.2.3 Recurrent Neural Networks 172

5.12.2.4 Time Delay, Wavenet, and Transformer Networks 175

5.12.2.5 Training of Neural Networks 175

5.12.2.6 Stochastic Gradient Descent (SGD) 176

5.12.2.7 Adaptive Moment Estimation Method (ADAM) 176

References 177

6 Linear Prediction 181

6.1 Vocal Tract Models and Short-Term Prediction 181

6.1.1 All-Zero Model 182

6.1.2 All-Pole Model 183

6.1.3 Pole-Zero Model 183

6.2 Optimal Prediction Coefficients for Stationary Signals 187

6.2.1 Optimum Prediction 187

6.2.2 Spectral Flatness Measure 190

6.3 Predictor Adaptation 192

6.3.1 Block-Oriented Adaptation 192

6.3.1.1 Auto-Correlation Method 193

6.3.1.2 Covariance Method 194

6.3.1.3 Levinson-Durbin Algorithm 196

6.3.2 Sequential Adaptation 201

6.4 Long-Term Prediction 204

References 209

7 Quantization 211

7.1 Analog Samples and Digital Representation 211

7.2 Uniform Quantization 212

7.3 Non-uniform Quantization 219

7.4 Optimal Quantization 227

7.5 Adaptive Quantization 228

7.6 Vector Quantization 232

7.6.1 Principle 232

7.6.2 The Complexity Problem 235

7.6.3 Lattice Quantization 236

7.6.4 Design of Optimal Vector Code Books 236

7.6.5 Gain-Shape Vector Quantization 239

7.7 Quantization of the Predictor Coefficients 240

7.7.1 Scalar Quantization of the LPC Coefficients 241

7.7.2 Scalar Quantization of the Reflection Coefficients 241

7.7.3 Scalar Quantization of the LSF Coefficients 243

References 246

8 Speech Coding 249

8.1 Speech-Coding Categories 249

8.2 Model-Based Predictive Coding 253

8.3 Linear Predictive Waveform Coding 255

8.3.1 First-Order DPCM 255

8.3.2 Open-Loop and Closed-Loop Prediction 258

8.3.3 Quantization of the Residual Signal 259

8.3.3.1 Quantization with Open-Loop Prediction 259

8.3.3.2 Quantization with Closed-Loop Prediction 261

8.3.3.3 Spectral Shaping of the Quantization Error 262

8.3.4 ADPCM with Sequential Adaptation 266

8.4 Parametric Coding 268

8.4.1 Vocoder Structures 268

8.4.2 LPC Vocoder 271

8.5 Hybrid Coding 272

8.5.1 Basic Codec Concepts 272

8.5.1.1 Scalar Quantization of the Residual Signal 274

8.5.1.2 Vector Quantization of the Residual Signal 276

8.5.2 Residual Signal Coding: RELP 279

8.5.3 Analysis by Synthesis: CELP 282

8.5.3.1 Principle 282

8.5.3.2 Fixed Code Book 283

8.5.3.3 Long-Term Prediction, Adaptive Code Book 287

8.6 Adaptive Postfiltering 289

8.7 Speech Codec Standards: Selected Examples 293

8.7.1 GSM Full-Rate Codec 295

8.7.2 EFR Codec 297

8.7.3 Adaptive Multi-Rate Narrowband Codec (AMR-NB) 299

8.7.4 ITU-T/G.722: 7 kHz Audio Coding within 64 kbit/s 301

8.7.5 Adaptive Multi-Rate Wideband Codec (AMR-WB) 301

8.7.6 Codec for Enhanced Voice Services (EVS) 303

8.7.7 Opus Codec IETF RFC 6716 306

References 307

9 Concealment of Erroneous or Lost Frames 313

9.1 Concepts for Error Concealment 314

9.1.1 Error Concealment by Hard Decision Decoding 315

9.1.2 Error Concealment by Soft Decision Decoding 316

9.1.3 Parameter Estimation 318

9.1.3.1 MAP Estimation 318

9.1.3.2 MS Estimation 318

9.1.4 The A Posteriori Probabilities 319

9.1.4.1 The A Priori Knowledge 320

9.1.4.2 The Parameter Distortion Probabilities 320

9.1.5 Example: Hard Decision vs. Soft Decision 321

9.2 Examples of Error Concealment Standards 323

9.2.1 Substitution and Muting of Lost Frames 323

9.2.2 AMR Codec: Substitution and Muting of Lost Frames 325

9.2.3 EVS Codec: Concealment of Lost Packets 329

9.3 Further Improvements 330

References 331

10 Bandwidth Extension of Speech Signals 335

10.1 BWE Concepts 337

10.2 BWE using the Model of Speech Production 339

10.2.1 Extension of the Excitation Signal 340

10.2.2 Spectral Envelope Estimation 342

10.2.2.1 Minimum Mean Square Error Estimation 344

10.2.2.2 Conditional Maximum A Posteriori Estimation 345

10.2.2.3 Extensions 345

10.2.2.4 Simplifications 346

10.2.3 Energy Envelope Estimation 346

10.3 Speech Codecs with Integrated BWE 349

10.3.1 BWE in the GSM Full-Rate Codec 349

10.3.2 BWE in the AMR Wideband Codec 351

10.3.3 BWE in the ITU Codec G.729.1 353

References 355

11 NELE: Near-End Listening Enhancement 361

11.1 Frequency Domain NELE (FD) 363

11.1.1 Speech Intelligibility Index NELE Optimization 364

11.1.1.1 SII-Optimized NELE Example 367

11.1.2 Closed-Form Gain-Shape NELE 368

11.1.2.1 The NoiseProp Shaping Function 370

11.1.2.2 The NoiseInverse Strategy...