Audio as numbers: samples, frames and why speech is latency-sensitive

#Sound as numbers

A microphone turns air pressure into a voltage; an analog-to-digital converter measures it many times per second. Each measurement is a sample. The number of samples per second is the sample rate, and the precision of each one is the bit depth.

Setting	Typical value	Used for
Sample rate	16,000 Hz	Speech recognition models
Sample rate	44,100 or 48,000 Hz	Music and high quality audio
Bit depth	16-bit	Standard (2 bytes per sample)
Channels	1 (mono)	Speech

#A runnable example

python

import math
rate = 16000                     # samples per second
secs = 1.5
n = int(rate * secs)
wave = [math.sin(2 * math.pi * 440 * i / rate) for i in range(n)]   # a 440 Hz tone
print("samples:", n)             # 24000
frame, hop = 400, 160            # 25 ms windows every 10 ms
frames = 1 + (n - frame) // hop
print("frames:", frames)         # 148
print("bytes at 16-bit mono:", n * 2)   # 48000
print("peak:", round(max(wave), 3))     # 1.0

So 1.5 seconds of speech is 24,000 numbers and 48,000 bytes. A speech model slides a 25 ms window every 10 ms over it, producing 148 frames; each frame is turned into features (often a spectrogram) and then into vectors. That is about 100 frames per second of audio, so a minute of speech is thousands of positions, and recognition or audio-understanding models compress or downsample them.

#Frames, features and tokens

Step	What happens
Resample	Convert to the model's sample rate
Frame	Cut into short overlapping windows
Features	Turn each window into a spectrogram-style vector
Encode	A neural encoder compresses the sequence
Decode	A language-model-style decoder writes text, or audio tokens for speech output

Speech output works in reverse: a model produces audio tokens or spectrogram frames, and a vocoder turns them into a waveform.

#Why speech is latency-sensitive

People expect an answer within a fraction of a second in conversation. The delay is the sum of stages:

1
Person stops speaking
2
End-of-speech detectionwaits to be sure they are done
3
Speech recognitionaudio to text
4
Language modeltime to the first token
5
Speech synthesistext to the first audio
6
First sound plays

Every stage adds delay, and the gap before the first sound is what users notice. Real-time systems therefore stream every stage (start recognising while the person talks, start speaking while the model writes) and measure the time from end of speech to first audio (real-time voice pipelines).

#Practical notes

Sample rate mismatches are a classic bug: a model expecting 16 kHz fed 44.1 kHz audio produces nonsense.
Mono versus stereo: mix down to mono for speech models.
Background noise and accents change accuracy; test with your real audio.
Privacy: audio is personal data. Local models keep it on the device (local-first privacy).

#Exercise

How many samples and how many bytes are in 10 seconds of 16 kHz, 16-bit mono audio?
With 25 ms windows every 10 ms, about how many frames per second of audio?

Answers: 1. 160,000 samples and 320,000 bytes. 2. About 100 frames per second (one every 10 ms).

Next: how vision-language models work, and what images cost.