Get Streaming STT Results Faster with RTZR FINALIZE

By Hazel

Streaming STT returns interim results in real time as it receives audio. To receive a final transcript, you must wait for the server to complete end-point detection and determine that the current utterance has ended.

FINALIZE is a request that tells the server when the client has determined that an utterance has ended. It lets you receive the final transcript without waiting for the server's end-point detection, so you can begin generating the next response sooner.

This article explains how FINALIZE works and how to use it, then presents an experiment comparing how long it takes to receive the final transcript with and without FINALIZE.

The client is the application that sends audio and control requests to the RTZR Streaming STT API. The server is the RTZR STT server that recognizes the audio and returns the results.

1. What Latency Does FINALIZE Reduce?

Let's compare the two flows when the client already knows that the utterance has ended.

Without FINALIZE

Application determines that the utterance has ended
  → Wait for the server's end-point detection  ← Latency reduced by FINALIZE
  → Receive the final transcript
  → Begin generating the next response


With FINALIZE

Application determines that the utterance has ended
  → Client sends FINALIZE
  → Receive the final transcript
  → Begin generating the next response

In streaming STT, the arrival time of the final transcript depends on when the server determines that the current utterance has ended. Unless it receives a separate request, the server uses end-point detection to determine whether the utterance is complete. Consequently, even after the user has finished speaking, there is a period during which the client must wait for the server to make that determination before receiving the final transcript.

If the application uses VAD or a turn detector to determine the end of an utterance first, the client can send FINALIZE at that point to tell the server that the current utterance is complete. Doing so asks the server to finish recognizing the audio received so far without waiting for its own end-point detection.

This can reduce the time spent waiting for the server's end-point detection after the utterance ends, allowing the client to receive the final transcript and begin generating the next response sooner.

End-Point Detection and epd_time

One of the settings the server uses to determine the end of an utterance is epd_time. It specifies the duration of silence the server waits through after speech stops before deciding that the current utterance has ended. According to the official .proto definition, the default is 0.5 seconds, and the recommended range is 0.5 to 1.0 seconds. With the default setting, the server waits so that pauses of approximately 0.5 seconds or less can remain part of the same utterance. If the silence lasts longer, the server finalizes the result for the current utterance.

Setting epd_time to a shorter value can produce the final transcript sooner, but it also increases the likelihood that hesitation or a brief pause in the middle of a sentence will be treated as the end of an utterance. A longer value is more tolerant of natural pauses but may increase latency. Choose a value appropriate for the intended domain and speech patterns.

2. The Roles of FINALIZE and EOS

Finalizing a single utterance and ending the entire audio input are separate operations. FINALIZE finalizes the current utterance without closing the streaming connection.

Over WebSocket, sending the EOS text message signals that there is no more audio to send. The client then receives the remaining recognition results and closes the connection. With gRPC, the client ends the entire input by half-closing only the sending side of the request stream, then continues receiving the remaining responses.

3. Sending FINALIZE over gRPC and WebSocket

The request has the same meaning in both protocols. After sending the audio for the current utterance, add a FINALIZE request to the same stream.

Send audio for the current utterance
  → Send a FINALIZE request
  → Confirm receipt of the final transcript
  → Send the next utterance over the same connection

The Python code below shows only how to add the request, assuming that authentication, the connection, and the recognition settings have already been configured. For complete connection code, see the gRPC integration example and WebSocket integration example.

gRPC

Add a control request to the request stream that was carrying the audio. Put the DECODER_CMD_FINALIZE command inside streaming_control.

# Add this to the same request stream when the client detects the end of the current utterance.

yield pb.DecoderRequest(
    streaming_control=pb.DecoderControl(
        command=pb.DecoderControl.DECODER_CMD_FINALIZE,
    )
)

After this request, you can continue sending audio requests for the next utterance over the same request stream. When all audio input for the conversation is complete, use a half-close to stop sending requests.

WebSocket

Send audio as binary messages and FINALIZE as a JSON text message. Run the code below inside an asynchronous function.

# Send this over the same WebSocket connection when the client detects the end of the current utterance.

await websocket.send(json.dumps({"type": "Finalize"}))

After sending FINALIZE, check the is_final value in a gRPC response or the final value in a WebSocket response to confirm that the result has been finalized.

Sending FINALIZE for an empty utterance or sending it repeatedly without audio does not generate a new result.

4. Comparison With and Without FINALIZE

This experiment assumes that the application already knows the exact end of an utterance and compares the time from that point until the final transcript is received.

Experimental Setup

We selected 50 utterances from AI Hub's Emotion-Tagged Free-Form Conversation (Adults) dataset. The selection criteria were as follows.

• Each WAV file had a corresponding transcription label, and the labeled utterance timestamps fell within the actual audio range.
• Neither speaker produced any additional speech for approximately three seconds after the utterance ended.
• The utterances were distributed across speakers and recordings and included a balanced mix of short, medium-length, and long utterances.
• The source audio segments used in the experiment did not overlap.

In addition, each speaker's voice is stored in a separate channel of the stereo source WAV files. If audio from both speakers is included in the input, utterance boundaries may become unclear when their speech overlaps or one speaker follows the other. To isolate the effect of FINALIZE in this experiment, we used audio from only the selected speaker's channel.

Based on these criteria, we selected 50 non-overlapping utterance segments from 35 source recordings.

The experiment compared a Baseline condition, in which FINALIZE was not sent, with a Finalize condition, in which FINALIZE was sent once at the transcription label's EndTime. Except for whether a FINALIZE request was sent, the settings were identical.

Both conditions used epd_time=0.5 in this experiment. The Baseline condition therefore waited for the server to determine the end of the utterance based on this setting, while the Finalize condition requested that the server skip that wait at the labeled EndTime.

Setting epd_time to a shorter value can make the Baseline condition faster, reducing the difference between the two conditions. Setting it to a longer value can increase the difference.

The measured value is the time from the labeled EndTime until the arrival of the final transcript containing the last part of that utterance.

Experimental Results

All 50 utterances completed successfully under both conditions. P50 is the median. P90 and P95 indicate that 90% and 95% of the utterances, respectively, received their results within the stated time.

Time to Receive the Final Transcript by Condition

Time Saved for the Same Utterance

The Finalize condition was faster for all 50 utterances, with a median latency reduction of 636.37 ms.

This experiment used the transcription data's labeled EndTime instead of an actual VAD. The results therefore show how much the server-side wait changes when the exact end of an utterance is known. They do not represent the end-to-end latency of a real-world voice agent, which would include VAD decision latency and detection errors.

5. Use Cases

FINALIZE is useful when an application can determine the end of a user's utterance using its own VAD or turn detector.

Callbot

In a callbot where the conversation flow and expected user responses are relatively constrained—such as for reservations, order intake, or identity verification—VAD can detect when the user stops speaking. The callbot can then send FINALIZE when the workflow indicates that the user's response is complete. This allows it to receive the final transcript without waiting for the server to determine the end of the utterance, so it can ask the next question or begin the next task sooner.

Voice Agent

A voice agent is a system that understands a user's speech, generates a contextually appropriate response or performs a necessary task, and then responds with speech. Voice agents are used in a variety of settings, including phone calls, customer support, and meetings. Unlike scenarios with fixed response formats, such as identity verification, open-ended conversations in customer support or general-purpose voice assistants may include pauses for thought or hesitation within a single user turn. In such cases, you can keep the server-side `epd_time` sufficiently long and send FINALIZE when the VAD or turn detector determines that the user's turn has ended.

Framework Integration Examples

This approach can also be used with open-source voice agent frameworks. LiveKit Agents is a framework that processes real-time audio through an STT–LLM–TTS pipeline and supports VAD, turn detection, and interruption handling. With the RTZR plugin for LiveKit Agents, you can integrate RTZR Streaming STT with LiveKit Agents and apply FINALIZE when the VAD or turn detector determines that the user's turn has ended.

TEN Framework also lets you build a real-time voice agent by combining extensions for VAD, turn detection, and other functions. In the proposed TEN integration, the turn controller sends asr_finalize to the RTZR ASR extension. The extension then sends FINALIZE over the existing WebSocket connection.

6. Conclusion

If your application can determine the end of an utterance before the server does, FINALIZE can reduce the wait and let the next response begin sooner. Apply it to your callbot or voice agent to create faster, more natural conversation flows.

References

Streaming STT | RTZR STT OpenAPI

This guide explains how to implement streaming speech-to-text. Two protocols are supported: 1) gRPC and 2) WebSocket. See Streaming STT - gRPC and Streaming STT - WebSocket for integration details.

RTZR Logo

rtzr-api/protos/rtzr-stt.proto at main · rtzr/rtzr-api

Contribute to rtzr/rtzr-api development by creating an account on GitHub.

GitHubrtzr