Table of Contents

Interface IStreamingInferenceEngine<TInput, TOutput, TStep>

Namespace
InferenceEngine.Core
Assembly
InferenceEngine.Core.dll

An inference engine that can also yield intermediate model-level steps (token ids / logits / chunks) as an asynchronous stream.

public interface IStreamingInferenceEngine<TInput, TOutput, TStep> : IInferenceEngine<TInput, TOutput>, IDisposable

Type Parameters

TInput

The raw input to the engine (e.g. a prompt string or token-id array).

TOutput

The fully-reduced one-shot result (e.g. the complete generated text).

TStep

One streamed model-level step (e.g. a token id, a logit row, or a text chunk).

Inherited Members

Remarks

Implementations continue to satisfy the one-shot IInferenceEngine<TInput, TOutput> contract; streaming is an additive capability. A consumer may program to either contract on the same instance.

Methods

PredictStreamingAsync(TInput, StreamingOptions?, CancellationToken)

Yield one model-level step (token id / logit row / chunk) at a time. The consumer decides how to render or accumulate the steps. Cancellation is honored at each step boundary.

IAsyncEnumerable<TStep> PredictStreamingAsync(TInput input, StreamingOptions? options = null, CancellationToken cancellationToken = default)

Parameters

input TInput

The raw input.

options StreamingOptions

Optional per-call streaming controls; null uses the engine default.

cancellationToken CancellationToken

Observed between steps; aborts promptly and leaves the instance clean.

Returns

IAsyncEnumerable<TStep>