Interface IStreamingInferenceEngine<TInput, TOutput, TStep>
- Namespace
- InferenceEngine.Core
- Assembly
- InferenceEngine.Core.dll
An inference engine that can also yield intermediate model-level steps (token ids / logits / chunks) as an asynchronous stream.
public interface IStreamingInferenceEngine<TInput, TOutput, TStep> : IInferenceEngine<TInput, TOutput>, IDisposable
Type Parameters
TInputThe raw input to the engine (e.g. a prompt string or token-id array).
TOutputThe fully-reduced one-shot result (e.g. the complete generated text).
TStepOne streamed model-level step (e.g. a token id, a logit row, or a text chunk).
- Inherited Members
Remarks
Implementations continue to satisfy the one-shot IInferenceEngine<TInput, TOutput> contract; streaming is an additive capability. A consumer may program to either contract on the same instance.
Methods
PredictStreamingAsync(TInput, StreamingOptions?, CancellationToken)
Yield one model-level step (token id / logit row / chunk) at a time. The consumer decides how to render or accumulate the steps. Cancellation is honored at each step boundary.
IAsyncEnumerable<TStep> PredictStreamingAsync(TInput input, StreamingOptions? options = null, CancellationToken cancellationToken = default)
Parameters
inputTInputThe raw input.
optionsStreamingOptionsOptional per-call streaming controls; null uses the engine default.
cancellationTokenCancellationTokenObserved between steps; aborts promptly and leaves the instance clean.
Returns
- IAsyncEnumerable<TStep>