Class BaseStreamingInferenceEngine<TInput, TOutput, TStep>
- Namespace
- InferenceEngine.Core
- Assembly
- InferenceEngine.Core.dll
Reuses BaseInferenceEngine<TInput, TOutput> for session/acceleration/warmup/ model-download/diagnostics, and adds an autoregressive decode loop driven by engine-supplied hooks. One-shot consumers see no change: PredictAsync(TInput) is routed through the streaming loop and reduced via BuildFinalOutput(IReadOnlyList<TStep>).
public abstract class BaseStreamingInferenceEngine<TInput, TOutput, TStep> : BaseInferenceEngine<TInput, TOutput>, IStreamingInferenceEngine<TInput, TOutput, TStep>, IInferenceEngine<TInput, TOutput>, IDisposable
Type Parameters
TInputRaw engine input (e.g. a prompt).
TOutputFully-reduced one-shot result.
TStepOne streamed model-level step.
- Inheritance
-
BaseInferenceEngine<TInput, TOutput>BaseStreamingInferenceEngine<TInput, TOutput, TStep>
- Implements
-
IStreamingInferenceEngine<TInput, TOutput, TStep>IInferenceEngine<TInput, TOutput>
- Derived
- Inherited Members
Constructors
BaseStreamingInferenceEngine(InferenceEngineOptions, ILogger?)
Creates the streaming engine, forwarding options + logger to the base.
protected BaseStreamingInferenceEngine(InferenceEngineOptions options, ILogger? logger = null)
Parameters
optionsInferenceEngineOptionsloggerILogger
Properties
CurrentOptions
The StreamingOptions for the stream currently in progress, or null outside a stream. Hooks (BuildStepInputs(TInput, object?, int)/ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?)) read this to honor IncludePrefillInStream, EosTokenIds, etc.
protected StreamingOptions? CurrentOptions { get; }
Property Value
DefaultStreamingOptions
Default streaming options used when a caller passes null.
protected virtual StreamingOptions DefaultStreamingOptions { get; }
Property Value
Methods
BuildFinalOutput(IReadOnlyList<TStep>)
Reduce the accumulated steps into the final one-shot TOutput.
protected abstract TOutput BuildFinalOutput(IReadOnlyList<TStep> steps)
Parameters
stepsIReadOnlyList<TStep>
Returns
- TOutput
BuildStepInputs(TInput, object?, int)
Build the ONNX inputs for a given step and return the carry-forward context.
step == 0 means prefill (full prompt); step > 0 means decode
(feed the previous step's present_key_values back as past_key_values).
protected abstract (List<NamedOnnxValue> Inputs, object? BuildContext) BuildStepInputs(TInput input, object? state, int step)
Parameters
Returns
CloneAsPastKeyValue(DisposableNamedOnnxValue, string)
Clone a DenseTensor<T> of float carried in a model output so it
survives disposal of the output collection, renaming it from a present_ name to the
matching past_ name for the next step. Engines call this from
ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?) to carry the KV cache forward. Other dtypes require extension.
protected static NamedOnnxValue? CloneAsPastKeyValue(DisposableNamedOnnxValue output, string pastName)
Parameters
outputDisposableNamedOnnxValuepastNamestring
Returns
DisposeNamedOnnxValues(IReadOnlyCollection<NamedOnnxValue>)
Dispose any NamedOnnxValue items in values that are disposable.
protected static void DisposeNamedOnnxValues(IReadOnlyCollection<NamedOnnxValue> values)
Parameters
valuesIReadOnlyCollection<NamedOnnxValue>
PostProcessWithContext(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?)
Not used by streaming engines.
protected override sealed TOutput PostProcessWithContext(IDisposableReadOnlyCollection<DisposableNamedOnnxValue> output, object? context)
Parameters
outputIDisposableReadOnlyCollection<DisposableNamedOnnxValue>contextobject
Returns
- TOutput
PreProcessWithContext(TInput)
Not used by streaming engines; one-shot PredictAsync(TInput) is routed through the stream.
protected override sealed (List<NamedOnnxValue> InputTensors, object? Context) PreProcessWithContext(TInput input)
Parameters
inputTInput
Returns
PredictAsync(TInput)
One-shot behavior is preserved: run the stream to completion and reduce it.
public override Task<TOutput> PredictAsync(TInput input)
Parameters
inputTInput
Returns
- Task<TOutput>
PredictStreamingAsync(TInput, StreamingOptions?, CancellationToken)
Yield one model-level step (token id / logit row / chunk) at a time. The consumer decides how to render or accumulate the steps. Cancellation is honored at each step boundary.
public IAsyncEnumerable<TStep> PredictStreamingAsync(TInput input, StreamingOptions? options = null, CancellationToken cancellationToken = default)
Parameters
inputTInputThe raw input.
optionsStreamingOptionsOptional per-call streaming controls; null uses the engine default.
cancellationTokenCancellationTokenObserved between steps; aborts promptly and leaves the instance clean.
Returns
- IAsyncEnumerable<TStep>
ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?)
Parse one forward pass's outputs into a streamed step, the next carry-forward state, and an
end-of-sequence signal. The engine is responsible for token-level EOS detection against
EosTokenIds (it holds the sampled token); the base enforces
the returned IsEos flag and MaxSteps.
protected abstract (TStep Step, object? NextState, bool IsEos) ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue> outputs, object? buildContext)
Parameters
outputsIDisposableReadOnlyCollection<DisposableNamedOnnxValue>buildContextobject
Returns
RunStep(IReadOnlyCollection<NamedOnnxValue>)
Run one forward pass. Virtual so tests can substitute a fake session without a real model.
protected virtual IDisposableReadOnlyCollection<DisposableNamedOnnxValue> RunStep(IReadOnlyCollection<NamedOnnxValue> inputs)
Parameters
inputsIReadOnlyCollection<NamedOnnxValue>
Returns
SampleNextToken(ReadOnlySpan<float>)
Samples one token id from a vocab-length logit row. Greedy argmax when Sampling's temperature is disabled (0); otherwise temperature-scaled softmax sampling with optional top-k / top-p filtering. Engines call this from ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?) when they want the default sampler.
protected virtual int SampleNextToken(ReadOnlySpan<float> logits)
Parameters
logitsReadOnlySpan<float>