Table of Contents

Class BaseStreamingInferenceEngine<TInput, TOutput, TStep>

Namespace
InferenceEngine.Core
Assembly
InferenceEngine.Core.dll

Reuses BaseInferenceEngine<TInput, TOutput> for session/acceleration/warmup/ model-download/diagnostics, and adds an autoregressive decode loop driven by engine-supplied hooks. One-shot consumers see no change: PredictAsync(TInput) is routed through the streaming loop and reduced via BuildFinalOutput(IReadOnlyList<TStep>).

public abstract class BaseStreamingInferenceEngine<TInput, TOutput, TStep> : BaseInferenceEngine<TInput, TOutput>, IStreamingInferenceEngine<TInput, TOutput, TStep>, IInferenceEngine<TInput, TOutput>, IDisposable

Type Parameters

TInput

Raw engine input (e.g. a prompt).

TOutput

Fully-reduced one-shot result.

TStep

One streamed model-level step.

Inheritance
BaseInferenceEngine<TInput, TOutput>
BaseStreamingInferenceEngine<TInput, TOutput, TStep>
Implements
IStreamingInferenceEngine<TInput, TOutput, TStep>
IInferenceEngine<TInput, TOutput>
Derived
Inherited Members

Constructors

BaseStreamingInferenceEngine(InferenceEngineOptions, ILogger?)

Creates the streaming engine, forwarding options + logger to the base.

protected BaseStreamingInferenceEngine(InferenceEngineOptions options, ILogger? logger = null)

Parameters

options InferenceEngineOptions
logger ILogger

Properties

CurrentOptions

protected StreamingOptions? CurrentOptions { get; }

Property Value

StreamingOptions

DefaultStreamingOptions

Default streaming options used when a caller passes null.

protected virtual StreamingOptions DefaultStreamingOptions { get; }

Property Value

StreamingOptions

Methods

BuildFinalOutput(IReadOnlyList<TStep>)

Reduce the accumulated steps into the final one-shot TOutput.

protected abstract TOutput BuildFinalOutput(IReadOnlyList<TStep> steps)

Parameters

steps IReadOnlyList<TStep>

Returns

TOutput

BuildStepInputs(TInput, object?, int)

Build the ONNX inputs for a given step and return the carry-forward context. step == 0 means prefill (full prompt); step > 0 means decode (feed the previous step's present_key_values back as past_key_values).

protected abstract (List<NamedOnnxValue> Inputs, object? BuildContext) BuildStepInputs(TInput input, object? state, int step)

Parameters

input TInput
state object
step int

Returns

(List<NamedOnnxValue> InputTensors, object Context)

CloneAsPastKeyValue(DisposableNamedOnnxValue, string)

Clone a DenseTensor<T> of float carried in a model output so it survives disposal of the output collection, renaming it from a present_ name to the matching past_ name for the next step. Engines call this from ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?) to carry the KV cache forward. Other dtypes require extension.

protected static NamedOnnxValue? CloneAsPastKeyValue(DisposableNamedOnnxValue output, string pastName)

Parameters

output DisposableNamedOnnxValue
pastName string

Returns

NamedOnnxValue

DisposeNamedOnnxValues(IReadOnlyCollection<NamedOnnxValue>)

Dispose any NamedOnnxValue items in values that are disposable.

protected static void DisposeNamedOnnxValues(IReadOnlyCollection<NamedOnnxValue> values)

Parameters

values IReadOnlyCollection<NamedOnnxValue>

PostProcessWithContext(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?)

Not used by streaming engines.

protected override sealed TOutput PostProcessWithContext(IDisposableReadOnlyCollection<DisposableNamedOnnxValue> output, object? context)

Parameters

output IDisposableReadOnlyCollection<DisposableNamedOnnxValue>
context object

Returns

TOutput

PreProcessWithContext(TInput)

Not used by streaming engines; one-shot PredictAsync(TInput) is routed through the stream.

protected override sealed (List<NamedOnnxValue> InputTensors, object? Context) PreProcessWithContext(TInput input)

Parameters

input TInput

Returns

(List<NamedOnnxValue> InputTensors, object Context)

PredictAsync(TInput)

One-shot behavior is preserved: run the stream to completion and reduce it.

public override Task<TOutput> PredictAsync(TInput input)

Parameters

input TInput

Returns

Task<TOutput>

PredictStreamingAsync(TInput, StreamingOptions?, CancellationToken)

Yield one model-level step (token id / logit row / chunk) at a time. The consumer decides how to render or accumulate the steps. Cancellation is honored at each step boundary.

public IAsyncEnumerable<TStep> PredictStreamingAsync(TInput input, StreamingOptions? options = null, CancellationToken cancellationToken = default)

Parameters

input TInput

The raw input.

options StreamingOptions

Optional per-call streaming controls; null uses the engine default.

cancellationToken CancellationToken

Observed between steps; aborts promptly and leaves the instance clean.

Returns

IAsyncEnumerable<TStep>

ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?)

Parse one forward pass's outputs into a streamed step, the next carry-forward state, and an end-of-sequence signal. The engine is responsible for token-level EOS detection against EosTokenIds (it holds the sampled token); the base enforces the returned IsEos flag and MaxSteps.

protected abstract (TStep Step, object? NextState, bool IsEos) ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue> outputs, object? buildContext)

Parameters

outputs IDisposableReadOnlyCollection<DisposableNamedOnnxValue>
buildContext object

Returns

(TStep Step, object NextState, bool IsEos)

RunStep(IReadOnlyCollection<NamedOnnxValue>)

Run one forward pass. Virtual so tests can substitute a fake session without a real model.

protected virtual IDisposableReadOnlyCollection<DisposableNamedOnnxValue> RunStep(IReadOnlyCollection<NamedOnnxValue> inputs)

Parameters

inputs IReadOnlyCollection<NamedOnnxValue>

Returns

IDisposableReadOnlyCollection<DisposableNamedOnnxValue>

SampleNextToken(ReadOnlySpan<float>)

Samples one token id from a vocab-length logit row. Greedy argmax when Sampling's temperature is disabled (0); otherwise temperature-scaled softmax sampling with optional top-k / top-p filtering. Engines call this from ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?) when they want the default sampler.

protected virtual int SampleNextToken(ReadOnlySpan<float> logits)

Parameters

logits ReadOnlySpan<float>

Returns

int