Table of Contents

Class TextGenerationEngine

Namespace
InferenceEngine.Core.Text
Assembly
InferenceEngine.Core.dll

Streaming text-generation engine for the Phi-3-mini-4k-instruct ONNX export. Owns SentencePiece tokenization end-to-end: callers pass a prompt string and receive streamed text chunks (string in -> string out).

public class TextGenerationEngine : BaseStreamingInferenceEngine<string, string, string>, IStreamingInferenceEngine<string, string, string>, IInferenceEngine<string, string>, IDisposable
Inheritance
TextGenerationEngine
Implements
Inherited Members

Remarks

Decode strategy: full-recompute per step. Each step re-feeds the entire growing token-id sequence (plus attention_mask and position_ids) and runs a fresh forward pass. This is model-agnostic and robust across ONNX export variants (which differ in their KV-cache I/O names/shapes), at the cost of O(n²) compute. A KV-cache optimization is a follow-up.

Phi-3 external-data note. The cpu-int4 export is split across a graph file (.onnx) and an external-data file (.onnx.data). ModelDownloadUrls references the .onnx only; a session created from that single file will fail to initialize without its sibling .onnx.data. For real runs, point ModelPath at a complete local bundle (both files co-located).

Constructors

TextGenerationEngine(InferenceEngineOptions, ILogger<TextGenerationEngine>?)

Creates the engine. The ONNX model and tokenizer are resolved lazily during LoadAsync().

public TextGenerationEngine(InferenceEngineOptions options, ILogger<TextGenerationEngine>? logger = null)

Parameters

options InferenceEngineOptions

Engine options. GPU acceleration defaults on; pass UseGpuAcceleration = false for CPU.

logger ILogger<TextGenerationEngine>

Optional typed logger.

Properties

DefaultStreamingOptions

Default streaming options: greedy, capped at 256 steps, terminating on the Phi-3 <|endoftext|> (32000) and <|im_end|> (32002) tokens.

protected override StreamingOptions DefaultStreamingOptions { get; }

Property Value

StreamingOptions

ModelDownloadUrls

Download URL for the Phi-3 graph file. See the class remarks: this references the .onnx only; the co-located .onnx.data external-data file is required for a successful session but is not fetched here.

protected override string[]? ModelDownloadUrls { get; }

Property Value

string[]

ModelName

The model file name used for cache keys and registry lookups.

protected override string ModelName { get; }

Property Value

string

Methods

BuildFinalOutput(IReadOnlyList<string>)

Reduces the streamed chunks into the final one-shot text.

protected override string BuildFinalOutput(IReadOnlyList<string> steps)

Parameters

steps IReadOnlyList<string>

Returns

string

BuildStepInputs(string, object?, int)

Step 0 tokenizes the prompt and seeds a TextGenerationEngine.GenerationState. Steps > 0 receive a state whose id list already includes the previously-sampled token; the full id list is rebuilt into input_ids, attention_mask (all-ones), and position_ids (0..n-1) — only for inputs the model actually declares.

protected override (List<NamedOnnxValue> Inputs, object? BuildContext) BuildStepInputs(string input, object? state, int step)

Parameters

input string
state object
step int

Returns

(List<NamedOnnxValue> InputTensors, object Context)

LoadAsync()

Loads the Phi-3 SentencePiece tokenizer (best-effort) and then initializes the ONNX session. A tokenizer load failure is logged as a warning and does not abort the session load; the first generation call will surface a clear error if a tokenizer is genuinely required.

public override Task LoadAsync()

Returns

Task

ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?)

Reads the logits output, greedy-samples the last position, appends the sampled id to the state, detokenizes the full id list, and yields only the new text suffix since the last yield (correctly handling SentencePiece subword merges).

protected override (string Step, object? NextState, bool IsEos) ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue> outputs, object? buildContext)

Parameters

outputs IDisposableReadOnlyCollection<DisposableNamedOnnxValue>
buildContext object

Returns

(string Step, object NextState, bool IsEos)