Class TextGenerationEngine
Streaming text-generation engine for the Phi-3-mini-4k-instruct ONNX export. Owns SentencePiece
tokenization end-to-end: callers pass a prompt string and receive streamed text chunks
(string in -> string out).
public class TextGenerationEngine : BaseStreamingInferenceEngine<string, string, string>, IStreamingInferenceEngine<string, string, string>, IInferenceEngine<string, string>, IDisposable
- Inheritance
-
TextGenerationEngine
- Implements
- Inherited Members
Remarks
Decode strategy: full-recompute per step. Each step re-feeds the entire growing token-id
sequence (plus attention_mask and position_ids) and runs a fresh forward pass. This
is model-agnostic and robust across ONNX export variants (which differ in their KV-cache I/O
names/shapes), at the cost of O(n²) compute. A KV-cache optimization is a follow-up.
Phi-3 external-data note. The cpu-int4 export is split across a graph file (.onnx)
and an external-data file (.onnx.data). ModelDownloadUrls references the
.onnx only; a session created from that single file will fail to initialize without its
sibling .onnx.data. For real runs, point ModelPath at
a complete local bundle (both files co-located).
Constructors
TextGenerationEngine(InferenceEngineOptions, ILogger<TextGenerationEngine>?)
Creates the engine. The ONNX model and tokenizer are resolved lazily during LoadAsync().
public TextGenerationEngine(InferenceEngineOptions options, ILogger<TextGenerationEngine>? logger = null)
Parameters
optionsInferenceEngineOptionsEngine options. GPU acceleration defaults on; pass
UseGpuAcceleration = falsefor CPU.loggerILogger<TextGenerationEngine>Optional typed logger.
Properties
DefaultStreamingOptions
Default streaming options: greedy, capped at 256 steps, terminating on the Phi-3
<|endoftext|> (32000) and <|im_end|> (32002) tokens.
protected override StreamingOptions DefaultStreamingOptions { get; }
Property Value
ModelDownloadUrls
Download URL for the Phi-3 graph file. See the class remarks: this references the
.onnx only; the co-located .onnx.data external-data file is required for a
successful session but is not fetched here.
protected override string[]? ModelDownloadUrls { get; }
Property Value
- string[]
ModelName
The model file name used for cache keys and registry lookups.
protected override string ModelName { get; }
Property Value
Methods
BuildFinalOutput(IReadOnlyList<string>)
Reduces the streamed chunks into the final one-shot text.
protected override string BuildFinalOutput(IReadOnlyList<string> steps)
Parameters
stepsIReadOnlyList<string>
Returns
BuildStepInputs(string, object?, int)
Step 0 tokenizes the prompt and seeds a TextGenerationEngine.GenerationState. Steps > 0 receive a
state whose id list already includes the previously-sampled token; the full id list is rebuilt
into input_ids, attention_mask (all-ones), and position_ids
(0..n-1) — only for inputs the model actually declares.
protected override (List<NamedOnnxValue> Inputs, object? BuildContext) BuildStepInputs(string input, object? state, int step)
Parameters
Returns
LoadAsync()
Loads the Phi-3 SentencePiece tokenizer (best-effort) and then initializes the ONNX session. A tokenizer load failure is logged as a warning and does not abort the session load; the first generation call will surface a clear error if a tokenizer is genuinely required.
public override Task LoadAsync()
Returns
ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?)
Reads the logits output, greedy-samples the last position, appends the sampled id to the state, detokenizes the full id list, and yields only the new text suffix since the last yield (correctly handling SentencePiece subword merges).
protected override (string Step, object? NextState, bool IsEos) ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue> outputs, object? buildContext)
Parameters
outputsIDisposableReadOnlyCollection<DisposableNamedOnnxValue>buildContextobject