Text inference engines
InferenceEngine.Core.Text adds two string-in/string-out engines that own tokenization and model
asset fetching end-to-end, so consuming apps need no tokenizer code of their own. The dependency
Microsoft.ML.Tokenizers (1.0.3) provides the tokenizers.
TextEmbeddingEngine
TextEmbeddingEngine : BaseInferenceEngine<string, float[]> produces L2-normalized embeddings from
the all-MiniLM-L6-v2 ONNX model.
- Tokenizer: uncased BERT WordPiece (
BertTokenizer,LowerCaseBeforeTokenization = true), vocab fetched fromXenova/all-MiniLM-L6-v2/vocab.txt. - Inputs are assembled from the model's declared input metadata (
input_ids,attention_mask, andtoken_type_ids/position_idswhen present), each typed to match its declared element type (int64orint32). - Output: the embedding tensor (
sentence_embedding/pooler/pooled, else the first output) is masked-mean-pooled over the sequence axis (for 3D[1,seq,dim]) and L2-normalized. The vector dimension is read from metadata, never hard-coded.
using InferenceEngine.Core;
using InferenceEngine.Core.Text;
var options = new InferenceEngineOptions { UseGpuAcceleration = false };
using var engine = new TextEmbeddingEngine(options);
await engine.LoadAsync();
float[] emb = await engine.PredictAsync("docker container");
The deterministic math (L2Normalize, MeanPool, CosineSimilarity) is exposed as internal static
helpers so it can be unit-tested without a model.
TextGenerationEngine
TextGenerationEngine : BaseStreamingInferenceEngine<string, string, string> streams text from the
Phi-3-mini-4k-instruct ONNX export (cpu-int4). TStep is a decoded text chunk.
- Tokenizer: SentencePiece model read via
LlamaTokenizer.Create(the public SentencePiece-backed factory in Tokenizers 1.0.3; see the note below),addBeginOfSentence = addEndOfSentence = false, special tokens<|endoftext|>=32000,<|im_start|>=32001,<|im_end|>=32002. - Default
StreamingOptions: greedy,MaxSteps = 256, EOS{ 32000, 32002 }.
Decode strategy: full-recompute per step
Each decode step re-feeds the entire growing token-id list (input_ids, attention_mask = ones,
position_ids = 0..n-1) and runs a fresh forward pass. The last position's logit row is greedy-sampled,
the sampled id is appended to the state, and the full id list is re-detokenized; only the new text
suffix since the previous yield is emitted (correctly handling SentencePiece subword merges).
This is model-agnostic and robust across ONNX export variants (which differ in KV-cache I/O
names/shapes), at the cost of O(n²) compute. A KV-cache optimization that reuses present_* /
past_* tensors is a planned follow-up.
var options = new InferenceEngineOptions { UseGpuAcceleration = false };
using var engine = new TextGenerationEngine(options);
await foreach (string chunk in engine.PredictStreamingAsync("What is 2 + 2?"))
{
Console.Write(chunk);
}
Phi-3 .onnx.data limitation
The cpu-int4 export is split across a graph file (...int4-rtn-block-32.onnx) and an external-data
file (...onnx.data). ModelDownloadUrls references the .onnx only; an InferenceSession
created from that single file will fail to initialize without its sibling .onnx.data. For real runs,
point InferenceEngineOptions.ModelPath at a complete local bundle with both files co-located.
Tokenizer factory note
Microsoft.ML.Tokenizers 1.0.3 exposes no public factory on SentencePieceTokenizer itself. The
public factory that reads a SentencePiece model and returns a SentencePieceTokenizer-derived
instance is LlamaTokenizer.Create(Stream, bool, bool, IReadOnlyDictionary<string,int>), which is
what TokenizerAssets.LoadPhi3TokenizerAsync uses. Phi-3's tokenizer.model is a SentencePiece
model.
Tokenizer assets
TokenizerAssets (internal) downloads/caches tokenizer vocab and model files via
DownloadingModelProvider under the tokenizers cache directory
({LocalApplicationData}/InferenceEngine/Tokenizers), then constructs the configured Tokenizer over
a read-only file stream. Both engines load their tokenizer best-effort in the constructor; if the
asset is unavailable at construction (e.g. offline), the engine still constructs and surfaces a clear
error on the first inference call.