Table of Contents

Namespace InferenceEngine.Core

Classes

BaseInferenceEngine<TInput, TOutput>

The core abstract base class that handles ONNX Runtime session management, cross-platform hardware acceleration detection, and resource cleanup. Supports state passing between Pre and Post processing steps via a context object.

BaseStreamingInferenceEngine<TInput, TOutput, TStep>

Reuses BaseInferenceEngine<TInput, TOutput> for session/acceleration/warmup/ model-download/diagnostics, and adds an autoregressive decode loop driven by engine-supplied hooks. One-shot consumers see no change: PredictAsync(TInput) is routed through the streaming loop and reduced via BuildFinalOutput(IReadOnlyList<TStep>).

BaseStreamingInferenceEngine<TInput, TOutput, TStep>.KeyValueCacheState

Convenience IDisposable holder for a list of carry-forward KV-cache tensors. Engines that use CloneAsPastKeyValue(DisposableNamedOnnxValue, string) should wrap the cloned values in an instance of this type (or any other IDisposable) and return it from ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?) as the next state. The base loop disposes the previous state on step rollover and the final state on stream end, releasing the cloned tensors.

InferenceEngineOptions

Configuration options for inference engine initialization.

OutOfProcessBaseInferenceEngine<TInput, TOutput>

Host-side proxy that offers the same contract as BaseInferenceEngine<TInput, TOutput> but runs the ONNX session in a child worker process over a length-prefixed JSON pipe. The host never loads libonnxruntime/comgr; the worker never loads Mesa/GL — avoiding the Mesa libLLVM-19 ↔ comgr libLLVM-20 collision that aborts an in-process MIGraphX + GL-rendered UI. BaseInferenceEngine<TInput, TOutput>.EngineInfo is mirrored from the worker's Load response, so the diagnostics surface (provider, fallback reason, ordered checks) is identical to the in-process engine.

OutOfProcessEngineOptions

Options for an OutOfProcessBaseInferenceEngine<TInput, TOutput>.

SamplingParams

Sampling parameters. Disabled/identity values mean "off".

StreamingOptions

Per-call streaming controls. Defaults are safe/greedy.

Interfaces

IInferenceEngine<TInput, TOutput>

A generic contract for an AI Inference Engine. Implementation details (ONNX, TensorFlow, Mock) are abstracted away.

IStreamingInferenceEngine<TInput, TOutput, TStep>

An inference engine that can also yield intermediate model-level steps (token ids / logits / chunks) as an asynchronous stream.