Namespace InferenceEngine.Core
Classes
- BaseInferenceEngine<TInput, TOutput>
The core abstract base class that handles ONNX Runtime session management, cross-platform hardware acceleration detection, and resource cleanup. Supports state passing between Pre and Post processing steps via a context object.
- BaseStreamingInferenceEngine<TInput, TOutput, TStep>
Reuses BaseInferenceEngine<TInput, TOutput> for session/acceleration/warmup/ model-download/diagnostics, and adds an autoregressive decode loop driven by engine-supplied hooks. One-shot consumers see no change: PredictAsync(TInput) is routed through the streaming loop and reduced via BuildFinalOutput(IReadOnlyList<TStep>).
- BaseStreamingInferenceEngine<TInput, TOutput, TStep>.KeyValueCacheState
Convenience IDisposable holder for a list of carry-forward KV-cache tensors. Engines that use CloneAsPastKeyValue(DisposableNamedOnnxValue, string) should wrap the cloned values in an instance of this type (or any other IDisposable) and return it from ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?) as the next state. The base loop disposes the previous state on step rollover and the final state on stream end, releasing the cloned tensors.
- InferenceEngineOptions
Configuration options for inference engine initialization.
- OutOfProcessBaseInferenceEngine<TInput, TOutput>
Host-side proxy that offers the same contract as BaseInferenceEngine<TInput, TOutput> but runs the ONNX session in a child worker process over a length-prefixed JSON pipe. The host never loads
libonnxruntime/comgr; the worker never loads Mesa/GL — avoiding the MesalibLLVM-19↔ comgrlibLLVM-20collision that aborts an in-process MIGraphX + GL-rendered UI. BaseInferenceEngine<TInput, TOutput>.EngineInfo is mirrored from the worker's Load response, so the diagnostics surface (provider, fallback reason, ordered checks) is identical to the in-process engine.
- SamplingParams
Sampling parameters. Disabled/identity values mean "off".
- StreamingOptions
Per-call streaming controls. Defaults are safe/greedy.
Interfaces
- IInferenceEngine<TInput, TOutput>
A generic contract for an AI Inference Engine. Implementation details (ONNX, TensorFlow, Mock) are abstracted away.
- IStreamingInferenceEngine<TInput, TOutput, TStep>
An inference engine that can also yield intermediate model-level steps (token ids / logits / chunks) as an asynchronous stream.