Table of Contents

Class BaseInferenceEngine<TInput, TOutput>

Namespace
InferenceEngine.Core
Assembly
InferenceEngine.Core.dll

The core abstract base class that handles ONNX Runtime session management, cross-platform hardware acceleration detection, and resource cleanup. Supports state passing between Pre and Post processing steps via a context object.

public abstract class BaseInferenceEngine<TInput, TOutput> : IInferenceEngine<TInput, TOutput>, IDisposable

Type Parameters

TInput
TOutput
Inheritance
BaseInferenceEngine<TInput, TOutput>
Implements
IInferenceEngine<TInput, TOutput>
Derived
Inherited Members

Constructors

BaseInferenceEngine(InferenceEngineOptions, ILogger?)

protected BaseInferenceEngine(InferenceEngineOptions options, ILogger? logger = null)

Parameters

options InferenceEngineOptions
logger ILogger

Fields

_logger

protected readonly ILogger _logger

Field Value

ILogger

_options

protected readonly InferenceEngineOptions _options

Field Value

InferenceEngineOptions

_runOptions

protected RunOptions? _runOptions

Field Value

RunOptions

_session

protected InferenceSession? _session

Field Value

InferenceSession

Properties

EffectivePlatform

Platform used to select the execution-provider branch. Override to force a branch in tests.

protected virtual HostPlatform EffectivePlatform { get; }

Property Value

HostPlatform

EngineInfo

Snapshot of the engine's loaded state. Returns Empty until LoadAsync() completes. Thread-safe: reference assignment is atomic and InferenceEngineInfo is an immutable record.

public InferenceEngineInfo EngineInfo { get; }

Property Value

InferenceEngineInfo

InputMetadata

public IReadOnlyDictionary<string, NodeMetadata>? InputMetadata { get; }

Property Value

IReadOnlyDictionary<string, NodeMetadata>

IsRocmSessionActive

Returns true when the ONNX session was successfully initialized with the MIGraphX execution provider on AMD ROCm hardware. false on non-Linux-x64 platforms, when MIGraphX initialization failed, or when CPU fallback was used.

public bool IsRocmSessionActive { get; }

Property Value

bool

ModelCacheDirectory

Base directory for model cache. Defaults to LocalApplicationData. Derived classes can override to use a custom cache location.

protected virtual string ModelCacheDirectory { get; }

Property Value

string

ModelDownloadUrls

URLs where the model can be downloaded from, in priority order. Derived classes can override to provide custom model sources. If not overridden, the base engine will use the default ModelRegistry to find URLs.

protected virtual string[]? ModelDownloadUrls { get; }

Property Value

string[]

ModelName

The name of the model file (e.g., "yolo11n.onnx"). Derived classes must implement this to specify which model they require.

protected abstract string ModelName { get; }

Property Value

string

ModelRegistry

Model registry for looking up download URLs by model name. Override to use a custom registry; defaults to the built-in registry.

protected virtual ModelRegistry ModelRegistry { get; }

Property Value

ModelRegistry

OutputMetadata

public IReadOnlyDictionary<string, NodeMetadata>? OutputMetadata { get; }

Property Value

IReadOnlyDictionary<string, NodeMetadata>

PrimaryInputName

public string? PrimaryInputName { get; }

Property Value

string

Status

Lifecycle + runtime snapshot (EngineLoadState + partial-or-full InferenceEngineInfo + load error). Transitions fire OnStatusChanged when set. Out-of-process proxies mirror the worker's terminal states onto the same property.

public InferenceEngineStatus Status { get; }

Property Value

InferenceEngineStatus

Methods

AppendCoreMlProvider(SessionOptions, CoreMLFlags)

Appends the CoreML execution provider. Override to inject a throw in tests.

protected virtual void AppendCoreMlProvider(SessionOptions options, CoreMLFlags flags)

Parameters

options SessionOptions
flags CoreMLFlags

AppendDirectMlProvider(SessionOptions, int)

Appends the DirectML execution provider. Override to inject a throw in tests.

protected virtual void AppendDirectMlProvider(SessionOptions options, int deviceId)

Parameters

options SessionOptions
deviceId int

AppendMIGraphXProvider(SessionOptions, int)

Appends the MIGraphX execution provider with the compiled-model (.mxr) cache directory resolved by ResolveMIGraphXModelCacheDirectory(). The cache persists compiled MIGraphX programs across process restarts; without it the ORT provider compiles the full model on every session construction (an empty cache path explicitly means "always compile" in the EP). Override to inject a throw in tests.

protected virtual void AppendMIGraphXProvider(SessionOptions options, int deviceId)

Parameters

options SessionOptions
deviceId int

ApplyLoadedState(InferenceEngineInfo, IReadOnlyDictionary<string, NodeMetadata>?, IReadOnlyDictionary<string, NodeMetadata>?, string?)

Apply engine state that was loaded by an out-of-process worker (no in-process InferenceSession). Used by OutOfProcessBaseInferenceEngine to mirror the worker's EngineInfo/metadata so the diagnostics surface (provider, fallback reason, ordered checks) behaves identically to the in-process engine. Does not create a session; the proxy's PredictAsync(TInput) marshals to the worker instead.

protected void ApplyLoadedState(InferenceEngineInfo engineInfo, IReadOnlyDictionary<string, NodeMetadata>? inputMetadata, IReadOnlyDictionary<string, NodeMetadata>? outputMetadata, string? primaryInputName)

Parameters

engineInfo InferenceEngineInfo
inputMetadata IReadOnlyDictionary<string, NodeMetadata>
outputMetadata IReadOnlyDictionary<string, NodeMetadata>
primaryInputName string

CreateInferenceSession(string, SessionOptions)

Constructs the ONNX session. Override to inject a recoverable native failure in tests.

protected virtual InferenceSession CreateInferenceSession(string modelPath, SessionOptions options)

Parameters

modelPath string
options SessionOptions

Returns

InferenceSession

Dispose()

Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.

public void Dispose()

Dispose(bool)

protected virtual void Dispose(bool disposing)

Parameters

disposing bool

GetWarmupInput()

Generates dummy input for model warmup. Default implementation assumes float tensors. Override this for int64/string inputs.

protected virtual List<NamedOnnxValue> GetWarmupInput()

Returns

List<NamedOnnxValue>

IsKfdPresent()

True when the ROCm KFD device node is present. Override in tests.

protected virtual bool IsKfdPresent()

Returns

bool

LoadAsync()

Initializes the inference session with the best available hardware provider.

public virtual Task LoadAsync()

Returns

Task

PostProcessWithContext(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?)

Takes raw ONNX output and the context object from PreProcess, returning the final result.

protected abstract TOutput PostProcessWithContext(IDisposableReadOnlyCollection<DisposableNamedOnnxValue> output, object? context)

Parameters

output IDisposableReadOnlyCollection<DisposableNamedOnnxValue>
context object

Returns

TOutput

PreProcessWithContext(TInput)

Takes raw input and returns ONNX tensors plus an optional context object. The context object is passed to PostProcessWithContext, allowing you to share state without overriding the main pipeline.

protected abstract (List<NamedOnnxValue> InputTensors, object? Context) PreProcessWithContext(TInput input)

Parameters

input TInput

Returns

(List<NamedOnnxValue> InputTensors, object Context)

PredictAsync(TInput)

Runs the inference pipeline: PreProcess -> Inference -> PostProcess.

public virtual Task<TOutput> PredictAsync(TInput input)

Parameters

input TInput

The raw input data.

Returns

Task<TOutput>

The processed prediction result.

ResolveMIGraphXModelCacheDirectory()

Resolves the MIGraphX compiled-model (.mxr) cache directory: an explicitly configured MIGraphXModelCacheDirectory wins (empty string = disabled), otherwise a per-user default under the same root as ModelCacheDirectory. Returns null when caching is disabled or the host has no user-profile directory to anchor the default. Pure path computation — directory creation happens in AppendMIGraphXProvider(SessionOptions, int) only.

protected virtual string? ResolveMIGraphXModelCacheDirectory()

Returns

string

TryLoadNativeLibrary(string, out string?)

Attempts to dlopen a native library; returns success and an error message on failure.

protected virtual bool TryLoadNativeLibrary(string name, out string? error)

Parameters

name string
error string

Returns

bool