Class BaseInferenceEngine<TInput, TOutput>
- Namespace
- InferenceEngine.Core
- Assembly
- InferenceEngine.Core.dll
The core abstract base class that handles ONNX Runtime session management, cross-platform hardware acceleration detection, and resource cleanup. Supports state passing between Pre and Post processing steps via a context object.
public abstract class BaseInferenceEngine<TInput, TOutput> : IInferenceEngine<TInput, TOutput>, IDisposable
Type Parameters
TInputTOutput
- Inheritance
-
BaseInferenceEngine<TInput, TOutput>
- Implements
-
IInferenceEngine<TInput, TOutput>
- Derived
- Inherited Members
Constructors
BaseInferenceEngine(InferenceEngineOptions, ILogger?)
protected BaseInferenceEngine(InferenceEngineOptions options, ILogger? logger = null)
Parameters
optionsInferenceEngineOptionsloggerILogger
Fields
_logger
protected readonly ILogger _logger
Field Value
_options
protected readonly InferenceEngineOptions _options
Field Value
_runOptions
protected RunOptions? _runOptions
Field Value
_session
protected InferenceSession? _session
Field Value
Properties
EffectivePlatform
Platform used to select the execution-provider branch. Override to force a branch in tests.
protected virtual HostPlatform EffectivePlatform { get; }
Property Value
EngineInfo
Snapshot of the engine's loaded state. Returns Empty until LoadAsync() completes. Thread-safe: reference assignment is atomic and InferenceEngineInfo is an immutable record.
public InferenceEngineInfo EngineInfo { get; }
Property Value
InputMetadata
public IReadOnlyDictionary<string, NodeMetadata>? InputMetadata { get; }
Property Value
IsRocmSessionActive
Returns true when the ONNX session was successfully initialized with the MIGraphX execution provider on AMD ROCm hardware. false on non-Linux-x64 platforms, when MIGraphX initialization failed, or when CPU fallback was used.
public bool IsRocmSessionActive { get; }
Property Value
ModelCacheDirectory
Base directory for model cache. Defaults to LocalApplicationData. Derived classes can override to use a custom cache location.
protected virtual string ModelCacheDirectory { get; }
Property Value
ModelDownloadUrls
URLs where the model can be downloaded from, in priority order. Derived classes can override to provide custom model sources. If not overridden, the base engine will use the default ModelRegistry to find URLs.
protected virtual string[]? ModelDownloadUrls { get; }
Property Value
- string[]
ModelName
The name of the model file (e.g., "yolo11n.onnx"). Derived classes must implement this to specify which model they require.
protected abstract string ModelName { get; }
Property Value
ModelRegistry
Model registry for looking up download URLs by model name. Override to use a custom registry; defaults to the built-in registry.
protected virtual ModelRegistry ModelRegistry { get; }
Property Value
OutputMetadata
public IReadOnlyDictionary<string, NodeMetadata>? OutputMetadata { get; }
Property Value
PrimaryInputName
public string? PrimaryInputName { get; }
Property Value
Status
Lifecycle + runtime snapshot (EngineLoadState + partial-or-full InferenceEngineInfo + load error). Transitions fire OnStatusChanged when set. Out-of-process proxies mirror the worker's terminal states onto the same property.
public InferenceEngineStatus Status { get; }
Property Value
Methods
AppendCoreMlProvider(SessionOptions, CoreMLFlags)
Appends the CoreML execution provider. Override to inject a throw in tests.
protected virtual void AppendCoreMlProvider(SessionOptions options, CoreMLFlags flags)
Parameters
optionsSessionOptionsflagsCoreMLFlags
AppendDirectMlProvider(SessionOptions, int)
Appends the DirectML execution provider. Override to inject a throw in tests.
protected virtual void AppendDirectMlProvider(SessionOptions options, int deviceId)
Parameters
optionsSessionOptionsdeviceIdint
AppendMIGraphXProvider(SessionOptions, int)
Appends the MIGraphX execution provider with the compiled-model (.mxr) cache directory resolved by ResolveMIGraphXModelCacheDirectory(). The cache persists compiled MIGraphX programs across process restarts; without it the ORT provider compiles the full model on every session construction (an empty cache path explicitly means "always compile" in the EP). Override to inject a throw in tests.
protected virtual void AppendMIGraphXProvider(SessionOptions options, int deviceId)
Parameters
optionsSessionOptionsdeviceIdint
ApplyLoadedState(InferenceEngineInfo, IReadOnlyDictionary<string, NodeMetadata>?, IReadOnlyDictionary<string, NodeMetadata>?, string?)
Apply engine state that was loaded by an out-of-process worker (no in-process
InferenceSession). Used by OutOfProcessBaseInferenceEngine to mirror
the worker's EngineInfo/metadata so the diagnostics surface (provider,
fallback reason, ordered checks) behaves identically to the in-process engine. Does not
create a session; the proxy's PredictAsync(TInput) marshals to the worker instead.
protected void ApplyLoadedState(InferenceEngineInfo engineInfo, IReadOnlyDictionary<string, NodeMetadata>? inputMetadata, IReadOnlyDictionary<string, NodeMetadata>? outputMetadata, string? primaryInputName)
Parameters
engineInfoInferenceEngineInfoinputMetadataIReadOnlyDictionary<string, NodeMetadata>outputMetadataIReadOnlyDictionary<string, NodeMetadata>primaryInputNamestring
CreateInferenceSession(string, SessionOptions)
Constructs the ONNX session. Override to inject a recoverable native failure in tests.
protected virtual InferenceSession CreateInferenceSession(string modelPath, SessionOptions options)
Parameters
modelPathstringoptionsSessionOptions
Returns
Dispose()
Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
public void Dispose()
Dispose(bool)
protected virtual void Dispose(bool disposing)
Parameters
disposingbool
GetWarmupInput()
Generates dummy input for model warmup. Default implementation assumes float tensors. Override this for int64/string inputs.
protected virtual List<NamedOnnxValue> GetWarmupInput()
Returns
IsKfdPresent()
True when the ROCm KFD device node is present. Override in tests.
protected virtual bool IsKfdPresent()
Returns
LoadAsync()
Initializes the inference session with the best available hardware provider.
public virtual Task LoadAsync()
Returns
PostProcessWithContext(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?)
Takes raw ONNX output and the context object from PreProcess, returning the final result.
protected abstract TOutput PostProcessWithContext(IDisposableReadOnlyCollection<DisposableNamedOnnxValue> output, object? context)
Parameters
outputIDisposableReadOnlyCollection<DisposableNamedOnnxValue>contextobject
Returns
- TOutput
PreProcessWithContext(TInput)
Takes raw input and returns ONNX tensors plus an optional context object. The context object is passed to PostProcessWithContext, allowing you to share state without overriding the main pipeline.
protected abstract (List<NamedOnnxValue> InputTensors, object? Context) PreProcessWithContext(TInput input)
Parameters
inputTInput
Returns
PredictAsync(TInput)
Runs the inference pipeline: PreProcess -> Inference -> PostProcess.
public virtual Task<TOutput> PredictAsync(TInput input)
Parameters
inputTInputThe raw input data.
Returns
- Task<TOutput>
The processed prediction result.
ResolveMIGraphXModelCacheDirectory()
Resolves the MIGraphX compiled-model (.mxr) cache directory: an explicitly configured MIGraphXModelCacheDirectory wins (empty string = disabled), otherwise a per-user default under the same root as ModelCacheDirectory. Returns null when caching is disabled or the host has no user-profile directory to anchor the default. Pure path computation — directory creation happens in AppendMIGraphXProvider(SessionOptions, int) only.
protected virtual string? ResolveMIGraphXModelCacheDirectory()
Returns
TryLoadNativeLibrary(string, out string?)
Attempts to dlopen a native library; returns success and an error message on failure.
protected virtual bool TryLoadNativeLibrary(string name, out string? error)