Table of Contents

Class OutOfProcessBaseInferenceEngine<TInput, TOutput>

Namespace
InferenceEngine.Core
Assembly
InferenceEngine.Core.dll

Host-side proxy that offers the same contract as BaseInferenceEngine<TInput, TOutput> but runs the ONNX session in a child worker process over a length-prefixed JSON pipe. The host never loads libonnxruntime/comgr; the worker never loads Mesa/GL — avoiding the Mesa libLLVM-19 ↔ comgr libLLVM-20 collision that aborts an in-process MIGraphX + GL-rendered UI. BaseInferenceEngine<TInput, TOutput>.EngineInfo is mirrored from the worker's Load response, so the diagnostics surface (provider, fallback reason, ordered checks) is identical to the in-process engine.

public abstract class OutOfProcessBaseInferenceEngine<TInput, TOutput> : BaseInferenceEngine<TInput, TOutput>, IInferenceEngine<TInput, TOutput>, IDisposable, IAsyncDisposable

Type Parameters

TInput
TOutput
Inheritance
BaseInferenceEngine<TInput, TOutput>
OutOfProcessBaseInferenceEngine<TInput, TOutput>
Implements
IInferenceEngine<TInput, TOutput>
Inherited Members

Remarks

This base overrides LoadAsync() and PredictAsync(TInput) to marshal to the worker; it never creates an in-process InferenceSession. The consumer supplies the worker executable path (its own app launched with worker args) and an OutOfProcessEngineOptions.

Congcurrency (H1): all load/predict IPC is serialized by InferenceEngine.Core.OutOfProcessBaseInferenceEngine<TInput, TOutput>._inFlight so that concurrent calls never interleave write/read on the shared pipe (which would silently pair the wrong request with the wrong response). Dispose(bool) (M6) also takes that lock so it cannot tear down the pipe while a call is mid-flight.

Constructors

OutOfProcessBaseInferenceEngine(OutOfProcessEngineOptions, ILogger?)

protected OutOfProcessBaseInferenceEngine(OutOfProcessEngineOptions options, ILogger? logger = null)

Parameters

options OutOfProcessEngineOptions
logger ILogger

Properties

ModelName

The name of the model file (e.g., "yolo11n.onnx"). Derived classes must implement this to specify which model they require.

protected override string ModelName { get; }

Property Value

string

Methods

CreateSerializer()

Serializer used for the IPC payload. Override to customize; defaults to SystemTextJsonWorkerSerializer<TInput, TOutput>.

protected virtual IWorkerSerializer<TInput, TOutput> CreateSerializer()

Returns

IWorkerSerializer<TInput, TOutput>

Dispose(bool)

protected override void Dispose(bool disposing)

Parameters

disposing bool

DisposeAsync()

Asynchronously tears down the worker: sends a graceful shutdown frame without blocking, then disposes the pipe/process and base session. Prefer await using over the synchronous Dispose(bool) — the synchronous path cannot (and must not) perform a sync-over-async write.

public virtual ValueTask DisposeAsync()

Returns

ValueTask

HandleWorkerStderrLine(string)

Single choke point for every worker stderr line: logs it, and while a load is in flight maps the native MIGraphX execution provider's compile markers onto the status object model (Detail) so consumers can show what the slow first-run kernel compile is doing instead of a bare "loading". The markers are pinned against the ORT 1.24.1.36 ROCm native build (libonnxruntime_providers_migraphx: "Model Compile: Begin" / "Model Compile: Complete"); any other line is logged only. Virtual test seam — never spawns a process itself.

protected virtual void HandleWorkerStderrLine(string line)

Parameters

line string

LoadAsync()

Spawn the worker, send load, and mirror the worker's BaseInferenceEngine<TInput, TOutput>.EngineInfo onto this proxy via ApplyLoadedState(InferenceEngineInfo, IReadOnlyDictionary<string, NodeMetadata>?, IReadOnlyDictionary<string, NodeMetadata>?, string?). Does NOT build an in-process session. Idempotent and thread-safe.

public override Task LoadAsync()

Returns

Task

PostProcessWithContext(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?)

Takes raw ONNX output and the context object from PreProcess, returning the final result.

protected override sealed TOutput PostProcessWithContext(IDisposableReadOnlyCollection<DisposableNamedOnnxValue> output, object? context)

Parameters

output IDisposableReadOnlyCollection<DisposableNamedOnnxValue>
context object

Returns

TOutput

PreProcessWithContext(TInput)

Takes raw input and returns ONNX tensors plus an optional context object. The context object is passed to PostProcessWithContext, allowing you to share state without overriding the main pipeline.

protected override sealed (List<NamedOnnxValue> InputTensors, object? Context) PreProcessWithContext(TInput input)

Parameters

input TInput

Returns

(List<NamedOnnxValue> InputTensors, object Context)

PredictAsync(TInput)

Marshal a predict to the worker and return its deserialized result.

public override Task<TOutput> PredictAsync(TInput input)

Parameters

input TInput

Returns

Task<TOutput>

StartWorker()

Spawn the worker process and wire its stdin/stdout. Override in tests to bypass process creation and inject in-memory streams (call SetWorkerStreamsForTesting(Stream, Stream)).

protected virtual void StartWorker()