Table of Contents

Class BaseStreamingInferenceEngine<TInput, TOutput, TStep>.KeyValueCacheState

Namespace
InferenceEngine.Core
Assembly
InferenceEngine.Core.dll

Convenience IDisposable holder for a list of carry-forward KV-cache tensors. Engines that use CloneAsPastKeyValue(DisposableNamedOnnxValue, string) should wrap the cloned values in an instance of this type (or any other IDisposable) and return it from ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?) as the next state. The base loop disposes the previous state on step rollover and the final state on stream end, releasing the cloned tensors.

protected sealed class BaseStreamingInferenceEngine<TInput, TOutput, TStep>.KeyValueCacheState : IDisposable
Inheritance
BaseStreamingInferenceEngine<TInput, TOutput, TStep>.KeyValueCacheState
Implements
Inherited Members

Constructors

KeyValueCacheState(List<NamedOnnxValue>)

public KeyValueCacheState(List<NamedOnnxValue> values)

Parameters

values List<NamedOnnxValue>

Properties

Values

The carry-forward KV-cache tensors.

public IReadOnlyList<NamedOnnxValue> Values { get; }

Property Value

IReadOnlyList<NamedOnnxValue>

Methods

Dispose()

Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.

public void Dispose()