Class BaseStreamingInferenceEngine<TInput, TOutput, TStep>.KeyValueCacheState
- Namespace
- InferenceEngine.Core
- Assembly
- InferenceEngine.Core.dll
Convenience IDisposable holder for a list of carry-forward KV-cache tensors. Engines that use CloneAsPastKeyValue(DisposableNamedOnnxValue, string) should wrap the cloned values in an instance of this type (or any other IDisposable) and return it from ProcessStepOutputs(IDisposableReadOnlyCollection<DisposableNamedOnnxValue>, object?) as the next state. The base loop disposes the previous state on step rollover and the final state on stream end, releasing the cloned tensors.
protected sealed class BaseStreamingInferenceEngine<TInput, TOutput, TStep>.KeyValueCacheState : IDisposable
- Inheritance
-
BaseStreamingInferenceEngine<TInput, TOutput, TStep>.KeyValueCacheState
- Implements
- Inherited Members
Constructors
KeyValueCacheState(List<NamedOnnxValue>)
public KeyValueCacheState(List<NamedOnnxValue> values)
Parameters
valuesList<NamedOnnxValue>
Properties
Values
The carry-forward KV-cache tensors.
public IReadOnlyList<NamedOnnxValue> Values { get; }
Property Value
Methods
Dispose()
Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
public void Dispose()