Table of Contents

Class StreamingOptions

Namespace
InferenceEngine.Core
Assembly
InferenceEngine.Core.dll

Per-call streaming controls. Defaults are safe/greedy.

public sealed class StreamingOptions
Inheritance
StreamingOptions
Inherited Members

Properties

ChunkSize

Batch >=1 decoded steps per yield (null/1 = one step per yield). Default null.

public int? ChunkSize { get; set; }

Property Value

int?

EosTokenIds

Token ids that terminate generation when sampled.

public IReadOnlyList<int> EosTokenIds { get; set; }

Property Value

IReadOnlyList<int>

IncludePrefillInStream

If true, also yield prompt tokens from the prefill pass. Default false.

public bool IncludePrefillInStream { get; set; }

Property Value

bool

MaxSteps

Maximum decode steps (prefill counts as step 0) before the loop stops.

public int MaxSteps { get; set; }

Property Value

int

Sampling

Sampling strategy. Honored by BaseStreamingInferenceEngine.SampleNextToken; temperature 0 (default) is greedy argmax; >0 enables softmax sampling with top-k/top-p.

public SamplingParams Sampling { get; set; }

Property Value

SamplingParams