Class StreamingOptions
- Namespace
- InferenceEngine.Core
- Assembly
- InferenceEngine.Core.dll
Per-call streaming controls. Defaults are safe/greedy.
public sealed class StreamingOptions
- Inheritance
-
StreamingOptions
- Inherited Members
Properties
ChunkSize
Batch >=1 decoded steps per yield (null/1 = one step per yield). Default null.
public int? ChunkSize { get; set; }
Property Value
- int?
EosTokenIds
Token ids that terminate generation when sampled.
public IReadOnlyList<int> EosTokenIds { get; set; }
Property Value
IncludePrefillInStream
If true, also yield prompt tokens from the prefill pass. Default false.
public bool IncludePrefillInStream { get; set; }
Property Value
MaxSteps
Maximum decode steps (prefill counts as step 0) before the loop stops.
public int MaxSteps { get; set; }
Property Value
Sampling
Sampling strategy. Honored by BaseStreamingInferenceEngine.SampleNextToken;
temperature 0 (default) is greedy argmax; >0 enables softmax sampling with top-k/top-p.
public SamplingParams Sampling { get; set; }