DriftEngineDriftCapture · function
clipTokenizer
A tokenizer over merges, [count, 2] token ids in rank order.
Explained in DriftCapture.
function clipTokenizer(merges: Float32Array): ClipTokenizerimport { clipTokenizer } from '@driftengine/capture';Parameters
| Parameter | Type | Description |
|---|---|---|
merges | Float32Array |