Struct HeuristicTokenizer 

pub struct HeuristicTokenizer {
    ratio: f32,
    model_family: String,
}

Heuristic tokenizer using character-to-token ratios.

This provides fast, approximate token counts without requiring an actual tokenizer model. Good for quick estimates.

Fields

ratio: f32

Tokens per character ratio (model-specific)

model_family: String

Model family identifier

Implementations

impl HeuristicTokenizer

pub fn for_model(model: &str) -> Self

Create a new heuristic tokenizer for a specific model.

Uses empirically-derived ratios based on model family.

pub fn with_ratio(ratio: f32, model_family: impl Into<String>) -> Self

Create with a custom ratio.

pub fn ratio(&self) -> f32

Get the current ratio.

Trait Implementations

impl Clone for HeuristicTokenizer

fn clone(&self) -> HeuristicTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HeuristicTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for HeuristicTokenizer

fn default() -> Self

Returns the “default value” for a type.

impl Tokenizer for HeuristicTokenizer

fn count(&self, text: &str) -> i32

Count tokens in the given text.

fn model_family(&self) -> &str

Get the model family this tokenizer is for (e.g., “gpt-4”, “claude”).

fn encode(&self, _text: &str) -> Vec<u32>

Encode text to token IDs (for advanced use cases). Returns empty vec if not supported.

fn decode(&self, _tokens: &[u32]) -> String

Decode token IDs back to text. Returns empty string if not supported.