estimate_tokens

cellstate_core::context

Function estimate_tokens 

Source 
fn estimate_tokens(text: &str) -> i32
Expand description

Estimate tokens using the default heuristic tokenizer.