pub fn estimate_tokens(text: &str) -> i32Expand description
Estimate token count using the configured tokenizer if present.
Falls back to estimate_tokens_heuristic when no global tokenizer has
been registered.
pub fn estimate_tokens(text: &str) -> i32Estimate token count using the configured tokenizer if present.
Falls back to estimate_tokens_heuristic when no global tokenizer has
been registered.