truncate_to_token_budget

Function truncate_to_token_budget 

Source
pub fn truncate_to_token_budget(text: &str, budget: i32) -> String
Expand description

Truncate text to fit within token budget. Prefers sentence boundaries, falls back to word boundaries.

§Arguments

  • text - The text to truncate
  • budget - Maximum token budget

§Returns

Truncated text that fits within the budget