Skip to main content

tokenize

Function tokenize 

Source
fn tokenize(text: &str) -> Vec<String>
Expand description

Tokenizer([tone_marks, period_comma, colon, other_punctuation]).run: re.split on the combined, case-insensitive pattern.