Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(text: &str) -> BTreeSet<String>
Expand description

Lowercase, stopword-stripped, singularised token set.