Contains Tokenizer and related classes for tracking the sequence of tokens in a document. Maintaining a token sequence is essential to many features in programmers' editors.

These classes are generally intended for the support of features associated with an editor; they are not necessary to understand for the sake of using the features.