.. -*- rst -*-

.. highlightlang:: none

Tokenizers
==========

TODO: Write me.

Here are the list of built-in tokenizers:

* TokenBigram
* TokenBigramSplitSymbol
* TokenBigramSplitSymbolAlpha
* TokenBigramSplitSymbolAlphaDigit
* TokenBigramIgnoreBlank
* TokenBigramIgnoreBlankSplitSymbol
* TokenBigramIgnoreBlankSplitAlpha
* TokenBigramIgnoreBlankSplitAlphaDigit
* TokenDelimit
* TokenDelimitNull
* TokenTrigram
* TokenUnigram

