Package emolib.tokenizer

Provides a high-level class to perform the tokenisation process.

See:
Description

Class Summary

Tokenizer

The Tokenizer abstract class defines the general structure to perform the tokenisation process, which splits a text string into individual units, called tokens.

Package emolib.tokenizer Description

Provides a high-level class to perform the tokenisation process.