net.siefkes.nlstego.util
Class TokenizerFactory

java.lang.Object
  net.siefkes.nlstego.util.TokenizerFactory

public class TokenizerFactory
extends Object

Factory for creating TextTokenizers of different types.

Version:

$Revision: 1.4 $, $Date: 2005/07/12 17:02:22 $, $Author: siefkes $

Author:

Christian Siefkes

Field Summary

static String

WHITESPACE_CONTROL_OTHER Pattern string capturing whitespace and control/other characters.

Constructor Summary

TokenizerFactory()

Method Summary

static TextTokenizer	createAlnumTokenizer(CharSequence text) Static factory method to create an instance for tokenizing alphanumeric and symbol sequences and puntuation.
static TextTokenizer	createCategoryTokenizer(CharSequence text) Static factory method to create an instance for tokenizing according to Unicode categories.
static TextTokenizer	createThoroughTokenizer(CharSequence text) Static factory method to create an instance that uses the "thorough" patterns listed below.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WHITESPACE_CONTROL_OTHER

public static final String WHITESPACE_CONTROL_OTHER

Pattern string capturing whitespace and control/other characters.

See Also:

Constant Field Values

Constructor Detail

TokenizerFactory

public TokenizerFactory()

Method Detail

createAlnumTokenizer

public static TextTokenizer createAlnumTokenizer(CharSequence text)

Static factory method to create an instance for tokenizing alphanumeric and symbol sequences and puntuation. Token types:

• sequences of letters and digits (incl. marks, Unicode categories "L", "M", "N") -- full token is captured for TextTokenizer.capturedText()
• sequences of symbols ("S" category) -- nothing is captured
• a single punctuation sign, possibly repeated ("P" category) -- nothing is captured

When you are only interested in words and numbers (e.g. for indexing), you can use the captured text -- it will contain the full token for alphanumeric sequences, it will be empty for symbols and punctuation.

The whitespace pattern comprised a sequence of whitespace and control/other characters ("C" and "Z" categories).

Parameters:

text - the text to tokenize

Returns:

the created tokenizer

createCategoryTokenizer

public static TextTokenizer createCategoryTokenizer(CharSequence text)

Static factory method to create an instance for tokenizing according to Unicode categories. Token types:

• sequences of letters (incl. marks, "L" and "M" categories) -- full token is captured for TextTokenizer.capturedText()
• sequences of digits (integral numbers, "N" category) -- full token is captured
• sequences of symbols ("S" category) -- nothing is captured
• sequences of punctuation ("P" category) -- nothing is captured

When you are only interested in words and numbers (e.g. for indexing), you can use the captured text -- it will contain the full token for letter and digit sequences, it will be empty for symbols and punctuation.

The whitespace pattern comprised a sequence of whitespace and control/other characters ("C" and "Z" categories).

Parameters:

text - the text to tokenize

Returns:

the created tokenizer

createThoroughTokenizer

public static TextTokenizer createThoroughTokenizer(CharSequence text)

Static factory method to create an instance that uses the "thorough" patterns listed below.

• sequences of letters (incl. marks)
• sequences of digits (integral numbers)
• sequences of math symbols
• sequences of currency symbols
• sequences of other symbols (modifiers and misc)
• a single punctuation sign, possibly repeated

These patterns don't contain any useful information for TextTokenizer.capturedText().

The whitespace pattern comprised a sequence of whitespace and control/other characters.

Parameters:

text - the text to tokenize

Returns:

the created tokenizer