ambit2.core.io

Class QuotedTokenizer

java.lang.Object

ambit2.core.io.QuotedTokenizer

public class QuotedTokenizer
extends Object

The QuotedTokenizer class is used to break a string into simple tokens or quoted tokens. A quoted token has the following characteristics:

• starts with a double quote
• two consective double quotes are replaced by a single double quote
• a single double quote terminates the token

See Also:

SimpleTokenizer

Field Summary

Fields

Modifier and Type	Field and Description
protected int	maxPosition
protected int	position
protected String	text

Constructor Summary

Constructors

Constructor and Description
QuotedTokenizer(String text, char delimiter) Constructs a tokenizer for the specified string.

Method Summary

Modifier and Type	Method and Description
List	getAllTokens() Tokenize the remaining text and return all the tokens
String	getRemainder() Get the text that has not yet been tokenized.
boolean	hasMoreTokens() Tests if there are more tokens available from this tokenizer.
static void	main(String[] args)
String	nextToken() Returns the next token from this tokenizer.
String	nextToken(int tokenCount) Returns the nth token from the current position of this tokenizer.
protected String	parseToken() Handle parsing of a quoted token here.
void	setDelimiter(String delimiter) Set the delimiter for the next token.
void	setText(String text) Set the text to be tokenized.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

text

protected String text

position

protected int position

maxPosition

protected int maxPosition

Constructor Detail

QuotedTokenizer

public QuotedTokenizer(String text,
                       char delimiter)

Constructs a tokenizer for the specified string.

Parameters:

text - a string to be tokenized.

delimiter - the delimiter.

Method Detail

setDelimiter

public void setDelimiter(String delimiter)

Set the delimiter for the next token.

Parameters:

delimiter - the delimiter.

parseToken

protected String parseToken()

Handle parsing of a quoted token here. Parsing of a simple token is still done by the parent.

Returns:

the next token from this tokenizer.

main

public static void main(String[] args)

setText

public void setText(String text)

Set the text to be tokenized.

Parameters:

text - a string to be tokenized.

hasMoreTokens

public boolean hasMoreTokens()

Tests if there are more tokens available from this tokenizer. A subsequent call to nextToken will return a token.

Returns:

true when there is at least one token remaining; false otherwise.

nextToken

public String nextToken()

Returns the next token from this tokenizer.

Returns:

the next token from this tokenizer.

Throws:

NoSuchElementException - if there are no more tokens

nextToken

public String nextToken(int tokenCount)

Returns the nth token from the current position of this tokenizer. This is equivalent to advancing n-1 tokens and returning the nth token.

Parameters:

tokenCount - the relative position of the token requested

Returns:

the token found at the requested relative position.

Throws:

NoSuchElementException - if there are no more tokens

getRemainder

public String getRemainder()

Get the text that has not yet been tokenized.

Returns:

the remainder of the text to be tokenized

getAllTokens

public List getAllTokens()

Tokenize the remaining text and return all the tokens

Returns:

a List containing all of the individual tokens