Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.pentaho.di.core
Class SimpleTokenizer

    java.lang.Object
      org.pentaho.di.core.SimpleTokenizer

    public class SimpleTokenizer

extends Object

The SimpleTokenizer class is used to break a string into tokens.

The delimiter can be used in one of two ways, depending on how the singleDelimiter flag is set:

- when true, the entire string is treated as a single delimiter
- when false, each character in the string is treated as a delimiter

The total number of tokens in the text is equal to the number of delimeters found plus one. An empty token is returned when:

- two consecutive delimiters are found
- the text starts with a delimiter
- the text ends with a delimiter

You can use the tokenizer like the StringTokenizer:

         SimpleTokenizer st = new SimpleTokenizer("this is a test", " ");
         while (st.hasMoreTokens())
             println(st.nextToken());

Or, you can use the tokenizer like the String.split(...) method:

         SimpleTokenizer st = new SimpleTokenizer("this is a test", " ");
         List list = st.getAllTokens();
         for (java.util.Iterator it = list.iterator(); it.hasNext();)
             println(it.next());

See Also:
  `StringTokenizer`, `String.split(java.lang.String, int)`

Constructor Summary
`SimpleTokenizer(String text, String delimiter)` Constructs a tokenizer for the specified string.
`SimpleTokenizer(String text, String delimiter, boolean singleDelimiter)` Constructs a tokenizer for the specified string.

Method Summary
`List<String>` | `getAllTokens()` Tokenize the remaining text and return all the tokens
`String` | `getRemainder()` Get the text that has not yet been tokenized.
`boolean` | `hasMoreTokens()` Tests if there are more tokens available from this tokenizer.
`String` | `nextToken()` Returns the next token from this tokenizer.
`String` | `nextToken(int tokenCount)` Returns the nth token from the current position of this tokenizer.
`void` | `setDelimiter(String delimiter)` Set the delimiter(s) used to parse the text.
`void` | `setText(String text)` Set the text to be tokenized.

Methods inherited from class java.lang.Object
`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### SimpleTokenizer

    public SimpleTokenizer(String text,
                           String delimiter)

  Constructs a tokenizer for the specified string. Each character in the delimiter string is treated as a delimiter.

  Parameters:
    `text` - a string to be tokenized.
    `delimiter` - the delimiter.

### SimpleTokenizer

    public SimpleTokenizer(String text,
                           String delimiter,
                           boolean singleDelimiter)

  Constructs a tokenizer for the specified string.

  If the singleDelmiter flag is true, then the delimiter string is used as a single delimiter. If the flag is false, the each character in the delimiter is treated as a delimiter.

  Parameters:
    `text` - a string to be tokenized.
    `delimiter` - the delimiter(s).
    `multipleDelimiters` - treat each character as a delimiter.

Method Detail

### setText

    public void setText(String text)

  Set the text to be tokenized.

  Parameters:
    `text` - a string to be tokenized.

### setDelimiter

    public void setDelimiter(String delimiter)

  Set the delimiter(s) used to parse the text. The delimiter can be reset before retrieving the next token.

  Parameters:
    `delimiter` - the delimiter.

### hasMoreTokens

    public boolean hasMoreTokens()

  Tests if there are more tokens available from this tokenizer. A subsequent call to `nextToken` will return a token.

  Returns:
    `true` when there is at least one token remaining; `false` otherwise.

### nextToken

    public String nextToken()

  Returns the next token from this tokenizer.

  Returns:
    the next token from this tokenizer.
  Throws:
    `NoSuchElementException` - if there are no more tokens

### nextToken

    public String nextToken(int tokenCount)

  Returns the nth token from the current position of this tokenizer. This is equivalent to advancing n-1 tokens and returning the nth token.

  Parameters:
    `tokenCount` - the relative position of the token requested
  Returns:
    the token found at the requested relative position.
  Throws:
    `NoSuchElementException` - if there are no more tokens

### getRemainder

    public String getRemainder()

  Get the text that has not yet been tokenized.

  Returns:
    the remainder of the text to be tokenized

### getAllTokens

    public List<String> getAllTokens()

  Tokenize the remaining text and return all the tokens

  Returns:
    a List containing all of the individual tokens

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD