Stringtokenizer Guide in Java

By Harish Gowda | | 3 minutes

Introduction

Big Hello! In this tutorial we will learn how to use StringTokenizer in Java. As the name suggests it breaks down string into smaller sub parts or tokens.

Just like complex problems are broken into smaller subproblems, StringTokenizer breaks strings into smaller tokens. This can be useful in many usecases

It can be useful in below usecases:

UseCases

  1. Tokenizing user inputs
  2. URL Parsing
  3. Data transformation
  4. Parsing configuration files
  5. Parsing CSV files

Usage

Let us see how we can use StringTokenizer in Java

Step 1:

First we need to import the class

1

import java.util.StringTokenizer;

Step 2:

We can create StringTokenizer by passing it 2 arguments. First argument is the string we want to tokenize. Second argument is the token by which string should be divided.

After initializing the tokenizer, we can get token by using a while loop.

“stringTokenizer.hasMoreTokens()” checks if there are more tokens available from this tokenizer’s string.

If this method returns true, then a subsequent call to “nextToken” with no argument will successfully return a token as shown in below code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

package examples;

import java.util.StringTokenizer;

public class App {

    public static void main(String[] args) {

        StringTokenizer stringTokenizer = new StringTokenizer("Apple,Orange,Mango,Banana",",");

        while(stringTokenizer.hasMoreTokens()){
            String token = stringTokenizer.nextToken();
            System.out.println(token);
        }


    }
}
1
2
3
4

Apple
Orange
Mango
Banana

Basic tokenization

Counting tokens

We can count no of tokens available in the string using countTokens() method

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

package examples;

import java.util.StringTokenizer;

public class App {

    public static void main(String[] args) {

        StringTokenizer stringTokenizer = new StringTokenizer("Apple,Orange,Mango,Banana",",");

        System.out.println("No of tokens: "+stringTokenizer.countTokens());


    }
}
1

No of tokens: 4

Multiple Delimiters

Instead of 1 delimiter we can specify multiple delimiters as shown in below example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

package examples;

import java.util.StringTokenizer;

public class App {

    public static void main(String[] args) {

        StringTokenizer stringTokenizer = new StringTokenizer("Apple,Orange|Mango,Banana",",|");

        while(stringTokenizer.hasMoreTokens()){
            String token = stringTokenizer.nextToken();
            System.out.println(token);
        }


    }
}
1
2
3
4

Apple
Orange
Mango
Banana

Returning delimiters as tokens

By default StringTokenizer does not return delimiters as tokens. You can change this by using adding 3rd argument to StringTokenizer instance as true

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

package examples;

import java.util.StringTokenizer;

public class App {

    public static void main(String[] args) {

        StringTokenizer stringTokenizer = new StringTokenizer("Apple,Orange,Mango,Banana",",",true);

        while(stringTokenizer.hasMoreTokens()){
            String token = stringTokenizer.nextToken();
            System.out.println(token);
        }



    }
}
1
2
3
4
5
6
7

Apple
,
Orange
,
Mango
,
Banana

Limitations in StringTokenizer

StringTokenizer do not support regular expressions which is required for complex parsing. It can be used only for simple tokenization tasks.

Conclusion

In this tutorial we learnt how to use StringTokenizer class with practical examples.