Java Scanner tokens() Method

The tokens() method in Java, part of the java.util.Scanner class, is used to create a stream of tokens from the input. This method is useful for processing input data as a stream of strings.

Table of Contents

1. Introduction
2. tokens() Method Syntax
3. Understanding tokens()
4. Examples
   • Basic Usage
   • Processing Tokens as a Stream
5. Real-World Use Case
6. Conclusion

Introduction

The tokens() method returns a stream of tokens that the Scanner generates from the input. This method is useful when you want to process input data in a functional style using Java Streams.

tokens() Method Syntax

The syntax for the tokens() method is as follows:

public Stream<String> tokens()

Parameters:

• This method does not take any parameters.

Returns:

• A Stream<String> of tokens.

Throws:

• IllegalStateException: If the scanner is closed.

Understanding tokens()

The tokens() method converts the input data into a stream of strings (tokens). This is useful for processing large amounts of data in a functional programming style, leveraging the power of Java Streams for efficient and concise data manipulation.

Examples

Basic Usage

To demonstrate the basic usage of tokens(), we will create a Scanner object and use it to create a stream of tokens from a string.

Example

import java.util.Scanner;
import java.util.stream.Stream;

public class TokensExample {
    public static void main(String[] args) {
        String input = "Hello world! Welcome to Java programming.";

        // Create Scanner object in try-with-resources to ensure it closes automatically
        try (Scanner scanner = new Scanner(input)) {
            Stream<String> tokenStream = scanner.tokens();

            // Print each token
            tokenStream.forEach(System.out::println);
        } // Scanner is automatically closed here
    }
}

Output:

Hello
world!
Welcome
to
Java
programming.

Processing Tokens as a Stream

This example shows how to process tokens as a stream, such as filtering and mapping tokens.

Example

import java.util.Scanner;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class ProcessTokensExample {
    public static void main(String[] args) {
        String input = "apple banana cherry date";

        // Create Scanner object in try-with-resources to ensure it closes automatically
        try (Scanner scanner = new Scanner(input)) {
            Stream<String> tokenStream = scanner.tokens();

            // Process tokens: filter and collect to a list
            var filteredTokens = tokenStream
                .filter(token -> token.length() > 5)
                .collect(Collectors.toList());

            System.out.println("Filtered tokens: " + filteredTokens);
        } // Scanner is automatically closed here
    }
}

Output:

Filtered tokens: [banana, cherry]

Real-World Use Case

Parsing a CSV File

In real-world applications, the tokens() method can be used to parse CSV data into individual tokens for further processing.

Example

import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
import java.util.stream.Stream;

public class CSVParser {
    public static void main(String[] args) {
        File file = new File("data.csv");

        try (Scanner scanner = new Scanner(file)) {
            scanner.useDelimiter(","); // Use comma as delimiter

            Stream<String> tokenStream = scanner.tokens();

            // Process tokens: print each token
            tokenStream.forEach(System.out::println);
        } catch (FileNotFoundException e) {
            System.out.println("File not found: " + e.getMessage());
        } // Scanner is automatically closed here
    }
}

Output (Assuming data.csv contains comma-separated values):

value1
value2
value3
...

Conclusion

The Scanner.tokens() method is used to create a stream of tokens from the input. This method is particularly useful for applications requiring functional-style data processing. By understanding and using this method, you can efficiently parse and handle input data as streams. Always close the Scanner using try-with-resources to ensure proper resource management.