For and For-Each Loop: a tale of how I iterated, was iterated, but was not iterated

Introduction

Loops are one of the basic structures of programming languages. For example, on the Oracle website there is a section “ Lesson: Language Basics ”, in which loops have a separate lesson “ The for Statement ”. Let's refresh the basics: The loop consists of three expressions (statements): initialization (initialization), condition (termination) and increment (increment): Interestingly, they are all optional, meaning we can, if we want, write:

for (;;){
}

True, in this case we will get an endless loop, because We do not specify a condition for exiting the loop (termination). The initialization expression is executed only once, before the entire loop is executed. It is always worth remembering that a cycle has its own scope. This means that initialization , termination , increment and the loop body see the same variables. The scope is always easy to determine using curly braces. Everything inside the brackets is not visible outside the brackets, but everything outside the brackets is visible inside the brackets. Initialization is just an expression. For example, instead of initializing a variable, you can generally call a method that will not return anything. Or just skip it, leaving a blank space before the first semicolon. The following expression specifies the termination condition . As long as it is true , the loop is executed. And if false , a new iteration will not start. If you look at the picture below, we get an error during compilation and the IDE will complain: our expression in the loop is unreachable. Since we will not have a single iteration in the loop, we will exit immediately, because false: It’s worth keeping an eye on the expression in the termination statement : it directly determines whether your application will have endless loops. Increment is the simplest expression. It is executed after each successful iteration of the loop. And this expression can also be skipped. For example:

int outerVar = 0;
for (;outerVar < 10;) {
	outerVar += 2;
	System.out.println("Value = " + outerVar);
}

As you can see from the example, each iteration of the loop we will increment in increments of 2, but only as long as the value outerVaris less than 10. In addition, since the expression in the increment statement is actually just an expression, it can contain anything. Therefore, no one forbids using a decrement instead of an increment, i.e. reduce value. You should always monitor the writing of the increment. +=performs an increase first and then an assignment, but if in the example above we write the opposite, we will get an infinite loop, because the variable outerVarwill never receive the changed value: in this case it =+will be calculated after the assignment. By the way, it’s the same with view increments ++. For example, we had a loop:

String[] names = {"John","Sara","Jack"};
for (int i = 0; i < names.length; ++i) {
	System.out.println(names[i]);
}

The cycle worked and there were no problems. But then the refactoring man came. He didn't understand the increment and just did this:

String[] names = {"John","Sara","Jack"};
for (int i = 0; i < names.length;) {
	System.out.println(names[++i]);
}

If the increment sign appears in front of the value, this means that it will first increase and then return to the place where it is indicated. In this example, we will immediately begin extracting the element at index 1 from the array, skipping the first one. And then at index 3 we will crash with the error " java.lang.ArrayIndexOutOfBoundsException ". As you might have guessed, this worked before simply because the increment is called after the iteration has completed. When transferring this expression to iteration, everything broke. As it turns out, even in a simple loop you can make a mess) If you have an array, maybe there’s some easier way to display all the elements?

For each loop

Starting with Java 1.5, the Java developers gave us a design for each loopdescribed on the Oracle site in the Guide called " The For-Each Loop " or for version 1.5.0 . In general, it will look like this: You can read the description of this construct in the Java Language Specification (JLS) to make sure that it is not magic. This construction is described in the chapter " 14.14.2. The enhanced for statement ". As you can see, the for each loop can be used with arrays and those that implement the java.lang.Iterable interface . That is, if you really want, you can implement the java.lang.Iterable interface and for each loop can be used with your class. You will immediately say, “Okay, it’s an iterable object, but an array is not an object. Sort of.” And you will be wrong, because... In Java, arrays are dynamically created objects. The language specification tells us this: “ In the Java programming language, arrays are objects .” In general, arrays are a bit of JVM magic, because... how the array is structured internally is unknown and is located somewhere inside the Java Virtual Machine. Anyone interested can read the answers on stackoverflow: " How does array class work in Java? " It turns out that if we are not using an array, then we must use something that implements Iterable . For example:

List<String> names = Arrays.asList("John", "Sara", "Jack");
for (String name : names) {
	System.out.println("Name = " + name);
}

Here you can just remember that if we use collections ( java.util.Collection ), thanks to this we get exactly Iterable . If an object has a class that implements Iterable, it is obliged to provide, when the iterator method is called, an Iterator that will iterate over the contents of that object. The code above, for example, would have a bytecode something like this (in IntelliJ Idea you can do "View" -> "Show bytecode" : As you can see, an iterator is actually used. If it weren't for the for each loop , we'd have to write something like:

List<String> names = Arrays.asList("John", "Sara", "Jack");
for (Iterator i = names.iterator(); /* continue if */ i.hasNext(); /* skip increment */) {
	String name = (String) i.next();
	System.out.println("Name = " + name);
}

Iterator

As we saw above, the Iterable interface says that for instances of some object, you can get an iterator with which you can iterate over the contents. Again, this can be said to be the Single Responsibility Principle from SOLID . The data structure itself should not drive the traversal, but it can provide one that should. The basic implementation of Iterator is that it is usually declared as an inner class that has access to the contents of the outer class and provides the desired element contained in the outer class. Here's an example from the class ArrayListof how an iterator returns an element:

public E next() {
            checkForComodification();
            int i = cursor;
            if (i >= size)
                throw new NoSuchElementException();
            Object[] elementData = ArrayList.this.elementData;
            if (i >= elementData.length)
                throw new ConcurrentModificationException();
            cursor = i + 1;
            return (E) elementData[lastRet = i];
}

As we can see, with the help of ArrayList.thisan iterator accesses the outer class and its variable elementData, and then returns an element from there. So, getting an iterator is very simple:

List<String> names = Arrays.asList("John", "Sara", "Jack");
Iterator<String> iterator = names.iterator();

Its work comes down to the fact that we can check whether there are elements further (the hasNext method ), get the next element (the next method ) and the remove method , which removes the last element received through next . The remove method is optional and is not guaranteed to be implemented. In fact, as Java evolves, interfaces also evolve. Therefore, in Java 8 there was also a method forEachRemainingthat allows you to perform some action on the remaining elements not visited by the iterator. What's interesting about an iterator and collections? For example, there is a class AbstractList. This is an abstract class that is the parent of ArrayListand LinkedList. And it is interesting to us because of such a field as modCount . Each change the contents of the list changes. So what does that matter to us? And the fact that the iterator makes sure that during operation the collection over which it is iterated does not change. As you understand, the implementation of the iterator for lists is located in the same place as modcount , that is, in the class AbstractList. Let's look at a simple example:

List<String> names = Arrays.asList("John", "Sara", "Jack");
names = new ArrayList(names);
Iterator<String> iterator = names.iterator();
names.add("modcount++");
System.out.println(iterator.next());

Here is the first interesting thing, although not on topic. Actually Arrays.asListreturns its own special one ArrayList( java.util.Arrays.ArrayList ). It does not implement adding methods, so it is unmodifiable. It is written about in the JavaDoc: fixed-size . But in fact, it is more than fixed-size . It is also immutable , that is, unchangeable; remove won't work on it either. We will also get an error, because... Having created the iterator, we remembered modcount in it . Then we changed the state of the collection “externally” (i.e., not through the iterator) and executed the iterator method. Therefore, we get the error: java.util.ConcurrentModificationException . To avoid this, the change during iteration must be performed through the iterator itself, and not through access to the collection:

List<String> names = Arrays.asList("John", "Sara", "Jack");
names = new ArrayList(names);
Iterator<String> iterator = names.iterator();
iterator.next();
iterator.remove();
System.out.println(iterator.next());

As you understand, if iterator.remove()you don’t do it before iterator.next(), then because. the iterator does not point to any element, then we will get an error. In the example, the iterator will go to the John element , remove it, and then get the Sara element . And here everything would be fine, but bad luck, again there are “nuances”) java.util.ConcurrentModificationException will only occur when hasNext()it returns true . That is, if you delete the last element through the collection itself, the iterator will not fall. For more details, it’s better to watch the report about Java puzzles from “ #ITsubbotnik Section JAVA: Java puzzles ”. We started such a detailed conversation for the simple reason that exactly the same nuances apply when for each loop... Our favorite iterator is used under the hood. And all these nuances apply there too. The only thing is, we won't have access to the iterator, and we won't be able to safely remove the element. By the way, as you understand, the state is remembered at the moment the iterator is created. And secure deletion only works where it is called. That is, this option will not work:

Iterator<String> iterator1 = names.iterator();
Iterator<String> iterator2 = names.iterator();
iterator1.next();
iterator1.remove();
System.out.println(iterator2.next());

Because for iterator2 the deletion through iterator1 was “external”, that is, it was performed somewhere outside and he knows nothing about it. On the topic of iterators, I would also like to note this. A special, extended iterator was made specifically for interface implementations List. And they named him ListIterator. It allows you to move not only forward, but also backward, and also allows you to find out the index of the previous element and the next one. In addition, it allows you to replace the current element or insert a new one at a position between the current iterator position and the next one. As you guessed, ListIteratorit is allowed to do this since Listaccess by index is implemented.

Java 8 and Iteration

The release of Java 8 has made life easier for many. We also did not ignore iteration over the contents of objects. To understand how this works, you need to say a few words about this. Java 8 introduced the java.util.function.Consumer class . Here's an example:

Consumer consumer = new Consumer() {
	@Override
	public void accept(Object o) {
		System.out.println(o);
	}
};

Consumer is a functional interface, which means that inside the interface there is only 1 unimplemented abstract method that requires mandatory implementation in those classes that specify the implements of this interface. This allows you to use such a magical thing as lambda. This article is not about that, but we need to understand why we can use it. So, using lambdas, the above Consumer can be rewritten like this: Consumer consumer = (obj) -> System.out.println(obj); This means that Java sees that something called obj will be passed to the input, and then the expression after -> will be executed for this obj. As for iteration, we can now do this:

List<String> names = Arrays.asList("John", "Sara", "Jack");
Consumer consumer = (obj) -> System.out.println(obj);
names.forEach(consumer);

If you go to the method forEach, you will see that everything is crazy simple. There's our favorite one for-each loop:

default void forEach(Consumer<? super T> action) {
        Objects.requireNonNull(action);
        for (T t : this) {
            action.accept(t);
        }
}

It is also possible to beautifully remove an element using an iterator, for example:

List<String> names = Arrays.asList("John", "Sara", "Jack");
names = new ArrayList(names);
Predicate predicate = (obj) -> obj.equals("John");
names.removeIf(predicate);

In this case, the removeIf method takes as input not a Consumer , but a Predicate . It returns boolean . In this case, if the predicate says " true ", then the element will be removed. It’s interesting that not everything is obvious here either)) Well, what do you want? People need to be given space to create puzzles at the conference. For example, let's take the following code for deleting everything that the iterator can reach after some iteration:

List<String> names = Arrays.asList("John", "Sara", "Jack");
names = new ArrayList(names);
Iterator<String> iterator = names.iterator();
iterator.next(); // Курсор на John
while (iterator.hasNext()) {
    iterator.next(); // Следующий элемент
    iterator.remove(); // Удалor его
}
System.out.println(names);

Okay, everything works here. But we remember that Java 8 after all. Therefore, let's try to simplify the code:

List<String> names = Arrays.asList("John", "Sara", "Jack");
names = new ArrayList(names);
Iterator<String> iterator = names.iterator();
iterator.next(); // Курсор на John
iterator.forEachRemaining(obj -> iterator.remove());
System.out.println(names);

Has it really become more beautiful? However, there will be a java.lang.IllegalStateException . And the reason is... a bug in Java. It turns out that it is fixed, but in JDK 9. Here is a link to the task in OpenJDK: Iterator.forEachRemaining vs. Iterator.remove . Naturally, this has already been discussed: Why iterator.forEachRemaining doesn't remove element in the Consumer lambda? Well, another way is directly through the Stream API:

List<String> names = new ArrayList(Arrays.asList("John", "Sara", "Jack"));
Stream<String> stream = names.stream();
stream.forEach(obj -> System.out.println(obj));

conclusions

As we saw from all the material above, a loop for-each loopis just “syntactic sugar” on top of an iterator. However, it is now used in many places. In addition, any product should be used with caution. For example, a harmless one forEachRemainingmay hide unpleasant surprises. And this once again proves that unit tests are needed. A good test could identify such a use case in your code. What you can watch/read on the topic:

• Video from the conference: " #ITsubbotnik Section JAVA: Java puzzles "
• Video from the conference: " Java 8 Puzzlers. Zurab Bely "
• Good article on Java Rush: " Iterator Pattern "

#Viacheslav