Rerun failed testcases in selenium using TestNG

In software testing when we run testcases, failed tests are inevitable . The tests fail due to various reasons like server or network issues, scripting issues, unresponsive application or validation failure. Sometimes to optimize the next runs we need to run only failed test cases and also running only the failed tests validate the bug fixes quickly.

To achieve this we have two approaches in TestNG:

1. By using testng-failed.xml file in test-output folder

2. By implementing TestNG IRetryAnalyzer

Let's discuss these in detail:

First lets know about TestNG, TestNG is a open-source test automation framework inspired by JUnit and NUnit. It provides additional functionalities such as test annotations, grouping, prioritization, parameterization, and sequencing techniques in the code – features that were not provided earlier.

TestNG framework not only manages test cases, but also provides detailed reports of those tests. It provides a detailed summary that displays the number of test cases that have failed. The report also enables testers to accurately locate bugs and fix them at the earliest.

Rerunning failed testcases is simple because TestNG has inbuilt support for this. Whenever we run test suite using testng.xml, a testng-failed.xml file gets created in the test-output folder. Later we can run this xml file to rerun the failed testcases.

Rerun failed test cases by using testng-failed.xml:

Follow the steps below for this method:

• Create a java class with tests

Here we created 3 java classes with 2 tests each having one failed assertion to fail the test.

package testngDemo;
import org.testng.Assert;
import org.testng.annotations.Test;
public class RerunFailedTest1 {
	@Test
	public void test1() {
			Assert.assertTrue(true);
	}
	@Test
	public void failedtest1() {
			Assert.assertTrue(false);
	}
}

• Create a testng.xml file: Right click on project=> TestNG=> Convert to TestNG

• Now run the testng.xml file: Right Click=> Run As=>TestNG Suite

• Now the console output is as below with 3 failed tests.

In Test output folder we have testng-failed.xml file created. To see the test output folder refresh the project. This file keeps track of the failed tests, so running this file, runs the failed tests only.

Run this testng-failed.xml file: Right click=> Run As => TestNG Suite. The console output is as below:

Now changing the assertion to pass one test and rerunning the xml file again, the console output is as below

Rerunning failed tests by implementing TestNG IRetryAnalyzer:

Sometimes the tests fail due to different reasons other than bugs in the application. In order to make sure that the failure reported in the test report is genuine and not just one-off cases, we can retry running the failed test cases to eliminate false-negative test results in our test reports.

To rerun failed test runs automatically during the test run itself, we implement IRetryAnalyzer interface provided by TestNG. By overriding retry() method of the interface in your class, you can control the number of attempts to rerun a failed test case.

• Create an implemented class of IRetryAnalyzer interface and override retry method:

package testngDemo;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
public class RetryAnalyzer implements IRetryAnalyzer {

	//Counter to keep track of retry attempts
	int retryAttemptsCounter = 0;

	//The max limit to retry running of failed test cases
	//Set the value to the number of times we want to retry
	int maxRetryLimit = 3;

	//Method to attempt retries for failure tests
	public boolean retry(ITestResult result) {
		if (!result.isSuccess()) {
			if(retryAttemptsCounter < maxRetryLimit){
				retryAttemptsCounter++;
				return true;
			}
		}
		return false;
	} 
}

• Now add an attribute called “retryAnalyzer” to @Test annotated method with value as Implemented class name as shown below

package testngDemo;
import org.testng.Assert;
import org.testng.annotations.Test;
public class RerunFailedTest1 {
	int counter;
		
	@Test
	public void test1() {
		System.out.println("test run");
		Assert.assertTrue(true);
	}
	@Test(retryAnalyzer = RetryAnalyzer.class)
	public void failedtest1() {
		
			System.out.println("Failed test run");
			Assert.assertTrue(false);		
	}
}

• Now if we run TestNG.xml we see failed test case is executed one more time as we gave retry count= 1. Test case is marked as failed only after it reaches max retry count. In the remaining test runs it will be marked as skipped, the console output is shown below.

• Let's consider a scenario where failed test case passed after 2 reruns, the code is as shown below. so we are changing retry count = 3 in RetryAnalyzer class.

package testngDemo;
import org.testng.Assert;
import org.testng.annotations.Test;
public class RerunFailedTest1 {
	int counter;
		
	@Test
	public void test1() {
		System.out.println("test run");
		Assert.assertTrue(true);
	
	}
	
	@Test(retryAnalyzer = RetryAnalyzer.class)
	public void failedtest1() {
		counter++;
		if(counter <= 2)
		{
			System.out.println("Failed test run");
			Assert.assertTrue(false);
		}
		System.out.println("Failed test run passed");
	}
}

The console output is as below.

• But here we need to write attribute to every testcase we want to rerun. This will be cumbersome when we have 100's of testcases. To overcome this we can implement another TestNG interface IAnnotationTransformer that overrides transform method.

• Create an implemented class of IAnnotationTransformer interface and override transform method as:

package testngDemo;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import org.testng.IAnnotationTransformer;
import org.testng.IRetryAnalyzer;
import org.testng.annotations.ITestAnnotation;


public class AnnotationTransfer implements IAnnotationTransformer{

	//Overriding the transform method to set the RetryAnalyzer
	public void transform(ITestAnnotation testAnnotation, Class testClass, 
			Constructor testConstructor, Method testMethod) {
		IRetryAnalyzer retry = testAnnotation.getRetryAnalyzer();
		if (retry == null)
			testAnnotation.setRetryAnalyzer(RetryAnalyzer.class);
	}

}

• Add listeners to TestNG.xml file , so that we need not add attribute every test case.

• The console output is same as previous one.

If we want to add retry functionality for only limited testcases then no need of implementing the IAnnotationTranformer interface and listeners in xml file, just set the @Test annotations with retryAnalyzer attribute with RetryAnalyzer.class.