Verify statements in TestNG

If you are using TestNG framework for your Java tests then sometimes you might have faced a situation where you want your test execution to continue even if there are any assertion failures and throw all errors at the end. For this you need verify statements (also called soft assertions). Currently TestNG does not support verify statements. We have to implement them by our own.

There are several ways you can mimic verify statements in your test. One of the most widely accepted way is using StringBuffer where assertion statement will be placed inside try catch block and whenever assertion fails, the thrown error will be caught and stored in StringBuffer allowing your test method execution to continue. At the end of the test, buffered errors will be checked and logged if found, marking your test as failed. I personally don’t recommend using try catch block in your tests since it involves too much code writing and maintenance. Also, if you miss to check StringBuffer at the end of the test then the errors that you caught will not be logged making your test faulty.

We need to implement verify statements such that they look more cleaner and easy to use. For that we can leverage TestNG’s listener capability, so let us start implementing them,

Download complete source code here https://github.com/Gadigeppa-J/testng.git.

TestMethodErrorBuffer.java – Buffer class to hold your verification errors.

import java.util.List;

/* package access only */
class TestMethodErrorBuffer {

	// thread safe while running tests in parallel
	private static ThreadLocal<List<Throwable>> testErrorBuffer = new ThreadLocal<>();
	
	static List<Throwable> get(){
		return testErrorBuffer.get();
	}
	
	static void set(List<Throwable> errorBuffer){
		testErrorBuffer.set(errorBuffer);
	}
	
	static void remove(){
		testErrorBuffer.remove();
	}
	
}

VerificationError.java – Thrown to indicate that verification has failed. Error objects of this class will be buffered in TestMethodErrorBuffer.

public class VerificationError extends Error{

	private static final long serialVersionUID = 8247563849457669512L;

	public VerificationError(String message){
		super(message);
	}
	
}

Verify.java – Verify statements. Wrapper class around TestNG’s Assertion tool.

import org.testng.Assert;

public class Verify {

	protected Verify() {
		// hide constructor
	}

	static public void verifyTrue(boolean condition, String message) {		  
		try{
			Assert.assertTrue(condition, message);
		}catch(AssertionError e){
			addToErrorBuffer(e);
		}		  
	}

	static public void verifyFalse(boolean condition, String message) {
		try{
			Assert.assertFalse(condition, message);
		}catch(AssertionError e){
			addToErrorBuffer(e);
		}
	}

	static public void verifyEquals(Object actual, Object expected, String message) {
		try{
			Assert.assertEquals(actual, expected, message);
		}catch(AssertionError e){
			addToErrorBuffer(e);
		}
	}

	public static void verifyNotEquals(Object actual1, Object actual2, String message) {
		try{
			Assert.assertNotEquals(actual1,actual2,message);
		}catch(AssertionError e){
			addToErrorBuffer(e);
		}
	}

	/*
	 * Add other wrapper methods
	 */
	// ......
	// ......
	// ......
	// ......
	// ......

	private static void addToErrorBuffer(AssertionError e){	  

		try{				

			VerificationError verificationError = new VerificationError(e.getMessage());

			verificationError.setStackTrace(e.getStackTrace());

			TestMethodErrorBuffer.get().add(verificationError);

		}catch(NullPointerException ex){

			throw new RuntimeException("Please let TestNG know about " + TestMethodListener.class.getName() + " listener for verify statements to work. For more information go to http://testng.org/doc/documentation-main.html#testng-listeners");
		}

	}
}

TestMethodListener.java– Implementation of TestNG’s IInvokedMethodListener interface. Logic for this implementation is adapted from Dave’s blog (http://seleniumexamples.com/blog/guide/using-soft-assertions-in-testng). For verify statements to work you need to let TestNG know about this listener. For more information go to http://testng.org/doc/documentation-main.html#testng-listeners.

import java.util.ArrayList;
import java.util.List;

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
import org.testng.internal.Utils;

public class TestMethodListener implements IInvokedMethodListener{

	@Override
	public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {

		if(method.isTestMethod()){	
			
			if(TestMethodErrorBuffer.get()!=null){
				throw new RuntimeException("Stale error buffer detected!");
			}
			
			TestMethodErrorBuffer.set(new ArrayList<Throwable>()); // each test method will have its own error buffer
		}

	}

	@Override
	public void afterInvocation(IInvokedMethod method, ITestResult testResult) {

		if(method.isTestMethod()){

			List<Throwable> lThrowable = TestMethodErrorBuffer.get();

			/* if there are verification failures */
			if(lThrowable.size() > 0){

				/* set test result to failure */
				testResult.setStatus(ITestResult.FAILURE);

				/* if there is assertion error/exception then add it to throwable list */
				if(testResult.getThrowable() != null){
					lThrowable.add(testResult.getThrowable());
				}

				int size = lThrowable.size();

				/* if there is only one throwable then set it directly to test result */
				if(size == 1){
					testResult.setThrowable(lThrowable.get(0));
				}else{

					StringBuffer failureMessage = new StringBuffer("Multiple failures (").append(size).append(")\n");
					StringBuffer fullStack = new StringBuffer();

					for(int i =0 ; i < size-1; i++){	
						failureMessage.append("(").append(i+1).append(")").append(lThrowable.get(i).getClass().getName()).append(":").append(lThrowable.get(i).getMessage()).append("\n");						
						fullStack.append("Failure ").append(i+1).append(" of ").append(size).append("\n");	
						fullStack.append(Utils.stackTrace(lThrowable.get(i),false)[1]).append("\n");
					}

					fullStack.append("Failure ").append(size).append(" of ").append(size).append("\n");
					Throwable last = lThrowable.get(size-1);					
					failureMessage.append("(").append(size).append(")").append(last.getClass().getName()).append(":").append(last.getMessage()).append("\n\n");
					
					fullStack.append(last.toString());

					testResult.setThrowable(new Throwable(failureMessage.toString() + fullStack.toString()));
					testResult.getThrowable().setStackTrace(last.getStackTrace());
				}

			}
			
			TestMethodErrorBuffer.remove(); // remove stale
			
		}
	}

}

Now let us test out our new verification tool,

import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

/*
*  Using Listeners annotation (@Listeners) this is one of the way
*  to let TestNG know about your listeners
*/ 
@Listeners(TestMethodListener.class) 
public class VerifyTests {

	@Test
	public void verifyEqualsTest(){
		
		// verify pass
		Verify.verifyEquals("test1", "test1", "test1 doesn't match"); 
		
		// verify fail (continue)
		Verify.verifyEquals("test!", "test2", "test2 doesn't match"); 
		
		// verify pass
		Verify.verifyEquals("test3", "test3", "test3 doesn't match");
		
		// verify fail (continue)
		Verify.verifyEquals("test#", "test4", "test4 doesn't match");
		
		// verify pass
		Verify.verifyEquals("test5", "test5", "test5 doesn't match");
		
		// verify pass
		Verify.verifyEquals("test6", "test6", "test6 doesn't match");
		
		// assert fail (exit)
		Assert.assertEquals("test$", "test7", "test7 doesn't match");
		
		// assert not run!
		Assert.assertEquals("test8", "test8", "test8 doesn't match");
		
		// verify not run!
		Verify.verifyEquals("test9", "test9", "test9 doesn't match");
	}
	
}

// Output

/*		FAILED: test01
		java.lang.Throwable: Multiple failures (3)
		(1)org.muthu.VerificationError:test2 doesn't matches expected [test2] but found [test!]
		(2)org.muthu.VerificationError:test4 doesn't matches expected [test4] but found [test#]
		(3)java.lang.AssertionError:test7 doesn't matches expected [test7] but found [test$]

		Failure 1 of 3
		org.muthu.VerificationError: test2 doesn't matches expected [test2] but found [test!]
			at org.testng.Assert.fail(Assert.java:94)
			at org.testng.Assert.failNotEquals(Assert.java:494)
			at org.testng.Assert.assertEquals(Assert.java:123)
			at org.testng.Assert.assertEquals(Assert.java:176)
			at org.muthu.Verify.verifyEquals(Verify.java:90)
			at org.muthu.TestVerify.test01(TestVerify.java:15)
			at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
			at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
			at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
			at java.lang.reflect.Method.invoke(Unknown Source)
			at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
			at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
			at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
			at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
			at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
			at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
			at org.testng.TestRunner.privateRun(TestRunner.java:767)
			at org.testng.TestRunner.run(TestRunner.java:617)
			at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
			at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
			at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
			at org.testng.SuiteRunner.run(SuiteRunner.java:240)
			at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
			at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
			at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
			at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
			at org.testng.TestNG.run(TestNG.java:1057)
			at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
			at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
			at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)

		Failure 2 of 3
		org.muthu.VerificationError: test4 doesn't matches expected [test4] but found [test#]
			at org.testng.Assert.fail(Assert.java:94)
			at org.testng.Assert.failNotEquals(Assert.java:494)
			at org.testng.Assert.assertEquals(Assert.java:123)
			at org.testng.Assert.assertEquals(Assert.java:176)
			at org.muthu.Verify.verifyEquals(Verify.java:90)
			at org.muthu.TestVerify.test01(TestVerify.java:17)
			at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
			at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
			at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
			at java.lang.reflect.Method.invoke(Unknown Source)
			at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
			at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
			at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
			at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
			at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
			at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
			at org.testng.TestRunner.privateRun(TestRunner.java:767)
			at org.testng.TestRunner.run(TestRunner.java:617)
			at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
			at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
			at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
			at org.testng.SuiteRunner.run(SuiteRunner.java:240)
			at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
			at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
			at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
			at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
			at org.testng.TestNG.run(TestNG.java:1057)
			at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
			at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
			at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)

		Failure 3 of 3
		java.lang.AssertionError: test7 doesn't matches expected [test7] but found [test$]
			at org.testng.Assert.fail(Assert.java:94)
			at org.testng.Assert.failNotEquals(Assert.java:494)
			at org.testng.Assert.assertEquals(Assert.java:123)
			at org.testng.Assert.assertEquals(Assert.java:176)
			at org.muthu.TestVerify.test01(TestVerify.java:20)
			at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
			at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
			at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
			at java.lang.reflect.Method.invoke(Unknown Source)
			at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
			at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
			at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
			at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
			at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
			at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
			at org.testng.TestRunner.privateRun(TestRunner.java:767)
			at org.testng.TestRunner.run(TestRunner.java:617)
			at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
			at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
			at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
			at org.testng.SuiteRunner.run(SuiteRunner.java:240)
			at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
			at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
			at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
			at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
			at org.testng.TestNG.run(TestNG.java:1057)
			at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
			at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
			at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)


		===============================================
		    Default test
		    Tests run: 1, Failures: 1, Skips: 0
		===============================================*/

Selecting and uploading files in Selenium

Selecting and uploading files while running your tests on local machine

Uploading files in Selenium Webdriver is quite easy. You just have to sendkeys your local file path to file upload element.

This is how your file upload element looks like,

This is just a plan HTML input element just like textfield but with type ‘file’,

When you click on ‘Choose File’ button, OS native file select dialog will be opened. This dialog will not be recognized by your driver, so when uploading any file in Selenium you should never click on ‘Choose File’ button. Instead, just use sendkeys to select the file,

WebElement El = driver.findElement(By.id("'fileUploadField'"));
El.sendKeys("c:\\temp\\test.txt");

This will select your file without having to select it through file select dialog. This works flawlessly across all supported drivers.

Selecting and uploading files while running your tests on Selenium Grid

If you are running your tests on grid then you should let your remote driver know that the file that needs to be uploaded is residing on local machine and not on remote machine. This can be achieved by setting LocalFileDetector for RemoteWebElement,

WebElement El = driver.findElement(By.id("'fileUploadField'"));
((RemoteWebElement) El ).setFileDetector(new LocalFileDetector()); 
El.sendKeys("c:\\temp\\test.txt");

Here you are setting local file detector for file upload element, so whenever you use sendkeys on this element, your RemoteWebDriver uses this file detector object to detect the file. If the file is detected then it will be base64 encoded and uploaded to remote server through JSON Wire Protocol and the remote fixed path will be selected as file path.

Sorting string list in ascending and descending order in Java

Sorting string list in ascending order in Java is straight forward.

Below code will sort string list in ascending order,

	public static void main(String[] args){

		List<String> stringList = new ArrayList<>();
		stringList.add("Suzuki");
		stringList.add("Honda");
		stringList.add("Toyota");
		stringList.add("Ford");
		stringList.add("Renault");
		stringList.add("Hyundai");

		// before sorting
		System.out.println(stringList);

		// sort the list
		Collections.sort(stringList);

		// after sorting
		System.out.println(stringList);
	}

// before sorting
[Suzuki, Honda, Toyota, Ford, Renault, Hyundai]

// after sorting
[Ford, Honda, Hyundai, Renault, Suzuki, Toyota]

Here I’m adding car company names to string list and passing it to Collections sort method. The sort method will sort string list items in ascending order. Notice that sort method will return void, instead it will manipulate the original list object that we passed to it so when we print the list object, string items are printed in ascending order.

But sorting string list in descending order is not straight forward as ascending order. To understand sorting in descending order first we need to understand how sorting mechanism work. To understand sorting mechanism please go through my another post Understanding string list sorting in Java.

To sort string list in descending order we have to use overloaded sort method of Collections class.

public static <T> void sort(List<T> list, Comparator<? super T> c)

It takes two parameters, list to be sorted and comparator object. Comparator object is the object where our sorting logic resides. First we need to write our own Comparator class where our sorting logic resides for string sorting and then pass its object to sort method

public class MyComparator implements Comparator<String>{
		@Override
		public int compare(String o1, String o2) {
			return o2.compareTo(o1); // magic line
		}
	}

The highlighted line in the above code is where all the magic happens. This line is responsible for sorting list in ascending and descending order. o2.compareTo(o1) will make sort method to sort list in descending order but If we change it to o1.compareTo(o2) then list will be sorted in ascending order.

Now pass MyComparator object to sort method along with string list to be sorted and run the code.

	public static void main(String[] args){

		List<String> stringList = new ArrayList<>();
		stringList.add("Suzuki");
		stringList.add("Honda");
		stringList.add("Toyota");
		stringList.add("Ford");
		stringList.add("Renault");
		stringList.add("Hyundai");

		// before sorting
		System.out.println(stringList);

		// sort the list
		MyComparator myComparator = new MyComparator();
		Collections.sort(stringList, myComparator);

		// after sorting
		System.out.println(stringList);
	}

// before sorting
[Suzuki, Honda, Toyota, Ford, Renault, Hyundai]

// after sorting
[Toyota, Suzuki, Renault, Hyundai, Honda, Ford]

Notice that string list items are printed in descending order.

Here is the helper method you can use for sorting string list in ascending and descending order,

	public void listSortEx(List<String> list, final Boolean descOrder){
		Collections.sort(list, new Comparator<String>() {

			@Override
			public int compare(String o1, String o2) {
				if(descOrder){
					return o2.compareTo(o1);
				}else{
					return o1.compareTo(o2);
				}
			}
		});
	}

Using this method is easy. Just pass it a string list to be sorted and a boolean value true if you want to sort list in descending order or false to sort list in ascending order. Notice that this function will return void, so the original list object you pass will be manipulated instead.

Understanding string list sorting in Java

Sorting in Java is not straight forward as we think. It has got its own mechanism to sort the list we provide. Here is the example for String sorting in Java,

	public static void main(String[] args){

		List<String> stringList = new ArrayList<>();
		stringList.add("Suzuki");
		stringList.add("Honda");
		stringList.add("Toyota");
		stringList.add("Ford");
		stringList.add("Renault");
		stringList.add("Hyundai");

		// before sorting
		System.out.println(stringList);

		// sort the list
		Collections.sort(stringList);

		// after sorting
		System.out.println(stringList);

	}

[Suzuki, Honda, Toyota, Ford, Renault, Hyundai]
[Ford, Honda, Hyundai, Renault, Suzuki, Toyota]

Here I’m adding car company names to ArrayList. When I print the list without sorting, company names are printed in the order I added to the list. When I print the list after sorting, company names are printed in alphabetical order.

Based on the sorted output, two questions comes to our mind Who decides on the sorting order of the String and Why strings are sorted in ascending alphabetical order and why not descending alphabetical order. Let me answer these questions.

When we look at Collections sort method declaration in Java docs this is what it says,

public static <T extends Comparable<? super T>> void sort(List<T> list)

Sorts the specified list into ascending order, according to the natural ordering of its elements. All elements in the list must implement the Comparable interface.

When it says all elements in list must implement Comparable interface, In our case it means String class should should implement Comparable interface since we are using String list.

Here is the String class declaration which implement Comparable Interface as per Java docs.

public final class String extends Object implements Serializable, Comparable<String>, CharSequence

Comparable interface has one only method compareTo(T o) which all implementing classes should implement. Since String class implements Comparable interface so it should implement compareTo method. The compareTo method will be invoked on all String objects in the list one by one by passing other String object as its argument during sorting. This method will return either negative number or positive number or zero based on the comparison result of two objects.

int compareTo(T o){
	// compare current object with o
}

• If current object is greater than o -> positive number is returned
• If current object is smaller than o -> negative number is returned
• If current object is equal to o -> zero is returned

Based on the return value sorting of object takes place inside the sort method. For Strings, logic for comparing String objects is already written in its compareTo method so we don’t have to worry about it.