web2.0goodies.com » testng http://web2.0goodies.com/blog Tips, Tricks, and How To's for Web 2.0 Tue, 03 Jan 2012 05:34:32 +0000 en hourly 1 http://wordpress.org/?v=3.3 MultiThreaded TestNG, Surefire Craziness. http://web2.0goodies.com/blog/java/multithreaded-testng-surefire-craziness/ http://web2.0goodies.com/blog/java/multithreaded-testng-surefire-craziness/#comments Tue, 30 Jun 2009 18:53:33 +0000 admin http://web2.0goodies.com/blog/?p=25 The Setup

Maven 2.0.9
Surefire 2.4.3
TestNG version 5.8

The Craziness

If you annotate a method to have an invocationCount of 2, and a threadPoolSize of 2, and your test fails in an @BeforeMethod or an @AfterMethod when run by surefire, the test run count reported by surefire will be inaccurate. Significantly innacurate at times.

For example:


package com.infusion.crm.application;

import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.AssertJUnit;
import org.testng.annotations.Test;

/**
* Created by IntelliJ IDEA.
* User: joey
* Date: Jun 29, 2009
* Time: 3:19:13 PM
* To change this template use File | Settings | File Templates.
*/
public class TestThreadPoolSize {
@BeforeMethod
public void before(){
AssertJUnit.assertFalse(true);
}

@Test
public void test1(){
AssertJUnit.assertFalse(true);
}
}

Returns the following from maven test:


-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestSuite
Tests run: 8, Failures: 1, Errors: 0, Skipped: 7, Time elapsed: 10.298 sec <<< FAILURE!

Results :

Failed tests:
before(com.infusion.crm.application.TestThreadPoolSize)

Tests run: 8, Failures: 1, Errors: 0, Skipped: 7

This problem is also manifested if you create an AnnotationTransformer that forces the invocationCount and threadPoolSize of a test to both be greater then 1.

This bug is only manifested when multiple threads are used. If you set the invocation count to 20 and the thread pool size to 1, the bug does not appear.

Sometimes it is unpredictable and worse then just 2x the right number of tests. I ran a test, with 9 methods, and 4 threads, and a very exhaustive @BeforeMethod that takes several seconds to run. It failed repeatedly and said: 142 test run, 3 failed, 139 skipped.

]]> http://web2.0goodies.com/blog/java/multithreaded-testng-surefire-craziness/feed/ 0