Test Distribution And Dependencies Proposal

issue #54161

Colours in text:

• todo
• done
• doing

Contents

1. Add test dependencies tests to project.xml
   1. New Tags Elements in project.xml
   2. ParseProjectXML ant tasks
   3. Converting classpaths from nbproject/project.properties to project.xml
2. Name convention for files
3. Compilation
4. Running tests
   1. CVS layout
   2. Test distribution
5. XTest Backward compatibility
6. Module project support
7. Extensibility

1. Add test dependencies to project.xml

1.1 New Tag Elements in project.xml

At first we need to specify the dependency between tests in the project.xml file. We need to specify dependecies for tests. For netbeans module can exists more test types. Currently we have qa-functional and unit test's types. Format of test dependency xml element is below

<test-dependencies> 
   <test-type>
   <name>unit|qa-functional</name>
    <dependency type="test|module" recursive="yes|no">
        <code-name-base>org.openide</code-name-base>
    </dependency> 
   </test-type>
</test-dependencies>  

• test-dependencies : defines testtype. There are test type for historical reason. QA team writes qa-functional. Developers write unit tests.
• dependency: single dependency either on test of module or module. It defines compilation and runtime dependency. The recursive property is used only for module dependency. It adds on classpath all modules which needs the module defined code-name-base to be enable in running IDE.
• code-name-base : code name base of the module

Example

            <test-dependencies>
               <test-type>
                   <name>unit</name>
                   <test-dependency>
                       <code-name-base>org.openide.filesystems</code-name-base>
                   </test-dependency>
                   <test-dependency>
                       <test/>
                       <code-name-base>org.openide.filesystems</code-name-base>
                   </test-dependency>
                   <test-dependency>
                       <code-name-base>org.openide.nodes</code-name-base>
                       <compile-dependency/>
                   </test-dependency>
                   <test-dependency>
                       <code-name-base>org.netbeans.modules.queries</code-name-base>
                       <recursive/>
                   </test-dependency>
               </test-type>
            </test-dependencies>     

1.2 ParseProjectXML ant tasks

ParseProjectXML ant tasks parses nbbuild/project.xml file. It sets classpaths for module. it was added testtype nested element in order to set classpath for tests. testtype element consists of these attributes:

• name - test type it can be either unit or qa-functional
• folder - set to the property value folder of test in test distribution
• compilecp - set to the property value compilation classpath
• runtimecp - set to the property value runtime classpath

1.3 Converting classpath from nbproject/project.project to project.xml

It was created fix-test-dependencies ant target in order to convert classpath from project.properties to project.xml in netbeans module project. It will convert test dependencies it the project.xml doesn't contains <test-dependencies> xml tag. The fix-test-dependencies will be run in testing mode (without modifications) if is defined test.fix.dependecies ant property.

2. Name conventions for files

2.1 Test folder for a module and test type

Target folder of test files for a module in binary test distribution is below.

    ${test_type}/${cluster_name}/${code_name_base}/
Example :
    unit/platform7/org-openide-filesystems/

The folder contains copy of test folder and build tests:

  tests.jar - compiled class
  data.zip - zip file with datafiles, the data file is created if in test  folder in sources exists data folder
  cfg-<test-type>.xml  - xtest configuration file 
  build-<test-type>.xml - xtest build file
  build.xml - xtest build file
  unit-unit.xml - junit build file (supported only for unit tests)
  xtest-unit.xml - xtest build file 
  test.properties - contains classpaths

The test.properties file contains these properties:

• test.unit.run.cp - runtime classpath for unit tests
• test.qa-functional.run.cp - for qa-functional tests
• extra.test.libs.cp - classpath with external libraries (nbjunit.jar,insanelib.jar)

The classpath contains merged classpath from nbbuild/project.properties with classpath evaluated from dependencies (project.xml).

Build scripts for running all tests(${test.dist.dir}/build.xml) with these targets:

• unit - runs all unit tests
• qa-functional - runs all qa-functional tests

2.2 Extra libraries

Extra libraries (nbjunit.jar, junit.jar,insanelib.jar, jemmy.jar,etc.) are stored in extralibs folder.

2.3 nbbuild/templates

Few buildscripts from nbbuild/templates are needed for running tests form test distribution by using xtest harness. These templates are stored in nbbuild/templates folder in root folder of test distribution.

3. Compilation

3.1 Single module

The tests are compiled by apisupport harness. No xtest is needed for building test distribution. Few targets are added to project's buildscript:

• test - runs unit tests (already exists)
• test-build - builds unit tests (already exists)
• test-build-deps - builds unit tests with dependencies
• test-build-qa-functional - builds qa-functional tests
• build-all-tests - builds all testtypes if cfg-<testtype>.xml files are present.

Extra libraries which are not modularized (${tools.jar}, etc.) are in nbproject/project.properties of module project.:

  test.unit.cp.extra - compilation classpath for unit tests
  test.unit.run.cp.extra - runtime classpath for unit tests
  test.qa-functional.cp.extra - compilation classpath for qa-functional tests
  test.qa-functional.run.cp.extra - runtime classpath for qa-functional tests

3.2 Whole tests distribution

build-test-dist ant tasks has been added to nbbuild/build.xml in order to build whole test distribution. Usefull properties:

• config.modules.test - list of building modules (for example "openide/loaders,openide/explorer")
• test.fail.on.error - if it is false the building errors of tests are ignored. The property has been created because in NetBeans cvs are uncompilable tests.

It builds tests for all modules defined by modules config. The default value is config.modules.stable.

4. Running tests

The tests can be run either from cvs or test distribution layout. The tests can be run by using xtest or junit harness (supported only for unit test type).

4.1 CVS layout

4.1.1 Single module for junit

  ant test (in project folder) 

4.1.2 All modules for junit

 ant run-test-dist (in nbbuild folder) 

4.1.3 Single module for xtest

     ant runtests -Dxtest.testtype=... -Dxtest.attribs=... -Dnetbeans.dest.dir=...  (in test folder) 

4.1.4 All modules for xtest

By Using master config

4.2 Test distribution

4.2.1 Single module:

 cd ${testtype}/{cluster}/code-name-base
      

• ant runtests -Dnetbeans.dest.dir=... (build scripts for xtest must be modified)
• ant -f unit-unit.xml -Dnetbeans.dest.dir=... (for unit tests by junit)

4.2.2 All modules:

For unit tests by junit

ant -f unit/unit-all-unit.xml -Dnetbeans.dest.dir=.. 

It runs all modules which has not <ide> executor.

For unit tests by xtest

ant -f unit/xtest-all-unit.xml -Dnetbeans.dest.dir=.. 

It runs all modules for testbag with "stable" attribs property . The attribs property can be override by xtest.attribs property.

Or from root folder of test distribution:

• ant unit   -- runs unit tests
• ant qa-functional   -- runs qa-functional tests
• ant   -- runs all tests

4.2.3 Custom properties:

• xtest.modules.list - specifies list of tests for execution in format "${cluster}/${code-name-base}".
  Example:

  ant -f unit/xtest-all-unit.xml -Dnetbeans.dest.dir=.. -Dxtest.modules.list="platform7/org-openide-filesystems"

  It runs unit tests for org-openide-filestems.
• test.required.modules - specifies required modules for test run on classpath, the modules are separated by ',' char. It is applied to only unit tests.
  Example:

  ant -f unit/xtest-all-unit.xml -Dnetbeans.dest.dir=.. -Dtest.module.filter="org-openide-loaders.jar,org-openide-explorer.jar"

  It runs all unit tests with org-openide-filesystems.jar and org-openide-loaders.jar on testing classpath.

5. Module/project support

• FileOwnerQuery and SourceOwner query for tests.jar
• Adding test libraries node

Actions in popup

• Run Tests - run unit tests
• • XTest|Compile QA Functional
  • XTest|Run QA Functional
  • XTest|Compile Unit
  • XTest|Run Unit

• Test Run Action is also available in popup of module suite.

6. Extensibility

Removing masterconfig.xml simplifies the extensibility of testing NetBeans modules. It was necessary to generate build script from masterconfig.xml by using xslt transformation for example to run code coverage for all modules. By using new design it will be used the property with modules list instead.

Some developers would like switch to TestNG. Into to test folder can be added testng-<testtype>.xml build script for unit tests. If it is desired to generatate xtest compatible xml results it can be implemented TestNG reporter class for this purpose. Task for running tests in modules is added to harness build script.