NoteThe Intel® Compilers Test-prioritization tool enables the profile-guided optimizations to select and prioritize application's tests based on prior execution profiles of the application. The tool offers a potential of significant time saving in testing and developing large-scale applications where testing is the major bottleneck. The tool can be used for both IA-32 and Itanium® architectures.
This tool enables the users to select and prioritize the tests that are most relevant for any subset of the application's code. When certain modules of an application are changed, the test-prioritization tool suggests the tests that are most probably affected by the change. The tool analyzes the profile data from previous runs of the application, discovers the dependency between the application's components and its tests, and uses this information to guide the process of testing.
The tool provides an effective testing hierarchy based on the application's code coverage. The advantages of the tool usage can be summarized as follows:
Minimizing the number of tests that are required to achieve a given overall coverage for any subset of the application: the tool defines the smallest subset of the application tests that achieve exactly the same code coverage as the entire set of tests.
Reducing the turn-around time of testing: instead of spending a long time on finding a possibly large number of failures, the tool enables the users to quickly find a small number of tests that expose the defects associated with the regressions caused by a change set.
Selecting and prioritizing the tests to achieve certain level of code coverage in a minimal time based on the data of the tests' execution time.
The syntax for this tool is as follows:
tselect -dpi_list file
where -dpi_list is a required tool option that sets the path to the DPI list file that contains the list of the .dpi files of the tests you need to prioritize.
The tool uses options that are listed in the table that follows.
To run the test-prioritization tool on an application’s tests, the following files are required:
The .spi file generated by Intel Compilers when compiling the application for the instrumented binaries with the -prof_genx option.
The .dpi files generated by Intel Compilers profmerge tool as a result of merging the dynamic profile information .dyn files of each of the application tests. The user needs to apply the profmerge tool to all .dyn files that are generated for each individual test and name the resulting .dpi in a fashion that uniquely identifies the test. The profmerge tool merges all the .dyn files that exist in the given directory.
Note
It is very important that the user makes sure that unrelated .dyn files, oftentimes from previous runs or from other tests, are not present in that directory. Otherwise, profile information will be based on invalid profile data. This can negatively impact the performance of optimized code as well as generate misleading coverage information.
User-generated file containing the list of tests to be prioritized.
Note
For successful tool execution, you should:
Name each test .dpi file so that the file names uniquely identify each test.
Create a DPI list file: a text file that contains the names of all .dpi test files. The name of this file serves as an input for the test-prioritization tool execution command. Each line of the DPI list file should include one, and only one, .dpi file name. The name can optionally be followed by the duration of the execution time for a corresponding test in the dd:hh:mm:ss format.
For example: Test1.dpi 00:00:60:35 informs that Test1 lasted 0 days, 0 hours, 60 minutes and 35 seconds.
The execution time is optional. However, if it is not provided, then the tool will not prioritize the test for minimizing execution time. It will prioritize to minimize the number of tests only.
The chart that follows presents the test-prioritization tool usage model.
Here are the steps for a simple example (myApp.f90) for IA-32 systems.
Set
PROF_DIR=c:/myApp/prof_dir
2. Issue command
ifort -prof_genx myApp.f90
This command compiles the program and generates instrumented binary myApp as well as the corresponding static profile information pgopti.spi.
3. Issue command
rm PROF_DIR /*.dyn
Make sure that there are no unrelated .dyn files present.
4. Issue command
myApp < data1
Invocation of this command runs the instrumented application and generates one or more new dynamic profile information files that have an extension .dyn in the directory specified by PROF_DIR.
5. Issue command
profmerge -prof_dpi Test1.dpi
At this step, the profmerge tool merges all the .dyn files into one file (Test1.dpi) that represents the total profile information of the application on Test1.
6. Issue command
rm PROF_DIR /*.dyn
Make sure that there are no unrelated .dyn files present.
7. Issue command
myApp < data2
This command runs the instrumented application and generates one or more new dynamic profile information files that have an extension .dyn in the directory specified by PROF_DIR.
8. Issue command
profmerge -prof_dpi Test2.dpi
At this step, the profmerge tool merges all the .dyn files into one file (Test2.dpi) that represents the total profile information of the application on Test2.
9. Issue command
rm PROF_DIR /*.dyn
Make sure that there are no unrelated .dyn files present.
10. Issue command
myApp < data3
This command runs the instrumented application and generates one or more new dynamic profile information files that have an extension .dyn in the directory specified by PROF_DIR.
11. Issue Command
profmerge -prof_dpi Test3.dpi
At this step, the profmerge tool merges all the .dyn files into one file (Test3.dpi) that represents the total profile information of the application on Test3.
12. Create a file named tests_list with three lines. The first line contains Test1.dpi, the second line contains Test2.dpi, and the third line contains Test3.dpi.
When these items are available, the test-prioritization tool may be launched from the command line in PROF_DIR directory as described in the following examples. In all examples, the discussion references the same set of data.
tselect -dpi_list tests_list -spi pgopti.spi
where the /spi option specifies the path to the .spi file.
Here is a sample output from this run of the test-prioritization tool.
Total number of tests = 3
Total block coverage ~ 52.17
Total function coverage ~ 50.00
|
Num |
%RatCvrg |
%BlkCvrg |
%FncCvrg |
Test Name @ Options |
|
1 |
87.50 |
45.65 |
37.50 |
Test3.dpi |
|
2 |
100.00 |
52.17 |
50.00 |
Test2.dpi |
In this example, the test-prioritization tool has provided the following information:
By running all three tests, we achieve 52.17% block coverage and 50.00% function coverage.
Test3 by itself covers 45.65% of the basic blocks of the application, which is 87.50% of the total block coverage that can be achieved from all three tests.
By adding Test2, we achieve a cumulative block coverage of 52.17% or 100% of the total block coverage of Test1, Test2, and Test3.
Elimination of Test1 has no negative impact on the total block coverage.
Suppose we have the following execution time of each test in the tests_list file.
Test1.dpi 00:00:60:35
Test2.dpi 00:00:10:15
Test3.dpi 00:00:30:45
The following command executes the test-prioritization tool to minimize
the execution time with the
-mintime option:
tselect -dpi_list tests_list -spi pgopti.spi -mintime
Here is a sample output.
Total number of tests = 3
Total block coverage ~ 52.17
Total function coverage ~ 50.00
Total execution time = 1:41:35
|
num |
elapsedTime |
%RatCvrg |
%BlkCvrg |
%FncCvrg |
Test Name @ Options |
|
1 |
10:15 |
75.00 |
39.13 |
25.00 |
Test2.dpi |
|
2 |
41:00 |
100.00 |
52.17 |
50.00 |
Test3.dpi |
In this case, the results indicate that the running all tests sequentially would require one hour, 45 minutes, and 35 seconds, while the selected tests would achieve the same total block coverage in only 41 minutes.
Note
The order of tests when prioritization is based on minimizing time (first Test2, then Test3) could be different than when prioritization is done based on minimizing the number of tests. See example above: first Test3, then Test2. In Example 2, Test2 is the test that gives the highest coverage per execution time. So, it is picked as the first test to run.
The -cutoff option enables the test-prioritization tool to exit when it reaches a given level of basic block coverage.
tselect -dpi_list tests_list -spi pgopti.spi -cutoff 85.00
If the tool is run with the cutoff value of 85.00 in the above example, only Test3 will be selected, as it achieves 45.65% block coverage, which corresponds to 87.50% of the total block coverage that is reached from all three tests.
The test-prioritization tool does an initial merging of all the profile information to figure out the total coverage that is obtained by running all the tests. The -nototal option. enables you to skip this step. In such a case, only the absolute coverage information will be reported, as the overall coverage remains unknown.