CIBC:Documentation:SCIRun:Testing
From SCIRun Documentation Wiki
Contents |
Overview
SCIRun uses CMake and CTest for regression testing. CTest is CMake’s testing program. Users can submit build results and testing results to our dashboard via the ctest executable.
http://dashboards.sci.utah.edu:8081/SCIRUN_CORE/Dashboard/
Different Dashboard Builds
There are 3 submission modes with CTtest: Experimental, Nightly, and Continuous.
Experimental By default no update happens. Experimental is typically used for testing if the changes were succesful, or verifying everything is working for new dashboards that are not yet tested, or for trying new features.
Nightly CTest will update to the latest nightly start time specified in your CTest configuration file as NIGHTLY_START_TIME. Assuming the nightly start time is 9pm EST and currently it is before 9pm EST, anything between 9pm EST yesterday and 9pm EST today will be updated to 9pm yesterday. This way all nightly submissions are updated to the same version of repository.
Continuous The repository is updated to the latest version and if there are no updates, the processing will stop. If there are any changes, the process is similar to Nightly or Experimental.
The Regression System (Administrative Level)
The Dart Server
The cibc Dart Server is currently located on cibc-web.sci.utah.edu. Our server is running a version from June 2007 (Daniel Bletzek at GE sent me a non-release version). But it is basically running Dart 1.0.9 plus a modification to the SubmissionErrors section. This version of Dart is sometimes referred to as Dart2 and was re-written to be much simpler than the original Dart (Dart Classic) which used Tcl. It has been completely re-written in Java and uses its own embedded web server and servlet engine (Jetty) and its own database (Derby). A more thorough link to the Dart documentation can be found in the Dart Documentation
Login to cibc-web.sci.utah.edu as darbyb (please contact darbyb@sci.utah.edu for the password). The files for the SCIRUN_CORE server are in the /home/darbyb/SCIRUN_CORE. The main actions you will do on cibc-web is to start/stop the server.
Starting the Server
To start the server, in cibc-web.sci.utah.edu, from /home/darbyb type the following command:
bin/jre1.5.0_08/bin/java –jar Dart1.0.9/DartServer.jar –refresh –refreshServer DartServer SCIRUN_CORE Teem Manta &
You will see lots of text scroll by. NOTE: If you need to start the server with multiple projects, just list them (i.e. SCIRUN_CORE Teem ...)
Stopping the Server
Sometimes, the server might become unresponsive. Some dashboards work, but others are broken. If you try to run the server startup command it will spit out somethign like the following. In that case, you need to kill the server (as described below) and restart it.
...
<12 Feb 2008 12:16:14,606> [main] ERROR dart.server.Database
SCIRUN_CORE: Failed to get connection
...
To stop the server, make sure to be logged in as darbyb on cibc-web.sci.utah.edu, and type the following command:
bin/jre1.5.0_08/bin/java –jar Dart1.0.9/DartClient.jar –q SCIRUN_CORE
Sometimes the above command doesn't work, and you will have to manually kill by finding out which process the server is running on, kill it, and then restart the server.
ps -Af | grep java | grep DartServer kill -9 <process id>
or with less human intervention (provided you have only one Dart server running):
ps -Af | grep java | grep DartServer | awk '{print $2}' | xargs kill -9
User Accounts
Users can create their own accounts on the Dart Server and eventually will be able to customize the look and feel of their dashboard page. Users associate repositories with user ids to allow Dart to notify them in cases such as a missing submission or errors in a Nightly or Continuous build.
There are default accounts created for many of the developers. They were created using their SCI (or CVRTI) email, with the password being their username. For example, foo@sci.utah.edu would have a password of foo. Then I associated the username used during SVN operations, with the SCIRUN_CORE repository.
For more information on creating a new server or project, please see the Dart Documentation.
Regression Machines, Scripts, Accounts
The cibc regression machines are as follows:
| Machine | OS | Graphics | Compiler | Login Instructions | Status |
| cibc-rd1 | 64-bit Red Hat Enterprise | NVidia Quadro FX 4500/PCI/SSE2 | gcc 3.4.6 | Login as scirun-tester | Need to get cron jobs working w/ X-server. Need to run all things on /scratch. |
| cibc-rd2 | 32-bit Windows XP | NVidia Quadro FX 4500 | Visual Studio .NET 7 (2003) | Login as darbyb (local account) | Need to figure out why some of the modules don't load (hence tests arent' being run right now). |
| cibc-rd3 | 32-bit Windows Vista | ATI Radeon X1950 | Microsoft 2005 Express | Login as scirun-tester (local account) | Seems to be working. |
| cibc-rd4 | 32-bit SuSE 10.2 | ATI Radeon X1950 | Login as scirun-tester | Just went online 9/6. Haven't started anything. | |
| cibc-rd5 | 32-bit Fedora Core 7 | Nvidia 7300LE | gcc 4.1.2 | Login as scirun-tester | Just started getting builds running. Need to make them run on /scratch and make sure cron jobs work once cibc-rd1 is working. |
| cibc-rd6 | 64-bit Ubuntu | Nvidia GeForce 8800 GTX | Being sent in for hardware failure. | ||
| cibc-rd7 | 64-bit Windows Vista | NVidia GeForce 8800 GTX | Microsoft .NET 2005 | Login as scirun-tester (local account) | Just got .NET 2005 installed and need to try to compile SCIRun. |
| cibc-rdmac | Darwin 8.10.1 | ATI Radeon X1600 256 MB VRAM | gcc 4.0.1 | Login as scirun-tester | Working. |
Machines rd1-rd7 are accessible through a KVM switch, which is accessible via cibc-access. The login for the system is cibc and you can contact ncrr-develop@sci.utah.edu for the password. To use the system simply log on and click the machine name for the machine you wish to view. This will want to down load a kvm.cgi file. Please open this with the Java Web Start program that installed with java. On Mac OSX it can be found at /Applications/Utilities/Java/Java Web Start. On linux it is called javaws and if found next to java. Once you open the file, you will have access to the desktop of the machine.
This system allows you to access the desktop of any of the regression machines (except cibc-rdmac which is in Jeroen's office). You can login with the appropriate testing account, setup Nightly and Continuous builds, and then close the KVM session, leaving the testing account logged in. This allows the tests to run on the testing machine and use that machines graphics card (sshing will use the local machine’s graphics). Once logged in, all of the available regression machines can be seen by clicking the Target Devices on the left column of the screen. Click the KVM Session link to the left of the machine you want to connect to. To close the session, move the mouse to the top of the virtual desktop and a select the File->Exit from the drop down menu.
There are a couple of options to make the KVM experience better. Go to the center top of the applet and hover your mouse there. A menu bar will appear. Go to Tools->Session Options. You can make your system mouse disappear by selecting "None" for the local cursor. Under the General tab, check the Pass-through box. Another menu option that is useful can be found in the View->Scaling and View->Color menus. This can help with the responsiveness of the KVM based on the bandwidth you have. If your system mouse reappears, go back to the cursor selection menu select something, hit OK, and then go back and select "None".
The windows machines have local accounts. The linux and mac machine use a generic scirun-tester account, but all of the builds need to be changed to build in /scratch on the local machine so they build faster.
The general directory structure is as follows: a)Builds directory that contains a Nightly and Continuous directory. Each of these contains a SCIRun/src b)Thirdparty which should contain the 3p build and any additional 3p needed (i.e. ITK) c) DashboardScripts which contains all of the CMake initialization scripts for running ctest and posting builds to the Dashboard d)Data which contains the CIBCData checkout which stores SCIRunData and gold standard images for image comparisons.
It is not consistent where these main directories are stored and may be a little bit confusing (especially since we're transitioning linux builds to build on /scratch). When possible, the DashboardScripts directory is in the /home/sci/scirun-tester account. If there is a /scratch directory, then the Builds, Thirdparty, and Data directories are there--otherwise they are in the home directory. For windows accounts, the home directory can be in C:/cygwin/home/darbyb or C:/Users/scirun-tester.
CMake Scripting Files
All of the Nightly and Continuous builds using CMake script files. CTest has a built-in scripting mode, which significantly simplifies generating dashboards. The ctest executable can be run using the -S argument and a path to a CMake script file. All of the script files can be found in the DashboardScripts directory. They are named using the machine name and submission type (Nightly, Continuous). For more information regarding the content of CMake script files, see CMake Scripting of CTest.
In Windows, the builds are controlled via the Scheduled Tasks mechanism. In Unix environments, ctest is run with cron jobs. There is currently an issue with running cron jobs on newer versions of linux where a cron job doesn't have access to the X-server. In this situation, none of the SCIRun tests that run various networks will run properly. This affects cibc-rd1, rd4, rd5, and rd6. In general, Nightly builds start around 4:30 am MST and Continuous builds start at 9 am MST and run until 7 pm.
Eventually, we want scripts that will update the CIBCData, Thirdparty and SCIRun svn repositories, and configure and build and run all appropriate tests. These scripts haven't been written/finished yet.
The Regression System (User Level)
Code Committment Process
- NOTE: Do not check in any changes if the dashboard Continuous or Nightly builds builds are indicating errors. This means our trunk is unstable.
- Make desired changes to your local svn tree.
- Update your local tree to the current repository version (remember to check the dashboard to make sure it is indicating a stable tree).
- Compile your build using CMake, turning testing on (see the Configuring SCIRun Testing Using CMake section). Ideally, turn on RUN_CLASS_TESTS, RUN_UNIT_TESTS, and RUN_EXAMPLE_TESTS and have the regression nets COMPARE_IMAGES. This will do the most thorough tests available at this time that you can do on your local machine.
- Submit an Experimental build, and verify that at least your local build compiled without errors and that all the tests passed.
- Commit your changes.
- Continuous builds start every hour on the hour from 9-7 MST on the 3 main platforms: Linux, Windows, and Mac. They take on average 30-60 minutes to complete. If your commit breaks a Nightly or Continuous build, you should receive an email from Dart with a short description and link to the build errors. You will only receive this email if your username has been registered with the Dart System (see User Accounts section above).
- Fix any errors from your commits as soon as possible. Please be courteous when scheduling your commits. Don't check in massive changes to the repository and then leave for the day and not be available to fix any errors. If you do break the build, you can view the errors by clicking under the Errors column of the specific broken build. The default is for only 50 errors to be reported to minimize overloading the Dart server.
Configuring SCIRun Testing Using CMake
During the CMake process, make sure that the BUILD_TESTING variable is set to ON. There are three different testing categories: testing Core SCIRun classes, unit test networks that test the functionality of individual modules, and sample test networks. These testing categories can be turned on or off by the CMake variables RUN_CLASS_TESTS, RUN_UNIT_TESTS, and RUN_SAMPLE_TESTS, respectively.
There are currently only two class tests for example purposes but this will increase in the future. These tests will test all of the classes in SCIRun. For each class, each function should be tested.
The unit tests consist of the networks found in the src/nets/Modules directory. For SCIRun alone, this is around 160 test networks.
The sample tests are more complicated example networks such as the forward-fem.srn network built in the tutorial.
CIBCData
The CIBCData repository (https://code.sci.utah.edu/svn/CIBCData) contains the most up to date SCIRunData, along with baseline images for the various platforms. All testing should using the SCIRunData directory in this repository as the value for the SCIRUN_DATA environment variable.
In the future, the testing scripts will update this repository automatically.
Image Comparisons
NOTE: Our current image comparison program is dependant in Teem.
By turning on the COMPARE_IMAGES CMake variable, each network tested that contains a ViewScene module, will add a test attempting to calculate the difference. When users turn on the COMPARE_IMAGES variable, they will need to specify a BASELINE_IMAGES_DIR which should point to a directory containing gold standard images for the test networks. If the gold standard image doesn't exist, or the network test fails and doesn't create a test image, the image comparison program will still try to execute but will report a failure. This is to help SCIRun developers make sure that we have the appropriate gold standard images and that the image comparisons are working properly.
If the baseline images and test images are found to be different, both images will be submitted to the dashboard.
CMake Variables for SCIRun Testing
BUILD_TESTING: Boolean to turn testing ON/OFF.
RUN_CLASS_TESTS: When testing, run the class tests (does not run SCIRun executable).
RUN_UNIT_TESTS: When testing, run the unit tests which should be testing individual modules.
RUN_SAMPLE_TESTS: When testing, run the sample test networks.
DART_TESTING_TIMEOUT (NOTE: If you have to account for dynamic compilation, make sure this is long enough to handle that)
BASELINE_IMAGES_DIR: Directory where baseline or gold standard images are stored. This should be the BaselineImages directory of the SVN repository for CIBCData (https://code.sci.utah.edu/svn/CIBCData/BaselineImages). Within the BaselineImages directory, are subdirectories for the different platforms.
TEST_IMAGES_DIR: Defaults to the project_binary_directory/Testing
COMPARE_IMAGES: Boolean to determine whether image comparisons are done.
