HOWTO Port Tests to STP |
![]() |
![]() |
![]() |
Overview |
![]() |
![]() |
![]() |
The Scalable Test Platform is designed so that users can add new tests simply. This how-to shows how to add a new test. An STP test should be:
- self-contained
- Any code needed by the test beyond common Linux/GNU/Unix utilites should be provided by the test kit. We'll provide some examples below
- fully automated
- As explained below, the STP engine will expect a driver script to be supplied in your test kit. Everything your test does will be driven by this driver script. No human intervention is possible.
- legal
- The results and test code must be covered by a proper Open Source License, so OSDL can distribute them. Please don't give us unlicensed code, or code that is under a non-Open Source license. Discuss license issues on the mailing lists.
Tests do _not_ have to be unique to STP. We welcome all test code.
Communication and Organization |
![]() |
![]() |
![]() |
First, let's explain the organizational side of things. This bit only applies if you wish to submit a test to the OSDL's STP (www.osdl.org). If you are using your own local copy of STP, feel free to ignore this section. Mechanics will be covered below.
We communicate STP information via mailing lists:
- stp-tests@lists.sourceforge.net - The general announcement list
- stp-devel@lists.sourceforge.net - Information for test developers
You should join these lists. Announce your new test on stp-tests when you are ready to roll it out.
The STP test code is stored in CVS on Sourceforge. (http://sourceforge.net/cvs/?group_id=35146 ) We reccommend that you grab a copy of the STP tests to use as an example. The module name is 'tests' and we've provided an example test 'exampletest' which I'll be referring to in this HOW-TO. Follow the Sourceforge CVS instructions to get an anonymous copy.
An important note: In most cases, the actual test code (not the STP wrapper) will have its own real home, separate from STP. (STP CVS will include a copy) For example, LMBench is maintained by the BitMover people and lives at http://www.bitmover.com/lmbench/. The OSDL Database test lives on Sourceforge: http://sourceforge.net/projects/osdldbt If you wish to change the _test_ instead of the test wrapper, you should contact the actual owner of the test code. So changes to DBT-1 should be made through the OSDL DBT Sourceforge project, not STP. Then the new version of the test can be added/updated on STP. If a test has no other home, the STP version is definitive and we'll keep changes in our CVS. (Example here would be the AIM tests - I don't think they have a living owner. ) If you are creating a brand-new never-seen-before-on-earth test, we also recommend that you have a separate home (Sourceforge works for us) for the test code. Duplication is good here.
To make your test part of the OSDL repository, you have two choices:
- -
- Obtain write access to our CVS. Send us an email, and we'll give you permission. Sourceforge works with ssh, and you can upload your SSH key for easy access. Write access requires that you play well with others. Changes made on Sourceforge do _not automatically jump to the OSDL lab STP rig - if you have STP CVS access, you will be required to inform OSDL ops when you make a commit so we can push your changes to the Big Rig.
- -
- Send us email, and we'll do the CVS work and OSDL deployment for you. We don't mind, really. Any of the Project Admins will be glad to respond, and we're glad to help. This is a good way to go if you don't want to be a test maintainer, but wish to add a test.
In the future: We are working on a new test repository method which will allow us to keep multiple versions of test code, and have more flexibility. Join the stp-devel mailing list and help plan it.
The Test Environment |
![]() |
![]() |
![]() |
The test environment is really very simple.
- -
- Each test has its own directory. Check CVS for examples. The directory is named for the test. There should be one test or test sequence per directory. Regardless of how you submit the test, the test code should be one complete directory, capable of being tarballed. When run, the test code will be placed in this directory on the test machine If you have write access to CVS, you will create this directory as part of your CVS upload.
- -
- All paths are relative to the test code directory (Typically '/root/testname') Avoid hard-coded paths in your test code.
- -
- Each test must have a driver script, and this script must be named 'wrap.sh'. This script can be written in shell, Perl, or Python.
- -
- The script must create a 'results' directory and place results there. Results will be recovered automatically by the STP engine. The engine will only look in 'results'.
- -
- The script must provide or create an 'index.html' file to present the results. Again, see STP test results for examples of this. Also see our example script.
- -
- The test script will be run by the root user. Non-root users must be created by the test script - use 'su -c' to run commands as a non-root user.
Step-by-Step |
![]() |
![]() |
![]() |
PreliminaryFirst, verify that you have all the bits necessary to run your test. If possible, you should try your test on several different distros/installs to determine what bits you need to provide. For example, if your test depends on ReiserFS, you may wish to include a version of the reiserfsprogs with your test, and your wrap script should build and install those tools. (See tiobench for an example ) Or, you might first test for the presence of reiserfsprogs, then test for version, then build your version if those tests fail. If your test does have those types of dependencies, you should test for then and either fail with an error message or supply the needed bits. STP provides 'standard' distro installations, which should include all the standard tools you need, but when in doubt, test for it!
Next, decide what is needed to properly setup your test. In many cases, the test authors have chosen to compile their test code from source on the STP machine. This is a good choice for longevity - and portability. You may also choose to install your test pre-built.
Then you should determine what is necessary to automate your test. What inputs are required? What setup must be performed before the test is run? What resources are needed? What output does the test produce, and how is that output captured and parsed? Also, be certain you know how the test is supposed to run. What is a 'normal' result? What is a 'bad' result?
Finally, document any constraints you might have. STP provides a variety of platforms, from single-CPU to 8-way ( and beyond...) Your test may not be well suited for all platforms.
Here is a suggested work method: Create a development directory to hold your test code. Keep the test code in a tarball. Before each test run, delete all files in the development directory, and reboot your development machine. Get in the habit of starting a test run with an empty directory and a fresh boot. Then do what the STP engine does: unpack your tarball, and 'sh -x wrap.sh'.
I use a Makefile to keep my development setup constant:
Makefile Example
# This is so i can do multiple runs while developing DATE := $(shell /bin/date '+%d_%m_%H%M') results: mkdir run.${DATE}; cp results/* run.${DATE} clean: # remove the previous test rm -rf dbt1 rm -f sapdb-server* rm -f sapdb-web* #clean config files cat /etc/services | grep -v sql[6,30] > /tmp/foo mv /tmp/foo /etc/services run: wrap.sh # this is what STP will do sh -x ./wrap.sh > run.output 2>&1
The Test Code |
![]() |
![]() |
![]() |
For our example, we are performing a very trivial test. Here's our test tarball:
tar tvf example.tar drwxr-xr-x cliffw/staff 0 2002-10-25 09:14:56 example/ -rw-r--r-- cliffw/staff 292 2002-10-25 09:12:59 example/Makefile -rw-r--r-- cliffw/staff 26 2002-07-02 12:45:00 example/index_foot.html -rw-r--r-- cliffw/staff 234 2002-07-02 12:44:38 example/index_head.html -rwxr-xr-x cliffw/staff 179 2002-10-25 09:14:34 example/upt.sh -rwxr-xr-x cliffw/staff 156 2002-10-25 09:14:18 example/cpu.sh -rwxr-xr-x cliffw/staff 157 2002-10-25 09:14:25 example/mem.sh
Our 'test' is three shell scripts. We have a Makefile, so we can build and install the test code with make. Then we'll run the test, capture the output and create a results page.
Makefile
all: cpu mem upt cpu: cpu.sh cp cpu.sh cpu; rm -f cpu.sh chmod 755 cpu mem: mem.sh cp mem.sh mem; rm -f mem.sh chmod 755 mem upt: upt.sh cp upt.sh upt; rm -f upt.sh chmod 755 upt
Notice we are careful to set execution permissions on our test code.
The test package includes some boilerplate html for our results page.
(index_head.html and index_foot.html )
The Wrap.sh Script |
![]() |
![]() |
![]() |
We're going to take a simple wrap.sh apart and show you how it's done. The actual script is in CVS in the exampletest directory.
#!/bin/sh # # wrap.sh: for dummy workload # Copyleft 2002 Cliff White# # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # [ rest removed ] Always include your license!
echo "This is a dummy test for the STP " VERSION=0.0.1 echo "$0 version $VERSION" # The STP engine captures standard out and standard error from your # test script and stores it in a log file. Make sure your log file is # informative MYHOME=`pwd` RESDIR='./results' #We establish two environment variables. $MYHOME #will be used for all relative paths. echo "Cleaning up the environment" if [ -d $RESDIR ]; then rm -rf $RESDIR fi mkdir $RESDIR if [ -d $RESDIR ]; then echo "Directory created!" else echo "Error - no results directory" fi # Create the output directory, using the Paranoid Method. (check before, # act, check after) # Now we'll do the test setup section:# unpack the test tar xvf ./example.tar # go to our test directory and # check for success cd example if [ -f ./cpu.sh ]; then echo "Tarball successfully unpacked" else echo "Tarball failed to unpack" exit 1 fi # build the test make # Test for build success if [ -f ./cpu ]; then echo "Make succeeded" else echo "Make failed" exit 1 fiFairly standard stuff here. Again, be sure to check for success, and report error on failure. Now let's run the test:
echo "Starting tests" # be sure to capture raw data for your results ./cpu > ../results/cpu.r ./mem > ../results/mem.r ./upt > ../results/upt.r
Results Processing |
![]() |
![]() |
![]() |
First, the test submitter will receive an email with the test log. If your test is simple, be sure the important results are both output to standard out and stored in a results file. This will ensure the email the user gets provides enough information they don't have to lookup the results on the web site.
For the Web site, you must create an index.html, and that page must have the presentation logic for your test results. Try to create a page that is informative, and useful by non-experts. You may want to make your results available in multiple formats - text, comma-separated (CSV) format for spreadsheet import or even binary, if the binary can be easily interpreted. Or, you can cheat, like I do:
# I use some boilerplate to start cat ./example/index_head.html > ./index.html echo "" >> ./index.html cat ./results/mem.r >> ./index.html echo "
" >> ./index.html cat ./example/index_foot.html >> ./index.html
That's about as simple as you can get.
STP will provide some system configuration information automatically as part of the test run. Machine configuration will be stored in the directories 'environment' and 'environment/machine_info'. I add these three lines to all my index.html files: ( they are part of the example footer.html)
<a href="environment/machine_info">Summary Information on Machine</a> <a href="environment/">System Environment Documentation</a> <a href="COPYING">Results Copyright License</a>
Further Examples |
![]() |
![]() |
![]() |
The STP supplies disk information in the file 'disk.info'. This portion of the STP is going to be improved soon.
example disk.info
#/dev/hdb,/mnt/hdb,65000,0,4,D /dev/sda,/mnt/sda,68369,0,4,D /dev/sdb,/mnt/sdb,68369,0,4,D /dev/sdc,/mnt/sdc,85461,0,4,D
Currently, only the first field in this file is valid. This file will be found in the directory above your test directory, so your script can access it as '../disk.info'. We are currently working on some script tools to make mounting devices easy. Right now, we have two example methods.
The DBT1-1tier test uses a Perl script, 'mp_crt.pl'. which reads disk.info and a mount.info file. ( see the dbt1-1tier directory in CVS) The 'tiobench' test uses a pair of scripts 'getmountinfo.sh' and 'putmountinfo.sh' .
We're working on a better tool for this, join the newsgroup and help out.