Running the Tests
=================

All the tests are executed using the "Run" script in the top-level directory.

The simplest way to generate results is with the commmand:
    ./Run

This will run a standard "index" test (see "The BYTE Index" below), and
save the report in the "results" directory, with a filename like
    hostname-2007-09-23-01
An HTML version is also saved.

If you want to generate both the basic system index and the graphics index,
then do:
    ./Run gindex

If your system has more than one CPU, the tests will be run twice -- once
with a single copy of each test running at once, and once with N copies,
where N is the number of CPUs.  Some categories of tests, however (currently
the graphics tests) will only run with a single copy.

Since the tests are based on constant time (variable work), a "system"
run usually takes about 29 minutes; the "graphics" part about 18 minutes.
A "gindex" run on a dual-core machine will do 2 "system" passes (single-
and dual-processing) and one "graphics" run, for a total around one and
a quarter hours.

============================================================================

Detailed Usage
==============

The Run script takes a number of options which you can use to customise a
test, and you can specify the names of the tests to run.  The full usage
is:

    Run [ -q | -v ] [-i <n> ] [-c <n> [-c <n> ...]] [test ...]

The option flags are:

  -q            Run in quiet mode.
  -v            Run in verbose mode.
  -i <count>    Run <count> iterations for each test -- slower tests
                use <count> / 3, but at least 1.  Defaults to 10 (3 for
                slow tests).
  -c <n>        Run <n> copies of each test in parallel.

The -c option can be given multiple times; for example:

    ./Run -c 1 -c 4

will run a single-streamed pass, then a 4-streamed pass.  Note that some
tests (currently the graphics tests) will only run in a single-streamed pass.

The remaining non-flag arguments are taken to be the names of tests to run.
The default is to run "index".  See "Tests" below.

When running the tests, I do *not* recommend switching to single-user mode
("init 1").  This seems to change the results in ways I don't understand,
and it's not realistic (unless your system will actually be running in this
mode, of course).  However, if using a windowing system, you may want to
switch to a minimal window setup (for example, log in to a "twm" session),
so that randomly-churning background processes don't randomise the results
too much.  This is particularly true for the graphics tests.


Output can be specified by setting the following environment variables:

  * "UB_RESULTDIR" : Absolute path of output directory of result files.
  * "UB_TMPDIR" : Absolute path of temporary files for IO tests.
  * "UB_OUTPUT_FILE_NAME" : Output file name. If exists it will be overwritten.
  * "UB_OUTPUT_CSV"       : If set "true", output results(score only) to .csv.
============================================================================

Tests
=====

The available tests are organised into categories; when generating index
scores (see "The BYTE Index" below) the results for each category are
produced separately.  The categories are:

   system          The original Unix system tests (not all are actually
                   in the index)
   2d              2D graphics tests (not all are actually in the index)
   3d              3D graphics tests
   misc            Various non-indexed tests

The following individual tests are available:

  system:
    dhry2reg         Dhrystone 2 using register variables
    whetstone-double Double-Precision Whetstone
    syscall          System Call Overhead
    pipe             Pipe Throughput
    context1         Pipe-based Context Switching
    spawn            Process Creation
    execl            Execl Throughput
    fstime-w         File Write 1024 bufsize 2000 maxblocks
    fstime-r         File Read 1024 bufsize 2000 maxblocks
    fstime           File Copy 1024 bufsize 2000 maxblocks
    fsbuffer-w       File Write 256 bufsize 500 maxblocks
    fsbuffer-r       File Read 256 bufsize 500 maxblocks
    fsbuffer         File Copy 256 bufsize 500 maxblocks
    fsdisk-w         File Write 4096 bufsize 8000 maxblocks
    fsdisk-r         File Read 4096 bufsize 8000 maxblocks
    fsdisk           File Copy 4096 bufsize 8000 maxblocks
    shell1           Shell Scripts (1 concurrent) (runs "looper 60 multi.sh 1")
    shell8           Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 8")
    shell16          Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 16")

  2d:
    2d-rects         2D graphics: rectangles
    2d-lines         2D graphics: lines
    2d-circle        2D graphics: circles
    2d-ellipse       2D graphics: ellipses
    2d-shapes        2D graphics: polygons
    2d-aashapes      2D graphics: aa polygons
    2d-polys         2D graphics: complex polygons
    2d-text          2D graphics: text
    2d-blit          2D graphics: images and blits
    2d-window        2D graphics: windows

  3d:
    ubgears          3D graphics: gears

  misc:
    C                C Compiler Throughput ("looper 60 $cCompiler cctest.c")
    arithoh          Arithoh (huh?)
    short            Arithmetic Test (short) (this is arith.c configured for
                     "short" variables; ditto for the ones below)
    int              Arithmetic Test (int)
    long             Arithmetic Test (long)
    float            Arithmetic Test (float)
    double           Arithmetic Test (double)
    dc               Dc: sqrt(2) to 99 decimal places (runs
                     "looper 30 dc < dc.dat", using your system's copy of "dc")
    hanoi            Recursion Test -- Tower of Hanoi
    grep             Grep for a string in a large file, using your system's
                     copy of "grep"
    sysexec          Exercise fork() and exec().

The following pseudo-test names are aliases for combinations of other
tests:

    arithmetic       Runs arithoh, short, int, long, float, double,
                     and whetstone-double
    dhry             Alias for dhry2reg
    dhrystone        Alias for dhry2reg
    whets            Alias for whetstone-double
    whetstone        Alias for whetstone-double
    load             Runs shell1, shell8, and shell16
    misc             Runs C, dc, and hanoi
    speed            Runs the arithmetic and system groups
    oldsystem        Runs execl, fstime, fsbuffer, fsdisk, pipe, context1,
                     spawn, and syscall
    system           Runs oldsystem plus shell1, shell8, and shell16
    fs               Runs fstime-w, fstime-r, fstime, fsbuffer-w,
                     fsbuffer-r, fsbuffer, fsdisk-w, fsdisk-r, and fsdisk
    shell            Runs shell1, shell8, and shell16

    index            Runs the tests which constitute the official index:
                     the oldsystem group, plus dhry2reg, whetstone-double,
                     shell1, and shell8
                     See "The BYTE Index" below for more information.
    graphics         Runs the tests which constitute the graphics index:
                     2d-rects, 2d-ellipse, 2d-aashapes, 2d-text, 2d-blit,
                     2d-window, and ubgears
    gindex           Runs the index and graphics groups, to generate both
                     sets of index results

    all              Runs all tests


============================================================================

The BYTE Index
==============

The purpose of this test is to provide a basic indicator of the performance
of a Unix-like system; hence, multiple tests are used to test various
aspects of the system's performance.  These test results are then compared
to the scores from a baseline system to produce an index value, which is
generally easier to handle than the raw sores.  The entire set of index
values is then combined to make an overall index for the system.

Since 1995, the baseline system has been "George", a SPARCstation 20-61
with 128 MB RAM, a SPARC Storage Array, and Solaris 2.3, whose ratings
were set at 10.0.  (So a system which scores 520 is 52 times faster than
this machine.)  Since the numbers are really only useful in a relative
sense, there's no particular reason to update the base system, so for the
sake of consistency it's probably best to leave it alone.  George's scores
are in the file "pgms/index.base"; this file is used to calculate the
index scores for any particular run.

Over the years, various changes have been made to the set of tests in the
index.  Although there is a desire for a consistent baseline, various tests
have been determined to be misleading, and have been removed; and a few
alternatives have been added.  These changes are detailed in the README,
and should be born in mind when looking at old scores.

A number of tests are included in the benchmark suite which are not part of
the index, for various reasons; these tests can of course be run manually.
See "Tests" above.


============================================================================

Graphics Tests
==============

As of version 5.1, UnixBench now contains some graphics benchmarks.  These
are intended to give a rough idea of the general graphics performance of
a system.

The graphics tests are in categories "2d" and "3d", so the index scores
for these tests are separate from the basic system index.  This seems
like a sensible division, since the graphics performance of a system
depends largely on the graphics adaptor.

The tests currently consist of some 2D "x11perf" tests and "ubgears".

* The 2D tests are a selection of the x11perf tests, using the host
  system's x11perf command (which must be installed and in the search
  path).  Only a few of the x11perf tests are used, in the interests
  of completing a test run in a reasonable time; if you want to do
  detailed diagnosis of an X server or graphics chip, then use x11perf
  directly.

* The 3D test is "ubgears", a modified version of the familiar "glxgears".
  This version runs for 5 seconds to "warm up", then performs a timed
  run and displays the average frames-per-second.

On multi-CPU systems, the graphics tests will only run in single-processing
mode.  This is because the meaning of running two copies of a test at once
is dubious; and the test windows tend to overlay each other, meaning that
the window behind isn't actually doing any work.


============================================================================

Multiple CPUs
=============

If your system has multiple CPUs, the default behaviour is to run the selected
tests twice -- once with one copy of each test program running at a time,
and once with N copies, where N is the number of CPUs.  (You can override
this with the "-c" option; see "Detailed Usage" above.)  This is designed to
allow you to assess:

 - the performance of your system when running a single task
 - the performance of your system when running multiple tasks
 - the gain from your system's implementation of parallel processing

The results, however, need to be handled with care.  Here are the results
of two runs on a dual-processor system, one in single-processing mode, one
dual-processing:

  Test                    Single     Dual   Gain
  --------------------    ------   ------   ----
  Dhrystone 2              562.5   1110.3    97%
  Double Whetstone         320.0    640.4   100%
  Execl Throughput         450.4    880.3    95%
  File Copy 1024           759.4    595.9   -22%
  File Copy 256            535.8    438.8   -18%
  File Copy 4096          1261.8   1043.4   -17%
  Pipe Throughput          481.0    979.3   104%
  Pipe-based Switching     326.8   1229.0   276%
  Process Creation         917.2   1714.1    87%
  Shell Scripts (1)       1064.9   1566.3    47%
  Shell Scripts (8)       1567.7   1709.9     9%
  System Call Overhead     944.2   1445.5    53%
  --------------------    ------   ------   ----
  Index Score:             678.2   1026.2    51%

As expected, the heavily CPU-dependent tasks -- dhrystone, whetstone,
execl, pipe throughput, process creation -- show close to 100% gain when
running 2 copies in parallel.

The Pipe-based Context Switching test measures context switching overhead
by sending messages back and forth between 2 processes.  I don't know why
it shows such a huge gain with 2 copies (ie. 4 processes total) running,
but it seems to be consistent on my system.  I think this may be an issue
with the SMP implementation.

The System Call Overhead shows a lesser gain, presumably because it uses a
lot of CPU time in single-threaded kernel code.  The shell scripts test with
8 concurrent processes shows no gain -- because the test itself runs 8
scripts in parallel, it's already using both CPUs, even when the benchmark
is run in single-stream mode.  The same test with one process per copy
shows a real gain.

The filesystem throughput tests show a loss, instead of a gain, when
multi-processing.  That there's no gain is to be expected, since the tests
are presumably constrained by the throughput of the I/O subsystem and the
disk drive itself; the drop in performance is presumably down to the
increased contention for resources, and perhaps greater disk head movement.

So what tests should you use, how many copies should you run, and how should
you interpret the results?  Well, that's up to you, since it depends on
what it is you're trying to measure.

Implementation
--------------

The multi-processing mode is implemented at the level of test iterations.
During each iteration of a test, N slave processes are started using fork().
Each of these slaves executes the test program using fork() and exec(),
reads and stores the entire output, times the run, and prints all the
results to a pipe.  The Run script reads the pipes for each of the slaves
in turn to get the results and times.  The scores are added, and the times
averaged.

The result is that each test program has N copies running at once.  They
should all finish at around the same time, since they run for constant time.

If a test program itself starts off K multiple processes (as with the shell8
test), then the effect will be that there are N * K processes running at
once.  This is probably not very useful for testing multi-CPU performance.


============================================================================

The Language Setting
====================

The $LANG environment variable determines how programs abnd library
routines interpret text.  This can have a big impact on the test results.

If $LANG is set to POSIX, or is left unset, text is treated as ASCII; if
it is set to en_US.UTF-8, foir example, then text is treated as being
encoded in UTF-8, which is more complex and therefore slower.  Setting
it to other languages can have varying results.

To ensure consistency between test runs, the Run script now (as of version
5.1.1) sets $LANG to "en_US.utf8".

This setting which is configured with the variable "$language".  You
should not change this if you want to share your results to allow
comparisons between systems; however, you may want to change it to see
how different language settings affect performance.

Each test report now includes the language settings in use.  The reported
language is what is set in $LANG, and is not necessarily supported by the
system; but we also report the character mapping and collation order which
are actually in use (as reported by "locale").


============================================================================

Interpreting the Results
========================

Interpreting the results of these tests is tricky, and totally depends on
what you're trying to measure.

For example, are you trying to measure how fast your CPU is?  Or how good
your compiler is?  Because these tests are all recompiled using your host
system's compiler, the performance of the compiler will inevitably impact
the performance of the tests.  Is this a problem?  If you're choosing a
system, you probably care about its overall speed, which may well depend
on how good its compiler is; so including that in the test results may be
the right answer.  But you may want to ensure that the right compiler is
used to build the tests.

On the other hand, with the vast majority of Unix systems being x86 / PC
compatibles, running Linux and the GNU C compiler, the results will tend
to be more dependent on the hardware; but the versions of the compiler and
OS can make a big difference.  (I measured a 50% gain between SUSE 10.1
and OpenSUSE 10.2 on the same machine.)  So you may want to make sure that
all your test systems are running the same version of the OS; or at least
publish the OS and compuiler versions with your results.  Then again, it may
be compiler performance that you're interested in.

The C test is very dubious -- it tests the speed of compilation.  If you're
running the exact same compiler on each system, OK; but otherwise, the
results should probably be discarded.  A slower compilation doesn't say
anything about the speed of your system, since the compiler may simply be
spending more time to super-optimise the code, which would actually make it
faster.

This will be particularly true on architectures like IA-64 (Itanium etc.)
where the compiler spends huge amounts of effort scheduling instructions
to run in parallel, with a resultant significant gain in execution speed.

Some tests are even more dubious in terms of host-dependency -- for example,
the "dc" test uses the host's version of dc (a calculator program).  The
version of this which is available can make a huge difference to the score,
which is why it's not in the index group.  Read through the release notes
for more on these kinds of issues.

Another age-old issue is that of the benchmarks being too trivial to be
meaningful.  With compilers getting ever smarter, and performing more
wide-ranging flow path analyses, the danger of parts of the benchmarks
simply being optimised out of existance is always present.

All in all, the "index" and "gindex" tests (see above) are designed to
give a reasonable measure of overall system performance; but the results
of any test run should always be used with care.