BenchKit

Do you want to evaluate the time and memory consumption of any software? BenchKit is made for you. For each run, it measures:

  • Run time Wall-clock time — the “user experience” point of view.
  • CPU usage Average CPU consumption of your software.
  • Peak memory Maximum memory used by your software.
  • Evolution How time and CPU evolve over the execution (as a percentage of what is allocated).

BenchKit is particularly useful when you need these measures over many inputs for a given program, and it can confine each execution within a maximum time or memory budget.

Example outputs

BenchKit provides a summary of the execution in a CSV file:

tool,test 1,ltl,2.68151048738,17.5,7.23868918419,normal,135055571400033_n_1
tool,test 2,ltl,2.55138092947,16.1,6.51366186142,timeout,135055571400035_n_1

Each column means:

# Column Meaning
1 Tool name identifies the tool, so files can be merged later
2 Test name the input/test being run
3 Examination the examination performed on the test
4 Peak memory maximum memory used during the execution (percentage)
5 CPU average CPU used during the execution (percentage)
6 Time time required to execute the program
7 Status normal, or timeout when the time limit is reached
8 Run id unique identifier giving access to additional data

BenchKit also samples memory and CPU usage throughout the execution in a CSV file:

1350560133.4,2.18068239019,13.6
1350560133.98,2.18020601597,13.5
1350560134.57,2.23562421696,13.6
1350560135.17,2.17822112339,13.9

Each column means:

# Column Meaning
1 Timestamp time of sampling (seconds since 1970)
2 CPU CPU sampled at this time (percentage)
3 Memory memory sampled at this time (percentage)

From such a sampled file, one can generate a chart (such as the ones selected for the MCC’2015).

How to use it

Current version

BenchKit is free software, distributed under the GPL. Documentation is ongoing.

View on GitHub

Personal BenchKit (old version)

Distributed under the GPL and no longer maintained. Provided as-is so you can operate and evaluate your own tool:

Version history

  • 1.1
    May 2, 2015

    Third published version, for the Model Checking Contest @ Petri Nets 2015. Adds the ability to assign several cores to a given virtual machine.

  • 1.0
    May 1, 2014

    Second published version, for the MCC @ Petri Nets 2014. Supports several virtual machines per host and is suitable for machines with a large number of cores.

  • β2
    Apr. 14, 2013

    Minor corrections and improved stability — in particular for the use of top in the VM to extract memory and CPU information (still one VM per remote host).

  • β1
    Feb. 5, 2013

    First published version (one virtual machine per remote host), for the MCC @ Petri Nets 2013.

Citation

If you use BenchKit, please cite:

@inproceedings{benchkit:2013,
  Author    = {F. Kordon and F. Hulin-Hubard},
  Booktitle = {14th International Conference on Application of Concurrency to System Design, {ACSD} 2014, Tunis La Marsa, Tunisia, June 23-27, 2014},
  Title     = {{BenchKit, a Tool for Massive Concurrent Benchmarking}},
  Pages     = {159--165},
  Url       = {http://dx.doi.org/10.1109/ACSD.2014.12},
  Year      = {2014}
}