CS 410 Top: Measuring Computer Performance

Credit Hours: 4
Course Coordinator: N/A
Course Description: We will cover the fundamentals of computer performance measurement at a level suitable for advanced undergraduates and beginning graduate students. Topics include measurement tools and techniques, measurement error, benchmarks, and experiment design. Projects focus on hands-on experience conducting performance measurement and analysis in a variety of contexts.
Prerequisites: CS 200, 201, 202, and 333 (or equivalent). Students must be comfortable programming in C or C++ in a Unix environment before course start.
Goals: Understand the main techniques of performance analysis: measurement, simulation, and modeling; and when to apply each method.

Learn aspects of computer architecture, operating systems, and compilers as necessary to study the related performance issues. Understand the relationship and interactions between the three as related to application performance.

Learn in depth the methods, challenges, and available tools for application performance measurement.

Introduction to recent and ongoing research in performance tools.

Upon the successful completion of this course students will be able to:

  1. Evaluate published performance studies.
  2. Design and implement performance studies.
  3. Work effectively in a group on a technical project.
  4. Use statistics to effectively interpret performance data.
  5. Choose appropriate measurement tools for a given task.
  6. Continue to advanced study in performance evaluation.
Textbooks: Required Text: "Measuring Computer Performance: A Practitioner's Guide" by David J. Lilja. Cambridge University Press. ISBN: 0 521 64105 5.
References: Approximately 3 additional readings will be assigned from the research literature, to be announced each quarter.
Major Topics: Performance metrics
Average Performance and Variability
Errors in Experimental Measurements
Comparing Alternatives
Measurement Tools and Techniques
Benchmark Programs
Linear-regression models
Design of Experiments
Simulation and random-number generation
Intro to Queueing Analysis

Case Studies (e.g., measuring web server performance, measuring cluster performance, building performance tools; includes architecture, compilers, OS topics as needed)

Laboratory Exercises: Project #1: Measuring Performance on a UNIX workstation (3 weeks; completed independently) Students complete 5 specified experiments on a UNIX workstation:
1. Determining clock precision
2. System call overhead
3. Context switch overhead
4. Page fault latency
5. File read/write time.
Then they present their results in a short research paper (max. length 6 pages) containing results and discussion of results.

Project #2: Performance Analysis Case Study (6 weeks; performed in groups of 4-5 students) Students choose from a variety of possible projects. Example topics:
1. Comparison of measurement technologies Students will choose a set of benchmark applications; and collect data for the benchmarks using both gprof and a dynamic instrumentation based profiler. They will analyze the results and discuss the differences between the two tools.
2. Network performance Students will develop techniques for combining application performance information with underlying system performance, such as available network bandwidth. Students may implement their ideas by extending an existing performance measurement research tool.

Each project team will present their results to the class; and write a report describing their approach and results.


CAC Category Credits Core Advanced
Data Structures
Algorithms
Software Design 0.3
Computer Architecture 0.5 0.5
Programming Languages 0.3

Oral and Written Communications: Every student is required to submit at least 2 written reports (not including exams, tests, quizzes, or commented programs) of typically 7 pages and to make 1 oral presentations of typically 5-10 minutes duration.
Social and Ethical Issues: None.
Theoretical Content: Probability theory; central limit theorem; Introductory queuing analysis. (3/20 lectures total)
Problem Analysis: Students design performance analysis studies, and review by case study performance studies conducted by others. Students learn statistical techniques for analyzing and comparing measured results and accounting for error.
Solution Design: Students learn design of experiments techniques; students design experiments to complete project 2; students read about design decisions made by others.