The Standard Performance Evaluation Corporation (SPEC) is an American non-profit corporation that aims to produce, establish, maintain, and endorse a standardized set of performance benchmarks for computers.
SPEC was founded in 1988. It was originally called System Performance Evaluation Cooperative. Their first benchmark standard was SPEC [1989] or SPEC Benchmark Suite Release 1.0, October 1989. SPEC benchmarks are widely used to evaluate the performance of computer systems; the test results are published on the SPEC website.
Over time, SPEC evolved into an umbrella organization comprising four main groups: the Graphics and Workstation Performance Group (GWPG), the High Performance Group (HPG), the Open Systems Group (OSG), and the most recent addition, the Research Group (RG).
Structure
Open Systems Group (OSG) High-Performance Group (HPG) Graphics and Workstation Performance Group (GWPG) SPEC Research Group (RG)
Membership
Any interested company or entity willing to commit to SPEC's standards can become a member. Membership benefits include:
Participation in benchmark development
Participation in review of results
Complimentary software based on group participation
The list of members is available on SPEC's membership page;.
Membership Levels
Sustaining Membership requires payment and typically includes hardware or software companies. SPEC "Associates" pay a reduced fee as nonprofits; typically includes academia and research organizations.
SPEC Benchmark Suites
The benchmarks aim to test "real-life" situations. There are several benchmarks testing Java scenarios, from simple computation (SPECjbb) to a full system with Java EE, database, disk, and network (SPECjEnterprise).
The SPEC CPU suites test CPU performance by measuring the run time of several programs such as the compiler GCC, the chemistry program gamess, and the weather program WRF. The various tasks are equally weighted; no attempt is made to weigh them based on their perceived importance. An overall score is based on a geometric mean.
Cloud
Measuring and comparing the provisioning, compute, storage, and network resources of IaaS cloud platforms.
SPEC Cloud IaaS 2018
SPEC Cloud IaaS 2016
CPU
Measuring and comparing combined performance of CPU, memory and compiler.
SPEC CPU2006 contains two suites: CINT2006 ("SPECint") - testing integer arithmetic, with programs such as compilers, interpreters, word processors, chess programs etc. CFP2006 ("SPECfp") - testing floating point performance, with physical simulations, 3D graphics, image processing, computational chemistry etc. SPEC CPU 2017 package contains four suites. The SPECspeed 2017 Integer and SPECspeed 2017 Floating Point suites are used for comparing time for a computer to complete single tasks. The SPECrate 2017 Integer and SPECrate 2017 Floating Point suites measure the throughput or work per unit of time.
Handheld
Graphics and Workstation Performance
Measuring performance of an OpenGL 3D graphics system, tested with various rendering tasks from several popular 3D-intensive real applications on a given system.
SPECwpc
High Performance Computing, OpenMP, MPI, OpenACC, OpenCL
OMP
The SPEC OMP (OpenMP) is the first one for evaluating performance based on OpenMP applications, for measuring the performance of SMP (Shared memory Multi-Processor, i.e. UMA) systems.
Java Client/Server
JBB
evaluates the performance of server side Java by emulating a three-tier client/server system (with emphasis on the middle tier).
j Enterprise
A multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) technology-based application servers.
Mail Servers
Storage
SPEC SFS is for measuring file server throughput and response time supporting both NFS and SMB protocol access.
Power
Virtualization
Web Servers
SPEC Tools
Server Efficiency Rating Tool (SERT). Intended to measure server efficiency, initially as part of the second generation of the US Environmental Protection Agency (EPA) ENERGY STAR for Computer Servers program. SPEC Chauffeur WDK Tool. Designed to simplify the development of workloads for measuring both energy efficiency and performance. PTDaemon. The SPEC PTDaemon software is used to control power analyzers in benchmarks which contain a power measurement component.