QPack: A cross-platform quantum benchmark-suite

Abstract

As the technology of quantum computers improves, the need to evaluate their performance also becomes an important tool for indexing and comparing of quantum performance. Current benchmarking proposals either focus on gate-level evaluation, are centered around a single performance metric, or only evaluate in-house quantum computers. This gives rise to the need for a holistic, application- oriented, and hardware-agnostic benchmarking tool that can provide fair and varied insight into quantum computer performance. This thesis continues the development of the QPack benchmark, which collects quantum computer data by running noisy intermediate-scale quantum (NISQ)-era applications and transforms this data into an overall performance score, which is decomposed into four subscores.

These scores are quantitative metrics of quantum performance that allow for easy and quick comparisons between different quantum computers. The QPack benchmark is an application-oriented cross-platform benchmarking suite for quantum computers and simulators, which makes use of scalable Quantum Approximate Optimization Algorithm and Variational Quantum Eigensolver applications. Using a varied set of benchmark applications, an insight into how well a quantum computer or its simulator performs on a general NISQ-era application can be quantitatively made. QPack is built on top of the cross-platform library |Lib⟩ (pronounced: libket), which allows for a single expression of a quantum circuit and execution on multiple quantum computers.

Using the QPack benchmarking scores, a comparison is made between various quantum computer simulators, running both locally and on vendors’ remote cloud services. Tested local simulators include Qiskit Aer, Cirq, Rigetti QVM, and QuEST. For remote simulators, the IBMQ, IonQ, and Rigetti simulators have been benchmarked. The QPack benchmark is also executed on the Rigetti Aspen-M-1 and a selection of available quantum hardware from the IBMQ aviary, namely the Nairobi, Jakarta, Perth, Lagos, Quito, and Manila processors. For all quantum computers, an analysis is made of their individual performance in the QPack benchmark, as well as an evaluation of how these simulators or hardware implementations compare to each other. Based on the results of the QPack benchmark, the local QuEST simulator, the remote IBMQ QASM simulator and the IBMQ Nairobi and Quito quantum computers achieve best performance compared to the other tested backends.

This work shows that the QPack benchmark is capable of providing holistic quantum computer performance for quantum computers, be it physical implementation or their simulator counterparts. The latest version of the QPack benchmark and all the results collected can be found in the repository: https://gitlab.com/libket/qpack/-/tree/stable.