Claims of leaps in quantum computing are made almost daily, but progress is hard to judge when each research group uses its own mixture of hardware, algorithms and evaluation metrics, making it near impossible to compare systems. Now, researchers are trying to make it easier to chart the performance of quantum machines.

As part of an ongoing effort, a consortium of UK researchers has created a suite of metrics that they say is a holistic way to measure the performance of quantum computers. They have published the work alongside a library of open-source software called QCMet¹. Separately, a group including tech giant IBM and Helsinki-based quantum-software company Algorithmiq launched the Quantum Advantage Tracker last month as a way to compare experiments that purport to show ‘quantum advantage’ — that is, an efficiency or accuracy better than that of a classical machine.

“People ask, ‘When is quantum computing going to be useful?’ I think it’s a little bit of a farcical question to answer without some data. And I think having data-driven answers exactly what initiatives like this are pushing for,” says James Whitfield, a quantum physicist at Dartmouth College in Hanover, New Hampshire.

Many possible metrics

Classical computers encode information in bits that can be either 1 or 0. Quantum computers are instead based on quantum bits, or ‘qubits’, using any system — such as light or tiny superconducting loops — that can be put into a state of being both 1 and 0 at once. Physicists make qubits interact in ways that they hope will allow them to perform certain kinds of calculation much faster than would otherwise be possible.

Some groups have created quantum computers containing than 1,000 qubits. But maintaining ever-larger systems while operating the computer is challenging.

When publicizing their results, quantum-computing companies often highlight the number of qubits in their machine as an indication of how advanced it is. But this doesn’t tell the full story: other metrics are often relevant, such as error rates and how many qubits can be connected or harnessed in each calculation.

And it is not always clear how the firm calculated the figure they report. The QCMet project — led by researchers the National Physical Laboratory (NPL) in Teddington, UK — has built a list of metrics and algorithms to capture the performance and capabilities of all kinds of quantum computer, as well as publishing software to help standardize how each measurement is performed.

“You shouldn’t just use one (metric) to tell you which is ‘better’ — you need to have the larger picture,” says Abhishek Agarwal, a quantum physicist at the NPL, who is part of the project. One often-ignored factor is stability, he says. In some systems, “if you run something now, versus even two minutes later, you can get quite different results”.

Using a range of metrics could help to harmonize what types of result that researchers report, improving comparisons between quantum computers, as well as interoperability, says Whitfield. Each type of qubit has its own strengths, and sometimes companies report only the metrics that suit them. “It’s almost like they don’t want you to compare across devices,” Whitfield says.

Scrutinizing quantum advantage

The Quantum Advantage Tracker aims to help compare quantum machines by gauging how well quantum computers perform at particular applications compared with classical computers. Claims of quantum advantage — often for little-known and very specific applications — are usually quickly overturned once external researchers improve the performance of classical methods. The tracker could help to monitor these ever-moving goalposts, says Whitfield.